Transcript
[0:00] Music.
[0:08] Well, it's that time of the week again. It's time for Chit Chat Across the Pond.
This is episode number 778 for November 18th, 2023.
And I'm your host, Alison Sheridan. Back after a normal hiatus that we cannot explain.
Our guest is Bart Bushottz with Programming by Stealth, installment 155.
How are you doing today, Bart? All right. If it wasn't for the fact that I'm sure we have noise cancelling, I was going to make a blowing noise on my microphone to get the dust off.
But I think that would probably just get noise cancelled out.
So that joke would fall very flat.
We were talking about it before we started up is we take a break every year.
It's unannounced and of indeterminate length at indeterminate time.
It appears to have been our fall break this year, but it's been often it's been our summer break, but we don't know what causes it.
It just doesn't we don't do programming by stealth 12 months of the year.
It appears to be about a nine month show.
Yeah, we don't plan it. You might think as I go, because I know podcasts that have a planned summer hiatus and they lead up to it and they say, hey, there's three shows left. Season.
We just, yeah, we just vanish. We just go poof.
[1:14] How rude. Hey, it's our show and we're having fun. So we are back with 155.
What are we going to do today, Bart?
So we are doing a little intermission. So I am waiting on the very final approval for what is going to be my first ever in my entire life as a working person, a pseudo-career break, I will be taking seven weeks of annual leave consecutively, which is not something I have...I don't think I've done that since I was a child in school. I don't remember.
So when that happens, I'm going to basically deep, deep, deep dive into getting XK Passive D rewritten.
But that's not now. That starts on the 15th of December if I get my final approval.
So, what do we do between now and then?
Well, by pure coincidence, both my personal me and my work me have found themselves in great need of processing JSON files on the terminal.
Work.me because there's an awful lot of security-related things.
[2:17] APIs and things, that basically throw JSON at you.
You query them and they give you JSON. And then if you want to act on them, you need to understand that JSON. And for personal.me, I am...
Actually, no, I'm going to keep my powder dry on what it is I'm trying to do.
But I've ended up with a giant big database in JSON format that I need to process automatically, because otherwise it will take me from now until the end of the universe.
And I have discovered I know just enough JSON to swear very loudly at it and get about 80% of what I need.
And you taught me, and everyone else who was at MaxDoc, or everyone who was at MaxDoc, because I wasn't, that if you want to learn something, the best thing to do is to teach it.
So we're going to learn how to process JSON from the command line.
Because you can't use grep or any of those tools, because JSON is far too structured.
And that is that way madness lies. If you try to grep your way to getting data out of a JSON file, you will go insane because you'll always get the wrong answer.
In other words, you tried that first.
Of course I did because I'm a regular expression person and that is the answer to life, the universe and everything, except for when it isn't.
And when it isn't, it's usually quite infuriating.
[3:31] So the JQ was written specifically for this job.
That is its one function in life. I would love to think it stands for JSON Query, but officially it doesn't stand for anything.
The official website has no mention of what the name means. The official website has, in fact, nothing like that. Nothing so frivolous.
Neither does the Wikipedia page. So JQ is JQ. So I'm kind of surprised.
We've looked at JSON several times, and I happen to have just been working with Homebridge and all of the configuration files are in JSON. And it's perfectly reasonable, human, readable file. I mean, it's plain text.
It's got little bracket structures. It's got things in quotes and colons, and it's very easy to read.
I'm just shocked that it's hard to work with. I would have expected it to be trivial.
Well, okay, so how do you find a value that's inside a specific name property inside the fourth element of an array using grep?
[4:34] Okay. I was going to bring it up in a text editor and type command F is what I was going to do.
You have to do it a few more times. But your shell script, right?
Yeah, your shell script isn't going to work like that.
So to get at this, because the data is very structured, you described it yourself, right? you have lists of things, you can have dictionaries, key-value pairs, and it's all very structured.
So you can represent very complicated data in JSON and retain all of its structure, all of its meaning effectively.
But in order to pull that meaning out, you need a tool that speaks JSON, that actually understands that this is the fifth element of an array, which is inside, which is a key-value pair and so forth.
You need something that actually understands the JSON structure, which means you need a tool for the job.
Now, jq, despite having an extremely short name, is stupendously powerful.
And its syntax reminds me a lot of my good friend the regular expression.
Because you can say a lot by typing very, very little, which is easy on the fingers, but hard on the brain.
[5:49] It takes a lot of understanding to write JQ and it takes even more to read other people's JQ, but when you get good at it, it is magic.
You know how I love a dense language part. Don't expect any whining at all from me on that part, right? Nope, not at all.
Especially not when square bracket has about five meanings. Oh, yay! I love that.
[6:13] So, we're not going to do everything JQ can do, because it is a full-featured language.
It is spectacularly powerful.
But we're going to do three things with it, which are the three most useful things to be able to do.
The first thing we're going to do is its simplest superpower.
It can take unformatted JSON and make it pretty.
So a lot of web service APIs and things will give you back your JSON as one giant big line.
No spaces, nothing. Oh, man, I've seen that.
And it's just this wall of text. Could you give me a carriage return here or there? Help me out. Right, exactly.
Yeah. So how on earth can you tell what's inside what, what the key value pairs are when it's just one big line of text?
So you pipe it to JQ and you get back syntax highlighted pretty JSON on your terminal.
So that is one of the most one of the things I do with it more than anything else is like, what is this API telling me? Oh, OK. It's decided to wrap everything in an array named answer.
OK, then I'll just answer dot whatever it is I really wanted. Fine, thank you.
The other thing you can do with jq, the next most powerful thing, is that you can dive into the structure.
So go into the array, find the fourth element, in there find the key value pair called surname and then give me just that value.
[7:31] So you can pick little pieces out. Imagine you have a config file and you're trying to write a script that goes to the same folder, well, how do you pull the folder name out of the config file?
Well, if you know that it's under the key value pair called the home deer or whatever, then you can just say, JQ, go fetch me the value for home deer from this file over here.
And then it will read into it and pull it out for you. So just pulling out pieces of data is very powerful.
[7:55] And then, then magic and or insanity happen.
The last thing you can do with JQ is you can actually take JSON as input.
And process it like you would a database with SQL and produce entirely new JSON that is pieces of the old one completely reassembled maybe you do some math so you can do things like take this giant big list of sales collapse it down into my top 5 products with the average retail price and it can do all of that math probably within 10 characters of JQ.
[8:36] And so who sends that kind of data in JSON? That's kind of surprising.
A lot of web services. No? There is.
Sorry, we had a freeze there. That was weird. That hardly ever happens to us.
So what I asked was who... I've never seen that kind of data, like sales data, come in a JSON format.
A lot of enterprise systems, when you ask them to talk to each other, they used to talk in XML, but XML is evil.
So now an awful lot of them talk in JSON.
Oh, I didn't know we hate XML now.
XML was a good first step, but JSON is better.
[9:19] So yeah, so a lot of things that need to talk to each other that are written in like five different languages, they're common.
The lingua franca of the enterprise is JSON these days. So you actually do get a lot of JSON data floating around.
And also if you have one of my favorite phrases I ever learned from a linguist is lingua franca. for anybody who doesn't know what it means, it means common language.
It's the common tongue. It's a great phrase.
Yeah, so in science at the moment the lingua franca is English.
It used to be Latin. Thank goodness it's English now because my Latin sucks.
[9:53] It's a little bit easier. By the way, since the phrase we are starting to step on each other I think we got out of sync a little bit.
So I may raise my hand to Bart to get so I don't keep stomping on him.
And Zoom may correct itself, it may figure it out. Okay, so basically, we can make it pretty, we can pull out pieces, and we can do a full-on transformer jobby.
And we're going to look at those with three common use cases, the most common places we as regular folk are likely to bump into JSON.
So one of the places is in those wonderful free web services APIs that we played with a lot in our JavaScript days, where we did our currency converter.
That was a bunch of JSON data that came back.
When we were querying about our IP address, that was JSON data.
When we were getting that cool weather API, that was JSON data.
So a lot of web services talk JSON.
And on the terminal, that means we take curl to get us the stuff from the web, and then we shove it through JQ, and then we can get answers out.
So we can take the weather.
[10:54] Curl is the command line tool for doing an HTTP request.
Okay, so you say, curl URL, and then you get back to data.
Yeah, it has. It has been a while. That may even be taming the terminal territory.
We may be crossing our streams a bit here.
[11:08] So that's one thing we're likely to do. The other thing we're likely to come across is reading config files.
Because if you're trying to automate something that's part of a bigger system, if you can read the config from your shell script, then you don't have to remember where it is you put things.
You can go read it from the config file where you put things.
What password is on that database? I know the config file knows, I'll get it from the config file.
And then the last thing is, there is actually, if someone wants to publish a free database, and they don't want to run a database server and have to pay a big server from, they just want the database as a single file, you could make it an Excel file.
But the problem is, you're now in a proprietary thing, only a few programming languages can handle Excel.
A lot of these things are published as JSON because it's completely open.
So if you have a giant big JSON file, you're going to need a way to query that giant big JSON file and actually pull out the information you want because that's kind of what you want from a database is to pull things out.
So there are the three use cases. So three features, three use cases.
And that's what we're going to focus on. So, a JSON file can be considered a database?
[12:22] It's a bunch of information in a format. Yeah, so if you... Okay, so this installment has some examples in its resources folder.
And one of those examples is a config file. The other example is a JSON file that contains a database of every Nobel Prize ever given. Oh, so I can go learn how to query for Andrea Ghez.
She must be in there. She absolutely must be in there.
Yes, you absolutely can query for her and figure out, because it has the citation, so you can figure out what wording it is that they gave her the Nobel Prize for.
And it also tells you what share of the prize. Did she get a third, a half, a quarter? Because they're not always evenly divided.
Oh, yeah. I definitely am going to want to do that.
Yeah, there you go. So yeah, it is, I mean, it is information in a structure that can be queried.
So that is a database. It's not a relational database.
Ah, okay. Okay. Okay.
[13:21] All right. Okay, so a little prologue, remind ourselves of some JSON.
So many, many moons ago I did a blog post on it on bartb.ie when I still used to write blog posts.
So that's linked in the show notes. There is a nice tutorial from FreedCodeCamp that I've also linked in the show notes, but honestly, it's quite straightforward.
So everything in JSON is one of six things. It can be a number.
42 is a number. 3.1415 is a number. They're just written as is.
You can have a string in JSON. They are written in double quotes.
JSON doesn't give you this luxury of choosing your quote. No, no, no. In JSON, it is a double quote.
You can have a boolean, which in JSON is T-R-U-E or F-A-L-S-E.
[14:08] No quotation, marks, or anything. Just true or false. You can also expressly say nothing with the keyword null.
So you can have a json file that says that the, I don't know, the number of, I don't know, you can have something mean null.
That can be meaningful. It can be meaningful to say there is nothing.
And then you can have an array, or a list if you prefer, which is square bracket, value comma, value comma, value comma, close your square bracket. bracket.
So very javascript-like syntax.
It's basically a subset of the javascript syntax.
The other thing you can have is a dictionary, sometimes called an object.
Open curly bracket, the key must be in quotation marks, which is different to javascript, followed by the colon and then the value.
Comma, key, colon, value. Comma, key, colon, value. Close your curly bracket.
And you can nest anything inside an array, including another array or a dictionary.
And you can nest anything inside a dictionary, including another dictionary or an array.
So with those six pieces, you can assemble basically any piece of information. And that's JSON.
That kind of is all there is to it. Okay, good.
[15:26] So I have here an example of the Nobel Prize for Chemistry for 2023 from that data file.
And so you can see it says key value pair year 2023 category chemistry laureates is a list of dictionaries.
They're then indexed by who the laureates are. Each laureate has an ID, which I guess means if the same laureate wins multiple years.
Oh, Einstein won a few times. He probably has the same ID each time he won.
Oh, we could find out when you're done teaching us. We could find out.
It's in the data file. Exactly.
So this is an array. And then a first name. You call that a dictionary?
[16:05] No, no. Okay. So laureates is an array of dictionaries. Yeah.
You said laureates was an array, or was a dictionary, and I want to make sure it was an array of dictionaries.
It's an array of dictionaries. So each laureate gets to be a little dictionary where they have an ID, a first name, a surname.
Then there is the motivation for giving them the prize. So for Mugi Bawendi, who won the chemistry prize this year, for the discovery and synthesis of quantum dots. They make great monitors, by the way.
And yeah, so... Actually, all three of them are the same. They are, and they also have the same share.
They're each getting a third, a third, a third. So it's interesting, share is three.
There are three shares, and they got three?
That's interesting. I think it's a fraction without the one over.
Looking at the data set, it seems to be a fraction, but without the one.
Yeah. So whatever you call that bottom side of a fraction, it has a name. Denominator.
Sounds plausible. Definitely a word. Definitely in my math class.
It's definitely denominator.
Okay, good. Good, good, good. Oh, the blindy and the blind on mathematics.
Well, I'll certainly take your word for that. Wait a minute. I'm not blind on that.
I'm blind on... I'm bad at arithmetic. Let's make sure we keep that straight. I'm bad at both.
[17:21] How did I ever make it through physics? Anyway, I did. Just about.
Right. So that's a little primer on JSON. So now let's look at JQ.
So the first thing is I would like to take the creator of JQ, whose name is in the show notes somewhere, and tell him that he's a bloody idiot for naming the terminal command command and the language the same thing.
He didn't have the jq command speaking jql or something so that you could actually look up jq and jq and not get the wrong... Sometimes you want to know about the command.
Sometimes you need to know about the language. They have the same name.
So it's like googling for apple photos.
You look for help with photo. Well, everything's a photo.
Same problem. So, to try save everyone's sanity, I'm going to use jq in a fixed width font when I mean the terminal command, and just the letters jq when I mean the language.
And also because this extremely intelligent and wonderful human being who wrote this amazing language makes me mildly cranky because he didn't capitalize it.
So it's lowercase jq. And I checked the documentation.
It's always lowercase jq in both cases.
So anyway, I've kept... It's his language, he has the right.
So I have kept it. But as I say, I'm using fixed width font when I mean the terminal command. command.
[18:44] The other thing we should say is that JSON is much newer than the shells like bash, sh, or even zsh.
So there is no built in support for JSON in the shell.
The jq command is a third party addon.
It's a thermal command not part of bash, sh, or zsh. It's a completely separate thing. You have to install separately.
So we're going to have to do that. We'll jump to that in a sec.
The other thing is I have put a list of useful resources here in the introductory episode because I figured that's the easiest place to refer back to them.
So we have a link to the official JQ homepage.
We have the official docs. We have a very nicely formatted HTML version of the JQ man page, which is very useful.
The languages Wikipedia page is actually quite nice. It gives a nice summary of the JQ syntax.
For now it's going to look like complete gobbledygook, but when you start to speak a bit of JQ, that Wikipedia page is actually quite useful.
And the other cool thing is there's a thing called jqplay at jqplay.org.
[19:56] Which is a tester for your JQ syntax.
So you have a text box at the top where you put your JQ syntax, a text box underneath that where you put your JSON, and then it gives you the answer. Oh, I love it.
So yeah, that is very convenient when you're trying to figure stuff out.
So it really helps you hone your syntax.
So in order to play along here, you definitely need to install jq.
And if your operating system doesn't ship with curl out of the box, then you also need to install curl.
The Mac does come with curl out of the box. Most Linux distributions come with wget instead.
So if you're on Linux, you should be able to get both jq and curl without any problem using your standard package manager. So if you're one of the Red Hat-like, Linuxes, that'll be yum install jq.
And if you're one of the Debian-style Linuxes, apt-get install jq.
[20:48] If you're in Windows-land, what you need is the Windows equivalent of Homebrew, which is something called the Chocolaty Package Manager. Oh man, I want chocolate.
Yeah, there you go. I mean, I like beer. Yeah, I like beer, but I prefer chocolate. Right, right, me too.
So, once you have installed Chocolaty, you can install jq by saying choco install jq-y.
I guess if you leave off the minus y you'll have to type yes.
Which is no, not the end of the world.
And then for those of us in Mac land, homebrew is what you really want to get jq easily.
So once you've installed the homebrew package manager, you can say brew install jq.
And in Windows land, you definitely need to install curl because Windows definitely doesn't ship with curl, so that's choco install curl minus y.
And then you're good to go.
Awesome. So the jq command, its input is going to be some JSON and some jq syntax, and then it's going to give us whatever it is we asked for with jq syntax.
So the input will come from standard in if you like, so you can pipe things to jq, or you can give it a list of files at the end of the command and it will read the JSON from the files.
[22:06] What they call the filter, so the bit of JQ syntax is called the JQ filter.
That is the first argument.
So the first argument is your search query, which is just the way grep works.
You say grep what you're looking for, where to go look.
JQ what you're looking for, where to go look. And there are some command line options, but we will leave those for now and we'll bump into those when they become useful.
I have one question. I actually read this far in the show notes and no further, so I don't know quite where we're going to go.
But why do I care about there being a language and a terminal command?
Do we care about the language?
Yes, because the language is how we say what we want. I thought we'd say that in the terminal command.
[22:55] Okay, so the terminal command takes as its first argument, the filter written in the language.
Okay, that sounds kind of like the same thing. Like they're really a distinction without a difference for us?
Okay, so the grep command takes as its first argument a regular expression written in Perl-style regular expression, PCRE.
Oh, okay. The jq command takes the jq filter in jq.
But had you never told me, I never probably would have noticed, right?
Until you start googling for things. And then it starts to tell you to use minus minus whatever, only you want the other JQ, where there are no minuses.
Because the language doesn't have command line flags, that's the terminal command has flags.
Oh man, I think I'm definitely confused on what the two of these things are.
Maybe this will come out in the proverbial wash as we go through this.
[23:53] Well, yes. So the filter is just the what you want. So the what you want is one string that is in a completely different language, right, that is in the JQ language.
The JQ terminal command takes command line arguments and flags and options and all that kind of stuff.
Oh, I see what you're saying. OK. Yeah.
OK, so you just Google for JQ. How do I do whatever in JQ? you, are they telling you how to make the terminal command do something?
Or are they telling you how to express it in the language that goes into the first argument?
[24:24] Do they even know? Sometimes, yeah, anyway, it gets very confusing.
The second thing that, okay, Stephen Dolan, that's the chap's name.
Look, he's obviously a genius, it's a really powerful language, but he made a decision that I think he probably regrets himself.
A very important piece of syntax in the JQ language involves the pipe symbol.
The pipe symbol is rather meaningful on the terminal.
[24:51] Meaning how we separate different terminal commands.
So the single most common way that people get completely tied in a mess with JQ is that they do not put single quotes around their JQ filter.
Because the filter will contain pipes a lot of the time. If you haven't quoted it. Oh, and you're piping stuff all over. Oh no.
Yeah, you think the pipe is JQ, but because you haven't quoted it, the terminal says, oh, I know that, that's for me.
So always, always, always, always, always quote your filters.
Because while there isn't a pipe in every JQ filter, if you don't get into the habit of quoting it, it is not a matter of if you will get a complete mess thanks to the pipe, it is a matter of when you will get a complete mess thanks to the pipe.
So rather than have us just get lucky and screw up early using a pipe, you're telling us now. Got it.
Yeah. Just always, always quote it. And that way sanity lies.
So for today, we don't want to do too heavy a lift, but I do want to show you the JQ command in action before we call it a day here.
So the first thing I want to do is show you how to make it pretty print.
So, there is a lovely free online API called FreeIPAPI that will tell you your IP address and some information about your IP address. So the URL is https://freeipapi.com.api.json.
[26:19] So if you just curl that, you will see the raw output.
So if you say curl space that URL, and it's all on one line.
It's just this one big line of glob.
Yeah. So what if we wanted it a bit prettier?
Well if we take that curl command. Now the minus s flag in the curl command is for silent.
If you don't use the minus s, curl will show you an ascii progress bar while it downloads the json, which is very annoying.
So I hate that ascii progress bar so I just use curl minus s all the time.
And then you pipe it to jq with no arguments. Because we don't want jq to filter parts of this JSON, we just want it to show it to us.
And if you give jq no arguments, you just pipe something to it, it will pretty print it in color.
Oh my gosh. If we use syntax highlighting. It's beautiful.
IP version 4. IP address blah, latitude and longitude. Hmm, wonder where that is. Yeah.
Yeah, now I did this on my server. I was about to do this on my home Mac and I was like, I'm not sharing that with the world. So that is the server that hosts BartB.ie.
I was in the Netherlands, which is the data center I chose. I was thinking of just turning on a VPN.
Make it more interesting. That would have done it too. Yeah, they could be anywhere. Where is Alison now? Tasmania. Yay.
[27:39] That's South of N.C. Rowley. Yeah. Immediately you can see the power of this.
When an API gives you back garbage, just pipe it to JQ and don't even bother with any arguments.
Just pipe the JQ and hey presto, pretty JSON.
So that immediately is a win. Straight away we have a win.
We can also pull some information out of a config file.
So I took the package.json for the website this-time.me, which.
[28:07] Is in GitHub, so it was easy to get to. I open-sourced it.
So the npm package.json file is sitting in the folder here.
I've renamed it so that things don't get weird inside our Git repository for these show notes.
So I've just called it this-time.me dash package.json. But it's just a package.json file.
And so that contains information about the this-time.me npm.
[28:34] Project.
So what if I wanted to know how many dependencies does that project have?
Well I can use jq to pull that information out, by giving it the filter inside single quotes dot dependencies pipe length.
So that will find the dependencies key inside that json file, and that dependencies is an array, so I can then pipe that array to JQ's length function, which will tell me the length of the array named dependencies.
So wait a minute, so pipe does mean pipe inside JQ, it's just not the pipe that the terminal command is allowed to use?
Right, we're not saying take the output... I thought it was going to have a totally different meaning.
No, it is how you chain parts of the JQ pipeline together.
So it is still for chaining parts together. So yes, you're right, it does have the same meaning.
But if you don't quote it, bash will be off with it instead of jq.
And bash also has a length command. But that's the length terminal command.
That is completely different to jq's length function.
Didn't you tell us that we had to use double quotes?
[29:46] No, so double quotes are if you have a variable that you want to expand the variable. So a double quote is an interpolated quote.
Okay, well we'll come back to that then. Okay.
Well, if you go back one episode, we recorded months ago, the last thing we did was a table summarizing all the different kinds of brackets.
So if you're ever wondering what the difference is in bash between a single and a double, because they both have their uses, then that table will tell you straight away.
Okay. Okay, and it's the same, because we are sort of in bash?
Jq is a terminal command.
Well, no, jq is a terminal command. So this is a bash command.
The first argument is the string, dot dependencies pipe length.
Okay, so what does dot mean?
Okay, so inside the string, we are now in jq land.
So I'm going to say, park that until next week, because this is a JQ filter.
It does something useful. It contains the pipe. It's a nice example.
The meaning of that JQ is, well, it's going to be the future.
It's going to be the focus of our next quite a few installments, actually.
Okay. There's a lot of learning to do here. You've asked it to tell us the dependencies.
Oh, do you have to spell it? How many dependencies are there?
Do you have to spell it correctly?
[31:05] Yes. Otherwise, the answer is probably zero. It is. So I was going to ask you...
Dependence... okay, jeez, I'm just going to copy it.
Because I got zero and I was thinking, what am I doing wrong?
Yeah, I think it's 12, if memory serves.
[31:24] Yes, now it gets 12. Yeah, I had an E, I where an E should go. Good, good.
Okay, so we gave it JQ, which is a bash command.
And then we gave it a terminal command, so works in Bash.
And then we quoted our filter, and we asked it for the length of the number of dependencies.
And then we fed it the file that we wanted it to look at. Okay.
Bing, bing, bing. Cool. That's it, exactly. Yeah. All right.
So it's a terminal command with two arguments.
And then the last thing is, how do we pull information out of a database?
So the file Well, nobelprizes.json contains all of our information about Nobel Prizes.
So what if we just want to see the 2023 prizes, or actually, sorry, what is it I have here?
I want to see the 2020 prizes because that's when Andrea won.
Okay, park that one for a while, we will get there.
So this example command shows us the first prize in the list.
It's basically very simple syntax, give me the first prize. So, it's jq with the filter dot prizes open square bracket zero close square bracket, which is the first prize in the array of prizes.
And that prints it out, nice and pretty. The first prize, which is the chemistry prize, I guess, they must be doing them alphabetically.
[32:47] Okay, because the winners are in a dictionary, we've said, I'm not a dictionary, in an array, we said we want the first one, which is zero.
So it's jq quote dot prizes square bracket zero, and then fed at the file.
Okay. And the colon following it in the show notes is not part of the command. I figured that out.
Oh, yes. Yeah. I guess that's poor typing on my part.
It's not in the fixed with highlight he style, but still. but it's close.
Yeah. So now I have a pretty list of year 2023, category chemistry, and the laureates are, and it gives the three laureates.
Very cool. Now, I'm now going to blow your head slightly.
So this is pulling information out. But remember, where we are going in this series is transforming the information.
So we don't have to get back JSON and the same structure of what we put into it.
So the very large and very cumbersome filter you're going to copy and paste without worrying about too much for now, is going to take all the prizes for 2023.
[33:53] Throw away almost all of the information, and just remember what the prize is and how many recipients there were.
Okay, so don't worry my pretty little head about it.
Yeah, it's a lot of pipes in there, right? That is a lot of syntax, a lot of pipes, select statement, double equal signs, there's a lot going on in that little bit of jq there.
It will all make sense to us in two or three installments time.
But the end result is that we now get an array of dictionaries that look nothing like the original.
It just has two key value pairs, prize, numRecipients.
Neither of those two key value pairs were in the original data.
We have built new data based on the old data, and we've applied math. Oh, cool!
Yeah, that's interesting. Quite powerful. Yeah.
So we have processed a database. Yeah.
[34:51] So here we are. Right. So the last thing we're going to do today, which is still an easy lift, is we're going to learn all the litty gritty details we need to know about making our pretty JQ a little bit prettier.
So there's nothing difficult here, but sometimes you want slightly different to the defaults.
So this is another excuse for me to remind us of how cool the free wttr.in web services API is.
Wetter.in, I guess. I think of it as wetter, as in more wet.
But I think it's supposed to be weather. But, you know, I'm in Ireland.
[35:26] And so that will return to you... By default, actually, it returns to you like ASCII art of the weather, which is just cool because it's ASCII art of the weather.
But if you say ?format equals J1, you get JSON for your current weather.
[35:42] And so if you do a curl minus s and you pipe that into jq, you can give it the jq filter dot current underscore conditions zero, which gives you the current conditions this hour, like not next hour, or the hour after.
So basically the current conditions right now is current underscore conditions zero, and it will print you them out in syntax highlighted pretty form and it does that clever thing where if you pipe it to another terminal command it will do it in plain text but if you pipe it to a terminal it will do it in pretty colours.
But what if you don't want the pretty colours? What if you wanted to output to the terminal completely vanilla?
Well then you can use either minus minus monochrome minus output or minus capital M.
One of those is shorter to type. Where is this the weather for?
Is that here at my house? Or at your house? Yes.
If you don't put anything between the forward slash and the question mark, it uses your IP address to determine your location.
If you do put something between the forward slash and the question mark, you can get the weather there.
So if you pop in Paris, you can see what it's like in Paris or Amsterdam or whatever. Or Dublin if you want to see what it's like here.
Dark, wet, cold. All right, let's see.
I tried that a silly way. Okay.
[37:06] All right, I've got the same thing, but it's not as pretty. So that's the boring monochrome version. Got it. Okay. Yeah.
Now, if you're doing something with that JSON, so imagine you spent 500 characters of JQ transforming your data a million ways from Sunday, and you want to send that data to another program.
Maybe you do not want it pretty printed. Maybe you actually want it all squished together.
The minus C flag, or minus minus compact minus output, it, we'll put it all on one line for you. It'll squish it all back together for you.
[37:40] Ah, ok. Can you combine, can you make it minus cm? You can indeed.
Then you get it monochrome and all squished together. Yeah, now I can't read it at all. Perfect!
Yeah, which may be what you want, right? If you're using a terminal command to feed another terminal command, that may be what you need.
Another thing you can do, by default, it uses two spaces as the indent.
What if you want a genuine actual tab character?
Well, ''--tab'' will give you an actual tab character, which may be important for something you're processing with a regular expression or something.
Or what if you just want a little bit more space because two isn't obvious enough?
You can also say ''--indent'' and give it a number, and that would be the number of spaces.
So minus minus indent four, we give it to you with four spaces instead of two.
Oh, but by the way, you can only indent between minus one and seven.
I happen to try eight. Minus one? What does minus one mean? I don't know.
Let's see. Probably no indent.
Oh, zero, surely. No.
[38:47] No, that's odd. It actually looks like eight. It's one more than when I asked it for seven. Huh.
[38:55] That's some interesting binary math going on there or something.
I just happened to get lucky and pick outside of the range.
[39:01] Yeah, and I guess a tab is eight characters by default, isn't it? Uh, maybe.
Yeah, I think it is. Interesting. Anyway, so that's all of the different ways you can have it pretty, but to be honest, I just use it without arguments, which gives it to me two spaces and pretty.
And that's usually exactly what I wanted to try to figure out what this weird API is trying to tell me.
Because that's actually what you often want, right? You know this API is going to give you some data and then you want to program that data into something.
Well, what are the key value pairs?
The easiest way is just show it to me, but show it to me pretty so that silly human me can read it.
So do you use this when you're trying to diagnose, is the word that comes to mind, but you're trying to figure out what you're looking for, where it is, and how it's going to look, and then you go, like, okay, now I know how to attack it to actually do what I need to do? Yes.
Yeah, so with my work hat on, a lot of security tools, when you query them, will give you back a bunch of JSON, and they'll tell you, call this URL to get the answer in JSON.
What it won't necessarily tell you is what the structure of that JSON is.
So it's, okay, great, I now have all of this JSON. How do I actually use this?
Just show it to me. Just let me look, and then I should be able to see what on earth it's doing. And that's what I use JQ for a lot.
Oh, cool. I really like this, because you gave us enough little fun at the end to give us the motivation. Why do we care?
[40:30] And not, but even though we didn't actually learn how to use it, we got a feel for, oh, look, that was cool. That's really nifty.
And by the way, it's just occurred to me, there's a reasonable chance that a percentage of the audience greater than zero listens to programming by stealth, but may never have heard my interview with Dr.
Andrea Ghez, the fourth woman in history to ever win the Nobel Prize in physics.
So that's who I keep referring to. If you've heard me say, oh, let me look up Andrea. That's who I've been talking to, talking about.
So not that I'm excited about it, but you should really go watch that interview.
Anybody who hasn't, you've got to go watch it. It's amazing.
Look for it on podfeed.com.
[41:09] Absolutely got it there. She is an extremely good science communicator and clearly an extremely good scientist, because you don't get a Nobel Prize for communicating.
You've got to actually do the science as well. So there's that part too.
Fascinating conversations.
[41:24] Absolutely fascinating. Well, this is good. So where we're going next, so as you could probably tell by that final example where we constructed a whole different piece of data from the original data, it can get complicated.
And I don't want to jump straight to complicated because your head will explode, my head will explode, it will be a mess.
So we're going to build up to it by that middle use case, by the extracting existing information sort of with surgical precision.
So that's what we're going to do first. And then after we've done the extraction, then we're ready for the transforming.
And the more advanced stuff. Because there's a difference between extracting and querying, right?
Give me the fifth element in the array is different to find every element in the array which has a length greater than four.
They are quite different things. And so that's going to be our progress, basically.
Precision strike, query, complete and utter madness.
That is the order we're going to tackle this in.
And by building it up that way, I am hoping that the density, the denseness of the syntax is never overwhelming, because we're going to build it like a little Lego fort, layer by layer by layer. That's a theory.
I shall see how it goes. Also, you need a little more time to learn the rest of it, right?
[42:44] I do. You have no idea. You know what took me longest in these show notes?
Those few examples, because I know just enough to nearly get what I want.
And the amount of time I spent on that bloody Nobel Prize one, trying to just get it to count me the number of recipients, that was 40 minutes of my life.
And a few choice words were uttered. Well, what do you always tell us?
It's the practice that does it, right? So now you know it. Ten thousand errors.
[43:14] The other thing I want to tell people to appreciate is, Bart and I were talking about this in the back channel for the last couple of days, was he was struggling with how to tell the story. And I think that's what BART really brings to this, is obviously the technical knowledge.
It's sort of like you had to be a good scientist to win the Nobel Prize, but you also have to be a good communicator.
But it's the way BART puts the story together that brings us along and helps us know why do we care, and then brings us into it.
So I think that was time well spent getting the story done. This was great. I loved it.
Yay! Good first episode back. Always nice to get back on the saddle.
Without falling off the horse.
Yeah, but I'm jumping over and landing on the other side.
All right. There can't be a better place to end the episode than right there, Bart.
Indeed. Until next time, happy computing. If you learn as much from Bart each week as I do, I'd like you to go over to lets-talk.ie and press one of the buttons over there to help support him.
He does 98% of the work here. I'm just the stooge that listens to him and asks the dumb questions.
If you go over to lets-talk.ie, you can support him on Patreon, you can donate via PayPal, or you can use one of his referral links.
[44:30] Music.
[44:31] I really hope you'll go over and help him out. In the meantime, you can contact me at Podfeet, or check out all of the shows we do over there over at podfeet.com. Thanks for listening.
[44:42] Music.