Transcript

[0:00] Music.

Introduction to Chit Chat Across the Pond

[0:08] Well, it's that time of the week again. It's time for Chit Chat Across the Pond.
This is episode number 782 for December 20th, 2023.
And I'm your host, Alison Sheridan. This week, our guest is Bart Bouchats with Programming Myself and Small Event 158, where we continue our journey with JQ.
And in an odd series of events, what you might hear in the background is rain at my house.

[0:31] It is dumping out there. Wow. Do you need some? Is it too dry?
I think it rained in February.
Okay. Maybe March. Maybe the Adda River will be a river.
Yeah, yeah, yeah. It's kind of different for us. As you described it, you said we call that Wednesday.
Yeah, pretty much. I got quite wet today. But anyway, that's neither here nor there.
So. All right. Where are we going today, Bart? We are.
Today is kind of a pause and gather our thoughts-y-ish sort of an episode.
So I said to you in the big picture that JQ has three big jobs in life.
It has pretty printing JSON, which was very easy, and we did that in the first installment. That didn't take us any time at all.
And then I said we can use it to pull information out of JSON files.
And so we started off by just pulling sort of data by its address, you know, go into the array, find the first element, go in there, grab me the value that matches the key name, waffles or whatever.
And so we were specifying a specific address within the data set to go and pull out something. thing.
And then last time, we met a whole bunch of concepts and we had to do a lot in one episode.
And I believe I told you that one of those concepts would be the most difficult thing in the entire series and that it's all downhill from here, which I hope you find reassuring.

[1:51] But we had to learn about operators. Actually, we had to start by learning about how we express data as literals, which is the same rules we use for JSON.
So basically, you have You have booleans, you have numbers, you have strings, you have null as a special thing, and then you have arrays and dictionaries as we call them.
And then we went on to look at operators, which do something with, you know, something on the left, something on the right, and they do something to produce something else.
And then we looked at functions and how if they don't take any arguments, you just use the name of the function and you don't have to bother with any brackets.
If it takes one argument, you just wrap that one argument in brackets.
And then we learned that if you have multiple arguments, you have to separate them with a semicolon.
And you joke that you're going to get this wrong all the time.
And I said, not I will, I do.

[2:39] And that is true while I was working on today's show notes I ended up making some very naughty words for a while and not understanding why my thing wasn't working and then I realised I had a comma instead of a semicolon and that makes a very big difference, because the comma oh yeah absolutely because the comma of course is the and also operator not the second argument operator they are very different things, second argument versus and also so.
And then we learned about the amazing select function for letting us filter our data, basically search for things based on straightforward conditions.
Is it equal to this? Is it not equal to this? Is it less than this?
Is it greater than this? And that was where we stopped.
Which gets us a lot of comparisons, right? You get to find anything where something's equal to something else.
But quite obviously, when I ask you to search for something in the general sense, You want to be able to do more than just equal to, not equal to, less than or greater than.
There's clearly a need for more searching ability.
And that's the driving force behind today's installment, really, is searching better.

Introduction to the topic of searching better in JQ

[3:47] And that then gets us ready next time to transform data, which is the third big thing that JQ can do.
So we can print it, query it, transform it.
And so the transform is the final part of this journey. And that's not for now.
We're getting ourselves ready for that.

[4:05] Okay, what piece are we going to do today? So today is, we're finishing off the querying.
So at some extent, I'm slightly trying to mirror last time's structure.
So we're going to start by looking at some more things related to type, because there's a couple of functions in JQ to do with types, which are useful to know.
And then we're going to move on to one new operator, which is going to save our bacon later in the show.
And many, many times in reality, because the real world is full of very questionably formatted JSON. on.
That is something that I have discovered by pure accident.
Our data set of Nobel laureates has some quirks, and that is very in keeping with what I find in the real world with my work hat on. There's a lot of quirky JSON out there.
And then we're going to finish up by looking at some more powerful comparisons.
So instead of just less than, greater than, or equal to, we are going to look at some other criteria we can give, and we're going to re-meet our our best friend in the world, the regular expression, which is, of course, an amazing way of searching for things. Does it match this pattern?

[5:10] Yeah, it sort of smelled like we would have to end up there.
Yeah, of course we did. Of course we did. Yes.
But of course, I started by setting a challenge last time, which was intended to, because I knew last time was a heavy lift, I had basically intended to make sure that we had three questions which touch on each of the core concepts that I had hoped to try to get across.
And i didn't expect necessarily for these to be easy because jq is what was the phrase we like to use dense dense jq is as dense as the terminal was and the terminal's denseness can be quite tricky so so i sent part a message before we started talking that i was really sad about the challenges because I really like what I'm learning. I really like data.
I really felt like I understood what he taught last week or two weeks ago.
And I spent a ton of time, I mean, hours and hours, probably four hours trying to do the first solution.
And I did not succeed. And I did not get close to succeeding because of some fundamental gaps in what I understood.
And I was sad because I really thought this was going to be fun.
And I never even got to to the second or third one. Never even got to start them because I couldn't get past the first one.

[6:30] So we've talked a little bit. We've talked me off the ledge, I guess is the way of putting it, before we got started.
But I was bummed because I thought I was going to love this.
And I did love trying, but towards the end, I was just flailing semicolons and commas and square brackets.
And I was asking ChatGPT for help.
Bart, I tried to read the JQ documentation, which is terrible, by the way. It was horrible. Horrible. Not even close.
Helping. I had to learn a lot of JQ before I could use the JQ documentation.
It was not, the JQ documentation is not a method of learning.
It is a method of looking up what you already know.

[7:06] At best. Yeah, I can see that. One of the things I've said on the Nocella cast is what I like about ChatGPT is I can describe in human form what I'm trying to do, and then it'll give me the right wording of what I'm supposed to do.
But what it answered with was nothing close to what you did.
So we're going to pretend I didn't even listen to the entire episode last week when you tried to tell me how to solve this first one.
Well, I'm going to start off with some very general tips actually.
So the first thing is when you're building this up, don't try to build everything at once.
I didn't. Okay, good. I didn't. Itty bitty little bites.
Okay, good. Because I sort of look at it as you start off with too much information and you're trying to filter it through multiple funnels to get less information out the other side.
And the less, you know you're done when the less contains only what you want and nothing more.
So you start off with everything and and you end up with only what you want.

Tips for solving the first challenge in JQ

[8:02] And in between, you may need to do some work.
And so if you write the JQ by, you know, you do one piece, and you run it, and you see the output, and what you should see is too much.

[8:17] And then you put a pipe, and then you try to solve the next piece of the problem and build it up that way.
We didn't actually talk about how to solve this, but I'm going to start right in.
The first question, the first challenge Bart gave us was, was what prize did Friend of the Nocella Cast Podcast, Dr. Andrea Giz, win?
List the year, category, and motivation.

[8:37] I can do pieces of that, but I can't do the year, category, motivation all in one fell swoop.
And the part I was stuck on was when we pipe one thing into the next thing into the next thing, we're kind of drilling down into the array laureates.
Once a laureate gets exploded and I'm down in there digging out the information about, for example, the motivation, I'm good. I can get that.
But I can't get back up out of the array in order to pull out the year and category.
I don't know how to do that. That's where I got stuck.
And the key is not backing out because you can't.
The key is not going in so you don't have to.
Okay. Okay, but I got to go in to get the motivation. You got to look in, but don't go in.
So imagine you've opened the door and you've had a look and you've decided, is this for me? No, it is not.
Throw that one away, but don't step in. Because once you step in...
But I do need something in there. I got to go in and get the motivation. Later.
Motivation's down in the... Right, but you want that later, after you've thrown away all the Nobel Prizes that are non-Andreas. Yes.

[9:55] Okay, let's step through it. Okay. Yeah, find the prize.
And then when you have the prize, then pull out the three pieces of information I asked for. But get the prize first. But how do I find the prize without looking in the door?
Okay, well, let's do that. No, you look in, don't step in. Look in, don't step in. It's my mental way of thinking about it. So let's start at the start.
We have a JSON data file that contains one dictionary with one top-level key-value pair named prizes.
And the value of prizes is an array. so that is one unwieldy thing so the first thing we actually want to do is explode that so we say dot prizes open square bracket close to a bracket, I'm with you so far Bart perfect, this is important because on the other side of the pipe what do we have?

[10:46] The other side of the pipe is now going to happen many times times, once for each element of the prizes array.
So each time the next filter gets called, it is working on an entire dictionary that represents a full prize.
So prizes zero, prizes one, prizes two, prizes three.

Working on each element of the prizes array

[11:08] Right. But each time, yes, but each time through it's seeing exactly one of those.
So each time through, the current value being processed is a full dictionary that contains a year, a category, a key value pair called laureates that is itself an array of more dictionaries.

[11:28] And it might contain something else, but I don't remember off the top of my head. But there, that's... No, objects. Objects, not dictionaries.
We call them dictionaries because we've been doing that since our JavaScript days. Well, objects.
But they're... Oh, did you say dictionaries? I'm sorry. I said dictionaries.
Okay. Sorry. Okay, I'm with you. Sorry. Objects, dictionaries, same thing. Got it.
Yes. Yeah, no, I've been very careful not to use the word object because Jill will kill me.
I've been very consistent about that the whole way through. Okay.
Okay. So we're going through a year category and laureates over and over and over again.
Exactly. So that is what we now have. So after we pass that first pipe, we're now doing the second pipe of the chain once for every dictionary that represents a full prize.
So we are now looking at the door of the full prize. So it contains our year, our category and our array of laureates. And we want to throw away every prize that doesn't have at least one laureate with the surname Gez.

[12:36] Now, our first attempt to do this would be to put another pipe and to say dot laureates, open square brackets, close square brackets.
And then we have then stepped into that array. and now the current thing being processed is a dictionary with a surname, a first name and a motivation and an ID, but let's ignore the ID.
And we can very easily find the current thing with the surname of Gez.
But what falls out of that pipe is now a laureate, not a Nobel Prize.
Because we stepped, we exploded the Nobel Prize. We took the Nobel Prize and we blew it up.
That's what we've got to not do. But how did you teach us not to? Yes, I did.
Because the entire reason that the three argument any function exists is to avoid having to step through the door.
It lets us look through the door, but not step through the door.
So inside our second pipe, right?
So everything between the second, sorry, inside our second filter.
So after the first pipe between the first pipe and the second pipe is our second filter it's a select open bracket a whole bunch of stuff close bracket.

Introduction to the select function and its purpose

[13:59] Okay. Is select a function by any chance? Yes. Yes, it is. Okay.
Did you tell us select was a function?
Yes, I did. Because you went through the functions, but you only talked about not and you did the booleans and stuff.
And then I said that select was a function that processes the input.
Okay. And it takes one argument that is a filter, and if that filter returns true, then select passes the value to the next step in the pipe, and if that filter returns false, select silently swallows the current value.
So that is how select reduces our number of things we're processing.
So we start off with, once for every single prize, we run it through select and we say, if this criteria matches, you go on to the next thing in the chain.
If this condition fails, you evaporate into the ether, you are destroyed.
So obviously at the end of that middle pipe, there are far fewer Nobel Prizes left.
Because everyone that doesn't meet a criteria is evaporated into nothingness.
Okay. So that is the point of select. So the question is, how do we decide which Nobel Prizes stay and which Nobel Prizes do not stay?

[15:24] Okay. So we need to write a condition that will be true when we have what we want.
Now, we need to check our surname, not against one thing, but against many things.
And that's why we use the any function. Because the any function checks one condition against many inputs.
And if any of those are true, any returns true.

The concept of multiple Nobel Prize winners discussed.

[15:53] Okay. So if she had won twice, for example, that's why you want any?
No, no. So the thing we are searching is a Nobel Prize.
It contains one year, one category, many laureates.
No, it's got lots of categories. No, a specific Nobel Prize. A specific.
You have taken the array of Nobel Prizes and broken them into multiple dictionaries.
And every time we loop through, we are processing a single dictionary, which contains a year, a category, and an array of laureates.

Explaining the process of processing a Nobel Prize

[16:35] Okay. I think I see what you're saying. Right. So we are processing a Nobel Prize.
Okay. So there is a Nobel Prize for 2023 in chemistry.
There is a Nobel Prize for 2023 in physics.
Yes, because when you look at that array in the raw JSON file, you will see it's open curly bracket, some stuff, close curly bracket, comma, open curly bracket, some stuff.
So each of those is a separate dictionary. Okay.
So the select is happening once for each dictionary.
And the question being asked is, does this one dictionary I'm looking at meet a criteria?
If the answer is yes, you continue. If the answer is no, you evaporate into the ether.

[17:22] All right so we are and what you used was any at this point and why wouldn't you want all like what if she'd won twice okay but annie will catch twice if i give you five playing cards and i say do any are any of these black if two of them are black it's still black, that's what annie means right but but i would answer yes which is true two cards which is true True, right?
So any answer is yes or no, true or false.
Right. Oh, and the true or false is what tells the select to send the answer now through the next pipe. Exactly. Right?
So we are interested if any of the laureates in the one prize we are looking at have the surname Gez.
Okay. So the any function in its two-argument form, the first thing you tell it is, what do you want me to explode so that I can look at many things?
And then a semicolon, what condition do I apply to each of the things?
So we are saying, take the laureates array, and for each one, look at the surname, is it equal to Gez?
If any of the surnames are Gez, the any function returns true.

[18:48] Okay. Which means the entire Nobel Prize has gotten a true on the select.
And the select then passes the prize, not true?
Exactly. It doesn't pass true. Exactly. That is the function.
Select either prizes the original thing or nothing.
So if select, if the condition works out to true, the entire prize gets to pass.
Okay. And you said select has one argument, and the one argument in this case is the any function.

Exploding laureates within the "any" function

[19:17] Yes and the any function has two arguments one is where do you want me to look yeah and then what do you want me to look for yep okay so in english if any of the laureates have a surname gets the entire prize gets to continue to the next stage so even though we did explode laureates, but we'd exploded it inside the any right so at the very top level, right at the top level we are still in we're the select function is working on the whole prize, okay you did not do an example that covered this by the way it's the last thing you say but you don't have an example that that i do actually because i show it not working because we can't see what marie curie got her prize for and then we show oh yeah and this is actually what marie curie got her prize for we explicitly do it because otherwise we can't tell why she got her prize.

The Mystery of Marie Curie's Nobel Prizes

[20:17] First time we do it we just get told Marie Curie Marie Curie Pierre Curie and we don't know why they got their prizes and the second time we do it we see the two prizes come out.

Understanding the Select Function

[20:28] I read your show notes at least six times, and I still didn't get that from it.
This, to me, looked like we exploded it, drove down inside, because what we returned was stuff that was inside laureates.
It wasn't up above laureates.
I thought once we exploded it, we were inside it.
I'm not sure how I'd know when I'm not inside it. Well, it's right.
So at the top level, what is the select working on? What is to the left?
The select is the one thing in that second filter in the chain.
So it is receiving whatever the filter before it produced.
So prizes, open square bracket, close bracket is sending a full prize dictionary.
So select is working on the prize. whatever happens inside select is inside select whatever's going to fall out of the other end here is based on what select received so select will either pass exactly what it got in or nothing, so what it got in is the full prize so either the full prize goes through or nothing goes through, okay yeah all the words that you're saying you did i'm remembering pieces of that i also said it was very difficult to keep your head around it because i'm asking you to imagine.

[21:52] What the current value is and that takes some getting used to yeah yeah okay so at this point we have successfully uh exploded prizes we've gone we've using the select we've gone through every prize that looked for any laureates with surname Gez.
But now you're going to pipe it through and you're going to do some more stuff.
I think it gets easier after this. It does get easier, but before we go further, can you run that without the remaining pipes?
Just run that .prizes pipe select close bracket, close bracket.
Just run that bit of it and then see what we have.
I can't. Oh, come on. Can I select the right window?
I believe you can. Whether you shall, I don't know.

Exploding the prize dictionary into its components

[22:44] No, not with the way it's... No, I've got to get some quotes and stuff.
There's one single quote that's left in the second bracket and the name of the file.
So yes, I have run that. and it came out with the prize, the 2020 physics prize, but it's got Penrose and Ginzel in there along with Andrea.
Right. So, but that is the entire prize dictionary.
And so now we're left with a much simpler problem.
I just wanted to know, what did I want to know? I wanted to know the year, the category, and the motivation was the little curve ball I threw in.

[23:19] So if we want to get the year. The motivation was the easy part, Bart, because I kept being inside there. I knew what the motivation was.
It was the category and the year that was hard because that was up above.
But now that we're not inside there yet, right? We're not inside there yet.
So now the category, now the motivation is actually a curveball.
So if we want to get the year, that's easy. We pipe dot year.
And that will just tell us the year.
And also with a comma. Bing, bing, bing. And also the category.
Great. That gives us two out of three. And we know we have an and also because I've told you I want the third thing. So we know what the next thing is going to be, comma.
But then the question becomes, what comes after the comma?
Well, what we need to do now is we need to actually go in, not look in, we need to go in to the laureates, get just the one for Andrea, and then get just the motivation.

[24:14] So we need to use brackets to say, everything I'm about to do here is one little piece.
Is that third thing? It's that third thing, right? So we wrap it all in brackets.
And then we start and we say dot laureates, open square bracket, close brackets.
So we are now actually exploding the laureates from the one prize that is Andrea's prize. Right? Because that's all that's left now.
Okay. So what's in here now is Penrose, Ginzel, and Gez.
Exactly. And then we pipe that to select. And select is now being handed a laureate.
It's no longer being handed a prize.
It's now being handed a laureate because we've exploded the laureates.
And then it's looking in with the simple one argument version of select, which means it's going to go through its input and just look for a simple property, dot surname double equals guess.
Because we don't have to dig deep into this Lariat anymore.
We're in the Lariats. They only have names and motivations. So there's nothing deep here to go in. We just say, give me the surname equals guess.
Ah, but what you've really done, you've selected, So you're looking...

[25:21] So there were three laureates, and only one of those three is going to make it through the gauntlet.
So what order are they in? So who's first in the array?
It's Penrose, Gonzalo, and then Gez. Okay. So the first time that that select happens, it looks and it goes, dot surname double equals Gez, Penrose.
False. He evaporates. Penrose is gone. Poof.
Second time through the array, it's whoever the second name is.
Evaporates. Poof. Oof. Third time's through. Guess. Yes.

[25:54] So now... So at that point, what is the input to the next pipe?
Is it that third object inside the prize that was physics for 2020?
Yes. Because dot laureates has exploded into its separate dictionaries for each winner.
And so we've thrown two of those dictionaries away. way and so we are now left with the dictionary with first name andrea surname gez and the motivation is what we really care about so when you say select dot surname double equals gez, with this one prize the 2020 physics you've got one prize it's one element in this giant array of prizes when you say select dot surname gez how does it know to take that whole object I thought it would have just surname Gez at that point.
Okay, no. So that is the condition, not what, that is how you want to look, not what you want to get back.
So what you get back is whatever was to the left of that pipe, right? So you're piping something into select.
So what is to the left of that pipe? You haven't said select any, you've said select surname.
Correct. You said, I want surname Gez. No. So all you should have is surname Gez.
No, that's not saying what you want. That's saying what the condition is.

[27:14] So it must be saying any object or dictionary sorry any dictionary within laureates within this little piece of laureates that has guess in it returned the entire dictionary yes whatever it is weird but that is that is the fundamental leap select returns the exact thing it was given all of it or nothing so we are exploding the laureates so the thing select is given is a dictionary with the keys surname, first name, motivation, and ID.
So the only thing that can come out of that select is that dictionary or nothing.
Whether or not it gets to come out is what's determined by the condition inside the parentheses.
It's only saying whether or not it passes. It's not filtering.
It's not extracting information. It's gating the information.
Either you all get through or nothing gets through.

[28:12] Hmm. Right? It's a condition, not a select, not a cherry picking.
It's the condition for letting it all through.
Yeah. That's odd. It is odd.
And I think that's what's caught you up. I think that we have now hit the nugget of what has tripped you up the entire way. No, I was stuck long before this part.
That's not where there's a whole other penny still floating.
But I wasn't even this far in, remember. I...
I couldn't get to this point, to where I would be confused.
I was not yet confused on this. It is the same penny.
Maybe a penny with multiple sides, so maybe it's not a penny, maybe it's some sort of a die that's falling in something, there's six sides, but it is, they are very, very connected. I guess it's sort of, yeah, I guess it's sort of similar, yeah.
Okay. All right, we're done with question number one in 28 minutes, 29 minutes. Bart asked me up front whether maybe this would be a two-parter. I think it will be.
Right, but it's important, right? So the reason we do the challenges is so that we can have these deep conversations, because it was easy to write that, but actually the fundamental questions are deep.
There is a lot going on here.

[29:28] So I don't think it's time misspent. I think it's time very, very well spent. No, no, no.
I mean, whether it was or not, I needed to have this conversation.
I hope the next one's easier.
I believe it is, actually. So the next question is, how many laureates were there for each prize? And I want the year, the category, and the number of winners.
So again, we're going to start with, well, we need to get each prize one by one.
So we're going to start our train of work by exploding out the dot prizes dictionary into its little pieces.
Sorry, the dot prizes array into its pieces. So arriving at the next step, well we want the year, well every prize has a year so we can just take it.
Year comma and also we want the category.
Great that's easy. Category comma and also.
Now we want how many winners were there.
So we now need to actually count the length of the laureates array. array.
So the answer we now want as our third and also is a length.
And I've told you that the length function expects to be passed an array.
So we say dot laureates pipe length.
And we just have to wrap it in brackets so that that whole, so that that whole thing becomes our third and also. Roundy brackets. So we're grouping those together.

Problem with .Lorets not exploded, need to define explode.

[30:49] So I have a couple of problems here. Why is .Lorets not exploded?
Because .Lorets pipe length. Okay, so the length function, by definition, needs to receive as its input an array.
Not a piece of an array, the whole array.
So if you explode the array, you do not have an array. You have pieces of an array. Maybe you need to define explode, Bart.
I thought explode meant open it up and let me look at it.
No, open it up means, okay, so remember what I said was that each filter has one or more inputs and produces one or more outputs.
And the number of outputs can be different to the number of inputs.
So when you explode, when you have a filter that explodes an array, the input is one array.
And the explode, square bracket, square bracket, means I am giving you many outputs.
So the next filter in the chain happens in parallel.
Not in series, in parallel.
Once for everything in the array. No, but it's not an array anymore.
It's not an array. You're telling me because I can't ask the length of it once it's exploded, so it can't be an array anymore.
Which is why I'm saying don't explode it. If you want the length of an array, you do not want pieces of an array. You want an array.

[32:15] Okay. Okay. So tell me, so this now, and saying this more quickly in one piece, dot prizes, double square bracket, pipe it, dot year, and also dot category, comma, and then open roundy brackets, dot laureates, pipe length.
Yes. Those roundy brackets. So that, what do you expect dot laureates pipe length to return?
The number of laureates. Is all, the number of laureates in what?
Okay so remember a year by year is that what you mean no no go back what is to the left of the pipe, there's two pipes okay the main top level pipe we have dot prizes explode pipe so everything from that pipe until the end of that string is working on.

[33:12] Is happening once for every element inside the prizes array.
So there is a dictionary with a year, a category, and laureates at the first element in the array.
And there's one in the second element of the array. And there's one in the third element. So everything from .year to the end happens once for every single prize.
So the first time through the loop, the year is going to be 2023.
And 23 the category is whatever's first in the list and then the array of laureates is okay so how many laureates are there in the first prize? I don't know.

[33:48] Three. Okay, then laureates would be three. Yeah, it's that array of laureates that's being checked.
And then it does it again for the next prize, and again for the next prize, and again for the next prize, and again for the next prize.

Exploding prizes, finding prizes won by one person.

[34:02] Okay, so it's showing, I'm looking at the bottom of the answer, but it says 1901 literature one, 1901 piece two, 1901 physics one, 1901 medicine one.
So that's one person won the medicine prize in 1901. Yeah. Okay.
Because it's happening once for every prize.

[34:25] So this explanation we're having here is not in the show notes, so I'm wondering what we're going to...
I will be asking this again because I won't be able to... This is a rephrase of last week's show notes.
The text of this is last time's show notes. We're saying it in a different way, so maybe I'm misunderstanding you.
You well i'm just saying when i look at these answers i'm not going to know why if when i go back to these show notes but because you say it's in the other show notes but i'm telling you bart i read it and read it right i believe you but i didn't get it at all so uh i understand it right at this moment if i know to look at this moment in time in the audio i need to understand it again, so uh i don't know how to say it differently to how i said it last time this is the problem i'm I wonder if I could put notes to sections or something in there to say look back at this piece but maybe that's my problem all right what's what's question number three okay so the the last question then is well so we know because we've just done it that sometimes there's three winners and sometimes there's two winners and sometimes there's one winner so I wonder which prizes were won by one person who was good enough to win it all by themselves so which prizes were won by just just one person, who got to win, you know, which prizes were given outright.

[35:45] And what we would like to know is the year, the category, the first name, the last name, and the motivation.
So you won it outright. Okay, so who were you? When did you get it? What did you get it for?
So as always, we start by exploding out our prizes.
So the next piece of the chain is going to happen once for every single prize.
And what we're interested in to decide whether or not the prize continues to the next step in the process is, do you have exactly one winner?
Right. That is our criteria for continuing.
We start off with all the prizes and we only want the ones that have exactly one winner.
So we have a select. What is the condition we want?
Well, we want the laureates pipe to length to be exactly one.

[36:34] But you put a question mark on L'Oreal suggesting don't look at the null ones, but the null ones would pass as not having a length of one. They would.
So why do you have to be specific?
Because otherwise you would get an error, because the length function will receive an input that it doesn't like.
It will receive null, and it doesn't like calculating the length of null.
Well, but null doesn't meet the criteria of length double equal one.
So it would be false, and it would just skip right over it.
No, based on the documentation, the length function insists you either give it genuinely nothing or you give it an array.
If you give it null, null is not an array.
Null is a value. The length function will throw an error if you give it null.

[37:18] So why not... Oh, was it only laureates that are missing? But aren't there years there's no prize?
Or did the prize go somewhere, just not to any laureates?
The prize exists as a dictionary.
But it contains no key value pair with the key laureate. Okay.
So putting the question mark on the prizes wouldn't do you any good.
It would not because the prize, the prize is a dictionary and it contains a year and a category.
And then it doesn't contain laureates. Got you. Got you. Okay.

Question about missing laureates and prizes without laureates.

[37:52] So that's why you have to put the question mark after laureates.
I'm going to remember a question I want, but I don't want to interrupt this anymore than I already have.
So we've got dot prizes piped. we want to select dot laureates with a question mark pipe equals pipe to length double equals one so at this point it's laureates pipe length i thought it's just.

[38:15] The length function works on the current input so how do we give it current input okay we pipe something to it okay so after that select function what has happened is we started it off with all the Nobel Prizes and now all that is left are the prizes with a laureates array of length 1.
So we now have the full dictionaries for each of our answers but I wanted you to get a bit more specific and hand me the year, the category, the surname, sorry the first name, the surname and the motivation.
So we know we want the year so dot year comma and also dot category comma and and also.
Now, I want the firstName, but Lariat is an array.
But I now know, for a fact, because I've proven it, that it is an array of length 1.
So how do I get the firstName from the firstLariat?
I say Lariat, open square bracket 0, close square bracket dot firstName.

Exploring data structure and accessing specific elements

[39:21] Oh, because the next time through, it's a different laureate, and it will also be in position zero. Yeah, exactly.
What would happen if you just said laureate dot first name instead of laureate square bracket zero close square bracket dot first name?
Null, because laureates is an array, not a dictionary.
Otherwise, you'd have to loop through it or something again.
Right, but we're looping through a thing of one. But we don't need to because we know it as one.
Okay. Yeah, we just want the first name from the first element in the array.
Okay. And we want the surname from the first element in the array.
And also, we want the motivation from the first element in the array. That is our answer.

[40:07] That one actually makes more sense than the other two. One of the things that bothers me about all of this is that it appears that you have to understand, understand everything about your data set before you start doing any of this.
So when I started this, I said, I don't know what the data set looks like.
I don't know where, what, I know it's prizes because Bart's been saying it over and over again in his example, but I don't know what the structure is below that.
And I didn't, I knew I could open up the file and maybe pretty printed or something, but I wanted to know, how do I find out what is the structure of the data I have? And I had trouble finding a way to do that.
I invented one. I just said, jq.prizes0 NobelPrizes.json Just show me what the first one is.
How do you know that you have to put a question mark in? You only know because it's screwed up. Because you've got a null.
Correct, yeah. And that is the only way you know that? Is when your data misbehaves?
Either you scroll through all the data by pre-printing it or you deal with problems when they arise.
And so we have a few times in the show notes, like last time we had, in the last episode, we had an interesting example where we first discovered, and I say we, I, while writing the show notes, first discovered there are not always laureates because I got an error.
So I then wrote a JQ query to show me all the prizes where the laureates is null.

[41:35] And that listed out all of those prizes. And that query was in the show notes as an example of how we find out which prizes don't have laureates.
And that then showed us the structure, pretty printed.
And I was like, okay, so that's what my data looks like. And then I adjusted.

[41:52] I feel like you go in blind to start with, though. I stumble around in the dark.
Yeah, and then pretty print the pieces I want. But I mean, it seems like there ought to be a way to tell me what does it look like.
But I didn't want to look at this thousands of line long JSON file.
The whole point of this is if, I mean, if it's short enough that I can scroll through it, then I don't need to do any of this nonsense. I can go, oh, there's Andrea.
Right, but picking out element zero is your answer. Okay.
That is your answer. And the other thing is at every point in the pipe, you can just stop, let it print out what it currently has, and then go deeper.

[42:29] Every time that you're meeting a pipe, just stop there, run it, and see what's coming out at that point in the chain.
And then the next time you put a pipe, you're going to filter it down further and filter it down further. So print as you go.
Right. Again, back to the problem is I couldn't get into the first part of it.
I was stuck at the beginning. Okay.
All right. We're ready to start new stuff, Bart. We're definitely going to have to split this. That's a long scroll bar coming up. Okay.

Transition to discussing data types and type conversion in JQ

[42:58] Where are we now? It is now 22 minutes. We're 42 minutes in.
We're 42 minutes in. Let's go like 15 minutes and see where we are.
Okay. So we spent a lot of time last time talking about data types, right?
So JQ is a language for querying JSON. so we discussed the fact that it has inherited, very cleverly, JSON's approach to types.
So there are strings, numbers, booleans, arrays, what it calls objects, what we call dictionaries, and the special value null.
And one of the things you can do to try to explore these things is there is a function named type which will tell you the type as a string.
So if you pipe something to type, it will give you back one of the following six strings.
Null, Boolean, Number, String, Array or Object.
And so you can use that to probe what something is.
Or you could use it inside your queries. Say select whatever, and the condition could be type double equals number.
To say basically, unless this thing is a number, don't pass it on to the next element in my chain.
So, select can be thought of as an if.

[44:21] Yes. If you find this, then pass it along. If that works for you, yes.
Okay, okay. The other way to look at it is as a filter. So, it either goes through or it doesn't go through. But either way, yeah, that's perfectly fine.
We call it an if if you like. Yeah, a conditional pass.
Yeah, whatever works for your brain.
So, that is, how do I tell what it is? the type function will tell you it's type great that's useful but we also know that all of our equality and stuff is done through strict typing so the string 42 and the number 42 are not the same because one of them's a string and one of them's a number so the double equals operator will say nope you're not the same so it would actually be darn useful to be able to convert from one to the other, And that is also true when we have operators like less than, greater than.
So if you give less than a pair of numbers, it will do a numeric check.
But if you give less than a pair of strings, it will do an alphabetic check.
And in alphabet land, 23 is greater than 2001.

[45:42] Because alphabetically, it comes after 2001. Because 2, 3 is greater than 2, 0.
So having numbers as strings and then doing a comparison will cause logic to cease to exist.
And we have this in our data set because our years are all strings.

Converting between numbers and strings using "toNumber" and "toString"

[46:03] Our entire JSON file of Nobel laureates is full of string numbers for the years.
Now, if you're always dealing in four-digit numbers, the fact that it's alphabetic doesn't make any difference. But if you try to say, what are all the Nobel Prizes after the year four?
Well, that's going to do very odd things, because it actually thinks none of them are after the year four, even though they're all really a long way after the year four.
So we need to be able to convert these things. And thankfully, JQ provides us two very sensibly named functions, toNumber and toString.

[46:36] And they will take the input, and they will convert it as they describe.
Or they will throw an error, if you give them something they don't like.
So, if we say 2001 pipe to string, double equals the string 2001, we will get a true.
Because the number has now been converted to a string, and so now those paths are strict.
Type check. So let me remind myself what you've written says jq minus n and then you've done this two string thing.
The minus n means don't expect an input there's nothing coming in from the left hand side.
Yes there is no input here. It's like null for some reason?
I think it's short for minus minus null input Okay. But minus n is easier Yeah.
Okay so jq minus n and then you've got inside single quotes because we single quote all this stuff stuff, open parentheses, 2001, pipe, two string, close parentheses.
So that's become one thing now.
That's the input, double equals, quote, 2001, that would be true.
Because we turned 2001 into, double quote, 2001, which is a string, so it's true.
Okay. Yeah. So we can say we've turned 2001 into 2001.

[47:52] And we've compared it to 2001, and they are indeed the same.
If we take the number 2001 and we double equals it to open our bracket the string 2001 pipe to number and then close that off we also get true because now we're comparing a number to a number and that is yeah those are indeed the same if we take the string 2001 and we pipe it to number and we say are you greater than the string 2 3 piped to number we now get the much more sane answer that 2001 is indeed greater than 23 because we've converted them both to numbers before we did the comparison and then we get the expected true.

[48:39] Okay. And if you try to convert something that is not a numeric string to number, JQ will get very cranky with you.
So if you take the string waffles and you pipe it to number, you get invalid numeric literal error.
Because waffles is not a numeric literal. Yeah, it does make sense.

Select-like functions filter data based on their type.

[49:01] The other thing we get out of the box is a bunch of, I call them select like functions.
They will take the input and they will just check its type.
They won't check to see whether it's equal to something else they'll just check its type and if the type is within is what it's supposed to be it gets the pass through otherwise it vanishes into the ether and these are does exactly what it says on the tin functions.
So they're named for what they filter. So to test these we're going to use a new data file called sampledata.json that contains a massive array of lots of random things.
It is an array that contains null, true, minus one, zero, eleven, three point one four one five, the string four two, the string waffles, an empty array, the array dogs comma cats, an empty dictionary and the dictionary apples colon.

[50:00] So that is a nice sampling of all of our different data types that Jason...
You just swept some data up off the floor, shook out the vacuum cleaner of data, got it.
Yeah. So the question becomes, how do we get only the things that match a certain type easily?
We could put it through select. So we could say, select, open bracket, dot, pipe, type, double equals, string.
And that would be a big select, and we could do it, right? Because we know that the type function will convert our type to a string, and then we can do a double equals.
But we don't have to. We just get these free functions that do that job as a shortcut.
And the first of them is nulls.
Whatever you pipe to nulls, if it is null, it goes through. If it is not null, it vanishes into the ether.
So if we say jq explode our input, so dot, open square bracket, and we tell it sampleData.json.
It will take that array, and once for everything in the array, it will run it through the nulls function, and all that will pass is things that are reading null.
So the entire array becomes one output, null.

[51:12] Everything else has been filtered away. Very boring one.
Well, okay, so the way you typed it in here, it says hash null. null.
So it's not actually going to say hash. It's just that you did that because that's a comment. So it's going to say null.
What if the second element in that array was null? Would it have said null comma null?
No, because it literally, well, no, because you notice it's not null comma blank, blank, blank, blank, blank.
It literally, like the select function, it is literally gone.
The stuff that doesn't meet the criteria is gone.
No, but I'm saying, what if you If you had two null in your array, what would the answer have been?
It would give me two outputs, null followed by another null.
Okay, that's what I asked. Yeah, okay. Yeah. Now, so nulls is kind of boring.

The "arrays" function filters and returns only arrays.

[52:01] It's like, okay, give me all the nulls. I guess you could count them, right? You could say pipe it to nulls and then count them or something. I don't know.
It's a very odd thing to want to do, but hypothetically, right?
What's much more interesting is the opposite of nulls. It's the values, which is everything that isn't null is a value.

[52:18] So the values filter will throw away the nulls, but let everything else pass through.
So when we take exactly the same structure, we say explode our input array, which is our big glop of every data type Bart could think of, and we pipe that to values, what comes out is true, minus one, zero, eleven, 3.1415, 42, waffles, empty array, the array dogs and cats, the empty dictionary, the dictionary apples 12 pairs 3.
The only thing that didn't come through was null, because null is not a value.
Well, null came through as... Oh, okay, I see.
So you don't list values as one of the select-like functions.
Oh, it is the second one. Okay, there it is. Okay.

[53:02] So values just means everything that's not null. Got it. Yeah.
Okay. The next one is Booleans. It only lets through true and false.
So in our data set, that means only true comes out.
The other one is numbers.
Numbers lets through minus 1, 0, 11, 3.15, 4, 1, 5.
It does not let through the string 4, 2.

[53:26] Because that is a string, not a number.
It makes sense, you say, but it's important to point that out, because I'm not sure that's obvious.
Well, it doesn't mean I'm not going to look at quote 42 and see the number 42.
Right, exactly. So it's worth pointing out.
There is also a function named strings, which unsurprisingly will give us 4, 2 and waffles from our big glop of data.
Another very useful one is arrays. It will give us all of the arrays.
In other words, the empty array and the array dogs, cats.
And it does return the empty array. The other one we have is objects, which gives us all the dictionaries.
It gives us the empty dictionary and the dictionary apples, 12 pairs, 3.
And then another interesting one is iterables. It gives us everything that you could loop over.
In other words, arrays and dictionaries.
So basically iterables are things that have more than one thing in them.
And the opposite of an iterable is a scalar. A scalar is a single-valued thing.
So the scalars are null, true, minus 1, 0, 11, 3.1415, the string for 2, and the string waffles.
I don't like that. You don't like that? Redefining the word scalar. No.

[54:44] Yeah, it's a programmer's definition, I guess. It's quite common within programming languages, but it is a programmer's definition of the word. I can see why the physicist, the engineer... It's supposed to mean it has magnitude, not direction.
It doesn't have anything to do with... It's like...
I don't like that. It's also a one-dimensional vector.

[55:04] Which in this case it is. Well, one-dimensional meaning not having direction. A dot?
Well, in math... It's the opposite of a vector. Yeah, exactly.
It has one dimension. It has a vector as direction and magnitude, yeah. So what...
What would happen if you put in waffles, not in quotes? It would give an error because that is not valid, Jason.
Okay. Gotcha. All right. Okay.

The "alternate operator" handles dirty data in JSON files.

[55:33] So the next thing I said we would have on our menu today is one new operator that exists.
And this operator exists for the sole reason that everyone's JSON is dirty. Dirty data is the norm.
And it's not always bad dirty data. Sometimes it's efficiency.
But the alternate operator lets you say, do this. And if that doesn't work because it doesn't exist, do this instead.
Did, which is very common to want. So very common examples are, there are data structures.
This is very common in Active Directory, where you could have one phone number.
Everybody might not know what Active Directory is, Bart.

[56:14] Okay. I only know because I worked in a corporate Microsoft world.
It's a directory of the users in a system.
And there is a thing equivalent to that inside your Mac, called LDAP.
But anyway, it is a bunch of, it's a giant big dictionary describing every person in an organization.
And if a person has one phone number, then LDAP or Active Directory will give you back a string when you query it.
If you have two phone numbers, it will give you back an array of strings.
It doesn't give you back an array of one string if you have one phone number.
It says, says, oh, it's just too much effort, I'm going to give you one phone number.
So that means that you're constantly and continuously dealing with the possibility of this could be a string, or it could be an array.
And so you always have these two possibilities flip-flopping inside your head.
So that's a very common approach that you'll see in lots of places.
A multi-valued attribute comes as a string or an array of strings.

[57:16] The other one we have is you may simply have keys that are optional.
Some users have a smart card.
Some users don't. That entire key could be optional. So it may not exist.
In our data set, a Nobel Prize where there were no winners has no laureates array.
So the laureates array does exist sometimes and doesn't exist other times.
But when the laureates array doesn't exist, there's actually a different thing that exists. It's the string overall motivation.
So that's actually a really good candidate for the alternative operator operator because either you have laureates or you have an overall motivation.
So that's a particularly nice example.

[57:59] So the way the operator works, so the operator, you're going to love this.
You're going to absolutely love this.
The operator is slash slash, which your brain, because you've grown up in JavaScript world, thinks means comment.
No, it means alternative. Alternative.
It's going to drive you absolutely mad because you're going to look at people's code on GitHub and you're going to think, oh, it's just a comment.
They've commented it out.
Nope. Nope, nope, nope, nope, nope. It means alternate. Well, we know JSON doesn't have comments, though. We do know that. Annoyingly.

How the // Operator Works

[58:34] I know. Very annoyingly. So the way it works is that it is an operator, so it has something on its left and something on its right.
If the thing... So when you have the slash slash, the very first thing that happens is whatever is to the left gets done.
And that will produce either null, sorry, that will produce a value, right? Whatever's to the left is going to have a value.
If that value is null or false, then the null or false are evaporated into non-existence, and whatever is to the right is evaluated, and that answer is the output.
So what that means in effect is, try what's on the left. If that gives you anything that's not false or null, then the thing on the right never happens.
If the thing on the left gives you false or null, do the thing on the right.

Cheap person's if-else implementation

[59:26] So it is kind of a cheap person's if-else.

Exploring Data with JQ

[59:33] So either the thing on the left or the thing on the right. what's still bothering me is, how do you know that this is happening inside your data the only way you know is by it not working right and then you can query your data and just look at the data so explode the dot prizes, and look at it and scroll and scroll and scroll and scroll and scroll and scroll or put it through select and put in the criteria so you're saying well I think sometimes that we don't have a laureates array so then you select dot laureates it's double equals null.
And then all that will come out of that pipe is all of the Nobel Prizes where there is a null laureate.
And then you can see the data. So you can use JQ to explore your data.

[1:00:19] You should describe what your code says there.
Well, I haven't even gotten to my code yet. So you asked me the question.
I was answering your question rather than trying to get onto my example. Okay.
So if you don't... Right, yeah. So you run into an error.
So then you can use JQ to figure out why you've run into an error, by using the select to look for things that are null or whatever.
So you actually can use jq to answer the questions about the data.
Which I'm not sure is the answer you were hoping for me to give you.
I think you have... No, because you have to know what you're looking for in order to query for it.
Right, but you can look for null. But you can look for null.
Well, it's going to tell you null. It's going to tell you null in the error.
So then you're going, okay, well, what is it?
So this failed because it's saying that the laureates are...
Can you query for null? Double equals null, yeah. That is a condition that will work. Yeah, okay.

[1:01:08] So my example of the alternate operator is to...
So for every prize, either there are laureates or there is a reason there are no laureates.
Which, in terms of our actual data structure, is either there is a dot overall motivation, which is the reason there are no laureates, or there's an array of laureates.
And so what I would like to print out is the number of winners or the reason there are none.
It seems like a valid thing to do. Either show me how many winners there were, or why there were no winners.
And so to do that, we take our dot prizes and we explode it out and then we pipe it into the alternate operator.
To the left, we have dot overall motivation.
So if there is an overall motivation, whatever's to the right will never happen.
So on the years where there are no prizes, it will just tell us why.
Most years do have a prize, so most of the time the thing on the left is going to evaluate to null, which means the thing on the right is going to happen instead, which is inside brackets dot laureates pipe length.
So give me the number of laureates. So the output is going to be 1, 2, 3.
There was no prize because of World War I. 4, 5, 1.
There was no prize because of World War II. Whatever the motivations are.
The overall motivations.

[1:02:26] Okay. This is interesting. And the reason I'm pausing as I'm listening to you is it's almost like this, what is this thing called again?
The alternate operator is almost like it undoes this concept of, I'm going to take everything and send it through if it meets my criteria.
Otherwise I'm going to vaporize it. In this case, it's going, no, no, no, no, no, don't vaporize it.
Give me something different if you don't meet that filter.
Filter yeah because sometimes you want to fall back to something else and sometimes you want it to evaporate so if you wanted to evaporate you just say dot laureates question mark and then all of the ones that don't have a laureates would evaporate but what if we actually do want to do something when there are no when there are no laureates right and that's what the alternate is for right if the answer is i want to do this or this well your alternate operator gives you a shorthand for do this or this very powerful shorthand your example makes me want to write more queries though and that and that's a good thing because uh what his so he said prizes explode it pipe it to dot overall motivation alternatively give me dot laureate slash uh pipe to length and so what we see is three one one one three three three so these are all the ones where there was a no there were laureates so it says for contributions to our our understanding of the evolution of the universe and Earth's place in the cosmos.

[1:03:46] Well, okay, but who won that? The next one talks about laser physics.
It's like, well, wait a minute, you had something that was a contribution for, but we don't know who it's for. I need to know more. It makes me want to run the queries more.

[1:03:59] Well, good, because that's where the and also operator comes in, right?
Because you can use that then to break out things more and give you your desired more information.

[1:04:11] So, yeah. Yeah. Now, this is a point where we can continue or not your choice as the host.

Conclusion and Teaser for Advanced Searching

[1:04:20] Well, it looks like how much more do you think we have here? We are in an hour.
I think I'd like us to call it here. This. OK, you want to do one more?

[1:04:32] If we're not going to do both, so basically we're now jumping into more advanced searching.
So we're either going to do our searching, which is actually just two ways of searching.
So it's whether or not something contains something and regular expressions.
Well, based on the fact that I've had a little internet glitch here, I'm nervous enough.
It's probably the fact that it's sprinkled in Los Angeles that I lost my internet connection for a minute there.
That I think we should probably cut it. You know, we did have a rule before that the challenge of solutions were so long that we would make them separate episodes.
But we've been barreling along so quickly and having easy examples.
I think we didn't do that. So I think we should tease this out right here and hold this one for next time.
Well, this is the perfect place to do that then. So we shall do that. Sorry, listeners.
I have no idea when you'll be hearing the rest of this because, well, yeah, because it's the silly season. So goodness only knows when we'll make this happen.
But, well, the show notes are written so I can record any time.
Yeah, exactly, exactly. Right, well, until next time, actually, and anyone who didn't succeed in the challenges, I guess maybe now is a good time to go back and do them again. Don't cheat, don't look at the answer.

[1:05:42] Try to redo them. Because it is good practice to get into the habit of understanding quite what is going on when you explode things and pipe things from one filter to another, because that's the key to it all.
So anyway, until next time, whenever it is, happy computing!
If you learn as much from Bart each week as I do, I'd like you to go over to let's-talk.ie and press one of the buttons over there to help support him.
He does 98% of the work here. I'm just the stooge that listens to him and asks the dumb questions.
If you go over to let's-talk.ie, you can support him on Patreon, you can donate via PayPal, or you can use one of his referral links.
I really hope you'll go over and help him out. In the meantime, you can contact me at Podfeet or check out all of the shows we do over there over at podfeet.com.
Thanks for listening and stay safe.

[1:06:34] Music.