Transcript

[0:00] Music.

Introduction

[0:08] What's that time of the week again? It's time for Chit Chat Across the Pond.
This is episode number 788 for March 2nd, 2024.
And I'm your host, Alison Sheridan. This week, our guest is Bart Bouchats, back with Programming by Stealth number 162.
And we're still having fun with JQ, right, Bart? we absolutely are um and it keeps getting slightly longer uh because i'm the reason i wanted to learn jq is because i needed it for my work life and i'm using it a lot since coming back from my extended leave a lot a lot and i'm discovering that there are more things to learn this this language keeps getting better um keeps getting more powerful and there was a thing i I thought would be like a paragraph and then I tried to write the paragraph and it became wash-o.
So episode 163 has completely changed. This new episode 163 just blipped into the middle of what I had already planned.
But I can see where we're going and I'm having so much fun with this.
So I'm hoping you're enjoying it too as I keep stretching it. I am.
I notice I get rusty real quickly that I forget things I knew.
So I'd just like to register here a whole bunch of dumb questions.
I mean, valuable questions to refresh the audience's memory as well, right?
Right. Yeah. I mean, that's literally your job, right? You're here to represent the audience. So, and you're very good at it, as I keep telling you.
So just to put us back into the big picture story.

[1:35] So we learned initially just the pretty print stuff, and then we learned how to query stuff, and then we learned how to filter stuff. and what we've been doing in recent installments is change stuff.
So not just build a new shape of a dictionary, but actually change things by doing math.
And by, in the last time we did math, we did assignment operators where we were changing the value of different keys in our dictionaries or different values in our arrays.
And then we had functions for messing with strings.

[2:09] And we're going to pick that up. We're going to pick that up today with the other two obvious types.
We have numbers, we have strings. so what's the next most logical thing?
Arrays and dictionaries. So we are going to learn about JQ's various functions for altering arrays and dictionaries.
So not just filtering them or creating them, but actually altering and give an array, give an array and then do something to it and then have a new array and dictionary as well.

Homework Recap and Progress

[2:36] And that will then set us up for next time where we're going to spend an entire episode talking about a special type of dictionary that it turns out is really powerful and really important and without learning what we're going to do next time i would not have been able to make use of the output from the have i been pwned api which kindly talks jason and believe it or not when you work for an organization with thousands of users a lot of them get caught up in data breaches shockingly shockingly when there's 20 million records leaked or is it 20,000 this week yeah anyway always big numbers so I had set you a challenge at the end of the last installment and what I noticed in the two weeks since we last recorded was an awful lot of amazingly cool activity on the git for xkpasswd which is kind of the excuse for learning most of what we've learned in this series so I guess that's a sign that the series is working, but I wonder, did you have so much fun doing that, that you may have forgotten to do your homework?

Working on Homework with Helma, Mike Price, and Dorothy

[3:47] Well, yeah. So when you reminded me of the homework yesterday.

[3:53] Helma and I and Mike Price and now Dorothy starting to poker head into it.

Collaborative Efforts on Homework

[3:58] And I think, did Steve Mattin do something in there too?
I know he's been working on the show notes for Programming by Stealth.

Working on Layout and Accessibility for Homework

[4:04] So he's been contributing as well, but yeah, we've been having a real ball oh, I've been working the layout stuff while Helma does the heavy lifting of the JavaScript side of it.
And then I went through and worked the accessibility pieces of it where it was pretty close, but there were some things, there were some contrast problems.

Using Accessibility Tools for Website Analysis

[4:22] And those are really hard to see on your own, but there's some tools you can employ.
There's the thing called the Wave Accessibility Tool that lets you analyze a website and it'll show you everything that's wrong with it from an accessibility standpoint.
And it highlighted the contrast errors that I can't see that looks fine to me but I don't have trouble with low contrast so.

[4:42] Anyway, it's been a lot of fun. So I did take a whack at it this morning, yesterday and this morning.
But I'm kind of glad I stopped because when I looked at your solution, it was so elegant that I was going down a rat hole of things that were not only incorrect, but really hard trying to get there the wrong way.

Reviewing Homework Solution

[5:03] So I've got a lot of questions about what you did. So I think I want to hear your solution.

Challenge Explanation for Homework

[5:08] Okay. Well, I guess we should say what the challenge was before we look at the solution. So what we had learned last time focused heavily on the assignment operators. They were the biggest new thing we learned.
So I focused the challenge on those and on doing something which I like to do, which is to correct bad JSON, because there is so much bad JSON in this world.
I really do spend a lot of time correcting other people's JSON, making it be the way I think it should be.
And so I asked you to make four changes to our Nobel laureates data set we've been using for most of this series on JQ.
So the first thing to do is to make it easier to detect whether or not a prize was or wasn't issued.

Adding a Boolean Key for Awarded Prizes

[5:49] So you can tell by two side effects whether or not a prize was issued.
It will have a field called overall motivation and it won't have an array of laureates.
And so either of those two side channels let you deduce that the prize wasn't given out. But I said, well, no, I actually want there to be a key in the dictionary that says awarded, true or false.
Because that way it's much easier to handle if that is in there as an explicit key. So the first one was add a Boolean key named awarded.

[6:18] The second thing I really don't like. Hang on. So are you going to talk about how you did it? Well, I'm going to say what we're doing first. Okay.

[6:27] Or do you want to look at them piece by piece? I was going to say the four things.

Explaining the Update Assignment for Prizes

[6:30] I'd actually like to go to piece by piece, because I understand the overall flow, and I think people will too, but all you wrote was dot square bracket, or it's prizes.
Well, let me step back a sec, yeah, because we need to jump back to a very important point, because the biggest penny that needs to drop for this assignment to be doable at all, without tying yourself in very large knots, is that what we need to do is we need to update the prizes.
So that means we're going to use an update assignment operator to manipulate the prizes.
So that means dot prizes pipe equals, which is the update assignment operator.
And then we're going to make a new value for prizes. So that means that from this point on, Oh!
Wait, a new value for all prizes? Or something inside prizes? Yes.
So... No. Okay. The top-level dictionary has one key named prizes, and that key is an array of dictionaries, one dictionary for each prize.
We need to put a new value into the prizes array, which means we have to explode the array, fix each prize, and then recollect our exploded array and put all of that back where we found it in .prizes.

[7:55] So the first thing I do is I start a square bracket to say, I am going to collect everything I do back into an array.
So the very first thing is open square bracket. And the very, very, very last thing at the bottom of my solution is the closing square bracket.
So I'm collecting all the pieces back together. I'm putting them into dot prizes.
Okay. So it's dot prizes, pipe equals, square bracket, gobbledygook, square bracket. Okay, so all the gobbledygook is going to replace what's in dot prizes or add to what's in dot prizes? Replace.
It's a new value for dot prizes. Because it's an assignment operator, not a, what's the other one called?
Well, we are using the update assignment operator so that we have access to the current prizes, which are now in dot.
So the current value of prizes are in dot. So if it's update, why is it not, how is it replacing everything?
I would think update would be additive, no?
No, no. Update means you have access to the current value, which you are going to be changing in some way.
So we're going to change everything in dot prizes. Zoom out a little.
So plus plus is an update operator.
It takes, it is based on the current value, but it is a whole new value, right?
If you increment three by one, it's a whole new value four, but your starting point is the old value. So it's an update assignment.
So our starting point is the original array and our finishing point is a whole new array.

[9:24] Okay. All right. I'll buy it. It seems counterintuitive, but okay.
So we've got dot prices, pipe equals. We're going to update assignment.
We're going to create a whole new array that's going to replace everything that we had.
Right. What makes it an update assignment is that from everything after that equal sign, the dot operator is the current value of the prices array.
So when I, on the next line, say dot, open square bracket, I am exploding the currently existing prices.

[9:55] Okay, I'm with you so far. Right. So now we're working on individual prizes because we've exploded them out.

Adding an Awarded Key for Laureates

[10:01] So now we can do the first part of our challenge, which is add an awarded key.
Okay, here's where I get stuck right away. Great, good. Okay, so you've exploded the array.
We've taken in dot prizes. We said we're going to update it.
We've exploded dot, which is now our new prizes.
Then you say pipe dot awarded equals has laureates.
How does .awarded equals create that key?
Okay. It is, so the equal symbol is a normal assignment operator.
So its job is to make keys.
It's to make or update a key. So if there was an awarded key, I would be replacing that value with a whole new value.
Or if it didn't exist, I'm making it. So I am literally creating.
So like you would, you don't, it's declaring and assigning all in one.
That's cheating. That should be, it should say, let dot awarded equals.
Like, I was sitting there going, well, how does he already talk into dot awarded if it's not there yet?
So just dot awarded equals says, if you got one updated, if you don't, just make it.
Right, because effectively everything exists with the value of null.

[11:15] Think of it that way, right? Every possible variable always exists.
No, no, no. That's my mole head hurt. Okay, fair enough. Everything, is that like philosophically in life Everything exists with the value of null.
That's probably very deep. I'm sure there's a philosopher somewhere can tell us that's very deep and worth an entire thesis and maybe a PhD.

[11:35] I'm going to use that in my Mastodon post to say, think deeply about this because we did.
Okay, so now I'm with you. So we're adding this key.
We already have everything that's in dot prizes. We haven't lost that stuff.
That's all there is. We haven't done any selects or any nonsense.
So we've added dot awarded equals has laureates and has laureates is is a boolean apparently, it is a boolean and i don't know why i wrapped it in an extra set of brackets oh i know why because i did that a long way and then i made it a short way so that extra set of brackets is entirely superfluous okay extra brackets don't bother me i'm okay with them they mildly bother me but yeah okay i'm okay with you being bothered okay so now we've got a key it looks at whether whether there's laureates, if it has them, it's going to set the key.awarded to true. If not, it's going to say false.
Correct. Because if you read the docs on has, it returns a Boolean, which makes sense because you would normally use it after an equals or something, right?
You'd normally use it in a condition of some sort. So it spits out a Boolean.
So that's one out of four.

[12:39] The second thing I said was, I really don't like it when important keys may or may not be present.
So in my mind, there should always be an array to hold the laureates.
If you don't have any laureates, it's an empty array. It's not no array.
Because that just makes you always have to put in question marks and all these kind of silly things everywhere.
Just make it an empty array. It's two bytes of data, an open square bracket and a closed square bracket. and it's not going to break the bank, right?
But Bart, everything exists with the value of null.

[13:14] Anyway. I'm going to keep bringing this up. Okay, so the way you do that is you say dot laureates and they use the, that's a ternary operator?
No, not the ternary, the or, the or one.
No, that is one we learned about last time. That is the conditional assignment, which means if laureates does exist, do nothing, If laureates doesn't exist, give it the value whatever's on the right.
So that allows us to set laureates where they don't exist become the empty array.
And where they do exist, it gets left completely alone.
If you take those two slashes out, you delete all the laureates.

[13:56] Okay. You know, when I went back and reread and reread and reread what slash slash equals is supposed to mean, and I don't get that from the description in the show notes from last time, that what you said makes perfect sense.
And I'm sure you said it last time, but it doesn't jump out at me.
So that's why I didn't catch it. Okay.
All right. So now we've said if there's no laureates, put it there.
If not, just leave it alone. Correct. So that's two out of four.

[14:23] So the third thing then was to go to descend.
The third and fourth things are inside our laureates because they need a bit of cleaning up too.
So that's cleaned up the top level dictionaries for the prizes, but the laureates also have a bit of a mess.
Now, we can use the same pattern we used for the prizes for the laureates.
So we say dot laureates pipe equals, open a square bracket, explode the laureates, work on them, and put them back together.
So we're now going to say that inside our prizes, we're now going to explode out the laureates with this update assignment so that we can put them back together and stick a new laureates array back where we found it.
And again, we're exploding the laureates, so now we can work on each laureate one by one.

Creating a Boolean Key for Laureates

[15:10] And the first thing I ask you to do is another Boolean key named organization to tell Tell us whether or not this prize was awarded to an organization.
Because again, the only way we can tell at the moment is by side effect, which is, do they have a surname?
If not, then they're an organization, which is very messy. It's a very unhuman readable code.
So we're simply going to say dot organization equals has surname piped to not.

[15:39] Yeah, what? That was my next question. Well, to not what? What?
If it has a surname, it's not an organization.
That's the rule. If it has a surname, it's a human. So to make it be the opposite, we knot it.

[15:55] So what does it mean? .organization equals has surname piped to not.
Right. So if the laureate has a surname, has surname is true.
But that's the opposite of it being an organization.
Because an organization has no surname. So we need to invert the true to a false to get the right answer. So has surname piped to not means doesn't have a surname.
No, it means if you have a surname, the answer is false.
So .organization gets assigned the value false if you have a surname.

[16:36] So hasSurname will return true if there is a surname.
So are you sending the organization to the surname? No. Okay, so hasSurname returns a Boolean.
So you now have Boolean piped to not.
So the outcome is going to be another Boolean.
And I said, has surname piped to not means doesn't have a surname, and you said no.
So I'm still lost. It means the opposite.
The way you said it sounded like you were saying something different, but it has the, yes, if you don't have a surname, then the value will be true.
Right. So if it doesn't have a surname, has surname piped to not, if it doesn't have a surname, then organization equals false.
Shoot.

[17:29] Okay, I'll go along with you. I'll say it's true. I buy it. Organization equals true. What does that do for us?
Did we just create a key called that organization? Yes.

Adding an Organization Key for Laureates

[17:40] So the thing was to make a new key named organization, which holds a Boolean to indicate whether the laureate is an organization or not.
So we want the organization to be a new key, which has the value of true or false, depending on whether the winner was or wasn't an organization.
And the way we can tell is if they do or don't have a surname.
If they don't have a surname, they are an organization, which is why we need that flip. Yeah, yeah.

Generating Display Names for Laureates

[18:10] So okay and the last thing the last the fourth challenge was to make a new key named display name so that we would always just be able to get the name to print easily because if they're a human they have a first name and a surname and so we want to have first name space surname and if they're an organization they only have a first name so we only want to print their first name and so instead of having to put all of that logic every time we want to print a laureate let's just make a new key named display name that has that value ready for us to use and the way i chose to make the value was to make a new array which contained first name and then either surname or nothingness depending on whether surname exists or not and then to join those with a space.

[19:01] Okay which is okay it's just i find i find it the easiest way to to make the space be there or not and then we now have the last one yeah then we now have a nice name and so when you run that what you get is our very very familiar nobel prizes but each one now has a new awarded field they will always have a laureates array even if it's empty and every laureate will have a new field named named Organization, and every laureate will have a display name.
And if you were to save that to a file, you could then call it Clean Nobel Prizes, and then you could use that file as your input for everything else.

Considering Using Cleaned Data for Future Work

[19:46] Gotcha. Gotcha. Okay. Should we just save it that way and start using that one?
I'm actually quite tempted to, although what I ended up doing in the examples later is play with a whole new data set because I needed different kinds of data.
But I'm not giving up on the Nobel prizes. They have been a lot of fun.
We're not quite done with them. They will show up in future.
I think the messiness made it more useful to you. If it had been a clean data set, it wouldn't have been as helpful.
Right. Because you've got to learn on realistic data and the The world is full of bad JSON.
This I know. And ironically, my new data set also contains bad JSON. String numbers.
It's a database with stock prices. And the prices are strings.

[20:27] Stock prices. Of course they are. Getting real good at pipe to number, aren't you?
Very good at pipe to number. Yeah, everywhere, all over the place.
Right, so some new stuff for our brains.

Moving on to Working with Arrays

[20:38] Let us start with arrays. So we've done numbers, we've done strings.
So arrays seem like the next thing to do.

Reordering Arrays with "Reverse"

[20:45] And the simplest kind of a transformation you would be likely to want to do to an array is to reorder it.
First thing that jumps to my mind is I wish the Nobel Prizes were in the opposite order. I'd like them oldest to newest instead of newest to oldest.
And the function to do that is simply reverse.
The reverse function takes as an input an array and gives you as an output an array. And what it does is it turns it back to front.
So if you send it the array 1, 2, 3, you get back the array 3, 2, 1.

[21:17] Just exactly what it says in the tin, as the British would say.
I like that one. I follow it.
And now use dash nc in your example, jq dash nc. What's the n is? No input?
Yes, because we're making the array and piping it to the reverse function.
And the c is for compact output. So instead of printing the output on five lines and opening bracket and opening square brace three comma two comma one comma closing square brace, it just prints them out in one line.
Because I, yeah, it just seemed very wasteful of show notes screen real estate to have just those three numbers splurted across five lines. Yeah.

Sorting Arrays with "Sort"

[21:54] The other thing then you're likely to want to do, other than reversing, is sorting.
And the function to sort is called sort. It takes as an input an array and gives you as an output an array.
And to be honest, it pretty much does what you expect it to do.
If you give it some numbers, it'll sort them numerically.
If you give them some strings, it'll sort them lexically or alphabetically.
And if you give them an array or a dictionary it will follow an algorithm that is in the official jq documentation and that is so complex i have decided to label it out of scope of this series.

[22:35] The other thing which it does which is interesting is that if you give it all numbers it's obvious what it's going to do and if you give it all strings it's obvious what it's going to do What happens when you give it a mixture?
Well the first thing it will do is it will sort them by type and then it will sort each type and it has a rule for the order of the types.
So nulls come first, followed by booleans, followed by numbers, followed by strings, followed by arrays, followed by dictionaries.
So if we we give it the array 1, 2, 3, 1, 4, 3, we get back 1, 3, 4, which is just a numeric sort.
Great. If we give it popcorn, waffles, pancakes, we get back pancakes, popcorn, waffles, which is an alphabetic sort.
And if we give it 42, true, 11, waffles, false, and pancakes, we get false, true, 11, 42, pancakes, waffles.
So the booleans, then the numbers, then the string.

[23:38] I guess they had to pick an order. Right, exactly. And they wrote it down in the documentation.
And that's kind of the important thing, right? It does what it says in the tin.
It doesn't need to make sense in this case.
It just needs to be written down. Okay, we're good.
And most of the time you sort things of the same type. So most of the time the behavior you get is what you expect.
And if you're wondering, well, how do I do a reverse sort? The answer is you pipe it to reverse when you're done.
There's no argument that says go backwards. You just pipe it to reverse.
Reverse sort of the terminal version of these things right everything does one thing and does it well and so if you want to reverse it well we have a function for that let's not clutter up the sort function by having an option for going forward or backward just shove it through reverse.

Sorting Arrays of Dictionaries

[24:20] Now there are rules for sorting dictionaries but actually if you have a dictionary, if you have an array of dictionaries there are probably some sort of a record it's probably like an array describing many items in a menu or something, or an array describing many Nobel Prizes.
It's going to be an array of related pieces of data, so you're probably going to want to sort them by some specific value.
So you might want to sort the Nobel Prizes by year.

[24:50] So there is a whole separate function to allow you to do that.
It's called sort sort underscore buy, and it takes one argument, which is a filter.
And that filter can be as complicated as you like, or it can be the name of a key.
So we could filter our menu in menu.json by price by saying sort underscore buy open parens dot price close parens, and it will sort it by the price.
If you do that, you will see that pancakes are the cheapest item in my menu, followed by hot hot dogs followed by waffles.
And that is most of the time when you do a sort by, you just say, give me a key and I'll sort it that way.
But that could be any filter.
So you can sort by a calculated amount. Right.
Which is where I turn to my new data set. I have discovered that there is a free JSON API where you can get stock information.
You have to jump through a very small hoop of registering for a free API key.
So it's not zero effort. You just register for an API key.

Exploring Stock Information API

[25:55] They need your email address and they will spam you a little, but that is it. That is the sum total of the cost. And you can filter the spam into your bin.

[26:03] It's from a place called... You can. You can. My spam filter just takes anything I really want, puts it in spam, and everything I don't want, it puts it right into my inbox. But that's a whole other story.
Charming. Yeah, it's awesome. I'm loving it.
So this is from a place called alphavantage.co, and they have a nice documentation, actually, describing all of the different things their APIs can do.
And one of their APIs is called Company Overview and it gives you this big giant glob of JSON for every company on the American Stock Exchange and the UK Stock Exchange and a whole bunch of other stock exchanges, and gives you more statistics than I understand.

[26:45] It gives me everything I expect, the name, the ticker symbol, a description, the latest earnings report, something called the PE ratio, which is the very, very edge of my understanding, thanks to the wonderfulness of the castaway Linda.
And then it goes on from there. Price to earnings ratio. Yes, which is a way of checking if a company is overvalued because if they make lots of money but their stock price is low, they'll have a high PE, which means they're good value for money.
And that is the sum total of my knowledge of stocks and shares.
We've kind of, we've bottomed out there.
But anyway, this data set is great fun, but it gives you back the information for one company at a time, and I wanted more, so there's a fun little aside in the show notes.
I wrote myself a shell script to fetch the stock information for a bunch of tech companies and then use JQ to assemble that into a new JSON file consisting of one array containing the dictionaries for each of my stock companies.
And so you will find a file named, what did I call it?
Techstocks.json or something? I think so, yeah.
How did you tell it which stocks were the ones you wanted?
So in my script, there's a bash array which has the values AAPL, MSFT, and IBM, which is how it knew to go for Microsoft.

[28:05] Yes, they're the ones I want. Yeah, because I didn't want too big of a data set or it's going to be a mess.
Okay. So the little script basically goes off and uses the curl command to fetch those three dictionaries and then uses jq to build an array out of those three dictionaries and save it to the file that you see, which is textdocs.json.
So that's the file we'll be using. But if you want a bit of, you know, how did Bart get that file, it's in the shell script.
And since we did shell scripting in this series, I thought, why hide my work? Why not share my work?
Let me enjoy. Anyway, all of that gets us back to this sort by thing.

Pondering a New Stock Selection Metric

[28:40] So what if we want to do some sort of weirdo stock calculation, and I was going to calculate the P-E ratio and then I discovered it's already in the data set, so I've decided, and this may be terrible advice, I am not giving you stock trading advice here, but imagine you decide that in my infinite wisdom I have discovered a new way to pick which stock to buy.
Imagine that I think the best stocks are the ones with the highest ratio of P.E. ratio to dividend per share.
Which seems like it might be sensible.
Because a high P.E. ratio means it's a stock that's undervalued and a high dividend per share means they pay out a lot for every share you have.
So a stock that's both undervalued and pays well sounds to this idiot like it might be a good investment.

[29:30] Right? Again, we're not advising in ratio for form.
I just needed something to calculate that was not already in the data set, and that wasn't. I don't even know what it's called. We'll call it the BART ratio, if you like.
Or actually, we'll call it the Linda ratio, because Linda told me everything I know about stocks.
She may hate me having it called after her, because this may be a silly idea, but anyway, it doesn't matter.
The point is, our data set contains the PE ratio, and it contains a dividend per share, but it does not contain this new product of them I've just invented.
So how do we sort our stock based on that?

Custom Sorting based on New Metric

[30:04] Well the answer is we use a filter that multiplies those two values together in sort underscore by.

[30:11] Before we do that, we need to actually figure out what the answer is.

Introduction to JQ Command

[30:16] So I have a JQ command in the show notes, which will print out the name of the company, the stock ticker, the PE ratio, the dividends per share, and their product.
So for Apple, that gives us AAPL as their stock ticker, and the product of the two is 26.54. four.
For IBM, it's 153.1-ish. And for Microsoft, it's 107.5-ish, which tells us that the smallest BART index, we shall call it, is Apple's, followed by Microsoft, and then the biggest is IBM.
So if our sort by is correct, it should give us back AAPL, MSFT, IBM.
So let's try. Because it's smallest to largest. Because it's smallest to largest.
I could have run it through, I could have piped it to reverse to get it the other way around, but I didn't want to clutter my example and it just sort of fit on one screen as it was.

[31:14] So sort by, open a bracket to give it a filter and then the filter we give it is open another bracket, dot PE ratio pipe to number, close bracket, star for multiply, open a bracket, dot dividend per share, pipe to number, close the bracket so we now have two numbers being multiplied and then we close the argument to sort by.
So sort by is going to be by this ratio I've invented.
And so if we run all of that through an actual jq command and then we pipe that to another filter that just replaces the giant big dictionaries with only their stock symbol we get back AAPL MSFT IBM.

[31:59] Which is sorting them based on this thing that doesn't exist.
It's just a number I mathematically made in the filter.
It existed while sorting and then vanished into the ether once the sorting was done.
So you can sort on anything, right? You can sort on some sort of weird average.
You can sort on anything you can express as a filter.

[32:19] That's pretty cool. Very powerful. Cool and powerful and quite useful in the real world.

Moving on to Array Manipulation

[32:26] Okay, so now let us move on to adding and removing things from our array.
So we can reverse them and we can sort them in very powerful ways.
So the other thing we can do is add things and take them away.
And one of the things we learned in the previous installment was that JQ is one of those languages where operators can be overloaded, is the jargon term.
In other words, an operator can do different things for different inputs.
So when we use a plus with numbers, we get addition.
When we use the same plus with strings we got concatenation you can also use plus and minus with arrays so you can have one array plus another array and what it will do is concatenate the arrays if you use plus so we can say the array one comma two plus the array three comma four will give us the array one two three four nope i don't like it i don't like it at all bart Really? That should be 4,6.
No, it should be 4,6.
Actually, if you wanted to do 4,6, you would use the add function, which allows you to add arrays together. But anyway, that's another hurdle there. Let's not do that.
I forget I said that. Plus does mean add sometimes.

[33:42] When you add arrays, you concatenate them. When you add strings, you concatenate them.
What would you guess minusing two arrays might do?
Well, I actually didn't read ahead. Well, then, not much of a guess.
So the array to the left of the minus is considered the input.
The array to the right of the minus is things you don't like.
Anything in the left array that exists in the right array gets removed from the output.

[34:11] What happens if you put something in the right array that doesn't exist in the input array? Does it just ignore it?
It just ignores it. So the example I have is 1, 2, 3, 4 minus 4, 5.
So 4 is in the input, 5 isn't. The answer I get is 1, 2, 3. So the 4 got stripped out, the 5 got ignored.
Well, I don't have to do anything, there already is no 5 there.
We're all good. And no, it doesn't shout at you. so you can basically say that there are these three or four values i need to veto and whether or not they're present i can stick them in that array and they will be pulled out if they are there and they won't be if they're not so maybe you have an array of words and there's some words you think are naughty and you might pick 10 random words and say take out the naughty ones you could always give the same array of naughty words and they'd only get sucked out i don't know why that example is in my head just anyway may or may not be related to a feature request for Or XKPassWD?
Exactly where my brain just went. Yeah, somebody was using XKPassWD in a school setting and wanted certain words that were, I don't know, trigger words or something.
Some of the words they suggested kind of made my head tilt, like the word woman was one of the words they didn't want in it.
But anyway, right away I thought about that, is you could just write out what you don't want and subtract it from the array. Yeah, exactly.
So that is two useful things you can do with the plus and minus operators, with arrays.

Deduplication of Arrays

[35:37] Next up for manipulating arrays is deduplication.
When you're processing a lot of arrays, particularly if you're combining lots of data sets, you will very often end up with duplicates, and you're very often not interested in the duplicates.
So an example for why it might be in my head is find out every data breach an organization has been involved in.
Well, if you get every breach your users were involved in and stick them all into one big array, you will have your answer, but it will be massively duplicated because 10 people were in this breach and 20 people were in that breach but you just want a list of all the breaches.
So I just basically took all of the arrays of every breach everyone was in shoved them into a giant big array and then went pipe unique, and out came just my uniques because that is the function unique deduplicate.
That made complete sense to me but what I don't understand is why it also sorted.
It's just what it does? It's because the easiest way at a computer science level to de-duplicate is to sort and then remove the duplicates. Because otherwise you have to remember everything that's gone before. It's very RAM intensive.
So a RAM efficient de-duplication sorts first.

[36:57] So that's what it does. That is why its side effect is sort and de-duplicate.
So Unique gives you a sorted de-duplicated array.
Which is probably what you want anyway. way that's me yeah because i found myself writing pipe unique pipe sort backspace backspace backspace, oh wait a second i don't have to do the sort it's already done for me i do have a question have you set up a text expander snippet yet for pipe to number i probably should you might as well, yeah with brackets around it and then jump the cursor back into the right point, yeah yeah actually you could yeah okay so that's unique and unique has a friend so unique is great for simple values like array like uh your boolean strings numbers unique will work with arrays and stuff but you may want to be a little pickier with a dictionary and say i just want one of of these data sets, I want one copy of everything in this array.
So you have an array of dictionaries and you only want one where a particular key has a particular value.
Unique underscore by lets you do the same thing as sort by.
So you basically provide a filter that says, do some calculation.
And then whatever the result of that calculation is, that's what we're going to use to deduplicate.

[38:22] Hmm. Okay. So 99.9% of the time, you're just going to use the name of a key, but you could de-duplicate based on some weird math.
So we're going to use our menu for this one because it's a nice, simple dictionary.
So if we wanted to, for arbitrary reasons, only get the items in our menu where the length of the name was different, we could do okay we could do unique by so first thing i'm going to print out the name pipe to the length which tells us that hot dogs and waffles have seven letters but pancakes have eight letters so that means if i do a unique on the length of the name i should lose either hot dogs or waffles and be left with pancakes in one of those two and so if we do unique by open around bracket dot name pipe length close around bracket we get back hot dogs pancakes so we have lost waffles because waffles and hot dogs are the same length, i had one quibble with uh part of your explanation on that on that you said um.

[39:37] Note that the elements in the output well wait a minute you said basically that you don't know which one's going to disappear, but I think you do.
I think it's the first one it finds that it gets to stay.
So since it's going alphabetically, it would find the first one would be alphabetically.
The documentation says don't count in it. It could be either.
I went back and looked at the documentation, Bart, and it didn't say that.
It had a section, they gave an example and the example they gave, it did it as it was a numerical example and it was the first one numerically.
I could have sworn there was a line there that says, do not count on the order.
Or that the order is not guaranteed. How can it be random, though?

The Power of Dictionaries

[40:20] It's got to be some... There's got to be an algorithm that it's following, right?

[40:25] Yes. There's got to be. I wouldn't... I'll put it this way. I wouldn't count on it being one or the other.
But generally speaking, whenever you... The unique by path expression function will keep only one element for each value obtained by applying the argument.
To give it as making an array by taking one element out of every group produced by group.
And then it gives you three examples, and the last one was unique by length, and it did it exactly the same way as the one we did here, where it was the first one it got to alphabetically.
I'm sorry, the first one as it went through the list. So it went through the array, and the first one it got to got to stay.
Right, so that says it uses the group by function to do the work, And if you look at the documentation for group buyouts, we do not promise which one gets kept.
I followed this one down the rabbit hole quite far. I was on the tarmac in Brussels Airport in the snow and had time to kill.
So I actually went and followed it to the docks for group by.
And then the docks were group by. So I don't see group by in this, so I didn't see.
Group by is another whole different function somewhere else up or down in the docks. It's a whole separate function. Okay.
And it has docks too. It's got to be knowable.
It has to be. But they said they wouldn't make a guarantee.
So I was like, well, if the docs don't make a promise, I'm not going to make a promise.
So I'm just going to be agnostic.

[41:54] Okay. It looks like... I'm uncomfortable with the concept that a computing platform would be able to say something was indeterminate in that case.
That just seems unlikely, that's all.
I'm sure it's the same output for the same input every time, But whether it's an obvious rule necessarily, it may be a little unobvious.
There may be some subtleties in there, particularly if you're dealing with complex data structures.

Exploring Lookup Tables

[42:27] Anyway, the point is we can deduplicate with simple values or even with complicated values using unique underscore by.
99.9% of the time you'll use unique by to just give the name of a key because you just want to make it. And then you don't care which one you get rid of because they're all the same. Exactly, exactly.
So the last thing on dictionary or on arrays is if you end up with lots of data sets like say a breached list of a list of breaches you're actually going to probably end up with an array of arrays because for every person in an organization who's breached you get an array of breach names and so you end up joining all of these array array array array array and then you you want to flatten them all into one master array.
And there's actually a function for that. It's called flatten.

[43:21] And if you give flatten an array of arrays, it will just give you back all of the values as if all the square brackets vanished into the ether.
And even if it's arrays all the way down, it will keep doing that recursively.
Unless you tell it not to go too far. So if you want to preserve some of your arrays at some depth, you can give it an argument, which is the maximum depth it will go until it stops flattening.
So if we give it the array one, one, followed by the array, two, three, followed by the array four, which contains the array five, six, and we flatten it.
So we now have an array that contains a number, an array, another array that contains another array.
Doesn't matter. It's like arrays all the way down here. It's very hard to say, but you end up with the output, one, two, three, four, five, six.
It just flattens all of the arrays. And you just get one, two, three, four, five, six.
Now, if we want to see the difference of flattening to different depths, If we say flatten 2 on that horrible data structure, it will go down to two levels within our array.
So that means we go into the first array, 2, 3, and we go into the 5, 6, which is inside the 4 array. So that's 2.

[44:36] So 1 square bracket to 4 and no square bracket to 5, we go into both of those, so we still get back 1, 2, 3, 4, 5, 6.
But if we give it 1, it doesn't flatten that inner, inner array.
So we get 1, 2, 3, 4, and then the array, 5, 6, because it stopped flattening.

[44:54] I looked at this one over and over again, and I'm sure this sounds like gibberish if you're just listening and not reading along.
And we're aware of that. But the way I like to think of it is, since we always count from zero, is if it just comes across a value, that's zero.
If it finds an array, that's one deep. If it finds an array inside of an array, that's two deep.
So that's how you know if you gave it two it would get both of those one it would only get the first one perfect that's so much better said than i could have it's exactly right and that that's probably what it's doing on the inside as well that's almost certainly how it's counting you've probably described its inner workings perfectly we got lucky so that is our array manipulation, we can reverse them we can sort them we can add things we can remove things we can unicify it or or de-duplicate as one would say in English, and we can flatten them, which is quite good.
So the last thing then is dictionaries, and we can add and remove keys from dictionaries.
And the plus operator is also overloaded here, which means if you want to add a new key into a dictionary, you can do that by plussing together two dictionaries.
So on the left, you have one dictionary, and on the right, you give it another dictionary with different keys and when you use the plus operator they get smushed together.

[46:17] Um if your dictionary contains dictionaries and you don't want one to replace the other you actually want to go recursively and merge them all the way down to infinity you use the multiplication operator so add will do it to one level deep and whoever's on the right wins multiply merges all all the way down so i read this one and i made a note can you elaborate on what you mean by recursive merge i'm gonna ask you to say it give me an example if i if i've got a dictionary that's one two three and a dictionary that's four five six okay well if it was a dictionary wouldn't be one two three so a dictionary could be monday one tuesday two okay right right okay and then we could add to it monday to wednesday three that's an easy one right that's let's start with the easy case.
So Monday 1, Tuesday 2, Wednesday 3 gets added to it. We get a new dictionary with Monday, Tuesday, Wednesday. All three of them. Okay.
If the one on the right hand side also had a dictionary Tuesday which had the value 4, then whatever I say in the show notes wins.
Because we have two values competing for Tuesday, and the show notes say whether it's the one on the left or the one on the right that wins. It's the one on the right.

[47:37] If both define the new value, the value on the right. Okay, so that would mean that the one on the right wins, so then in that case it's going to be the second Tuesday, which was four, would win.
But now, that's our simple example. So in that case, plus and multiplication don't do anything different.
Now... But I didn't hear multiplication in that whole explanation.
You just did the addition one. Right, because in this case, in this case, the two are the same.
And the reason they're the same is because our values were 1, 2, 3.
So our values are not dictionaries. Our values are just numbers.
If our values are dictionaries, right? So imagine our Monday dictionary contains total sales.

[48:24] And let's say profit and number of items sold.
So for Monday, we have a profit of 10 and a number of items sold of 2.
Now for Tuesday we have a profit of 11 and a number of items sold of 3 now we add to it another Tuesday which has a number of items sold of 3, Wait, don't do numbers. It's a bad example. It has a number of items sold.
So if the second thing is a dictionary, right? So if Monday contains another dictionary, if you use plus, then the dictionary on the right wins.
And everything in the first dictionary gets thrown away, and only the dictionary on the right gets kept.
Okay. If you use a multiply, it does a merge, not a throw away and replace.
Place so would emerge add the values of the add the values for the same key no so if the keys clash you still throw away the one on the left but if the okay imagine i think maybe you need to put an example in for this one because i'm not getting close on it i don't think you have an example i don't because it made my head hurt a lot um oh okay well that's okay we can stop with with it, making our head hurt. And if we ever need to use it, we'll go look it up.
But it better not be on the homework, on the final exam.
It is not. I promise you it is not on the homework.

[49:53] No, I can make it work. So if your dictionary, if you have a dictionary, which can, okay, you have two really simple dictionaries you're multiplying together.
The first dictionary has one key named Bob that contains two keys, A and B.
It doesn't matter what the values are. And you have a second dictionary that also contains the key Bob, but it contains the keys A, B, C.
If you add them, then no.
Now, OK, I have the perfect one. Bob. We're sticking to Bob because I like Bob, right?
Our dictionary has a key Bob that contains a dictionary with the keys A and B. And our second dictionary has a Bob that contains the key C.
If I add them, then Bob becomes C.
Because the one on the right wins. So it doesn't become ABC.
If I multiply them, it becomes ABC.
It merges at a deeper level. Oh.

[51:03] But if they both had a C in them, then the one on the right would win.
Then it would be just like addition.
Right, exactly. Sorry, just by plus.
Precisely, yes. So basically what happens if there's extra keys, if you multiply and they get merged together, if you add them, the one on the right wins.
But the entire dictionary doesn't do a deep merge.
Whatever it is, it replaces it. Yeah. Okay. Like I say, I have yet to use the multiply operator between two dictionaries, but I have used the add operator between two dictionaries.
On behalf of the class, we hope you never make us use that one.
Or you teach it to us again with an example when you do.

[51:45] I didn't spend too much time on it because I think it's a bit of an edge case, but I do want you to know it's possible because that's kind of the important thing, right?
Know what's possible so you can go look it up in the docs. because if you don't know it's possible you won't think to go looking for it now this is where we get to one mild gripe i have so we were able to remove things from an array with the minus operator doesn't work on dictionaries there is no minus for dictionaries as an operator, if you want to remove a key you have to use the function del so so you delete it not minus it Yeah.
So you pipe it to Dell and then you give Dell the argument of the name of the key you want to make go away.
So if we want to remove the stock from every item in our menu, we would explode our menu, pipe it to Dell.stock, and then wrap the whole lot in square brackets to re-implode?
I don't know.
Catch together our explodey bits back into an array.
Fuse. And then we get, fuse them back together.
And then we end up with hot dogs with a price, pancakes with a price, and waffles with a price.
But the stock has vanished because we said Dell.stock.

[53:02] It's kind of obvious. Why isn't it a minus? It should be a minus.
Yeah. Anyway. It was right there, ready to be used.
Yeah. Anyway, that's the way they baked it. So that's what it does.
So I have a challenge for you for next time, which does not involve the multiplication operator.

[53:19] So I would like you to take our Nobel Prizes data set, and I would like you to give me just a list of all the laureates, but each of them just once.
So Marie Curie does not get to be listed three times. She only gets to be listed once.
And I would like all of the laureates, not in the order they happen to appear in the data set, but I would like them in alphabetical order, please.
So no duplicates and alphabetic, please. And that is actually...
Alphabetic by surname, I hope. Ah.
So if you can get them alphabetic at all, then you get full marks.
If you'd like a bonus prize, now this, this, this, I've done this because I needed to make sure it was possible. So I have the challenge solution done.
And if you're naughty, you can peep at PBS 163 plus as a branch in Git.
But I know it's possible because I've done it. For bonus credit, I would like you to sort it such that human beings are sorted on surname, but organizations should continue to be sorted on their one and only name, which is their first name.

[54:26] And when you print out the human beings they should be printed first name surname even though they're sorted on surname i want you to print them so marie curie should sort as a c but you should be printed marie curie so she and pierre should be next to each other so it can it can be be done and the key... I'll give you one hint.
You're going to need to use sort by.

[54:57] So do you remember in the old days when we managed our music files in iTunes and we used to use the ID3 tags?
You would have remembered from making your podcast episodes.
There was an ID... Well, yeah, podcast episodes have ID3 tags.
Right. We still use them. There is an ID3 tag named sort order.
And if you wanted to have the Beatles sort as B, you would put Beatles in the sort order field, but leave the name as The Beatles. And then they would sort in your list by B, but be printed as the Beatles.
Huh. That principle is the key to this challenge. Bonus. Okay.

[55:40] Well, I hope I can figure this one out. This doesn't sound as hard, but they're both short. I'm often wrong.
The challenge solutions are both short.
So if you end up with a long answer, you've gone down a rabbit hole.
Might be on the wrong path.
Yes. Yes, they are both. Even if the second one sounds complicated, if you take out the spacing and stuff, it is definitely less than 10 lines.
Mine is slightly longer because I have comments and things.
But in terms of actual raw work, it is less than 10 lines for both of them. Okay.
Right. So a little sneak peek of where we're going with this.

Simplifying Array Editing

[56:19] So at this stage, we're doing pretty well because we can now manipulate each of the major data types.
We can do math on numbers, we can manipulate strings, we can manipulate arrays, and we can manipulate dictionaries.
But we have a little bit more fun to have with dictionaries because dictionaries are very powerful tools and there's a particular way of using dictionaries called a lookup table that is very common in data sets that will be given to you by APIs and which JQ has great trouble processing as they are.

[57:02] But JQ allows you to translate them into a form JQ likes.
Then you can process them. And then if you like, you can reassemble them back into a lookup table at the end.
Or you can use non-lookup table shaped data to make your own lookup tables.
And so it sounds like how can you have an entire installment about one type of dictionary?
But it was going to be a section three for today. and it isn't because we'd be here forever.
Because it got to be more complicated or at least more interesting.
And more interesting, frankly, yes. And very powerful. And then we are ready to move on to what I thought would be 163.
So you will notice in the homework and in our examples today.

[57:49] You will see me continuously go open square bracket, dot open square bracket, close square bracket, pipe some stuff because we are exploding loading, and then catching the pieces, right?
We've done it in so many examples today.
That looks inefficient, and we're doing it all the time.
The people who wrote JQ are very fond of making the things you do often simple.
We must be doing it wrong if our code is constantly full of this complicated structure.
We are doing it wrong. There is a way to edit an entire array or an entire dictionary without exploding it.

[58:32] We can go into it, edit it in place, which means that it can go into our pipeline, array to array to array to array to array, and yet we're doing microsurgery inside each piece.
All this time you've been making us do it the hard way? Yes, because otherwise two things would happen.
You wouldn't appreciate the power of the easy way and it would make your head explode.
Oh, okay. It will not make your head explode now. No.
Because we've done all this work. But if I'd started here, your head would have exploded and stayed exploded.
And this would have been a very short series and there would have been much crankiness. So that's why we're doing it in this order.

[59:16] And then we have a few more little bits of cleaning up. But that is, at that stage, if you wanted to tune out, you could.
And at that stage, you're getting the bonus material.
The really cool stuff JQ can do, which will take you from a JQ user to a JQ power user.
But after these next two installments you have done, you know the way it's the 80-20 rule, 80% of the time you only need 20% of the features?
That gets us to the 80-20 rule after those two installments, and then we are going to spend a little bit of time doing another 20% of the remaining 80%. We're still going to leave the really mad stuff out.
I've read every single word of the docs, and I know what everything does, and I've made a choice of what's worthy of our time, and there are some really fun things for us still to do.
But if you get if you find this is all too much you can tune out after these next two and then you have enough jq to do like almost everything you're likely to want to do but you could do it better if you stay tuned.

The Future of JQ and Show Notes

[1:00:18] Right well that sounds good bart i think you've also let people in on a little secret that bart started working ahead in the show notes so if you go to github and you look for his um programming by stealth show notes you can actually be pulling and seeing things as they get edited you don't have no guarantees it's what it'll end up as but you know they always get rewritten but it's kind of if you want to see the sausage being made there'll be a branch with a plus on the end that's the one for future stuff so at the moment is 163 plus as soon as i start working on 163 it'll become 164 plus but any branch that ends in a plus is the super secret sneak peek, Well, but they're all WIP, aren't they? They are. Sorry, they're WIP dash.
So right now there's a WIP 162, which is what we're recording right now, the second, and a WIP 163 plus.
Okay. Okay, that's what the plus means. Yeah, the plus has the sample solution to this challenge I just set because I always do the challenge before I write the show notes because I am terrified of setting you a challenge that's impossible.
That would be a bad thing. Because I wouldn't find out until I went to write the show notes two weeks later, you would have two weeks of hell and I would never hear the end of it.

[1:01:33] So the show notes are not complete until the plus is gone, the whip branches merged in, I've done a typo check, we put all the links to the audio files in there, and then Steve Mattan has come back and told us all the typos I missed.
Yeah, and Steve's going to hear this in about three months' time because he's about three months behind us, but he's doing a sterling job of cleaning up after us.
Oh, I don't know though, Bart. He's doing like eight a week.
I mean, I think he's going to catch up and pass us pretty soon.
He might be fixing stuff in the plus branch soon.
He might be. Okay, great.
Right, folks. Until next time, now that you're all gone wibbly wobbly timey wimey, happy computing.
If you learn as much from Bart each week as I do, I'd like you to go over to let's-talk.ie and press one of the buttons over there to help support him.
He does 98% of the work here. I'm just the stooge that listens to him and asks the dumb questions.
If you go over to let's-talk.ie, you can support him on Patreon, you can donate via PayPal, or you can use one of his referral links.
I really hope you'll go over and help him out.
In the meantime, you can contact me at Podfeet or check out all of the shows we do over there over at podfeet.com.
Thanks for listening and see you next time.

[1:02:50] Music.