CCATP_2024_06_07
An audio podcast where Bart Busschots is teaching the audience to program. Associated tutorial shownotes are available at https://pbs.bartificer.net.
Automatic Shownotes
Chapters
Introduction
Summer Break Announcement
Plans for Summer Break
Getting Started with JQ
Exploring Recursion in JQ
Handling Challenges with Map, Map Values, and Reduce
Challenge Overview
Exploring Complex Assignments
Understanding the Alternative Operator
Exploring Alternative Operator Logic
Mapping Breaches Data
Exploring the Alternative Operator
Exploring Alternative Operator Logic
Understanding the Alternative Operator
Exploring Alternative Operator Logic
The Challenge Continues
Understanding the Recurse Function
Exploring the Recurse Function
Recursing Over Nested Arrays
Recursing Over Nested Arrays Example
Applying Recursion to File Systems
Recursing Over File System Data
Exploring File System Data
Understanding Dot Dot Operator
Calculating Total Elements in Data Structure
Exploring Syntactic Sugar in JQ
Utilizing Syntactic Sugar for Shorter Code
Exploring Complex Assignments in JQ
Applying Complex Assignments
Long Summary
In the final episode of Programming by Stealth with Bart Buschatz on Chit Chat Across the Pond, we reflect on the extensive learning journey we've embarked on throughout the JQ series. The episode marks a bittersweet moment as we wrap up this chapter and look forward to a summer break before delving into the model view controller topic with Helma Vanderlinden.
During this episode, we explore the intricacies of recursion in handling nested data structures in JQ, particularly focusing on manipulating data from the "Have I Been Pwned" service. By utilizing map, map values, and reduce functions, we tackle the complexities of determining exposure scores based on breached passwords. Despite the challenges, JQ proves its prowess in effectively cleaning and structuring data for manipulation.
The discussion delves into the behavior of the recurse function in JSON, demonstrating its role in returning values from nested arrays. Practical examples showcase how to apply this concept effectively, from simple nested arrays to larger file system structures, emphasizing the power of JQ in data manipulation. We also explore shortcuts in JQ for streamlined code, from simplifying dictionary creation to updating values in nested arrays with practical applications such as modifying email addresses.
Moving forward, we explore the concept of variables in programming, highlighting the 'as' operator and destructuring for binding values seamlessly. The versatility of the 'as' operator is showcased in handling various data structures, along with the use of 'if' statements and 'try-catch' blocks for error resilience in JQ scripting. Practical examples illustrate the efficiency gained through these techniques in data manipulation tasks.
Continuing the conversation, we delve into error handling in JQ scripts, discussing options such as catching errors, providing default values, and enhancing error messages for effective debugging. We touch on controlling input data with 'input' and 'inputs' functions, looping techniques, and the possibility of importing external data and writing custom JQ functions. The addition of JQ syntax highlighting to KotEditor is shared as exciting news, underscoring the depth and power of the language. As we conclude the JQ journey, we express gratitude for the learning experience and set the stage for exploring YAML, a markup language that complements Markdown for structured data manipulation in the upcoming learning journey.
During this episode, we explore the intricacies of recursion in handling nested data structures in JQ, particularly focusing on manipulating data from the "Have I Been Pwned" service. By utilizing map, map values, and reduce functions, we tackle the complexities of determining exposure scores based on breached passwords. Despite the challenges, JQ proves its prowess in effectively cleaning and structuring data for manipulation.
The discussion delves into the behavior of the recurse function in JSON, demonstrating its role in returning values from nested arrays. Practical examples showcase how to apply this concept effectively, from simple nested arrays to larger file system structures, emphasizing the power of JQ in data manipulation. We also explore shortcuts in JQ for streamlined code, from simplifying dictionary creation to updating values in nested arrays with practical applications such as modifying email addresses.
Moving forward, we explore the concept of variables in programming, highlighting the 'as' operator and destructuring for binding values seamlessly. The versatility of the 'as' operator is showcased in handling various data structures, along with the use of 'if' statements and 'try-catch' blocks for error resilience in JQ scripting. Practical examples illustrate the efficiency gained through these techniques in data manipulation tasks.
Continuing the conversation, we delve into error handling in JQ scripts, discussing options such as catching errors, providing default values, and enhancing error messages for effective debugging. We touch on controlling input data with 'input' and 'inputs' functions, looping techniques, and the possibility of importing external data and writing custom JQ functions. The addition of JQ syntax highlighting to KotEditor is shared as exciting news, underscoring the depth and power of the language. As we conclude the JQ journey, we express gratitude for the learning experience and set the stage for exploring YAML, a markup language that complements Markdown for structured data manipulation in the upcoming learning journey.
Brief Summary
In the final episode of Programming by Stealth on Chit Chat Across the Pond, we wrap up the JQ series and look ahead to the model view controller topic. We delve into recursion, variables, error handling, and JQ syntax highlighting, highlighting the language's power. The episode concludes with gratitude for the learning journey and excitement for exploring YAML next.
Tags
final episode
Programming by Stealth
Chit Chat Across the Pond
JQ series
model view controller
recursion
variables
error handling
syntax highlighting
YAML
learning journey
Transcript
[0:07]
Introduction
[0:00]Music.
[0:07]Well, it's that time of the week again. It's time for Chit Chat Across the Pond. This is episode number 795 for June 7th, 2024. And I'm your host, Alison Sheridan. This week, our guest is Bart Buschatz with Programming by Stealth 167 of X, which I believe is our finale episode on JQ. I don't know if I'm happy or sad. Yeah, you got a real bittersweet look on your face.
[0:34]
Summer Break Announcement
[0:31]I mean, I thought this was going to be short and maybe not all that interesting. Thing and it's been amazing and I've learned so much and I think I've enjoyed writing these notes more than anything we've done in a long time because it just it scratched an itch I really had, and I'm really proud of them but finished now nearly tied up in a bow well maybe we should mention right now one of the things Bart and I've noticed over the years is that we've accidentally taken a summer break every year and we have never announced that we're taking a summer break because we didn't intentionally do it. It was just scheduling got difficult. And this year, we're going to, in between JQ and the next subject, we're going to have Helma Vanderlinden come on, and either she's going to come on alone, or you and me and her, or some combination. We're going to talk about the model view controller thing that she's been doing with our XAPAS WD application. And so she's going to come on, but it's not a good time for her. So we are officially going on summer break after this, which, I don't know, a couple months? Not sure. Not sure when we'll be back. But there will be a break.
[1:40]
Plans for Summer Break
[1:41]You see, I was kind of planning on squeaking in a little quick one on YAML. Oh, you are? Okay, just one. You know how you are when you get started, though, if you find something fun.
[1:51]No, no, it's two. And I'm going to know it's two because I need to tell you why you care and then show you. So that's why it's a two-parter. But I've already planned it. Okay. Okay, well, I alarmed everybody a little bit early. Yes. But we will... Not much early, because... You'll have a little yamble with your summer. How does that sound?
[2:09]
Getting Started with JQ
[2:10]Yes. All right, well, let's get started.
[2:14]By the way, dear listeners, we have some latency. So it's not that myself and Alison are terrible at this. We are doing our best, but we are slightly stepping on each other. So we're almost done. And 90% of today is going to be doing stuff we can already do, but nicer. So most of this is about ways in which JQ allows you to achieve the ends we can achieve now with way less typing and way shorter code and code that reads more like English and code that will be easier to maintain. But we have one more, I say it's an optional topic, but it's a heavy ish lift and it's to do with recursion. So most of the data in the world isn't recursive.
[3:04]But when it is recursive you have to use special tools so when i say recursive i mean, it's nested in a self-similar way potentially infinitely far down and the only example that we all meet day to day is our file system a file system contains files and folders a folder contains files and folders a folder contains files and folders right they can go on forever ever so based on what we know at the moment with jq i couldn't give you a json input in the same sort of a shape and tell you to find me all of x within that nested data structure because you could explode one layer and then you could explode two layers but you'd need to know how far to keep exploding so you could say well if i put question marks after it i'll go at least 10 deep but if it goes to 11 your query won't work anymore because so there actually are special tools in jq to tell it to infinity not beyond just just to infinity basically follow this all the way down so there's actually a separate set of tools within jq for that kind of data so that's the one new thing and
[4:22]
Exploring Recursion in JQ
[4:18]i've done it in this optional episode because if you don't have in nested.
[4:22]Data you're never going to need this but if you have nested data oh my god are you going to need this so basically i want everyone to remember that it exists and that it's in these show notes and if you never need it probably better for your sanity but if you do need it the tooling is right here and while it does always take my brain some time to get into that sort of infinite loopy sort of a way of thinking, I usually get there in the end, and I was able to do examples, and they work. So that's a thing.
[4:57]And then, as I say, from then on, it's easy. It's all nice, fun stuff. So we did have a challenge before, our last challenge last time, which was to make use of map, map values, and reduce. And I managed to find one challenge that allowed all three of those new things we learned last time to be used within one short example. So map and map values are about doing the same thing to an entire array, an entire dictionary. And reduce is about simplifying or distilling an entire array of numbers or something into one something. So an array of many things becomes a something.
[5:40]
Exploring Complex Assignments
[5:40]And so we're able to use all three.
[5:44]So what i wanted you to do was to take a an example output from the have i been pwned domain notification service which though the one in the show notes is in semi-fictional one for the non-existent domain demo.bartofusier.net and what i wanted to do is to first off just get rid of the dot breaches and dot paste shenanigans and just have the real information be at the top level of the data structure, then explicitly use map values to replace each key on this new top-level lookup table. With a dictionary that contains two keys. So at the moment, it's basically you have usernames mapping to an array of strings, which are just the names of the breaches. But instead, we wanted to map to breaches, which is going to be name, title, and data classes.
[6:40]And exposure score, which we're going to calculate, which is zero initially for every breach the user is caught up in that doesn't have password, that's plus one. And for every breach that does have passwords it's plus 10 and so that will then give you at a glance how pwned is this person and it's an excuse to use reduce is really what it is, Well I didn't get a chance to work on this homework unfortunately I really like just this whole concept of map, map values and reduce I sort of wish it existed in other languages you know it's a really slick feature and I like this example too because it is exactly what you'd like to know.
[7:23]
Understanding the Alternative Operator
[7:24]You're starting badness. Oh, it's got your password in it? Okay, we're going to start that one at 11, actually. I swear I would have done it.
[7:31]Well, Alison, you say you'd like it in other languages. Have a look at the array prototype in JavaScript. There is a .reduce and a .map function in JavaScript. Is there really? Good to know. I'll put that in the back of my mind. There you go. Okay so just sample solution is in the installment zip as pbs166challenge solution.jq it's also in the show notes it's mostly comments the first the first thing is just get rid of everything below the breaches well that is probably the single simplest piece of a challenge I've ever given you it is literally dot breaches that is it we have now thrown away everything EverythingBelow.breaches. So there we need to, we now have a local. Why does that throw away everything below breaches?
[8:23]
Exploring Alternative Operator Logic
[8:24]Above, I guess, is probably the right. Above, okay, that makes sense. Gotcha, okay. So we now have a lookup table that maps the username part of an email address to arrays of breach names. So we want to make that array of breach names a little bit nicer. So we're going to define ourselves a new object here, sorry, new dictionary with two keys, breaches and exposure score. And we're going to work on each one, one at a time. So for now, let's worry about exposure score later. So for the breaches, what do we want to do? We want to map our current array of just names and turn that into a dictionary. So we say map, open the round bracket to start the map, open a curly bracket to assemble ourselves a fresh dictionary. We say name colon dot, because we have an array of strings. So dot is the string. Hey, that's our name. Great. Pop that into the dictionary. Then we need the title of the breach. Well, we're going to get that with our data enrichment. So we're going to pull that in from breach details zero, because minus minus slurp file always wraps it in an array. So the data we suck in is actually at zero. So breach details zero is basically the top of the list of all the breaches that came from the Have I Been Pwned database.
[9:48]That's a lookup table by name. So the name is in dot. So square brackets dot gets us to the right entry.
[10:00]
Mapping Breaches Data
[9:56]And then the title is the title. And the data classes are the data classes. So that gives us our, using the map function, we've now done that for every single name of a breach in our original list of strings. It's just, we've done it every time now. All in one. Yay. That's so clean. That's what I love about the JQ stuff is it just, it's just clean. Yeah. Yeah, readable. So that brings us very, right, very maintainable over time, which is why it's so powerful. So the next thing then is the exposure score, where we need to do a little bit of math.
[10:30]So we need to reduce over our current list of stuff, which we're going to save as a variable called breach name. So we're going to say reduce the current list of names as dollar breach name. And then we're going to say start at zero so zero semicolon so what are we going to do well we're going to each time we go through we're going to say whatever the current the current exposure score is we're going to add to that either one or ten depending on how bad things are and to get the one or the ten is based on the jq we know at the moment really bloody annoying because the only thing we know is the alternative operator, which is the two slashes. And the alternative operator's rule is, if whatever's on the left of me is null, false, or empty, then and only then shall I do whatever is on the right of me.
[11:29]Now, zero would be kind of convenient because it evaluates to false.
[11:34]
Exploring the Alternative Operator
[11:34]But I want one or ten. Neither of those are false. So I needed to get really... You did that on purpose. No. I discovered that after I set the challenge.
[11:47]Okay. I spent a long time figuring out the answer. But anyway, the key to this is that if you send something to empty, then it vanishes. the chain stops. So empty pipe 10 means that if the empty happens, it never gets onto the pipe 10. So the actual string is basically say data classes contains passwords, or basically the alternative operator empty piped to 10. All of that is wrapped in brackets. it's alternatively 1. I promise you that works and to prove it works we're gonna have to we're gonna have to just do it logically we're gonna have to step through this. Oh okay I was just about to go wait what? I know there's no way to do this but to step through it so let us assume that our breach does not contain passwords so we arrive at the first alternative operator with false.
[12:49]
Exploring Alternative Operator Logic
[12:49]False is false or null. So whatever's on the right happens. What's on the right is empty pipe to 10.
[13:00]So empty means everything evaporates. So the pipe to 10 never happens because empty has just killed that whole chain. So then we arrive at the bracket. Wait, wait, wait. Why would you ever empty to 10? I mean... Well, we'll see what happens in a minute. I don't even understand why. No, no. If we had come to true, we would never have gone empty. We would have just piped true to 10. And if you pipe true to 10, you get 10 because you've piped... Wait, but that's inside parentheses together. The empty pipe to 10 is collected together in parentheses with the contains passwords. So it would always happen, no? No?
[13:42]
Understanding the Alternative Operator
[13:43]No. No, there's one less parenthesis there than you think. What's bracketed off is the pipe to contains. So the alternative operator is going to receive true, and then after that is empty. So you're going to have either true or empty going as far as the pipe 10. So right now, what's come to the pipe 10 is empty. I'm glad you told everybody we're not just stepping on each other on purpose. So if contains passwords is true, we go to the ternary operator, which is... Is it really a ternary operator? No, it's an... Alternative operator. Alternative operator. Okay, so contains passwords, if it's true...
[14:28]It doesn't go to empty because the thing is true, which means it goes straight to pipe to 10. Correct. It contains passwords if it's false. Then it's empty pipe to 10. So it's either true pipe to 10 or empty pipe to 10. True pipe to 10 becomes 10. Empty pipe to 10 becomes nothingness. It's gone. Okay, so they both pipe to 10. That makes me feel better. Okay, right. But what pipes to 10 if it's false is empty.
[14:57]
Exploring Alternative Operator Logic
[14:57]Yeah. Yeah, which means that the pipe effectively doesn't happen because you have nothing to pipe. It's run dry. The pipe is dry. It's like the Los Angeles River. There's no water left, so it doesn't matter what's at the end of the river. It's not happening. Right. Which means that if it was false there, we'd go to the other, the second alternative operator, which would give us the one. So if there are no passwords in the breach, the answer is one. If there was a password in the breach, we get to 10. Wait, so if contains passwords is false, how does it ever get to the one if it's already gone to empty to pipe to 10? Okay, so that's now wrapped in a, but there's a parenthesis there, right? Then we hit our closing parenthesis. So what arrives at that second alternative operator is empty. False, empty, or null, well, that's empty. So what happens is on the right, one.
[15:58]This is icky. Yes, it is. But it's really good. It's very convenient how icky this is. Because one of our little pieces of, and there's a better way, is going to solve this ick. We're going to fix this ickiness. So anyway, that's actually the really hard part of this challenge.
[16:20]
The Challenge Continues
[16:17]So the map is actually really nice. The reduce is really nice. The icky part is absolutely the simple bit of, add one or ten. How hard can that be? Really hard. Well, now we have a problem to be solved. Yes, we do. So anyway, that is, if you run through all the sample stuff, that is the challenge solution working perfectly. And the hard part was explaining those two alternative operators. And you can see that it works because poor, who is it here? Who is the person who got an exposure score of 32? M.W. Kelly? It's our, oh. Yes, yes, it is. Yes, it's our friend M.W. Kelly, who we've been using as a punching bag this whole way through. Poor Michelle.
[17:05]I should tell you offline, but anyway. Okay. I remember who I anonymized that from. Okay. Not Michelle. Anyway, so that is the end of our last challenge. So our one piece of heavy lifting are nested data structures. And another word for nesting is recursion. So it may not come as a surprise that our friend here is the recurse function.
[17:32]
Understanding the Recurse Function
[17:33]And the recurse function takes zero, one or two arguments, but I'm going to start by telling you its most verbose and chatty version, which is the two arguments. So, recurse, first argument is the so-called generator filter. In other words, this is the rule that tells me how to get from here to the next here. And it's going to apply that rule over over and over and over and over again until it runs out of here. So it's a rule to get from, say, folder to all child folders, which is, you know, in the case of a file system, it's just go into every folder would be what you'd put there. Again, we're going to see an example of this in a second, but it's basically, it's the rule to get from now to the next now.
[18:23]And that could, in theory, be infinity. In theory, that could go on forever. So the emergency break is the optional second argument. And that optional second argument, as long as that evaluates to true, it keeps going. And if the second argument is present and goes to false, it stops. And so if you're using, say, mathematics as your generator instead of, you know, you could basically say dot plus one, and it will recurse through every possible integer and your condition could be dot less than 1000 and then it will recurse its way, So you're putting a limit on it You're putting a limit on it I have never needed to but that is a thing you can do if you need to Just in case you hit a runaway condition somehow, accidentally.
[19:17]
Exploring the Recurse Function
[19:18]Exactly. So this is all a little esoteric. One last thing I will say before I get into explicit examples is you can use it on arrays. And if you use it on arrays, then, wait, that's not what I wanted to say here. What have I put in bold for myself? Oh, I put it in bold for a reason. So the recurse function makes many outputs. So it behaves very much like the explode operator. So you take one thing and it becomes many, many, many things. So if you would like to collect the answer in an array, you want to surround your call to recurse in square brackets, like we used to do explode and catch, recurs and catch.
[20:10]The other thing to say is that the recurse function returns every single value it meets.
[20:22]
Recursing Over Nested Arrays
[20:18]And so that may give you more values than you expect. And I'm going to prove this to you by recursing over a nested array. And a nested array is about as simple of a nested structure as you can get. And I'm going to show you with a really, really simple nested array. The array contains one, and then another array, which contains two and three. So square bracket one, comma, square bracket two, comma, three, close two square brackets. Close two square brackets. And so we would recurse over that with the generator dot open square bracket, close square bracket, question mark, which is the explode operator,
[21:04]
Recursing Over Nested Arrays Example
[20:59]and the question mark is so as not to throw an error when we run out of nested arrays. So that will keep exploding until there are no more arrays. So it will work for our very simple one array inside another. It would also work if we added a third array inside and then a fourth array deeper inside and a fifth array deeper inside. It would keep working to infinity. But you have to put the question mark because otherwise when you run out of arrays it will say error.
[21:25]So the question mark just says, yeah, ignore the error, stop.
[21:29]So when you recurse over that very simple array, you don't get three outputs. You get five outputs.
[21:39]The first thing it outputs is the whole array. So the first output you get is open square bracket one comma open square bracket two comma three close close. Then it dives into, then it explodes that array and it starts to work through. So the next thing it will spit out is 1, because that's the next thing it meets. Then it meets the second element of the array it's currently exploding, which is the entire second array. So then it outputs the entire second array. Now it recurses. And so it goes into the second array, and it gives us the 2. And then it gives us the 3. And now it's run out of stuff, so now it stops. But when we recursed that simple array, we got five values out. That makes a certain sense to me it is sensible but it may not be what you expect so i just wanted to be really explicit that you get a lot yeah yeah okay now the only example i could think of is a file system therefore i went looking and found that there is a node.js module called Dear2JSON that will, instead of running an ls command to give you some string output you run this command and you get your file system as a giant big JSON dictionary Oh, that's cool.
[23:00]That's really cool. And it's nested. So for every element in your file system, it has a key parent, a key path, a key name, a key type, which is directory or file, and then a key children, which may or may not be present, which is an array of exactly the same thing.
[23:23]
Applying Recursion to File Systems
[23:24]Over and over and over and over and over again, however deep your file system goes. It is a nested structure, potentially to infinity. It is perfect for recursing over. So that's what we're going to do. We're going to recurse over it. I pointed it at the entire installment resources directory in the Git repository. So not just the one for 167, which didn't have any nesting. I went back a step and I pointed it at the entire one for all of our installments ever. How'd that work out for you, Bert? Big. Big.
[24:01]
Recursing Over File System Data
[24:01]There is some sample output in the show notes. It is heavily truncated. Very heavily truncated. So how do we now recurse over it? Well, if you want to recurse... You can't leave us with this part. You got to tell me what these things mean. Parent, obviously, I would think would be the directory above you. But as I'm looking at, you've got parent PBS167. Path and name don't have anything. Our name is dot dot. Can you explain what these are? Okay. Is that out of the scope? So I generated.
[24:34]
Exploring File System Data
[24:35]No. The reason the first one is funny is because of how I made the listing. So ignore the first one and then start looking at them and they make much more sense. So it says PBS 100 doesn't have a parent because it's at the top level of the resources folder.
[24:52]It has the name PBS 100, the path PBS 100, and it is of type directory. Pbs 100 contains children the first child of pbs 100 has a parent of pbs 100 its path is pbs 100 slash pbs 96 sample solution allison its name is pbs 96 challenge release allison its type is directory it then has more children which includes an index at html the reason the very first one is funny is because i asked it to do the listing for dot dot slash i was in pbs 167 and i typed the command shown just below which says give me dear to jason of dot dot so it sees the parent of dot dot as my current folder which is pbs 167 then i asked to go back a level and list that, that's why it says the parent of nothing is pbs 167 it's because of how i asked it to go down and then back up okay okay.
[25:59]So it is sensible after you, apart from that top level one, which is really weird, it is actually very sensible, which is why its name is dot dot, which makes no sense. I like how you have a comment and the comment just goes dot dot dot. The other billion files that would have been found in here. Yeah. So we recurse over this with recurse dot children, open square bracket, close square bracket, question mark, because it's children is the bit that's nested all the way down, right? Wait, wait, wait. You're already asking it a question. I don't know what question we're asking.
[26:33]Okay. So the first thing I'm going to ask it to do is recurse everything into an array. So open square bracket, recurse, close square bracket. And then I'm piping that to length because otherwise the output would be very big. So I basically said, give me everything. And then just tell me how much of everything there is. What question are we asking?
[26:55]
Understanding Dot Dot Operator
[26:52]You're telling us the command, but what question are we asking? What are we trying to find out with this directory structure? Well, the first thing is just how much is in here. So the first question I'm asking is just how many things are in this nested structure. And then we can ask more specific questions. So Recurse is just going to break it to pieces. So Explode just takes an array and pulls it apart. Recurse takes a nested structure and pulls it apart at every level all the way down. But we've just learned that Recurse through an array, for example, would find five things instead of three, like you might think. So this wouldn't be how many files are actually in this directory, correct?
[27:37]It's how many files and directories are in this directory. But it could find more than that, right? Well, you say that, but the parent folder would be found as one thing. Well, that is a thing. The parent folder contains files and folders. Each of those would be found as one thing. But I'm saying, you just showed us that that array with the nested array found five things instead of three, and it counted things extra, kind of counted the unexploded second array as one of the elements it found. So you could end up finding a number bigger than the number of files and directories. Or is none of this nested arrays? Okay, but think about the nested array example. Surely an array is a folder. So basically an array that contains values and other arrays is just like a folder that contains files and other folders. So imagine that array was a folder structure. I see what you're saying. There actually are five things. Okay, all right, all right, I'll allow it. Yeah, so the first thing we do is we just recurse it and then pipe it to length just to see what's here and because I didn't want to paste in half of the internet.
[28:52]
Calculating Total Elements in Data Structure
[28:52]So, 2079 is the answer of how many things there are in here. So now let's be a little bit more useful. I'm going to pipe that, instead of to length, to select type equals file. So now we're going to ignore type equals directory. And so now we're only going to get back how many files are there in this file system structure. And that tells us there's 1,887. That's still a lot. so then i went okay how many index.html's are there so i piped it to select type equals file and name equals index.html and then i said only give me back the path so then i piped it to dot path so instead of giving me the whole big dictionary for each file it just gave me back the path value within each dictionary and that gives us a slightly more useful 51 paths of which I've just shown you the first four, which are PBS100, PBS96, Sample Solution, Allison, index.html, BART, index.html, and then PBS104A, index.html. And there's another 48 of them.
[30:02]So that is useful. We have successfully dealt with and searched a nested structure, which we couldn't do until we learned about the recurse function.
[30:19]
Exploring Syntactic Sugar in JQ
[30:16]Okay, that's pretty interesting. It is. Now, I said you could have zero, one, or two arguments. So far, we've used one argument every time. And the optional other argument we've never even touched, the condition, right? That's only if you're dealing with numbers. If you're dealing with an actual data structure, they all end. There is no such thing as an array that goes to infinity because you couldn't have it in a file. If I can save it in a .json file, it is not infinite.
[30:47]So what happens with the zero argument version? The zero argument version is actually equivalent to .open square bracket, close square bracket, question mark. The zero argument version assumes an array and explodes the array. So if you just say recurse, without giving it any arguments, it behaves exactly the same as recurse dot open square bracket close square bracket question mark. And that's so common, JQ has given you an operator for recursing arrays, dot dot. If you just say dot dot, that is a recurse of an entire nested array. So you can explode an entire nested array with dot dot why is it that's weird so dot dot means recurse recurse means recurse open square bracket question mark no so dot dot means recurse open round bracket dot square bracket square bracket question mark close round bracket so dot dot is go down into this array forever okay Okay. They really like to make it short. Maybe it happens enough that you want to have it be able to be typed just that quickly.
[32:01]Yeah, well, they mention in the show notes that there are other languages where dot dot is how you do that with an array. And therefore, for people who are living in those other languages, that JQ offers this as a now you'll feel at home. So I like that you call them in the show notes. Yeah. Oh, no, it's in the documentation, not the show notes part. Oh, that's brilliant. That's wonderful. Now, this is a perfect segue.
[32:32]
Utilizing Syntactic Sugar for Shorter Code
[32:33]Dot dot, instead of reduce open bracket dot square bracket square bracket question mark close bracket, replacing all of that with dot dot adds no new abilities, right? JQ can't do anything new because of this little shortcut. It just makes your code a little bit shorter and nicer. We have a name for that. It's called syntactic sugar because it just makes your code syntax a little bit sweeter. It's just, it's not new functionality, it's just nicer, and it's called Syntactic Sugar, and lots of languages do Syntactic Sugar. So we're going to spend some time showing you some fun ways that you can shorten your JQ.
[33:13]So the first thing we can do when we're making dictionaries is we can save ourselves wasteful typing. How often have we had, say, a dictionary, say from Have I Been Pwned's full list of breaches, which has like 20 key value pairs for every breach Troy's ever found. And we've only ever cared about the name, the title, and the data classes. And so I have written a million and one times, open curly bracket, name colon dot name, comma, title colon dot title, comma, data classes colon dot data classes, to pick out just the bits I want. That is spectacularly duplicative, right? What I've basically said is take the name from here and make it the name over here. If I just put open curly bracket name, title, data classes, it assumes I mean name, colon, dot name, title, colon, dot title, data classes, colon, dot data classes. It's a shortcut. Oh, that's beautiful. If you're not changing. Yeah. Yeah. So that's immediately fantastic. So we have a demo in the show notes of that in action. It just really saves you typing. It's very pretty. There is a similar thing because. I got to ask, how annoying was it for you to write it the other way so many times for us knowing, God, I could make this so much simpler. Yeah.
[34:42]Very. This happens to me a lot, Alison. This happens to me a lot when I'm writing show notes. It was the same when we were doing JavaScript and Git. I know what's coming, but I can't tell you because it would confuse everything. And I'm really, really careful not to be confused. I don't always succeed, but I do try. You're trying, right? There is a similar equally useful shortcut because when we make a variable, I'm very keen to name them sensible things. So if I have a variable, I am going to use it as a key value pair within a dictionary. I am probably going to name the key the same as I named the variable because it's the same information. So how many times have I written category colon dollar category comma year colon dollar year when working with our Nobel Prizes once we discovered variables? Lo and behold dollar Dollar category comma dollar year is exactly the same as category colon dollar category comma year colon dollar year. Again, it just shortens your syntax beautifully.
[35:48]
Exploring Complex Assignments in JQ
[35:48]Now, complex assignment is either going to make perfect sense and you're going to use it because it makes your code shorter, or it's going to confuse you so much you're going to pretend I never said this. And both are perfectly valid because this is syntactic sugar. There is nothing we can do here with these shortcuts we can't do explicitly with map or by exploding and catching.
[36:14]But the assignment operators in JQ are not for assigning to variables because that would make too much sense. They are for manipulating the data flowing through our filters. So we have a dictionary comes in and we use the assignment operators to change the value of an existing key or add a new key. So we are assigning into a dictionary or into an array, right? Just to remind ourselves that JQ uses the word assignment for the data flowing through our filters, not for variables. We'll come to variables again in a moment for our next piece of syntactic sugar. So when I say complex assignment, I am talking about updating the values in the data flowing through our filters, operators, not one thing at a time, but many things at a time. So we know that we can use equals to give a key a new value, or plus equals to increment a key, or minus equals to decrement a key. Well, the syntax is pattern on the left, operator in the middle, new value on the right.
[37:21]We have always just put, like, dot first name or array element zero. We've always put one thing on the left. If you put something that maps to many things, what does JQ do? It effectively loops it for you. So if you, say, take our have-I-been-pwned export and turn it into a list of entries with two entries, you now have an array of dictionaries, and every dictionary has a key named key. If I wanted to update all of them, at the moment I would use the map function. But what I can actually do is I can say dot open square bracket, close square bracket, dot key.
[38:09]And then use the pipe equals, which is the update assignment operator. And so I say the key becomes equal to the current value of the key plus the string at demo.bartofisser.net and that will loop over every key element in the entire array and update them all to be whatever they currently are with at demo.bartofisser.net added to it I have effectively looped over my entire array to update every key and I think I can see exactly how you're using this. Since you're responsible for this security of your university, you get the Aviv and Pohn database, but it doesn't tell you what the extension is, what the domain is for the email address. You can slap it on there. Now you've got a way to send it to those people.
[38:58]
Applying Complex Assignments
[38:59]Bing bing bing yeah that's that's it exactly yeah and smell it yeah but you can see basically what we're doing here is we're exploding the array and then saying where to go and jq will just follow all of the possibilities for us and apply the same update to all of those things at once it's called complex assignment and it's very powerful but you have to be very very careful about which which types of filter you use on the left and you can get quite clever about it so you can even believe it or not on the left hand side of an update assignment you can use a select to only update some of the keys so my second example i'm gonna prefix the word naughty, in front of anyone who's caught up in more than one breach.
[39:53]Mw kelly's going down yeah so all to the left of my assignment operator i have explode the array pipe it to select if its value has a length that's greater than or equal to two, then i want to take the key and make that key become equal to naughty plus its current value and believe it or not that works that will loop over only the matching entries and stick in the naughty, that's actually pretty slick it is magical, I like that. It makes my head explode. I don't know how they coded it. I don't know how they make this magic work. It's voodoo, but it's very powerful. Oh, M.W. Kelly isn't the only one. It looks like A. Hawkins is in trouble too. Bart's given them the naughty sticker. Yeah, I had fun doing that. How fair, I want to know what you really did with that.
[40:55]Okay, so I mentioned variables in passing. Well, the other, so the word we use for variables is not assigning, we bind values to variables. We have the as operator for binding values to variable names. And when we met it, we said something on the left makes one or more values. Then we say as some variable name and we loop over all the values. And every time we go through the loop, the variable has the new value. So it's many values go into one variable and we loop, is what the as does. Well, believe it or not, you can make many variables in one as statement and it will loop over them as a unit. So you take, for every input, you make three or four variables and then within your loop, those variables all exist. Instead of having to loop multiple times, you just do it all at once. And you do this with something called destructuring.
[42:02]What arrives at the left is not a single value like one or two, it's a dictionary or an array and what you basically say is pull these three or four variables out of the dictionary or out of the array and I'd like you to do it in this pattern. So this sounds really weird, So I'm going to make this very real with a very, very real world example.
[42:27]It is very common that you end up with data that's come from goodness knows where in some sort of a very simplistic file structure like CSV or tab separated value. And you want to process it with modern tooling, so you're going to convert it to JSON. And when you do that, there's a couple of things you can end up doing, but one of the most common ways is to turn it into a two-dimensional array. So you have a CSV file and you end up with a JSON file that has an array of arrays. And so every column in the CSV is in the child array. And for every line in the CSV, you have an entry in that. So you have an array of arrays. I didn't say that very well, but I think I'm hoping you get what I mean. I think so. So there is a file with only three entries called dummylog.csv, which contains three Three lines that are valid CSV. It's a date followed by a severity followed by an error message. It happens to be on December 25th. It starts off with an informational message that there are carrots and cookies placed on the mantelpiece. Then there is a notice that the vibration sensor on the roof has been triggered. And then the motion sensor in the chimney was triggered. I don't know what happened there. Interesting.
[43:44]I can hear Steve in my head now with Honda Bob's wonderful poem. No, the wonderful poem with Honda Bob in it. Yeah. Anyway, so shock and or horror, there is an NPM module for doing exactly this conversion. It's called CSV to JSON. So I took the dummy CSV file and turned it into dummy log dash 2D array dot JSON. And now it looks like an array that contains the array with the three entries date, info, carrot and cookies placed on mantelpiece. And second array is date, notice, vibration detected on roof, date, notice, motion sensor on chimney. So we have an array of arrays.
[44:24]And in each one, we have a time, a severity, and a message. This is very real world, apart from the content, which is not. Okay. So if we want to process that, we care about three things at every point here. We want the timestamp, the severity, and the message. So we use destructuring to pull out all three variables at once by saying, on the left hand side of the as is explode the top level array. On the right hand side of the as we have square bracket dollar timestamp comma dollar severity comma dollar message. What that says is you are going to be handed an array. Take the first thing and make it the variable dollar timestamp. Take the second thing and make it the variable dollar severity. Take the third thing and make it the variable message. So we have declared three variables at once by saying this will be an array ray. First thing gets this name. Second thing, this name. Third thing, this name.
[45:24]Okay. Sorry, you may or may not have heard something stupidly just ding because my do not disturb went off because we've been going for too long. I did hear that ring. I'm glad you noticed. Well, that's not going to happen again. Now just click the moon icon yet again. So we can now use our three variables to print out a message. So we're going to make a new string that's going to have the string severity followed followed by a colon, followed by the message, followed by in parentheses, the letters logged, the symbol at, and then the timestamp. And so that proves that if we then say, loop over that, you know, import that file into JQ and apply this filter, you will see that all three variables get created each time. And we get the output info colon, carrot and cookies, place on mantelpiece, logged at whatever, notice vibration detect on roof, logged at whatever, whatever, notice motion detected by chimney sensor logged out, whatever. So we were able to make three variables by giving it a pattern. So we destructured the array into three variables.
[46:25]It's incredibly clean. Isn't it? I mean, it's so hard to describe, but wow, is it nice when you see it written down. Right, right. Now, when you're converting CSV to JSON, there's actually a second, equally valid way of doing it. Instead of making an array of arrays, wouldn't it make more sense to make an array of dictionaries? So the actual input, when we take our CSV file, would become the array, dictionary, timestamp, colon, severity, colon, message, colon, second dictionary, timestamp, severity, message, third dictionary, timestamp, severity, message. Well, actually, yeah, that makes perfect sense. And lo and behold, the exact same JavaScript library I used for the first conversion I could use for the other conversion. So, hey, look, dummy log dash dictionary array dot JSON also exists in the installment zip. And we can destructure that as well. And the way we destructure that is by saying explode our array as, but instead of saying a square bracket this time, we say $timestamp, $severity, $message. So we're making variables out of the keys of exactly the same names. So because our keys are named timestamp, severity, and message, this pattern makes perfect sense to JQ, and it will work exactly like before.
[47:45]Can I ask a dumb question real quick? I probably should have asked this earlier, but so you've just got, you exploded the array, and then you say as squirrely bracket dollar timestamp comma dollar severity comma dollar message. If you'd called that dollar waffles, dollar pancakes, dollar scones, would that also have worked? With the array example, definitely, because you're saying whatever the first thing in the array is, it is now named waffles. Right, but now you've actually got names, timestamp severity message. Is it actually looking at that and saying, okay, since he said dollar timestamp, he must mean timestamp?
[48:21]Yes, but I could rename it if I wanted to. To rename it, I'd have to use the longer syntax. Timestamp colon dollar waffles. Yeah, but okay. So it really is looking at, he used the exact same name, stuck a dollar on it. I know what he's got. I know what he's doing. It's really doing that. That's cool. It really is doing that. And if you want different names, so my example is nowhere near as fun as waffles and pancakes. It's TS for timestamp, SEV for severity, and MSG for message. But that shows you that it's timestamp colon dollar TS, severity colon dollar sev message colon dollar msg okay i i really hadn't read ahead to ask that question i i actually was the straight man accidentally that time so that's cool and that works i don't know that doesn't sound right it has like a description of your sexuality more than anything else but anyway it does it does it is we get drunk now i have one i have one piece of this isn't just syntactic sugar. This is like syntactic caramel. It's like sugar only better.
[49:30]So imagine you're in the real world and you have some log files from the server in Japan and some log files from the server in, I don't know, Hawaii or something. I'm thinking very cool places I haven't been.
[49:43]Some people are going to convert them one way, some people are going to convert them the other way. You're going to get handed a bunch of JSON files. Analyze all of this. What shape is the data in? Well, it could be shape one or it could be shape two. So you could write two JQ scripts or you could sort of do everything twice and have some, you know, do a little bit of work with the alternate operator or whatever to figure out what shape it is and then do one thing if it's one shape and another thing if it's another shape. But no, that's built in. The as operator, you can specify multiple patterns. And you can basically say, try this pattern, then this pattern, then this pattern. And once you find one that works, go with that. The operator is called destructuring alternative operator, question mark slash slash. So I can say dot explode as open square bracket dollar timestamp comma dollar severity comma dollar message. Message close my square bracket or question mark slash curly bracket dollar timestamp dollar severity dollar message and it will make my three variables regardless of the shape of the data it got it will from that point forward in my jq script those three variables will exist it will either have found them in an array or it will have found them in a dictionary it doesn't care anymore more. Oh, nice.
[51:07]So I can run that command against dummy log dash star array dot JSON, which means it's going to get half of its input as an array and half of its input as dictionaries. And it just works. It outputs the log twice. Huh? Isn't that amazing? That really is cool. Right. Okay. So that's our syntactic sugar, which I think is pretty sweet. Now, I'm going to introduce you to some old friends.
[51:37]The JQ documentation, not show notes, documentation, goes out of its way to say, this exists, but you shouldn't use it unless you really need it. Don't do it this way because you're not doing it the JQ way, unless you really need it. And so I've been avoiding these things because I don't want people to get into the habit of going, oh, I've always used an if statement in JavaScript, script so I should just use if statements in jq. Most of the time you shouldn't but sometimes you should and you can. So this is where we go back to our example from the challenge solution that was really quite ugly our 10 or 1 example. That's an if statement. If we have a password it's 10 otherwise it's 1 if else right that's how we would say it in English. That's how we should be able to say it in jq and we can the syntax is if then else end so it's four keywords to make an if statement if condition then first thing to do else second thing to do end if sort of like most things yeah we don't have we're not using brackets right it's very much like shell script right if then else fee if then else end yeah.
[53:01]The else end is optional you can say if then end you can leave out the else sure now what would you imagine jq would do if you left off the else what's the null case right in javascript the null cases do nothing right if i write if something open curly bracket a bunch of code else open curly bracket, semicolon, close curly bracket, that has exactly the same effect as just leaving off the else. The else is do nothing. The default else is do nothing.
[53:37]What would the default else be in a language like JQ where it's all about plumbing? JQ's job in life is to stream data through itself. Should the else be... No, empty would be very destructive. It's do nothing, dot. dot the default else is past the input as the output oh okay okay so in my mind what i say is, alter if so i call it an alter if statement because if the if statement doesn't match the data flows through if the if statement does match i alter the data so in my mind i say alter What if some condition? Hmm. And that's how I handle it. So now we can go back to our very strange example.
[54:27]Yeah, in the show notes, by the way, I go into a lot more detail proving the implied else, but I've done a better job of saying it than reading the show notes out loud. This is where we get into the whole, what I say to you works really well in audio, and what's written in the show notes works well in the absence of us two talking. So I'm not going to go through this in spectacular detail, but let's go look at our example. So what does our really annoying thing become? Well, it just becomes if, what was the condition? It was if the, oh, if the data classes contains passwords, 10. So if data classes contains passwords, then 10, else one end. So instead of those three alternate operators, it just becomes a simple if condition, then 10, and else won't end. I like that a lot better, Bert. That is substantially simpler.
[55:21]Slightly. Yeah. So that is our first friend. Our next friend you may recognize is Trycatch. So we can work around errors. We know that if we use the question mark, we can basically tell JQ, this may or may not exist and JQ will figure it out. And we have been able to avoid errors while being very careful to stick our question mark all over the place. We have in our very odd data set with the Nobel Prizes that are sometimes missing laureates completely, sometimes missing their surnames. And we've worked around it with question mark or with the alternate operator, with the slash slash. But we had to be very, very careful not to forget a question mark or our data or our output would be truncated because we'd had an error throw away half of our input or whatever. Well, we don't have to be that careful if we don't want to. If we have a really complicated script, we can put a try, do all of the things that could go wrong, then stick a catch. And then any error that happens anywhere between the try and the catch, that error gets handed to the catch as the input to the catch. So within a catch, dot is the error message, not the current input, the error message. Yeah, yeah, yeah, yeah.
[56:42]And so basically we can take all of our logic to transform our Nobel prizes and then at the very end say, catch. Yeah, we're going to ignore this one. In fact, we can silently catch. We don't even have to catch. The default catch is, shut up, is the default catch. But if it succeeds in the try, there's no error message, so dot becomes whatever you were trying to pass through?
[57:10]Correct. So if you succeed, then the catch never happens. Right. So the data just flows through completely as it would normally. Right. It's only if there's an error. If it goes to catch, the dot becomes the error message. Correct. And the catch then has the ability to do something with that error message and then emit whatever the outcome is at that stage. So the catch can silence the error. And then you end up with nothing coming out the other end but you know you can choose to swallow errors so we have an example basically if we try to explore okay let me just read my own example here um i put the number of oh yeah sorry i got very creative here i needed something silly to do so I could make an error on purpose. So I'm going to tell you the number, the total number of letters in all of the names of all of the winners in every physics prize since 1940. Between 1940 and 19... In the 1940s. It's a very strange thing to want to do.
[58:26]We're going to start off... With the naive approach, we're not going to handle our errors at all. We're just going to let the error happen so that we see why we handle errors. So this first implementation is intentionally bad, right? We're naive. This is our first day of JQ. So we're going to say dot prizes, send that into map. Inside map, we're going to say dot year pipe to number as dollar year. So we've saved our year as a number, as God intended. Then we're going to select for dollar year greater than or equal to 1940 and dollar year less than or equal to 1950 and, sorry, less than 1950 and category is physics. So now our prizes have been filtered down to just the physics prizes in the 1940s. Then to calculate the number of letters in the laureates' names, we're going to say dot laureates, pipe that to map, where we're going to make a new string which is first name followed by surname and then we're going to pipe that new string to length and then we're going to send all of that to the add function to add up the length of all of our laureates. So if we run that, we get jq error at NobelPrizes.json0 cannot iterate over null, null.
[59:50]I wish I had $100 for every time I saw that message when I've been writing jq.
[59:57]I'll half it with you.
[1:00:02]So, this is because in the 1940s there was a wee thing called World War II and many of the prizes were not awarded. So there is no laureates array.
[1:00:14]So, how do we handle these errors? Well, we could have done it with a question mark as our old-fashioned way, but let's learn how to use a try-catch block instead. So what we're going to do is we're going to say, We're going to do everything exactly the same, but we're going to say try open brackets dot laureates pipe to map, yada, yada, yada. And then catch, and we're going to start by just, just show me the raw error. So we're just going to say catch dot.
[1:00:43]And if you run that, you get 12, 20, 17, 16, 13, 16, 9, cannot iterate over null, null. Cannot iterate over null, null. Cannot iterate over null, null. And then our array ends. How is that different to last time? Well, what I love about this is if I'd known how to do that, I'd know where was I running into that stupid null? The last three. Because on the first one, the error was effectively never caught, so it bubbled all the way up to the top, and the entire JQ process just went, no, I'm done, and you got no output. Whereas now the error has been turned into a string. So we're constructing an array, and we're getting back seven valid values, and then three strings, because we're just saying catch dot. So we're basically saying, you know, inside the catch dot is the error message. So we're just saying, yeah, spit out the error message. And so the answer becomes the error message. But hey, we got our seven real answers and we know exactly where the problem is. So that's not useless.
[1:01:51]Our other option is to enrich the error message. So inside of our catch, we can put some sort of a sensible message to ourselves. So the example I used is fail to calculate total name length for backslash open round dollar year close round with error colon and then I put in dot. So now when we run it it says 12 whatever whatever then it says fail to calculate total name length for 1942 with error cannot iterate over null. That's a lot more useful. 1942 you say. Now we know which year it is.
[1:02:29]Right, we go look at the JSON file. Oh, there is no laureates array. Huh, okay, I've got to handle that somehow. The other option is to put, instead of the error message, just put a sensible default value or some sort of sensible human message. Why does it have to say candidate over null? Why don't we just give something a human being would care about? So we can say catch no laureates in backstash round bracket dollar year. Well, that's not useless, right? We get our numbers followed by no Lariat's in 42, 41, 40. That's fabulous. The other thing we could do is output a useful default value. If there are no Lariat's, how many letters are in their names? Zero. Right? Now you've really solved the problem. Why break everything? Right, exactly. So why not just fill our array with 0, 0, 0? So catch, zero.
[1:03:28]Finally, we can just silently swallow the errors. You said that we wouldn't need any of this. I could have used almost everything you've taught us earlier in the season. Right. But you see the word you said there, could have, right? You were able to achieve everything without it. It just would have been easier with it. That's why these are the optional extras, that different ones would be really appealing to different people. So the final thing we can do is swallow the error completely completely, by just having no catch at all. This is the implied catch is do nothing. And so that just means that if there are years with no laureates, well, then they don't have any letters. So we just don't include them in our answer. So we get seven answers instead of 10. Now, the way you did that though is catch zero. No, no, that was the previous example. Catch zero gives us the zero, zero, zero in the end. Well, you said silently swallowing errors, and then you have catch zero below that in the show That's a pretty good point.
[1:04:27]That's a type, that's some copy and pasting went wrong somewhere. That should say catch empty. Okay, I'll change that. Yeah, so to suddenly swallow an error, you say catch empty. But actually, you don't have to. Because the, like the implied if, sorry, the implied else was dot, the implied catch is actually empty. If you don't include a catch, it's as if you did catch empty. Oh. So the error is just vanish.
[1:04:59]Oh so in your example then you say you see try dot laureates map to first name and surname pipe to length and then add and then you just don't say anything after that, it's just there's no catch it's so sorry just ignore whatever doesn't work i tried i got an error i just moved on yeah it didn't stop the script it didn't kill the script it just yeah i know this might throw some errors don't care give me what worked carry on but what's important What's important about you teaching us both, though, is in this particular example, we now know, because we've been playing in the Nobel laureates for 14 years now, that there are years there were no laureates, and we know why. It's because of World War II and what happened, and so we know that we can safely ignore those. But you might have data where it's actually pretty important that that's not working, and you still want to know, why did that not work? What was it, and what do I need to do to work around it? Maybe you get to skip it, but maybe you don't. Maybe you need to clean the data. Right, exactly. So while you're debugging, you're going to say, catch me and show me the errors. And then when you figure out what's going on, you may say, ah, okay, default value makes sense here. You might update your catch, but yeah. And there are valid reasons for all of these approaches.
[1:06:09]Okay. Honorable mentions, because I spent more hours on these show notes than any of the show notes I've ever done. And there were still things I was like, oh, that's cool. Oh, that's cool. Oh, that's cool. Or I've used this and this saved to be from problems. So I'm going to give honorable mentions and remind everyone that the manual is at jqlang.github.io slash jq slash manual. And the honorable mentions is there are two special functions called input and inputs which you can use to control where in your filter the input data actually arrives and so there are actually are times when it's useful to have the input not immediately come in but to maybe do a bit of setup work and then wrap the input in an array and then process the input as one big array instead of lots of separate values well you can't wrap the inputs in a square brackets if you don't have the inputs somehow so the input keyword lets you wrap the inputs in square brackets and turn them into an array which means you could then reduce every single input which you couldn't do otherwise.
[1:07:23]I ended up using it for a somewhat interesting reason. So you joked about the fact that adding in the email addresses was something that I probably had to do in the real world. But actually, my problem in the real world is more complicated than that, because I work at a large institution. We have rebranded a few times since I started working there. So we don't have a domain main name. We actually have a few. And we have, have I been pwned, monitoring a few. So my file names are domain name dot json.
[1:08:03]And we learned there's a special variable that will tell you the file name a piece of data came from. So what I want to do is take all of my breaches, put them them in one big array, process that one big array, and then use the special variable to get the name of the file to add the right domain name. And the only way to do that is by saying open square bracket, inputs, close square bracket, and then pipe that into my big letter JQ. And then I can add the right domain name into everyone's email address and do a mail merge with the right domain name. So inputs is very useful.
[1:08:39]Now, we already know that we can loop with as and that we can distill with the reduce keyword. And it is possible to achieve all of your ends with those two things but there are actually some flavors of looping available that may just be more convenient in specific situations.
[1:09:02]The foreach keyword doesn't do what you would think as a JavaScript programmer. This caught me out. The foreach keyword is like reduce but instead of only telling you the output at the very end it tells you every value on the way as an array so you don't get one output you get all of the outputs so with a reduce it's like do this do this do this do this do this and then the final one comes out with for each you see the total adding up and you get the whole array out at the end so if you're using it to calculate say the um what is it we did the factorial was it factory it was factorial we did then you would get back all of the numbers in the sequence okay not just the final one that's kind of there are times that's useful yeah there are also subtle variations on looping with repeat while and until and they're basically they give you shorter syntax depending on whether you want to do something while something is true or do something until something becomes true, it's syntactic sugar. It's just, it makes your code more English. Again, they all have their subtlety, so if you want to figure out the difference in repeat while and until, the documentation will tell you exactly how they differ from each other. Anything you can do with those three you can do with as, but your code may be more human-friendly with those three.
[1:10:31]We have seen that on the command line, we can use minus minus slurp file to pull in an external piece of data into a variable. That's how we enriched our data in the last stages of our have I been pwned fun. Now that means I have to remember from the command line what to pull in. The import keyword lets me do the pulling in within my script. So i can say go fetch the json file sitting in the same folder as me with the name blah blah blah by saying import name of file as dollar variable name okay then you're hard coding the path into the script instead of into the command line probably a good idea yeah yeah yeah now the fourth thing made my head explode believe it or not you can write your own jq functions, not only can you write your own jq functions you can collect them into your own jq modules, save them into a little separate file and then import those modules into any jq scripts you could start to build up a little library of allison's favorite jq features make all of your own functions then start importing them into your other jq scripts so if you're like i wish there was a jq function to do whatever you could actually write it wow have you done this.
[1:11:52]Uh no because not not yet not yet there is will there is a there is a data set where i where i'm going to be working with long term and i'm finding myself doing an awful lot of copying and pasting between different jq scripts and then i say oh that's a better way to do this function and then i go and open five or six files and copy and paste and that's a bad smell yeah yeah that tells me I should pull that function out separately. So yes, I am going to, the next time I refactor a little bit of real world code, but I haven't yet.
[1:12:25]But I have read the docs. I know how. Well, hey, before you give your final thoughts, I want to break in with something I think is really interesting. I use KotEditor as my light text editor for things where maybe it's not something I'm going to use long term or I'm not doing a big project or anything like that. So all of my homework, I've been trying to do it in KotEditor. But KotEditor, you can choose a language, but JQ was not one of the languages. Languages so in on march 2nd i wrote to the um to the developers on github i opened an issue and i said i'm starting to use jq to query json files quite a bit as cod editor is my editor of choice i'd love it if i could get syntax highlighting for jq thanks for any consideration and i got the answer back on april 20th currently not cod editor currently has a policy of including only syntax for well-known enough languages and is cautious about adding new ones.
[1:13:22]However, I feel that . slash jq is now quite popular. I can't say I'd adopt it right away, but it'd be a shame to reject it so soon. Let me see how it goes for a while longer. I'll keep this open. And on, where's the date? Two weeks ago, the syntax for jq will be added to the next version, COD editor 4.8.5, which is out. So now right when the home now now when we're done with jq i have caught editor ready to go.
[1:13:53]But hey i did my work for you but he was coming after me i paved the road so i was i was really pleased to see that i thought that was cool i'm really happy that an output from this series is that one of an editor we both like has gotten a little bit better that's kind of a cool outcome yeah yeah i thought that'd be a little present for you here at the end of the uh of the episode okay so my final thoughts here the first thing is i think we've joked that the teacher was only very slightly ahead of the class on this one right i was very well motivated to learn jq because i need it in the real world because just about every api endpoint these days spits out jason that that's how apis work now and i'm spending a lot of time getting information from apis because Because strangely enough, security is often quite a boring job. One of the things I have to do is verify that we are centrally receiving telemetry and stuff that we think we are. And the only way to do that is to query the central, query the list of every server we think we have.
[1:14:57]Do these tally up? The amount of time I spent tallying things is just ridiculous, but it's one API and another API, and they should both be the same, right? The amount of places running Defender should be the same as the amount of servers we have. The amount of places logging to syslog should be the same as the amount of Linux servers we have. Are they the same? Query this API, query this API. It's all JSON data that I'm comparing to each other. It's JQ all the way down right it's just jq everywhere and so i was really well motivated to learn this and i was very keen to learn it and i thought it would be kind of cool but i had no idea how deep this world would be i had no idea how powerful jq would be so i have had so much fun and i never thought we'd still be at it months and months later we started this before christmas and i thought it was two or three weeks it's june so you know but i've really enjoyed the journey and the really amazing thing is if you go to the jq manual and you read through the manual and you look at every function name even with the honorable mentions even at this potpourri episode we've still only covered about 70 percent of what the language can do wow but that's still that's still pretty amazing i mean I don't think we got close to 70% of JavaScript, for example. No.
[1:16:19]No, you're dead right. And it's the old 80-20 rule, right? 80% of the time you're going to need 20% of the functionality. I think we've done an extremely good dive into JQ. I think we know more JQ than most people use JQ. But it's kind of fascinating that even now there are still so many more things in the docs that I've made a mental note to myself, ooh, I should play with that. Ooh, I should figure that out. So it's kind of fun. So, our next topic is going to be a much more contained thing. So, instead of it being a programming language, which means it has infinite possibilities, we're going to look at a markup language. We haven't done a new markup language in ages, right? Markup languages, we've done HTML, we've done JSON. That's it. They're the only two markup languages we've done in this massive series. So Markdown has a friend.
[1:17:12]The two have become really synonymous with each other. Markdown and YAML have become best buddies because YAML is an even more human-friendly way of writing JSON. So it's strings, numbers, dictionaries, and arrays. That's all YAML can do. Oh, that's exactly what Jason can do. But instead of all of those curly brackets and square brackets, YAML lets us do it in an even more human-friendly way, in an even prettier way. So it's the perfect complement to Markdown, which lets us do formatted text in a human-friendly way. And so if you need to put metadata into a Markdown file, YAML has become the way it's done. The other thing that people have started to do a lot is if you want configuration files that are human-friendly, make them YAML. Because as we'll see, YAML is so easy to write. And so it's even easier to read. So we're going to learn why we care about YAML and then we're going to learn how to write YAML. And I predict you're going to quite like YAML because it's very elegant. Very pretty. Okay, so you say... And it's very powerful. It's caused me... Yeah, so I say, you didn't believe me about Markdown.
[1:18:29]Well, yeah, YAML has caused me specific problems. So we'll see. We'll see if you can sell me on it. But yeah, I've now gotten into the point of being irritated at Bart when he doesn't use Markdown. So this old dog can definitely learn new tricks.
[1:18:50]That's a wonderful turnaround, actually. Yeah, it's like, you actually got quite cranky recently because I didn't use Markdown for something. And there were reasons, mainly because Apple Notes. Oh, you used Apple Notes. Oh, you're killing me. You're killing me, Bart. I was listening to something and someone was asked, what feature do you want from WWDC? And I said, I don't care about AI. I just want Markdown and Notes. Well, I think I talked about it on Clockwise. So you might have heard it there that is quite possible it might have been me again.
[1:19:30]But I'm okay with that if you think it was twice right, absolutely right okay well that brings us to the end of our JQ journey like I say I'm a bit slow ending this episode because I am slightly sad to be finishing up with it but hopefully everyone enjoyed this journey as much as I did and until next time lots of happy computing, If you learn as much from Bart each week as I do, I'd like you to go over to let's-talk.ie and press one of the buttons over there to help support him. He does 98% of the work here. I'm just the stooge that listens to him and asks the dumb questions. If you go over to let's-talk.ie, you can support him on Patreon, you can donate via PayPal, or you can use one of his referral links. I really hope you'll go over and help him out. In the meantime, you can contact me.
[1:20:20]Music.