Transcript
[0:00] Music.
[0:08] Well, it's that time of the week again. It's time for Chitchat Across the Pond.
This is episode number 774 for July 29th, 2023. And I'm your host, Alison Sheridan.
This week, our guest is Bart Buschatz with Programming Myself 153.
And we're going to talk about my favorite subject, scope.
[0:26] Well, I think this is an episode I'm really looking forward to because after I thought this was going to be a three episode arc, But we're finally getting to the end of our journey in terms of learning new stuff.
So we are going to come back and do a sort of a farewell episode where we put everything into context. And what I'm hoping is the farewell episode will be like a quick reference for me and I'm assuming many others. Good idea.
Wrap it in a bow where we know how to get back to all the goodness.
This has been a great series.
I'll actually be sad to see it go. I've really enjoyed it.
I think I like these smaller, bite-sized pieces of something, you know, that's not—JavaScript was a long haul. What was that, two and a half years or something we were on that?
Right. And it was also your very first language, so that makes it a longer haul.
Our two more unexpected ones were Git and Bash, and they've both ended up really good.
Yeah, Git. People are writing to us now going, hey, I just found this Git series. This is really cool.
[1:28] So, very cool. I use my own show notes a lot. One of the things I'm really excited about is I have started opening the terminal when, I'm coding or when I'm working on the show notes, so that I can do my Git stuff right from the command line.
And that makes me feel really, really happy. I mean, I still bring out the big guns with the GUI when things get weird, but I mean, I set up a text expander snippet, git com, and that's my git commit message with git commit dash am and and so I'm just ready to go doing it from the command line makes me happy.
[2:06] Excellent. Well, with my work hat on, I do a lot of Git stuff over SSH, at which point you don't have a client, right? So you're in on the terminal all the time, and it really is very liberating, empowering even, to be able to do it on the command line.
I think that's a really great word to use for the terminal itself. It's liberating.
It's powerful. It gives you something you don't have. It seems like it's more limited, almost its limitations or its strength in some ways. I don't know.
[2:37] It can do pretty much everything. Yeah, finding the right magic word.
Right. So as I say, this one is kind of bittersweet. This is our last bit of new content, but it's actually rather important because as you start to write bigger things, you find yourself copying and pasting the same piece of code multiple times in your script.
And that sets off all of your, this is a bad smell from a software engineering point of view thing. And in JavaScript, you would have immediately gone, this is either a function or a class. And I have been finding it ever more difficult in examples and in sample homework and stuff, not to use functions, because I find myself doing the same thing multiple times. And then it's like, this is wrong, I should do it right. But I can't do it right, because we haven't talked about functions. And we can't talk about functions unless we we talk about scope and scope and bash is weird.
[3:29] And I've been a little bit afraid of it. I'm kind of glad I had a couple of extra weeks on this one.
So I'm much more comfortable now.
In my talk at MaxTalk this last week, it was about how I learned by teaching.
And I used you as an example of how you say that you're often one step ahead of the audience, you know, one week ahead as you're learning stuff. so you've learned a lot in this sequence too.
[3:54] I have, I absolutely have. So today we're going to learn how to use POSIX functions because POSIX is a superset of Bash and Bash has its own special syntax that only works in Bash or we can use the POSIX syntax that will work in Bash and SH and ZSH and lots of other places too.
So I was like, well I'm only going to learn one or I'll confuse myself.
So let us learn the POSIX way so that our knowledge is portable.
And then the other thing is, yeah, we'll get to lexical versus dynamic scope a little bit later, but let's start with functions. But before we even do that, I did set you a challenge last time, which is a bit of a cheat of a challenge, because the challenge was to take your existing sample solution for our multiplication table printer script and to rewrite it to make use of xargs and or arithmetic expressions as appropriate to your particular code. But of course, I used my example to show why xargs is great.
So what I'd already done the xargs part of the assignment.
Cheater.
[5:01] Cheater a bit. Yeah. So all I ended up doing now, I have included my full sample solution in the zip file, pbs152-challenge-solution.sh. So for me, it was about changing the mathematics into arithmetic expression. So instead of writing a string, of echoing a string and piping it to the basic calculator bc, which is a bit messy, instead I was just using the $$ and then writing out the mathematical equation properly and then catching the value that way, which does make nicer code, let's be honest. Right, right.
So, the other thing that that let me do in the challenge solution was to revisit one of those little things I told you to turn into a text expander snippet and not to think about too much, which is that every time we're finished with optargs, sorry, with getting optional arguments, yeah, with optargs, sorry, getops, why did my show note say optargs?
It should say getops. Oh, I can fix that.
[6:02] Yes, that's interesting. So every time we use getOpts, we have to finish afterwards with this little bit of boilerplate that basically says take all of the optional arguments that may or may not exist and take them off the front of the array of arguments so that they're out of our way and then continue the script so that $1 and $2 are the normal arguments.
And so we've been doing that with the shift keyword and then we've been telling the shift keywords to use the value which we compute by echoing opt-int, which is the index for getopt, minus one, and piping that to the basic calculator, which is really weird-looking code. We can replace that with the much nicer shift, dollar, paren, paren, opt-int, minus one, paren, paren.
I think you were really mean to teach us that basic calculator first and then show us, yeah, Well, you were already cranky that there were so many brackets and they meant so many things.
I was like, no, I'm going to keep this one on the back bubbler for a bit. There is too much confusion happening here. I stand by that decision.
So let us jump into functions. And the first thing I'm going to do is say that all of us are at a disadvantage. Because we know how to program in other languages. And we think we know what the function is.
[7:21] Because we've come from JavaScript or PHP or all these kind of languages and in those languages a function takes its input through arguments and gives its output as a value that is returned, and we use the return keyword to return that value but in bash functions don't really work like that think of them as a script within a script or think of them as a custom terminal command. They want their input primarily via standard in, they want their output via standard out, and they do return something. They return an exit code. They don't return a value, they return an exit code. And so you use the return keyword not to return your value, and that's going to drive you nuts.
So the normal way to return your value is just to echo it, so it goes to standard out.
And then whoever's using your function will use the standard terminal plumbing to take standard out.
And yeah, what do they want to do? Is this the middle of a pipeline?
Do we want this to go to a file? Do we want this to go to a variable?
You don't know. And arguably the function shouldn't care. The function's job is to make some data exist and put it on standard out.
[8:41] Okay. Now, just like terminal commands do have arguments, bash functions do have arguments.
But don't think you have to send everything in through an argument, right? Terminal commands really like to use standard in, so so should your functions. So think of your function as a custom terminal command. I wish there was a terminal command to do blah blah blah.
Let me make one. Function.
So that's how you should think of them. They're custom terminal commands. Okay. So.
[9:14] We're going to learn the POSIX syntax, and there's another thing we're at a disadvantage of.
So the POSIX syntax is name of function, space, open parens, close parens, space, open and curly bracket, all of the commands that make up your function, close your curly bracket.
Those two parens, to you and me and to everyone else who's done JavaScript or C or Java or Perl or PHP or I could list a million more languages, we think that the names of our arguments go in the parens. That's what we're used to. Think of those parentheses as one single character that means nothing more than, this is a function. It is just a marker to say, this is a function. Nothing goes in there. They are purely ornamental.
They serve no function, other than to say, I'm a function.
I have a little confession to make. It was a very long time at a programming by stealth before I actually ever put anything inside the parentheses.
I sort of always thought of it as just a, you know, this is just to tell me that that's a function.
So Dorothy was always throwing stuff inside her parentheses, and I was always like, I'm afraid to do that. I don't know what that means.
Well I guess if you go back to your old self, you'll be just fine, because in Bashland, in POSIX land, they are just there to demarcate this thing I'm about to make as a function.
[10:38] And so it's basically name of function, the two parens, and then curly brackets to contain our function. At least it's curly brackets! At least that much makes sense to us.
Yeah, yeah, no, that's good.
So the easiest way is to show. So let us start with a very simple function. So it's a simple function to do hello world, because I think that's compulsory.
Yeah, there are laws. We must obey all laws.
There are laws. So our function is going to simply print out hello world on one line, and then on the next line it's going to print out inside parens from inside a POSIX function.
Just so that our function isn't too tiny. So the script itself you will find in pbs153a-fn.sh. And the script itself will call our function, three times by default, or if we pass our script a number, it will call our function that many times. So we can see our function do its thing multiple times. And I say in the show notes I'm going to do it once by default, and then I think I wrote my code to do it.
But I fixed it in the show notes. You had it in the show notes as, you must be missing one poll, because it does say function encounter equals one, and that's what it said in the file. actually says one.
[12:05] Yes, OK, so the... OK, is it the file or the show notes? Because it should say 1.
Both. They both say 1 now.
Oh, good. Good, good, good. OK, I did actually pull... Yeah, I'm curious. We'll have to double check that.
I'll make a note for us to check that after the show. But yeah, it said 3 and I was like...
OK, good, because the sample output... I was like, how is it 3? But it says 1. But I fixed it. Yeah, it is 1.
Yeah. And the sample output in the show notes also clearly shows it as 1, so yeah. Yeah.
I mean, okay, so looking then at the definition for our function is simply hello w, which is what I named my function, space r2parens, space curly bracket, and then echo hello world, echo from inside the POSIX function. And that's it.
To call the function, we just treat it like any other terminal command on planet earth.
We just say the name of the function, and it will call the function.
Oh, that really is all there is to it. Okay. Well, that looks like it makes perfect sense.
[13:08] Sorry, I can't multitask. I was just sending Bart a little note telling him that something's dinging in his room.
Heard it a couple of times. Which it shouldn't be, because I have personal focus on, but I guess I'll switch you to do not disturb, which is a more restrictive personal focus. Right.
Oh, didn't mean to put my screensaver on. That definitely doesn't help.
The audience is like, what are they doing, these clowns, have they never done this before?
Yeah, you'd think not. Anyway, interesting.
Okay, so we just call the function by just saying the name of it.
There's no two parentheses for that.
Nope, because again, it's a custom terminal command. Just think of it as, I wrote a terminal command.
Okay, so like cat doesn't have parentheses on it. LS doesn't have parentheses.
One of the things, I saw Bart before we started, he named his function hello with a W, which, I thought was hilarious because I always misspell hello and my fingers just want to put a W on the end of it.
So I thought he had done that and I started fixing it all over the place and then I realized no, it was hello world.
It's going to make my brain hurt even more, so it'll be even harder.
[14:19] And the other thing, just to draw your attention to, since it's new in our exploration of Bash, is I am making a point in my examples, this installment, to make heavy use of arithmetic expressions to help bed them in.
So when I decrement my counter inside my while loop, I'm using double round parens without the dollar sign, because I don't want the value of, I just want it to do the action.
So take my counter and decrement it. That actually helped when I saw it again when I was pre-reading the show notes, to see that you're using the arithmetic expressions. I like it. Excellent.
And just a reminder that if you put the dollar in front of the arithmetic expression, you get the answer of the math, and if you don't put the dollar, you get the exit code.
So that means we can use the arithmetic expressions for conditionals, like, for example, only printing an empty line unless we're at the last hello world.
So it says, round paren, round paren, fn counter greater than zero, round paren, round paren, and and echo blank string.
So that means if the function counter is still greater than zero, and it started, say, at three, it goes three, two, one, it's going to echo it, and as soon as it gets to zero.
So we are getting the execution of the function.
[15:38] So what we're getting there, because we didn't put a dollar in front of it, we're basically getting true or false, or rather success or fail, from the greater than.
Oh, from the greater than. But function counter is a value that's getting decremented. So, it's just a variable. Well, that's the line above it. It's a variable.
Yeah. It's just a variable, right? Correct. Yes, function counter is a variable. Yes.
Okay, got it. Yes, because also, as a reminder, inside an arithmetic expression, you don't give functions the dollar sign, because it's like, this is math. If it isn't numbers, it's a variable.
Right. Because the only thing allowed inside your two parentheses is math.
So in math universe, you don't need to say dollar x, you just say x.
Right, right.
[16:20] Again, it's bash trying to be helpful, but it does have the side effect of sometimes being confusing, which is why I'm making a point of repeating it.
Okay, so. Every function is like a script within a script, I said.
Or like a custom terminal command.
Well, we know that our script gets its very own copy of standard in, standard out, and standard error.
And every time our script calls another script, or calls a terminal command, that command gets its own copy of standard in, standard out, and standard error, which is why we can use the pipes to manipulate these things. The standard out from the cat command is piped into the standard in of the grep command or whatever. So they each have their own.
Functions are like terminal commands. So every time you run a function, that running copy of the function gets its very own standard in, standard out, and standard error. And what they're connected to completely depends on how you call your function. So if you put your function into the middle of a pipe, then your function's standard in will be whatever the pipe has just connected to it.
Like if you just call a command without doing any terminal plumbing, then the standard in for the command is your standard in, the same is true with the function. If you just use standard in without a pipe, then the function has the same standard in your script hat.
[17:41] So it just inherits it, but it does have its own one. So that's just to note.
And also, it can write the standard out, and it's its own standard out. So you can use it inside a pipe, just like you can any other terminal command. So basically, it behaves like you expect, is what I'm trying to say in very convoluted ways. It works like it should.
So, the next example I'm going to give is a little function. So you're going to find that in pbs135b fnStreams() where we're going to write a bash function that uses the streams.
That uses standard in and standard out to be terminal-like. And we're going to write a function that is a terminal command called pal, which is short for palindrome, that's going to take some input from standard in, and it's going to print it the right way around once, and then immediately follow with it backwards.
So it becomes a palindrome because you print it forward and then backward.
[18:40] So, you're on mute, Alison, I hope that's intended. Ah, I was moving my chair a minute ago and didn't want to make a racket.
Oh, good. Okay. So our function is going to take its input from standard in and then write its output to standard out. So inside the function, again, we just say pal, open friends, close friends, curly. So we're saying make me a new function, name pal, and all the rest is just window dressing. So the first thing we do inside our function is we read from whatever happens to be the function's standard input, which will be different every time you call the function.
Sometimes you might call it inside a pipe. Sometimes you might call it inside a script.
The function doesn't know what it will be. It just knows it will have a standard input.
And we want to take that standard input and save it in a variable, which I'm choosing to call str for string, because it really could be anything.
Right? We're just going to palindromize.
So we're using the cat function, the cat command, which by default reads standard in and prints the the standard out and we're catching it with the $parens syntax. So we're basically saying the output of the cat command, save it in this variable called str.
Okay. And then we're going to print that variable as is, without a trailing new line. So echo minus n for minus no new line. And then we have to reverse it. So there is a terminal command called rev, which reverses things. So we're going to echo our string and pipe it to rev.
Why does that exist? That seems like such an odd thing. I mean, it's perfect for this little example, but why does it exist?
[20:09] I generally find that if something might be useful somewhere...
So, the terminal thinks of itself as a bunch of Lego bricks and you can build any hassle you want.
Someone thought that would be an idea, so there it is.
So, that's our entire function. So, it reads from standard in, and it just uses the echo command for its output, right?
So, you'd be used to, in JavaScript, creating a string, building up the string, and then returning the string.
But here, the primary output mechanism is standard out. So just echo the stuff. Just echo it. That is how you give output from your functions in the normal course of things.
Different. Not difficult, but different. So how do we use our function? Well, the easiest way to use it is to pipe something to it.
So we're going to say, echo your palindromic username. And then we're going to send, we're So the variable $USER in all caps is one of those variables that exist in every Unix-y system which is your username.
So we're going to take the result of running the command echo $USER PIPE PAL and then print that out after the words your palindromic username.
So basically the bit that's actually going to PAL is echo $USER PIPE PAL.
So we're going to send your username to our new function.
And it will write to its standard out, the $parens is going to capture that and print.
[21:29] It out as a string, as part of the echo command that came before it.
So when you run this script, you will see that it does what it says in the tin now.
We also then give another example of using our function within a pipeline.
So it's not at the end of a pipe, it's in the middle of a pipe.
And so we're going to write the hostname in all caps.
[21:48] So we're going to take the hostname command, which will just print your hostname, we're going to pipe that to our pal function. So our pal function receives our hostname. It will write to it standard out, which we're now going to pipe to the transliterate command tr, which is going to take all the lowercase a to z's and transliterate them to uppercase a to z's. So in other words, convert them to uppercase. And so when you echo that out, pal host, which is the variable we're saving all this into, is going to be the uppercase shouty version of our computer's name. So if you run that script, I guess you're going to say Allison Smlietz or whatever Allison Backwards is.
Oh, come on. You know what Allison Backwards is?
Oh, Nocella. Duh. There probably isn't anybody else whose name you absolutely should know what it is backwards.
Yeah. So it says Allison Nocella, and then it writes my computer name and then runs it right into itself backwards and the whole thing's in all caps.
[22:49] Yeah, that makes perfect sense. So, I mean, it's not an exciting function.
No, but I like it as an example, but it does just what you would expect it to do. Precisely.
So we have made a terminal command, right? Whenever we're writing functions, we're making terminal commands, and we're being all terminally, and we are using the streams.
So the other very terminally thing is exit codes, right? If you need to communicate success or failure, you do that through an exit code.
An exit code of zero means all good, and any other exit code means not happy.
So a very common thing you might want to customise your exit codes on is a function to test something.
Is this a duh duh duh duh duh? Well that's a very much a success fail sort of answer, so exit codes are a fantastic mechanism for communicating that.
So our third example, 153c fn return, is going to show how we can use the return keyword to do two things. It stops the function in its tracks and ends the function with the return code we give it. So it's exactly like the exit command for a script, but instead of ending the whole script, it just ends the function.
Okay. Okay. So if there were other code further down, it would never happen, right? So you might say if some condition returns zero, and that will just stop executing the function. Like Like in JavaScript, you can at any point say, return myString, and that will stop the function running.
But it's not going to stop the script, it's just going to stop this function from running.
[24:19] Correct, correct. So the exit command will stop the whole script, the return command stops the function with the appropriate exit code.
So we're going to write a very boring function called is underscore int, which is going to check if something is an integer.
So we're going to take our input from standard in, and we're not going to write anything to standard out. We're just simply going to return the exit code 0 if it is an integer, or 1 if it is not an integer.
So again, is int space, paren paren, space curly.
So we're going to start by taking whatever is in standard in, which is what the cat command does if you don't give it any arguments, and we're going to pipe that to egrep, which we're We're going to tell it to be quiet, so minus q for just be quiet egrep, just give me a return code, don't print anything to the screen.
And the pattern we'd like it to match is starts with an optional minus sign and then one or more digits.
Okay. And then we're going to do that ampersand ampersand thing and then return zero.
So if the EGREP evaluates to...
[25:27] True, then we will go ahead and return zero, which means all is okay. Because in Bash world, everything's upside down. Yeah, that was what that pause was, because you're going, wait a minute, it should be the other way around. So zero is true, one is false.
Yeah, so zero is success. I like to say success in my head. Success, success. Okay, good. Yeah, that's easier.
Or no error. Zero is absence of error. That's actually where it comes from.
Okay, that's better. So, assuming the grep matches, our function will cease at that point. It will say, I'm happy, and it will stop running.
Oh, because you did a return?
Because I did a return. If that has not happened, I am not happy, then the second line in the function happens, return one. But it still stops.
Yes, there is an error. It still stops, yeah, but it would have done now.
Okay. Yes. And in fact, it would stop even if we didn't do anything else, but, you know.
Oh, OK, I see. I was a little confused by just the way this was formatted.
It looked like this was everything that was in the script, but actually, you're running it.
No, is that all in the script?
So the whole script is what's in the show notes, which is just a tiny little four-line function followed by us actually using the function. And that's...
That's what's coming next, so check some values. That's inside the script.
[26:46] Inside the script, yes, not inside the function. Okay, okay. Oh, right, right, right, right.
So the function is stopped, but the script can have more stuff in it.
In case this is going to be the input. but that's odd, you did the input stuff afterwards.
[27:02] Well, right, you define a function, then you use it. Yeah, okay.
All we've done here is define the function, so now it exists.
You can't use a function until it exists. JavaScript does clever things.
JavaScript will read all of your file and then pretend you'd written the function definitions at the top.
It's called hoisting. So when it goes to run your code, all of your functions are at the top of your file.
It doesn't matter where you wrote them.
When JavaScript runs them, they're all at the top of the file.
But in Bash, you better define them before you use them, or Bash is going to go, I don't know. talking about. Isint? Never heard of that.
Okay, but you haven't walked through the second half of this, but you don't ever use the function.
I do. I do. Many times. Every time you see is underscore int for the rest of the file, we're calling our function.
I don't see is underscore int in the second half of this block of code that we're looking.
[27:52] At. Um, so inside the do, on the first line inside the do, we use it.
Oh, there it is. Okay. All right, good. It is there. Couldn't see it.
Again, you're looking for parentheses. I am. I am. Yeah, you're right. That's what's wrong. OK.
Yeah. So you've got to get into that. We're making custom terminal commands here.
Custom terminal command. Custom terminal command.
So in order to test our function, we should run it against some values.
And I got lazy of copying and pasting.
So I made an array with my test values and then I looped through my array.
So my array contains the values 42, which is an integer, 4.5, which is not an integer, Minus one, which is an integer, and waffles, which is most certainly not an integer. But I would like an integer of waffles. One, please. No minus one. No waffles for you.
[28:39] No. So we then loop over it with a standard for loop. So for val in, and then that syntax we all love so much to get all the values in an array into a loop. Dollar, parens, name of array, open square bracket, at close square bracket, closer curly brackets. It's a lovely syntax.
I can't walk across the keyboard syntax. Okay. Okay.
Pretty much. Anyway. Or noob.
[29:04] We're saying if, and then we're piping our value into our isn't function. So if echo$val pipe isn't. So we're making standard in for our function b, our variable that we're testing.
And then we're echoing out either the value is an integer, else the value is not an integer.
So that will then run four times. And when you run it, you will see that it says that 42 is an integer, 4.5 is not an integer, minus 1 is an integer, and waffles is not an integer.
[29:35] Good. So there we go. So that is an example of us using exit codes. Again, we're behaving just like a terminal command, because that's what we're doing here. We're making our own terminal commands. Now, you and I both know that terminal commands can take arguments. Sure, they work on standard in for a lot of their work, and they work on standard out for most of their output.
But they can take arguments. LS most certainly can take an argument of what folder would you like me to list. If you don't give it anything it will do something sensible, but it can take arguments. You can give it flags, optional arguments for flags, like "-a", for show me everything, "-l", for give me a long listing. So we know we can have arguments.
So of course our functions can do the same. And the way it works is that as well as each of our functions getting their own standard in, standard out, and standard error, they also get their own $octosorp for how many arguments was I given, $at for all of my arguments, and $1, $2, $3, etc. So all of those special variables we've been using as the arguments to our script, every function gets their own copy of that selfsame set of arguments.
Wait a minute, are you saying that these functions are like we've written our own terminal commands?
Hmm. They are. You should also think of your scripts the same.
[30:57] It's terminal commands all the way down, folks.
So that actually means we already know how to use our arguments, as we've done it before.
So let us now make our isInt function a little bit more clever.
So just like the cat command, we'll either print whatever's on standard in to standard out, Or, if we give the cat command some arguments, it will use those arguments instead of standard int.
So, why don't we have our isInt function say, did I get any arguments?
If I got arguments, then I should check if they're all integers.
And if they're all integers, then I'm going to be happy, so I'm going to return success.
If even one of them is not, I'm going to return error.
Okay, so. Let us start by first off... So our function is a little bit longer this time because we're making it do some things here. For the first time, our function actually stretches to about 15 or 20 lines here. So the first thing we do inside isInt is we make a variable named int or e for int regular expression so that I don't have to keep copying and pasting that lovely bit of gibberish that means starts with an optional minus followed by one or more digits.
Please tell me that's a TextExpander snippet by now.
[32:12] Do you know it isn't because I typed, I am so fluent in regex language that it's easier for me to type without remember an abbreviation.
I don't know if that's something to be proud of or ashamed of.
But I I really I have a T-shirt that has to be or not to be as a regular expression.
It's stupendously nerdy.
I love it. And I use it as like a one a one T-shirt test. And so far, four people have laughed.
I like you. You're my people.
Anyway, I have saved my regular expression. This is, again, avoid code reuse, because if I made a mistake, if I copied and pasted that instead of making it a variable, I would have to find everywhere I made the same mistake, whereas by saving it as a variable, I only make the mistake.
I have one place to correct my inevitable mistakes.
It's good practice. Anyway, so we save our regular expression.
And the first thing we need to do is figure out, do we have more than zero arguments? If we have zero, we want to read from standard in.
Otherwise, we want to do something else. So the first thing is we say if, and then inside of our square brackets, so we can test a condition, we're going to test if $octotherp, or $pound, or $hash, or whatever we're calling it this week, is greater than zero.
So the greater than is minus GT, if we're inside our test command, so the square brackets are our test command.
And which one is $octotherp? What does that mean again?
Is that all of the arguments? That is the number of. The number of arguments.
No, all of. It's got a, oh good, it's got a number symbol.
[33:40] Yeah, it's one I can remember. So if that is true, we're going to do something new.
So put a pin in that for about 15 seconds, else we want to do exactly what we did before, which is process standard in.
So I copied and pasted from the previous, I didn't copy and paste, the previous script I used as my starting point, and these are the two lines that remain, and all the rest was built around those two lines.
They are exactly what we had before, because processing standard in is just like it always was.
Other part of the if is new. Okay, so we did have an argument. There is something to process in the arguments. What shall we do? We shall loop over the arguments. So for val in, open double quote, dollar at, close double quote, which is our syntax for exploding our arguments into the loop. So each time we go through the loop, the variable $val will contain the the argument we're processing at the moment.
And so we're going to take the current value we're testing, we're going to pipe it to egrep, we're going to shove it through our regular expression, and now we're going to invert our logic.
[34:47] So previously we said AND AND return 0, because there was only one thing to test, so we could just do that.
We're like, well, this matches our regular expression, great, we're done.
Well if I want to say all the arguments have to be integers, then I can't just say, I can't declare victory when the first one checks out. I've won the battle, but not the war.
So I need to flip my logic around here. What I'm saying here is I'm only going to exit the function if I fail. Because if even one of them fails, we're done. They can't all be integers if one of them isn't.
If one of them isn't. First, second, third, fourth, it doesn't matter.
As soon as you hit one, return one.
[35:25] Precisely. So now instead of saying ampersand ampersand return zero, I'm saying or return, one. So I've inverted my logic.
Okay. I've done a little Boolean trick there. Now, if we make it the whole way to the end of the loop and we have not exited the function, then they must all have passed. They must all have been integers. So therefore, I now end that part of my if statement with return zero.
Okay. I just thought of something. Your else is taking the form of processing standard in that you did in your previous example. And in that case, I forget, did you have it be that they all had to be... No, your previous example, they didn't all have to be integers.
Right, because standard in is one thing. Standard in has a value, so it was simpler. We just had a value to test. Well, now we could have 1 arg, or 5 args, or a million args. So that's why it's more complicated.
Okay, but you're going to get two different answers from this. If you sent it the same four values, your 42, minus 1, pancakes, and whatever the other one was, 4.5, if you sent it those as arguments, you would get false. You would get, no, there's no integers. But The second test, if you don't, within the same function, if you, or the same script, I'm sorry, if you don't give it any arguments, it's going to ask you to answer you four times.
[36:51] It's not going to take the collective. Only if you call it four times.
Well, you're giving it- No, no. You're giving it for...
[36:57] Right, but we did it in a loop. So each time we called it last time, we called it with one value.
We piped a single value into standard in. If we pipe all the values into standard in, they'll arrive as one big string, which is gobbledygook. So it's actually going to behave exactly like it did last time. I know. I'm saying it is going to behave exactly like last time. But the first half of your function, you're going to get two different ways of looking at the problem. You could give it the same four arguments. If you gave it as arguments, it'll come back false. It'll say you failed, but if you don't give it any arguments, it'll give you one success and three fails, even though they were the same four values.
[37:38] Not quite. So last time we called the function four times with one value each time. If we do the same thing and call the function four times with one argument each time, we will get exactly the the same output. But if we're interested in answering a bigger question, so basically what we have now done is we've made our script more powerful. It can answer the question, is this one thing an int? Or it can answer a new question it couldn't answer before, are these all integers? Which is a more powerful question.
Yeah, yeah. I'm showing off that I understand how this works by saying that you're getting two different answers from the same set of data.
Given those four inputs, if you give them as arguments, you'll get one answer.
But if you don't give them as arguments, you're going to get four separate answers.
That is completely true. And it is also true that if you give them as an argument one at a time, you'll also get the four separate answers. True.
[38:36] True. Okay. Interesting. Yes. Now, OK, so we can use ourself, same for loop as last time.
But this time, we're going to say, instead of piping and stuff, we're just going to call our function directly. So the code is much cleaner.
If isn't $val, wow, that's a lot easier than saying if echo $val pipe isn't.
That's faffing about. We just say, if isn't $val, well, hey, that's much more English.
See, then echo yes, else echo no. It's nice and clean. We can also shove it multiple values now.
So we can say if isInt 42-1, then echo allIntegers, else echo not allIntegers.
And we can also send it 11 and waffles, and then echo allIntegers or not allIntegers.
So when you run that code, what you will get is the nice, clean answers, basically, that the first two are all...
You know, the first time...
Sorry, let me say all that again. The first loop through, we get exactly what we had before. Yes, no, yes, no.
The second time when we call it with 42 and minus one, we get, yep, they're all integers.
And then we call it with 11 and waffles, we get, nope, not all integers, because waffles is not an integer.
Okay.
[39:56] So, what I sort of wanted to draw your attention to is that the same function, isInt, within that example there, is called in three different ways, right?
We can say if echo $val pipe isInt, and it works just fine, because we're taking standard in and our function handles that.
Or we can say if isInt a single argument, and it works just fine.
Or we can say if isInt with two arguments, and it works just fine.
So we're now starting to write something which behaves awfully like a real-world terminal command. Yeah. Yeah.
[40:30] Okay. So. Now I need to bring us to VariableScope. Because.
We have unwittingly created spooky action at a distance.
We have been using variables inside our functions with gay abandon, and I have had to be stupendously careful in these examples.
[40:57] These examples work despite the fact that I have not taken into account how Bash's scope works.
[41:04] These functions... This is where you want to back up and give that description that we inserted at the beginning?
This is where I'm leading into that description in about 30 seconds.
So at the moment, our code works because I have been really careful to use unique variable names everywhere and never have a clash because all those variables are global. All of them.
They're all globals. So they would all stomp on each other. Because bash, by default, makes every variable a global variable, which is sort of the big picture view of a mechanism for handling scope that is known as dynamic scope, which nothing we have used before uses dynamic scope. We have lived in a universe of where I don't think I've ever told you we use lexical scope or static scope, as it's also known now, but we have been. And the reason I haven't said it is a because very few computer scientists even speak that kind of jargon. And almost every language that you're ever going to come across, with the sole exception to my knowledge of shell script, is lexically scoped. Because that's how us humans tend to work.
So what is lexically scoped? What's that definition? means.
[42:29] A lexical analysis is you're analyzing the word. So a lexical scope, where in the code your variable is defined, determines the variable's scope.
So you're used to me saying to you, if I define a variable inside a function, it comes into being at the opening curly bracket and ceases to exist at the closing curly bracket.
So it is lexically scoped in your source code within its containing curly brackets.
And no matter how deep you nest the variable, the curly brackets are always what's giving it its lexical scope.
So the line on the file tells you the scope. So lexical and slash static, which are synonyms in this case, static is the opposite of dynamic, but neither one of those sound like where I define it matters versus it doesn't matter where I define it.
Right, so what is the other world view? I have to come back to the definition every time because those words don't mean that.
I know. They do mean that. But again, they're hard to define.
Yeah. Yeah. It's hard to...
Like, I don't like the names, but I can't think of a better one.
Yeah. Okay, so let me see if I can say it again because we've been talking a lot about it.
Static slash lexical is where, like JavaScript, where you define it, that defines its scope.
But dynamic means it's always global.
[43:52] Yes, and. There's a very important and here. So what makes dynamic scope dynamic is that the scope is figured out by how your function is called. So if you have a variable called, waffles, and that variable is inside a function inside a script, depending on how you call that function, waffles may have a different value. So you could have another function called pancakes, and pancakes could set the variable waffles to 42, and then call the function waffles. And then waffles will see... I've mixed up my names here horribly.
I lost you in the waffles anyway.
Yeah. We're going to say the variable is called waffles, and the functions are called function1 and function2. How's that?
So we have a variable called waffles And if I say in my script waffles becomes equal to 42 And then inside function 1 I print out the value of waffles.
[45:01] It will get the 42 because it was set in the script before I called the function If in function 2 I have a line that says waffles becomes equal to minus 1 And then the next line is, call function one, then the value is actually going to be minus one.
Oh, come on. The code hasn't changed. Where I call the code affects the value it sees.
That's horrible.
It's horrible until you think about how shell scripts work, and then it makes perfect sense.
Because we have been using this dynamical scope all along, we just haven't known it.
So by default, everything is global, but it doesn't have to be.
You can expressly say, I want to make these variables local to me.
So inside your function, you say to Bash, these variables are mine, I want my own copy of these.
But you then create a new universe. So every function or terminal command you call inside your function sees your value.
[46:15] So it's like you're casting a shadow, right? So there was a global value and you made a new value and you're casting a shadow on everything you call.
OK. And if something you call changes the value, so it can make another, littler scope for everything it calls.
And cast a shadow inside a shadow inside a shadow. So you're projecting down.
So now think about how variables like the input file separator work. Ifs.
So I have told you to be really wary of messing with ifs, because ifs is global.
I haven't told you about functions. If you would like to use ifs within a function, you are 100% safe to do so as long as you localize it.
So if you say, local ifs, which we're going to get to the syntax in a second, but if you basically say, I want to make my own ifs, then your spooky action at a distance is gone.
Because now your change only applies inside your function and everything you call from your function.
So if you want to make the input separator be the comma instead of the newline character, if you do it inside a function and you expressly say, make it a local variable, spooky action at a distance evaporates. because you haven't messed with all the rest of your script.
Right? Okay. It's only being constrained by your function. So when you're writing functions that use variables, you always have to ask yourself, global, or do I want my own?
Okay. So probably be safer to do local unless you're sure you want it global?
[47:44] Bing, bing, bing. So by default, you should get into the habit of starting every function with a line that that localizes your variables.
So the syntax to say this one is local is the keyword local, followed by the names of one or more variables.
And just like with a for loop, we don't say for $val, we say for val in, you say local var one, var two, not $var one, $var two, it's just the names of the variables.
And so we can use those to localize our variables.
Now, I have jumped the gun slightly on the show notes because it just was, the English was better.
That flowed really well.
Okay. It did, and I wasn't gonna stop myself just because my show notes were in a different order.
I noticed that, and then I went, no, it's working.
So now I'm going to, I have told you what I want to tell you.
Now I'm going to show you that I'm not talking poop.
So, if you were unaware of this reality of how Bash works, and you naively wrote a function, to, say, add an optional argument to our isn't. So, the getops command manipulates $1, $2, etc.
Well, every function gets its own $1 and $2, so you would immediately assume that getops will work just fine within a function. And it does.
[49:13] But it won't work just fine without spooky action at a distance if you don't remember to localize things. So we're going to do it without localizing things, so we break everything.
So we're going to make our isInt function take one optional argument, a flag, minus a. And if that flag is set, then we will accept any integer instead of all integers.
So in our previous one, if you gave it three values, they all had to be integers.
If you say minus a, then any can be an integer.
So effectively, an or instead of an and.
Okay. Seems like a reasonable thing to do. So if we just take the syntax we're used to, and we shove it in our function before we do the test for whether or not we have any arguments, we do the normal getup stuff.
So I'm still saving my regular expression to a variable called int or e, because I still want that. I'm making another variable called any okay, and I'm setting it to the empty string.
That's just going to be my variable to know whether or not we're in anything or everything mode.
It's just my variable to hold that fact. Then we're going to use getOpts, and the pattern is colon, ooh, colon a, no colon, tut tut tut, typo Bart. What should it be?
Just a, because a doesn't need a value. And no colon after?
[50:33] You No, so colon a for I want my own error messages, but a is a flag that doesn't take a value.
Yeah, okay. So just colon a. Okay.
So while getOpts are patterned is colon a, and then we're going to save the current value in a variable called opt. Then we're going to do our usual case opt in. If it's an a, we set anyOK to be one. Otherwise, we print out our error message. And instead of saying exit one, we say return one, because we don't want to kill the whole script just because they used the function wrong.
We just want to end the function. So that's the only thing that's changed here from what we're used to. And then we have our code like before for the most part. But we've added in an extra if statement to say if any is okay, we need to do a third type of logic. So we're still going to have a loop. But this time we say if we find any one that is a valid integer, we can immediately jump to success without testing anymore.
So we now have a third different convoluted way of having this logic.
So we have if statements inside of statements, but it's just logic. It's nothing new to us.
[51:47] And if we run this function, the first time we call the function, everything is fine. It does exactly what we want. So we call the function with multiple arguments. We don't give it a minus a, and it quite correctly gives us the answer we expect.
And the second time we call it WITH the minus a, and it STILL works fine.
It still says, yeah, there's at least one integer in your list of test values.
And then the third time we call it WITHOUT the minus a, with exactly the same input we used before, and now it barfs.
It gives us a dumb error message.
[52:22] What? Why? How? No. The answer is because the way getOpt works is it uses a variable called optInd to count where it is in the list of arguments. How far have I gotten looking for my optional flags? That's how we can do that little optInd minus one trick at the end.
So the optInd is counting. It's a global variable. So the first time we call our function, optInd starts at 0 and moves forward until it meets the minus a.
Right. The second time we call our function, there is no minus, so optInd doesn't move.
But it's not at 0. It's still sitting out here in the middle.
The third time we call our function, optInd is not at the start of our argument list.
It's in the middle of our argument list. And the first thing it meets is minus 1, which is our test value.
And it goes, what? Minus 1?
That's not in my list. Error.
[53:21] Because it's a global variable. Okay. I think I lost you at how opt-in is being used as an input to this function.
You.
[53:34] Not to this function, to getOpt. getOpt uses opt-in. opt-in belongs to getOpt.
Okay. But we never... So when you say while getOpt...
Right.
GetOps says, start opt-in at zero. The first time GetOps runs, opt-in is at its default value of no value, so GetOps says, oh, there is no opt-in, great, make it zero.
And it uses opt-in to remember how far it's gotten on the list of arguments inside that while loop. We say, while GetOps.
It's using opt-in. Let me see if I get it. So when you're shifting it, you're doing that while it's also changing?
The reason we can shift is because getOpt has been changing the value of optInt.
So why is it barfing again?
The first time we call our function, everything is as getOpt expects.
So we call our function three times.
The first time with a minus a. So the first time with a minus a, getOpt does everything it normally would.
So it slowly counts optInt, and it goes, how far do I get?
I get as far as wherever minus a appears in the argument list.
And so the value of opt-in is going to be 2, I think.
[54:58] It doesn't matter. The value of opt-in is not zero. It is wherever minus a appeared in the argument list, and we shift by that amount, minus one.
The second time we call our function, we don't give it a minus a, so getOpts doesn't actually move opt-in because there is nothing, there are no minuses in this process.
Oh, sorry. Just got it. Okay. That's where the piece happens. The third time.
Okay. Yeah. The third time we go through the loop, oopsie daisies, we now start to look for our options and the first option we find is minus one.
Okay. Which is garbage. Right. It goes all wrong. So do we have to localize opt-in?
Yes, we do. Huh.
We have to localize every variable we use inside this function. Okay.
We have to localize int or re, every single variable, right?
We don't want any global variables leaking out of this function, right?
Like you say, unless you explicitly want it to be global, unless there's a reason you want to address something global, you shouldn't.
So, by default, localize all your variables unless you intentionally want to project something into the global scope.
[56:03] So basically, you go through your code, and every time you find a variable, you localize it.
So the working version of this script, which doesn't do any weird spooky action at a distance, is identical to that version, but the very first line inside the script is simply local space, and then all the variable names we use, which is intori, anyok, opt, optint, optarg, and val.
[56:29] Okay. Now, every time the function is called, it gets an entirely fresh intori, an entirely fresh anyok, an entirely fresh opt, an entirely fresh optint, an entirely fresh optarg, an entirely fresh val.
So I'm struggling with, we have a lot of the same code being replicated throughout this as you're doing these build-up of examples. Where am I looking for where you actually use the syntax of local?
So in the final version of this file, which is PBS153F, I'm in the show notes.
At the very first line of the function.
In the show notes, I just copied the one line, rather than copying the entire function again, because I thought it would be too confusing. So it says localize the appropriate variables, local, int or e, any okay.
[57:22] Just it's the two paragraphs above reusing functions with source.
Okay, I'm looking for localize the, and it's not in here.
Spooky action at a distance. I scrolled up and down. I'm lost. I'm lost.
We're past it. Use local. Okay, there. Oh, got it. Okay. Oh, and I know you said this for the audience, but I was busy trying to find it.
So local, and then you just list all the variables.
Enter re, any, okay, opt, opt in, opt arg, and val. Okay.
Yeah, just list them. And that's it. All the spooky x in the distance just evaporates.
Because now we have a proper scope on our function.
So do you start by localizing everything you're doing, and then when you realize you need something global, then you change it? Or you remove it from the local list? Correct. Okay.
Yeah. So when I start a function, I put the word local, and I just leave it there.
And then every time I go to use a variable, I scroll up, I type my variable name up there, and then I use it.
I pretend Bash needs me to declare variables. It doesn't, but if you pretend it does, then you'll get it right.
Do I just pretend that local is declaring my variables? Which it sort of is, it's declaring them as local.
So, the final piece of the puzzle for today is I've been saying that the reason we want functions is to avoid code reuse, which is very true.
[58:40] But, if you're working on a very specific script that does something really one-off, really one off, you're going to write a function that has no meaning anywhere ever again. It does this one weird thing and you need it in this one script and that's great.
But that's not how coding often goes. You find yourself doing something quite generic that you would like to use in 20 scripts.
[59:03] Do you write a function in your first script, then say oh great that one works and then copy and paste it into the other 19 scripts and then 3 weeks later you find that you've made a typo and then you have to remember the other 19 scripts. No, you don't, of course.
Or you don't want to. So you can pull the content of one bash file into another script as if you had typed it there.
Oh, really? Basically, you import it as if you had just typed it there. And the command is source followed by the file name to go get.
And so you can give it the full path. You can give it a relative path and it will be relative to the current working directory, which is dangerous, because if you have your script sitting in your home directory and you're off in some other directory, then your dot is not your home directory, and so a relative path would be dangerous.
So you can use the trick of dirname $0, which we used when we were doing our menu example, which is probably what you want to do 99% of the time, because that will basically say, this file next to my script. Not next to where I am, next to where my script is. So it gives you a relative path relative to your script. Or the other way to do it is to have a single folder in your home directory where you put all of your utility scripts and then use the the full path. Actually you could use tilde. You could say source tilde slash.
[1:00:21] Sure, but having a relative path lets you know where you are. Yeah.
Exactly. So I tend to go with the relative path. The point is, source followed by a file path will suck all of the code in as if you had typed it at that point in the file.
So it's just import, basically. So if you have all of your functions sitting in a folder and you just give them sensible names, then you can just say, source, you know, checkint or whatever, and then it'll pull in your checkint function or whatever. And you just, you know, you build up a little library of useful things.
I like it. Do you have a lot of these?
[1:00:54] I do have a few. I do have a few, yes. And I keep them all in sync across my Macs using shame.
I didn't tell you a couple of days ago, I was just doing, oh, I was working with you on making some changes to my Git config. And then I realized, oh, man, I have to go over to my now that I have two laptops, I had to go over to my other laptop and change it. I was, man, Man, I wish there was some way to like version control this bit.
Oh, he's going to spank me if I say that out loud.
[1:01:24] Well, it's an easy answer. So if you picked up this series while we were just on Bash, you might want to go back and take a look at the Chez Moi series.
Indeed, yes. Which will get you into the gist rabbit hole, because Chez Moi is very gitty.
Which is a good rabbit hole to be in.
Exactly. So, we have now arrived at a point where we really have covered not just the Bash basics.
I had intended this to be a very superficial view of Bash, but you know something? We've actually covered more Bash than most sysadmins will ever use. Because most sysadmins use other people's Bash and they don't quite know what it does. We now have truly covered all of the important features of Bash. And like I said, we're going to tie it all together in a neat little package in one final installment.
But in terms of new stuff, in terms of ideas and concepts, we actually have a really good grasp of shell scripting now.
And we have all the means we need to do this, as the English expression goes, in anger, because we can use functions and the source command to break apart our reusable pieces and stick them in such a place that we really can build up a library of shell scripts to do powerful things for us.
[1:02:40] And, you know, you could then combine that with something like TextExpander to call your various little scripts and things. But, you know, we're building up really powerful stuff here and we've learned a lot of very powerful tools here.
So, I think the big takeaway from today is that we know how to make functions, we know how scope works, and finally, I can stop saying, never mess with ifs.
I can change my advice to be, only mess with the localized copies of ifs.
Because once you localize it, you're not doing spooky action at a distance anymore.
You're doing very controlled action at a distance, because you are deciding that you need the input separator to be the comma for your purposes within this defined piece of your code base.
That is powerful, not spooky. That is good.
So three months ago, when I tried to use ifs and you said, no, now we know the answer why.
[1:03:33] Now we know why I was very... I didn't want to really go into too much detail of why it was dangerous, but I definitely said it could cause spooky action. This is that it might bite you.
And I worked very hard in my examples to avoid it, which sometimes was a bit circuitous.
Right, right. I know there's a right way to do this. It's the local keyword, but I can't say it yet.
No, I can't.
So I always like when we get to this stage of a series because I can stop remembering what it is I'm lying by omission about.
Because I never tell you an untruth, right? I work really hard never to give an untruth.
I just very judiciously don't mention things that would be confusing until we're ready.
And I'm done. They're all gone. I don't have any more hidden little mustn't say, mustn't say. And I love that because then I can just do stuff.
That's a lot of mental food you've been carrying.
[1:04:19] Yep. Well it's always the case. And it's like I say, I never want to lie, but I do want to not confuse. Which is difficult.
So we're going to wrap it up on the bow next time. If you would like a challenge, I'm afraid I wasn't very imaginative this time, but I can promise you that your code for printing out your multiplication table reuses some logic in a few places. It must have some bad smells of code reuse.
Find them. Replace them with functions.
There we go.
Easy enough. Well, this, I've got to give you a tip of the hat.
I know there have been a lot of times I've struggled with trying to understand things.
These show notes are amazing this week. These are, not that they aren't always, but the clarity and the simplicity of your examples really helped, that they weren't big, complex things.
They were small, bite-sized pieces that I could go, okay, I know what checking to see if something is an integer. Okay, let's just do that little tiny thing.
So I appreciate that. I know it takes extra work to be simpler than it does to be complex sometimes.
I'm just really happy we, by accident, had quite a few weeks to write these notes because they went through a few iterations.
Okay, good.
But I'm really happy where they ended up, actually. I went out for my cycle in a very good mood, and even though I got a puncture, I still came home in a good mood. Good, good. All right.
Okay, well, until we convene for one more bit of bashing, happy computing!
[1:05:48] If you learn as much from Bart each week as I do, I'd like you to go over to lets-talk.ie and press one of the buttons over there to help support him.
He does 98% of the work here, I'm just the stooge that listens to him and asks the dumb questions.
If you go over to lets-talk.ie, you can support him on Patreon, you can donate via PayPal, Or you can use one of his referral links.
[1:06:24] Music.