Episode 2: Parsing arguments in bash. In this video, Dave is demonstrating how bash processes command line arguments. As is so often the case in bash programming, it’s helpful to understand the underlying Unix/Linux concepts, this time the way a program launched from the exec() family of system calls supplies arguments to the program’s main() function: 1. argc, an integer. Indicates the number of arguments. 2. argv, a pointer to an array of strings, includes the list of arguments, with the first element in the array being the name of the executed program. This video compares the relationship between $* and $@, both of which are used to dereference the full array of positional parameters, _except_ for that first element of argv which has the name of the script. That’s accessed as $0. The number of arguments (roughly, the equivalent of args) is accessed as $#. In Dave’s initial cringe script, he’s looping over the unquoted $*, again showing how bash word splitting applies, even with arguments. He begins to invoke the script with different arguments, eventually giving a quoted string that contains integral whitespace, here showing how bash can supply embedded whitespace to programs. Remember, the exec() system call family isn’t messing around with word splitting at all, the arguments are just an array of strings supplied to it, and these arguments can contain anything you want, including non-printable characters, whitespace, different character sets, whatever. This is a very early introduction to a very important concept for shell users: How do you get bash to set up the precise argv array you want when you use it to invoke other programs? $* and $@, unquoted, are both expanding their internal concept of argv, using the first character in the bash field separator variable IFS as the separating character in the expansion. This expansion is then subject to bash word-splitting, which is governed by the whole of IFS, not just the first character. In the special case “$@“ or “${@}”, bash instead expands argv where the expansion is not subject to further word splitting. This special case is the final variation on the script Dave shows us, his based version. Along the way he also shows us how $* expands arg, still using that IFS character as the field separator, but with the expansion not subject to further word splitting by bash. This same * vs @ distinction is also available in arrays, for example, put a copy of Dave’s based script on your system and then try: - a=(a b c “d e”) - based “${#a}” - based “${a[1]}” - based ${a[*]} - based ${a[@]} - based “${a[*]}” - based “${a[@]” Another way of thinking about all of this is that in bash, the name of the variable containing the argv array is the empty string. This script does not demonstrate every way of interacting with bash positional parameters. $1 through $9 access the first 9 positional parameters, brace enclosures (e.g., ${10} , ${11}, ${20}) are needed for subsequent arguments. The section of the bash man page which discusses all this at length is the PARAMETERS section, especially in the subsection titled “Special Parameters.” You can alter the set of bash positional parameters in two commonly used ways: 1. The shift command is used to remove positional parameters, deleting elements starting from $1 and shifting the others leftward in the array. Often scripts will do something like: while [ $# -gt 0 ]; do echo “arg is: \”$1\””; shift; done 2. The set command is used to reset the positional parameters completely. A common pattern in scripts is: [ $# -eq 0 ] && set defaultarg1 defaultarg2 … defaultargN … in order to give a command a default set of arguments to act on. Dave’s focus in these early videos is to get you thinking right away about several key concepts: - bash word splitting, and how to control it using quoting - how variable expansion interacts with word splitting He’s starting here, in part, because the beating heart of any bash program is how it passes arguments to the other commands it uses exec() family system calls to invoke. But he’s also starting here for another important reason: To demonstrate how to do iterative debugging of bash scripts, not just by changing one thing at a time until you get it right, but also by being imaginative about what will happen if your script sees input you did not initially expect, especially in terms of how that input behaves if contains non-alphanumeric characters such as whitespace, newlines, or even non-printables such as terminal control sequences. As you learn bash using these videos as teaching aids, keep in mind that thinking like a bash programmaer involves always keeping these things in mind. Happy scripting!
Are the last four based examples you gave us supposed to give the same output? I tried to replace the double quotes with single quote while defining the array a, and now the examples act like they should (I guess). Why is that?
> $* and $@, unquoted, are both expanding their internal concept of argv, using the first character in the bash field separator variable IFS as the separating character in the expansion I tried < IFS="$(printf ' \t')" ./based foo bar 'chocolate ice cream' > hoping to get a single line output , but instead I got , and I expected the single line output because I changed the IFS so the "first character in IFS" would be the newline, which is not present in the arguments, so the arguments had to be interpreted as a single one. Sorry If my English is not perfect
@@killua_148 I did some experimenting with this as well. I was able to get the expected output with: set a b c IFS=$' \t ' echo "$*" ... but not with a number of other variants, like: set a b c IFS=$' \t ' echo "$*" or: set a b c IFS=$' \t ' echo $* I believe what's happening here is that the parameter expansion of $* due to bash word splitting is fighting with the word expansion of bash during CLI evaluation, resulting in some surprising results. I'm too lazy to check the GNU site for bash to see if there are existing bugs reported for this, but maybe I should. Being able to claim a bugfix to bash would be worth bragging about on my LinkedIn profile. 🙂
@@killua_148 Thinking about this some more, the two bugs I would file in bash are: 1, IFS=... as part of the same expression that expands a quoted context for $* should apply the new value of IFS to the expansion. 2, the bash builtin 'echo' should optionally separate its argv values with the first character of IFS instead of using a space. But the real implication here that I need to think harder about is how when you change IFS, positional parameters given as whitespace separated literals in a bash pipeline are still separated by whitespace, but positional parameters derived from word splitting done as part of a bash expansion/interpolation are done using IFS. So you have to use additional care about quoting and escaping whitespace, no matter how and when you set IFS itself. The lesson I'm taking away from this is that I need to carefully test any scripts which rely on a change to IFS in how "$*" is expanded, at least until I have internalized what is currently surprising to me about how expansion using IFS works. In particular, I'm a little iffy about the idea of ever setting IFS to $'\000' in a context where I would be expanding "$*". If I do any more reading on this, I'll respond again in these comments.
I needed this. Seriously. When all positive thinking fails, never underestimate the power of being mocked in a rapid and condescending voice. What does this say about me-god!?
![ตัวอย่างภาพยนตร์ Venom: The Last Dance [Official - Sub Thai]](http://i.ytimg.com/vi/DHPjvAwmnRE/mqdefault.jpg)
Came for mad bash skills, left with a dope tune!
🤘🤘
Episode 2: Parsing arguments in bash.
In this video, Dave is demonstrating how bash processes command line arguments. As is so often the case in bash programming, it’s helpful to understand the underlying Unix/Linux concepts, this time the way a program launched from the exec() family of system calls supplies arguments to the program’s main() function:
1. argc, an integer. Indicates the number of arguments.
2. argv, a pointer to an array of strings, includes the list of arguments, with the first element in the array being the name of the executed program.
This video compares the relationship between $* and $@, both of which are used to dereference the full array of positional parameters, _except_ for that first element of argv which has the name of the script. That’s accessed as $0. The number of arguments (roughly, the equivalent of args) is accessed as $#.
In Dave’s initial cringe script, he’s looping over the unquoted $*, again showing how bash word splitting applies, even with arguments.
He begins to invoke the script with different arguments, eventually giving a quoted string that contains integral whitespace, here showing how bash can supply embedded whitespace to programs. Remember, the exec() system call family isn’t messing around with word splitting at all, the arguments are just an array of strings supplied to it, and these arguments can contain anything you want, including non-printable characters, whitespace, different character sets, whatever. This is a very early introduction to a very important concept for shell users: How do you get bash to set up the precise argv array you want when you use it to invoke other programs?
$* and $@, unquoted, are both expanding their internal concept of argv, using the first character in the bash field separator variable IFS as the separating character in the expansion. This expansion is then subject to bash word-splitting, which is governed by the whole of IFS, not just the first character. In the special case “$@“ or “${@}”, bash instead expands argv where the expansion is not subject to further word splitting. This special case is the final variation on the script Dave shows us, his based version. Along the way he also shows us how $* expands arg, still using that IFS character as the field separator, but with the expansion not subject to further word splitting by bash.
This same * vs @ distinction is also available in arrays, for example, put a copy of Dave’s based script on your system and then try:
- a=(a b c “d e”)
- based “${#a}”
- based “${a[1]}”
- based ${a[*]}
- based ${a[@]}
- based “${a[*]}”
- based “${a[@]”
Another way of thinking about all of this is that in bash, the name of the variable containing the argv array is the empty string.
This script does not demonstrate every way of interacting with bash positional parameters. $1 through $9 access the first 9 positional parameters, brace enclosures (e.g., ${10} , ${11}, ${20}) are needed for subsequent arguments.
The section of the bash man page which discusses all this at length is the PARAMETERS section, especially in the subsection titled “Special Parameters.”
You can alter the set of bash positional parameters in two commonly used ways:
1. The shift command is used to remove positional parameters, deleting elements starting from $1 and shifting the others leftward in the array. Often scripts will do something like: while [ $# -gt 0 ]; do echo “arg is: \”$1\””; shift; done
2. The set command is used to reset the positional parameters completely. A common pattern in scripts is: [ $# -eq 0 ] && set defaultarg1 defaultarg2 … defaultargN
… in order to give a command a default set of arguments to act on.
Dave’s focus in these early videos is to get you thinking right away about several key concepts:
- bash word splitting, and how to control it using quoting
- how variable expansion interacts with word splitting
He’s starting here, in part, because the beating heart of any bash program is how it passes arguments to the other commands it uses exec() family system calls to invoke. But he’s also starting here for another important reason: To demonstrate how to do iterative debugging of bash scripts, not just by changing one thing at a time until you get it right, but also by being imaginative about what will happen if your script sees input you did not initially expect, especially in terms of how that input behaves if contains non-alphanumeric characters such as whitespace, newlines, or even non-printables such as terminal control sequences.
As you learn bash using these videos as teaching aids, keep in mind that thinking like a bash programmaer involves always keeping these things in mind. Happy scripting!
Are the last four based examples you gave us supposed to give the same output?
I tried to replace the double quotes with single quote while defining the array a, and now the examples act like they should (I guess). Why is that?
> $* and $@, unquoted, are both expanding their internal concept of argv, using the first character in the bash field separator variable IFS as the separating character in the expansion
I tried < IFS="$(printf '
\t')" ./based foo bar 'chocolate ice cream' > hoping to get a single line output , but instead I got , and
I expected the single line output because I changed the IFS so the "first character in IFS" would be the newline, which is not present in the arguments, so the arguments had to be interpreted as a single one.
Sorry If my English is not perfect
@@killua_148 I did some experimenting with this as well. I was able to get the expected output with:
set a b c
IFS=$'
\t '
echo "$*"
... but not with a number of other variants, like:
set a b c
IFS=$'
\t ' echo "$*"
or:
set a b c
IFS=$'
\t '
echo $*
I believe what's happening here is that the parameter expansion of $* due to bash word splitting is fighting with the word expansion of bash during CLI evaluation, resulting in some surprising results.
I'm too lazy to check the GNU site for bash to see if there are existing bugs reported for this, but maybe I should. Being able to claim a bugfix to bash would be worth bragging about on my LinkedIn profile. 🙂
@@extrageneity Let me know if you discover something. I'm new to scripting, I won't be of much help here.
@@killua_148 Thinking about this some more, the two bugs I would file in bash are: 1, IFS=... as part of the same expression that expands a quoted context for $* should apply the new value of IFS to the expansion. 2, the bash builtin 'echo' should optionally separate its argv values with the first character of IFS instead of using a space.
But the real implication here that I need to think harder about is how when you change IFS, positional parameters given as whitespace separated literals in a bash pipeline are still separated by whitespace, but positional parameters derived from word splitting done as part of a bash expansion/interpolation are done using IFS. So you have to use additional care about quoting and escaping whitespace, no matter how and when you set IFS itself.
The lesson I'm taking away from this is that I need to carefully test any scripts which rely on a change to IFS in how "$*" is expanded, at least until I have internalized what is currently surprising to me about how expansion using IFS works. In particular, I'm a little iffy about the idea of ever setting IFS to $'\000' in a context where I would be expanding "$*".
If I do any more reading on this, I'll respond again in these comments.
never thought i'd be learning better syntax from Bam Margera
apes gonna be so mad when she’s these videos
I needed this. Seriously.
When all positive thinking fails, never underestimate the power of being mocked in a rapid and condescending voice. What does this say about me-god!?
Just started learning bash. This whole channel should help a lot. Thank you.
welcome! #bash #debugging #programminglanguage #unix
last night I went to bed at 4 am watching all your tiktoks. I truly appreciated the videos man c: Hope one day to be as good as you
this is super nice - appreciate it. #bash #devops #unix
`in "$@"` is redundant, it's the default. `for arg; do echo ""; done` is enough
good point!
This entire premise is amazing.
I don't even care about bash, and all this stuff, but the style is marvelous.
🤘
Bro doesn't waste my time!
that’s right!
I'm not cool enough for this channel
"$*" and "$@" give me different results on GNU bash, version 5.2.21(1)-release . with "$*" I get with "$@" i get the same as in the video