วีดีโอ

Love the Xmas decorations! :-D

Episode 13, Formatting Dates with `strftime` in Bash. Key concepts in this episode: - Unix epochtime, sometimes called time_t - strftime from the standard C library - the bash printf builtin, EPOCHSECONDS, and EPOCHREALTIME. PART 1: Unix Epoch Time, or the impending computer apocalypse that may never happen. In computing, an 'epoch' is the zero-date on the clock and calendar. Different computer systems have different epochs, but the standard Unix epoch is midnight UTC (originally, GMT) on January 1st, 1970. The standard C library defines some functions for dealing with time relative to this epoch, including a data type called time_t, which was originally established as a signed 32-bit integer. This give a time horizon, from epoch until integer overflow, of 2 to the 31st power in seconds, which is 0x7fffffff seconds in hex, or about 2.1 billion seconds in decimal. To see the exact moment when we exhaust this allocation of seconds using the commands Dave demonstrates in this video, you can do: $ TZ=UTC printf "%(%c %z)T " '0x80000000' Tue Jan 19 03:14:08 2038 +0000 On a system with a bash shell built using a 32-bit time_t, this won't print that date, it will instead print this one: $ TZ=UTC printf "%(%c %z)T " '-0x80000000' Fri Dec 13 20:45:52 1901 +0000 ... because at that point, the 32nd bit in the integer, used to denote positive or negative, flips from 0 to 1. So, instead of being 2147483647 seconds after the epoch, you're -2147483648 seconds before it. The bitwise representation of -1 as a 32-bit signed integer is 0xffffffff. So, in less than 14 years, computers running Unix and Linux based binaries built with a 32-bit time representation will suddenly begin thinking the date is 139 years and change earlier. This is either going to be apocalyptic or no big deal, depending on how many mission-critical systems dealing with time are still building binaries around the Unix epoch in 32 bits without handling for integer overflow. PART 2: strftime, or how we convert from Unix epoch seconds into text humans understand. strftime is a C library function out of the standard C library specified as part of Unix and Linux. The function name means 'string format, time', and takes a printf-style format string where percent-escaped tokens are replaced with formatted values derived from the arguments, and literals interpolated between the tokens are retained as themselves. Unlike printf and its friends, where each token generally gets one distinct argument, strftime formats all its tokens with data taken from a single argument. This concept, that your computer system has an epoch, and a "clock tick" length, and that time is measured as an integer, the number of clock ticks since the epoch, is essential to understand if you're going to program on computer systems. It's how you do arithmetic involving time. Convert a date into seconds since epoch, using a library which understands leap seconds, time zones, and the rest, do integer math, then either convert back, or convert from clock ticks (seconds) to the units you would prefer to deal in (hours, days, years). If a language or an operating environment defines an epoch, it typically also provides a formatter to convert a time value (a count of clock ticks) into text, and often to do the reverse--produce a time value from text describing it. Not all computer systems share the Unix epoch time. The Java virtual machine, for instance, shares the same epoch date/time as Unix, but counts out from that date in microseconds rather than in seconds. Not all computer systems measure their clock ticks in seconds or even microseconds. All this is why textual representations of dates need to be not just human-readable, but computer readable. There is an entire standards document ISO-8601, which discusses this, as well as various RFCs including 5322 (which replaced 2822, which replaced 822) which borrow heavily from ISO-8601. For this reason the best practice is generally to store and transmit dates as strings rather than as epoch seconds. PART 3: I'm not actually a fan of how bash handles date and time manipulation, burying a format string within a format string. There's a certain logic to it, but I just don't love it. It's different in zsh, and the differences (including the presence of a direct command called strftime, which offers a command line switch which reverses the behavior of the function, such that it takes a formatted time string as its argument, and outputs the epoch time) are one of the reasons I decided to stick with zsh as my own shell environment when I first decided to start operating in it. To pick up where I left off in part 2, To get bash to render epoch seconds in the standard format for this, you want: %(%FT%T.000000%z)T which produces output like this: $ TZ=UTC printf "%(%FT%T.000000%z)T " -1 2024-06-05T02:59:08.000000+0000 In the above, %F gives the date in year, month, and day format, the T is a literal separating the date from the time, %T gives the time in hours, minutes, and seconds, I have hardcoded .000000 because bash strftime doesn't deal in microseconds, you have to patch those in separately from $EPOCHREALTIME if you want them, and %z gives the timezone offset from UTC in hundredths of hours. You can give 'Z' (or "Zulu time") instead of +0000 to denote UTC, and you can omit microseconds entirely, and date parsers from stuff like Splunk and Python will generally still be smart enough to auto-ingest it. EPOCHTIME and EPOCHREALTIME are special variables in bash, until you assign then as normal variables, bash instead assigns them any time you expand them in a shell expression. You can see the effects of this by comparing repeated executions of: echo $EPOCHSECONDS $EPOCHREALTIME with repeated executions of: declare | grep EPOCH There are a few other such variables, which bash treats specially unless you start messing with them, including BASHPID, DIRSTACK, RANDOM, and a number of others. The man page has a good list of them. This is mostly trivia but if you do start seeing strange behavior, be conscious of the possibilities here. Note about $EPOCHREALTIME: it only gives me microseconds on some platforms. When I use a Homebrew-built bash on MacOS, I instead get it in nanoseconds--but the man page warns that at those time scales, it's hard to use the data reliably because bash can't get it from the kernel for free, it has to do floating point arithmetic. There are times $EPOCHREALTIME can be helpful, but generally even a single microsecond is too much precision, in practice using a shell language like bash you generally want to round/truncate to something like milliseconds or even tenths of seconds in some cases. So, don't assume that you will always get exactly six digits after the dot; only use as many digits of precision as you need, and be conscious that taking extra precision is going to create inherent variance in your datestamps. There are also security implications to exposing too much microsecond/nanosecond data from your system to other users--a general topic called a "timing attack." I won't discuss that here, except to say, if you find yourself wanting to use EPOCHREALTIME, think carefully about whether or not you actually need it, and why. CONCLUSIONS: To be a more based and less cringe programmer, you need to understand how to store time, how to output it, how to do math with it, etc. Understanding epoch is an important part of that, and understanding enough as a programmer to understand that any language, even a shell language like bash, should include native ways of interacting with time functions instead of having to shell out to the date utility, is a good instinct to build. As you orient yourself to a new language, whether it's new or old, understand that time functions almost certainly exist and that it's going to serve you well to learn where they are and how to use them. Understand also that epochs end or change. One of the strangest outages I ever dealt with supporting a production system happened when a SQL database server my employer operated had its internal epoch change/reset because it ran out. Arithmetic comparing two SQL based dates produced unexpected results across the epoch boundary, and it was extremely vexing/confusing because the epoch itself was not easily visible to SQL developers! Whatever language you're writing in, even bash, you need to consider what happens if either the way time-since-epoch is encoded cannot represent a time you're dealing with, or if the meaning of the epoch changes at some point, either on your system or on systems it's talking with. You can insulate yourself against a great deal of uncertainty here by taking care never to store values as raw offsets from local epoch, but instead always reformatting them into some widely understood text based date string. Eric S. Raymond wrote in The Art of Unix Programming that in writing programs, be generous in what you accept, rigorous in what you emit. Understanding how to use strftime in bash and related languages is a great way of cleaving more closely to that principle. Happy timing!

Everybody has said "rej-ex" since the 1970s. Also people say GIF with a hard G, no matter what the creator may have intended. And nobody says JSON like the boy's name "Jason, everybody says it "jay-SON", sorry originator of the format. And correcting others' pronunciation is the cringiest of cringedom.

Don't like it ? 🚪

slay

Some of us learned on DOS and still lovingly remember it.

I would say, you look like Hagrid. 👍

Give the cheese guy his cheese

Cheese

I watched watchmen in lMAX when it was new. Great movie.

if anyone is searching for it: 'fields' are called 'capturing groups' in regex lingo. they also can be nested. cool stuff. substring and cut are forever cringe, even before this video dropped

I agree with this gentleman! I also think a quick prayer to ysap before running your scripts is never a bad thing 😅

let's do a little bash project dude

echoing works with strings. $ casts the variable to a string. what the hell is possibly confusing about this?

use the man pages

Couldn't you just put all the info into the thumbnail so I don't have to watch the video?

This channel is fucking great!

Episode 12, Order of commands on the Unix Shell Key concepts in this episode: - forking sudo, how does it work? - time and resource usage metrics TOPIC 1, sudo and what it does We've talked in my comments on previous episodes about fork() and exec(). sudo is one of many utilities which starts and then does an exec() call, passing control on to a command you specify, just like bash and env both do. sudo is different in one key way, which is that the binary itself is owned by a privileged user, typically root, and is typically given permissions including the setuid bit. This bit, when set on a program, causes the program to execute with the uid of the program's owner, rather than the uid of the user who executed it. This means that any utility with 'setuid root' permissions, including sudo, always executes with root privileges. sudo very carefully guards how it uses these privileges before deciding to exec() to the program you specified. It can further change its own uid, lowering privileges back down from root to a different user, if you ask it to do so. Dave gives two examples in this episode: - time sudo echo hi - sudo time echo hi Each calls sudo, which operates with setuid permissions in both cases. Dave already summarizes the difference: in one, the command line passed to exec() by sudo (after doing its security checks and possibly after changing uid a second time) is 'echo hi', and in the other, the command line is 'time echo hi'. In 'ls -l', setuid permissions on a program replace the 'execute' bit in the first permissions with 's', for example showing '-rwsr-xr-x' instead of 'rwxr-xr-x'. TOPIC 2, time and resource usage metrics One of the keys to good shell scripting is learning how to time out your shell operations. How slow is too slow? How do you measure that? One very common answer is the 'time' builtin to bash. As Dave demonstrates, it reports back real time vs user CPU time vs system CPU time. What are the differences between these three metrics? - real time is the actual elapsed clock time between when your process started and when it ended. - user time is the amount of time your process spent on CPU performing userspace logic (very broadly, math) - system time is the amount of time your process spent on CPU performing kernel logic (also very broadly, math abstracted away from you by system calls) Real time will often be greater than the sum of user time and system time, rather than exactly equal to it; this is because certain activity, such as network I/O and even in some cases disk I/O, is not accounted for as CPU usage. Real time can also be less than the sum of user and system time, especially if your program is CPU-intensive and utilizes multiple threads concurrently. For a simple example of this, you can do something like: time ((tar cfp - /usr/bin | gzip -c - | wc -c > /dev/null) & (tar cfp - /usr/bin | gzip -c - | wc -c > /dev/null) & (tar cfp - /usr/bin | gzip -c - | wc -c > /dev/null) && wait) This example spawns three bash pipelines in parallel, each one counting how many characters there are in a gzipped tarball of /usr/bin on your host. Since compression is computationally expensive, user time in your use case will exceed real time. On a system with pathologically complex ACLs, system auditing, etc., even system time might be high! (Note, this use case, tar to stdout, can be extraordinarily useful in another context: streaming file sets over ssh when interoperating with sudo. Rsync is powerful but getting it to do the right thing when you need to log in unprivileged and then escalate to root can be more trouble than it's worth.) All of these usage metrics are made available to either bash or to the external time utility that ships with your OS via a system call named getrusage(), or via one of the wrapping functions which exists for it such as the wait4() system call in Linux. If you read your OS manual for that getrusage() system call, you will see that user time and CPU time are far from the only resource usage metrics available. This is why the OS-specific utility in /usr/bin is so much more complicated than the equivalent bash builtin. Bash, which is designed to be portable between many systems, only assumes that getrusage() will implement what POSIX, the standard Unix specification, says it should implement. Both MacOS and Linux offer a variety of other metrics, especially around how the kernel is handling the memory demands of the process, and how the process is interacting with the inter-process communication subsystems. These can be critical sources of "ghost" latency which isn't part of any single system call but also isn't attributable to userspace CPU. You may never need to know that stuff for shell scripting but it comes up faster than you think if you're running anything performance-sensitive in production. This is a case where sometimes it's cringe to use the bash builtin instead of the OS equivalent command. But, very often, the builtin still gives you enough to go on if your process is running slowly. But, maybe more importantly, this difference between the builtin and the system version of the utility also demonstrates something which is very often true in computing disciplines: Often, to get portability, you have to give up either observability or performance. Learning what bash is doing when it does things is about learning how to work closer to the metal so that you can get more out of it. CONCLUSIONS This episode had no cringe program and no based program. So, how does it demonstrate why you might suck at programming? First, if you have any confusion at all about what sudo does, or about how it operates, your system privileges exceed your knowledge of how to operate safely. Learn what UID and EUID are in a Unix/Linux concept, how the setuid permission on a file owned by root interoperates with that, and _exactly_ how sudo interoperates with the other programs that execute it, and that it executes. Second, to be good at programming you need to understand how to tune what you're programming. You can't meanigfully tune what you haven't measured. To measure usefully, you have to understand how to collect metrics, what metrics you're collecting, and what those metrics are describing to you about how your program operates. Dave is demonstrating here that reversing only two words in a command can change both _which_ metrics you're collecting--time from the bash builtin, vs time(1) with its different output--and what those metrics signify. One measures full execution including sudo overhead, the other measures only the command which was ultimately executed with root privileges. Which were you trying to measure, and why? I used to tell my work trainees, long ago, Unix is not for the innocent. This video illustrates what looks at first to be a very simple concept, but it's one which hides a lot of complexity, risk, and power within it. When this example makes full sense to you, you'll be a lot closer to unlocking not just the power of the bash shell, but of the entire operating system underneath it. Safe travels!

I always love trying to explain the difference between "su user" and "su - user" People just don't get it... ;)

Honestly it annoys me that we still ship duplicated commands that have been replaced with shell internals for so long. I get that you have to include them if apps rely on them… but it just feels so wrong.

I'm not sure I suck at bash anymore now that I've been watching your videos but I do know your viewers suck at commenting :)

Duplicati to TrueNAS, pushes to offline, offsite via tailscale. I'd imagine your skills and "rsync" are way over what I'm imagining

the sheer amount of information in these two minute videos is mind boggling. Sometimes i have to watch at half speed to fully understand.

Richard Stallman open sourced the Unix beard in 1983. Now anyone can have one.

Nice sir But I don't understand that much 🤯

I don't think having a hairy heart is healthy

You don't need to escape the slash (/) character with backslash (\) in regular expression. Also, anchors are redundant here (^ at the beginning and .*$ at the end )

It depends on whether you're pronouncing it as "reg-ex" shortened from "reg"ular "ex"pression, or as a word by itself, "regex", which would have a G as in giraffe sound following common English phonology. No one says "ree" at the start of the word.

Risk is entirely too high for enterprise use IMHO. Especially with Microsoft being spyware and Copilot. LLM is good if heavily sanitized. Copilot is evil.

It's in the name "Regular Expression", not "Rejular Expression". But the point moreover is you knew what it was referring to and still decided to be pedantic

Ur 0 view shorts are the only shorts recommended to me

Yo mom hehe

Thanks. I had no idea. Usually I use a separate scripting language instead of the shell. This is handy to know.

Thanks Dave, great content

You read the man? 😂 PS: in real life this is slow as hell. This task is a great candidate for "find" with printf-style formatted output

I mean I would use python or if it has to parse lots of files C++

This is so cool! Great content man!

Glad you made the short, first time im seeing the channel! Liked and subbed

This was the video that convinced me I need to start moving from [ ] to [[ ]] in scripts.

Episode 11 - Regex Builtin to Bash Core concepts in this episode: - compound conditional expressions using [[ ]] instead of simple test conditionals using [ ] - the =~ operator in compound conditional expressions Topic 1: Compound Conditional Expressions. Compound conditional expressions are bash expressions on the form of: [[ <expression> ]] They're described in the bash man page at the 'Compound Commands' section of the SHELL GRAMMAR section, and are then expanded on under CONDITIONAL EXPRESSIONS. Compound conditional expressions are part of the basic bash grammar, and differ from the expressions accessible via [ <expression> ]. Note here that '[' is a bash builtin which does the same thing as the test builtin, but requires a closing square-bracket. The conditional expressions enclosed in [[ ]] can be thought of as a superset of the expressions enclosed in [ ], with =~, which does regex matching, only supported in the [[ ]] version. This operator accepts extended regular expressions as described in the regex(3) man page, which are the flavor of regular expressions used by egrep and grep -E, as opposed to the more popular PCREs (perl-compatible regular expressions) which most modern languages implement. A key thing to understand is that you do get access to match groups, and that the matching fields can be seen after =~ evaluation in the BASH_REMATCH. Dave covers this in his example too. Topic 2: The =~ operator for regular expression matching. The =~ operator takes a string on its left-hand side as input text to be parsed by regex, and takes the right-hand side to be a regular expression. It's important to note here, anything quoted, including variable expansions, is treated as literals by the regex parser, not as regular expressions. Return codes from the =~ operation are: 0 - the text matched the regex 1 - the text didn't match the regex 2 - the string provided as a regex could not be compiled as a regular expression. A sample command which worked for me in demonstrating this: re='^(uid=.*) (gid=.*) ?(groups=.*)?$' [[ "$(id)" =~ $re ]] After executing this, I had a $BASH_REMATCH array where [1] had my uid, [2] had my group id, and [3] had any supplemental groups assigned to me. There's also a [0] which contains the entire matching string. It's worth noting, $BASH_REMATCH is a global variable even when you call it from functions. End notes: This is the YSAP episode which persuaded me that I should begin to replace all use of [ ] with [[ ]] in my scripts, both in bash and in zsh. Happy matching!

The regex here is brittle as well. Should have been something like '([^/]*)\s*-\s*(\d{2}(\d{2}-){2}\d{2})\.jpg'. Can be written in a single line in bash: ls ./files/*| perl -ne"m{.*/([^-]*)\s*?-\s*?(\d{4}(-\d{2}){2})} and print $1,':',$2"

Where can i learn regex this good any resources?

I assume the [[ =~ ]] syntax supports regex fully, but does "grep" support the full range of regex functionality like this (e.g. capture groups)?

Well it turns out I in fact do suck at programming.

It all returns to associative arrays. I use one of these in the setup script for my dotfiles. Works for me.

This is awesome!

TIL I write cringe bash scripts