Awk: Hack the planet['s text]! (Presentation)
ฝัง
- เผยแพร่เมื่อ 25 ก.ค. 2024
- Awk: Hack the Planet['s text]! - Part 1: The Presentation
NOTE: This is the original video. It had some screen tearing that was distracting at best and OCD aggravating at worst. It has been re-recorded, with improved slides, some deeper explanations, and NO screen tearing. Watch it here: • Awk: Hack the planet['...
Original Description
Ready to take on the challenges? Head to Part 2: • Awk: Hack the planet['...
Awk has been around almost forever, yet so many today are unaware of it's power and elegance. It is an amazingly powerful tool that is it's own Turing complete programming language. Awk is so powerful that it can be used to create entire services (that process text). But, there's a lot of ignorance out there regarding Awk, and ignorance breeds fear. Come take the Awk red pill like that guy in the documentary "The Matrix" did. Awk can be a ton of fun! Let's make text processing fun again!
We start out by discussing what Awk is, and briefly reviewing the history of Awk. We'll then go over some examples of cool things we can do to whet our appetites. Then we'll go over the syntax and rules of the Awk language. Then we'll see real examples of Awk in action by doing some amazing text processing using only Awk.
Throughout the process, there will be lots of examples that you can run and test yourself (if you want to). Some text files will be provided so you can quickly and easily reproduce the results locally in real time. Source code is available on github: github.com/FreedomBen/awk-hac... .
By the end of this presentation, you will be ready to start using Awk to solve real world problems. You will be comfortable reading and understanding Awk programs and will be ready to slice and dice like a classic *nix hacker.
When you are ready to take on the challenges, you can find the source at: github.com/FreedomBen/awk-hac...
You can also watch my solutions (with in depth explanations) in Part 2: • Awk: Hack the planet['...
If you want to see more videos like this, be sure to subscribe!
UPDATE: There's a copy of the Aho/Kernighan/Weinberger book on the Internet Archive! archive.org/download/pdfy-MgN... - วิทยาศาสตร์และเทคโนโลยี
![Awk: Hack the planet['s text]! (Presentation) - 2023 Update](http://i.ytimg.com/vi/E5aQxIdjT0M/mqdefault.jpg)
![Awk: Hack the planet['s text]! (Presentation) - 2023 Update](/img/tr.png)
![Awk: Hack the planet['s text]! (Exercises)](http://i.ytimg.com/vi/4UGLsRYDfo8/mqdefault.jpg)
This is perhaps the most useful video on awk I've ever seen. Great job.
Great stuff. Learn a lot. Awk is really useful! The exercises are great, too.
For everyone watching: The screen corruptions are only temporary and fixed by the author during the talk.
Thank you, I was considering mentioning the corruptions to him until I saw your comment. Cheers!
I'm 18 minutes in and it's happened about thirty times with no fix, not even a comment on it. What does "only temporary" mean in the context of a video? The video itself is temporary, of course anything in it is temporary. Even more temporary if you click away to a good video.
it's linux 🤣
For reference 23:10 is where it's fixed
WOW !!! - At probably every example I stopped the video and quickly tried stuff out and explored it further - that's a very informative video Ben; Really good and interesting... A big thank you :-)
Hi from HN. Just finished your exercises after watching the Presentation video.
Thank you for doing this. I reckon I fit right in with your target audience as someone who often has to drop into the command line.
I'd always been intimidated by `awk` but now feel further empowered with even just the basics acquired from the video and exercises. The approach was great.
I tried reading books about AWK , You did an excellent job covering almost all features ... Thank You :)
Thank you for the presentation Benjamin. It is really well done and very informative.
Very nice introduction to awk. Thanks a lot for your effort to compile it in a video
Thank you very much for intro to AWK, great job. Appreciate it.
Thank you for introducing me to AWK with this video! I just wrote my first AWK program! It converts an XML file into a YAML fixture that is used to import contents into a database.
This was awesome! I'm so impressed by this wonderful presentation that I'm going to take the recommendation and buy that extremely expensive book.
Thank you, thank you and thank you! Job well done, I no longer need to review O'Reilly AWK book!
Simply superb!
Thanks a lot. Great stuff!
Thanks a lot ❤
Thank you for this lesson!
I took on an Awkful (silent K) challenge from an awk bigot. He designed the challenge - beat him hands down, both dev time and run time, using lexx and cpp.
Awk is completely awkful (silend k).
Nicely expalined. By the way, thanks for the humor at 24:35. Can't agree you more. haha.
Very cool
Small nitpick, just for fun (!)
You said in the talk that NR is number of processed records. But that would mean that for the first record it would be zero. But it is one. So, it’s more like an index of the current record, starting from one, as humans do
Great point!
I'm 12 minutes into this video and I haven't seen much of anything that would actually run on the command line.
What minute are you at now? Asking for a friend
24:22 The screen tearing is happening in the video itself, nothing wrong with your browser. I almost restarted my browser until I heard him say he restarted the computer to stop the screen tearing problem.
Wow, this needs to be on top. I spent almost an hour struggling to fix 'my issue'. smh
AWK is awesome, when you need to bulk filter/reformat/parse text...
gg. Thanks for the lesson
@56:00: The "{$1=$1}1" command doesn't quite do what is promised. Default awk removes the whitespace between and around _every_ field, and print only re-inserts a single space between fields. So it doesn't just remove initial whitespace, it also removes trailing whitespace, and reduces the remaining line down to a single-space separated string.
Edit: To definitively remove initial whitespace only, you should use the sub (or gsub) substitution function and regex match it (they operate on $0 by default).
"{ sub( /^\s*/ , "" ) }1"
Thanks David, you're absolutely right! That's super helpful.
Unfortunately awk can’t process all quirks of csv.
what is the name of the book your talking about? "the awk book" i could not find however i did find the "awk programing language" is that the book your talking about?
Hi Charles, sorry for the dfelayed answer. Yes it's called "The AWK Programming Language" by by Alfred V. Aho, Brian W. Kernighan, and Peter J. Weinberger. I don't know if TH-cam will let me link to amazon here, but it's this book: www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp/020107981X/ref=sr_1_1?crid=2K66EAG2M14MX
There's also a copy on the Internet Archive! `wget archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf`
333th like
if you installing/administering the server you surely have the needed permissions to install any programming language you want.
yeah maybe learning awk is useful if your typical use case is hacking netbsd routers (not yours), but in 99% of cases using awk just because it's more unix like is overated. on windows installing awk is more difficult than installing python or perl. writing in posix shell or awk does not give you any benefits apart of the ability to run it on like very old HP mainframe, or locked down BSD/unix-like system. for anything else just use modern language.
> if you installing/administering the server you surely have the needed permissions to install any programming language you want.
This is true from a "do I have root" perspective, but not necessarily from a "do I have permission/authorization" perspective. Generally you don't want to install anything not needed as it can increase attack surface area and increase storage costs, so in practice you end up with whatever the language stack is for the application you are running, and ideally nothing else. In practice though there will be everything that comes by default with the linux distro, so bash, sed, awk, etc. If you write generic system setup scripts (like I do) that you want to be able to run on every image in the organzation (and outside the org, such as on images from Docker Hub and other registries), awk is the only safe choice for portability. If you want to install ruby or python, that's a huge amount of image bloat and a huge increase in attack surface area just to allow you to do some quick string processing/manipulating.
But even if we did assume we could just install whatever we want, awk is still by far the winner of code golf. Of course it trades clarity for brevity in many cases, so nothing is free, but once you learn the implicit behavior of awk it's really not bad. 4 hours of effort you can pretty learn all you need to grok awk.
This is probably my weakest argument, but awk is also a lingua franca, much like bash. If there's zero chance anybody else will ever read your code, then it doesn't matter, but even if most of the time they won't, if you learn and use the lingua franca tool you're ready for the times when you do share the code.
Overall though, I just think awk is badass, and it makes me happy. It's super quick and easy to bang out command line stuff with it, and it works no matter where I am, whether in a stripped down container build, an old home router, or my modern Fedora 37 machine.
@@freedom_ben If by everywhere you mean linux ok, but it's not 'everywhere', windows doesn't have awk, macOS also needs to install gnu utils and homebrew, and even unixes (BSDs) are having Unix awk, not GNU awk which can affect portability of your scripts.
Anyway, if it suites you of course use it, I just wanted to point out that it is not a silver bullet, and not as universal as admins claim it to be.
@@JamesSmith-ix5jd by "everywhere" I mean posix. but yes windows doesn't have awk by default, but given how windows is primarily a GUI based OS with its own way of doing things, if you want to do non-windows-specific command line stuff anyway you're gonna have some things to install. macos at least used to have awk but I haven't used it in several years so they may have removed it. The BSDs do have awk (I just used it on my pfSense machine a couple weeks ago) and there are very few GNU-specific things that break portability. Even if awk doesn't come with windows, I can't think of any language that is more universally available.
But yes I would agree, there is no fully universal silver bullet, including awk. But awk is by far the closest and for me (primarily a linux user at home and work, with some BSDs) has been a silver bullet. If you're a primarily windows person, awk is certainly not the best choice. Please keep in mind, this talk was originally done at a LINUX conference (Linux Fest Northwest).
terrible screen terring.....
Yeah, that broke my heart when I discovered it :-(
You didn't explain what fields or columns are when you needed to. Are they separated by whitespace? You never bother to define it and it is clearly absolutely critical for anyone to understand what follows.
@Mark Scheck Shhhh … now he’s going to understand what follows. 😅
Mentioned at 28:25 too
And 30:30