I've been using jq for years. I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT. JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful
Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!
I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.
If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.
I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter). The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.
For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo
This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.
I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful. nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell. Definitely a good tool for data querying/manipulation though.
I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.
Prime pushed me get more knowledgable at my cli one of the 1st things after neovim was jq, surprise that you did not know this earlier. Im actually using to pipe from a curl and into a xargs to perform queries in a row
Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing
In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend
jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!
Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.
After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.
@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection. Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.
Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.
THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!
That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.
Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.
okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious there's jid, which allows interactive json filtering than there's jiq, which also allows interactive json filtering, but uses jq as its backend then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql
jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...
This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!
You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).
You can even pipe a curl to a rest api into jq that returns json.. awesome! Great for testing an api from the terminal.. thanks for sharing! If I could add jq to a bash script that’s part of a cron, you could do some interesting data extraction and save it to a new json file to be evaluated or transferred into a folder that an application actually can read or display through a web interface.
I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too
I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day
I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is. I actually started using powershell on mac because its support for objects and converting from json is so much better.
Honestly very surprised to see that so many devs don't know jq. If you've tried to work with any sort of web API in a shell script, jq is like the tool they teach in 101.
Nice 😊! If I wanna look for logs in a humongous log file with a certain timestamp for debugging, this is super super useful! Thanks 😊 prime! NGL this looks super familiar though, I may have used something similar, but this is neat - gonna try it! Yee haw 🤠! - Amy
I've been using jq for years.
I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT.
JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful
I thought only i knew about this XD
Stamps:
1:35 - Reading a file
1:39 - Compact (-c)
1:45 - Specific field (.)
1:50 - Raw (-r)
2:21 - Showing keys (keys)
2:31 - New object
3:00 - Select not null fields
3:31 - has()
3:47 - Math
4:59 - Contains
Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!
Look up one more character and search for jqp. Soooo nice. :)
I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.
If only I'd have known this a year ago. And it's been there on my computer for who knows how long! Very useful - thanks for sharing.
I use yq (the YAML equivalent) in my deploy jobs for a lightweight way of updating argocd applications in my IAC repos. Works great.
I used this to deal with a 9GB json file I was handed.
Split it into 90MB parts before my database ingest and it saved me so much effort.
Do you want the Earth to collapse into a black hole under the weight of 9 gigbyte JSON? Are you nut?
This is actually pretty sick, I wish I would have known about this literally 2 hours ago where I had a big json file I wanted to get some info from
for me was las week... fumbled like a mf...
If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.
Still learning and only understand 15% of what you talk about but you’re still entertaining and motivate me to grind, that’s talent.
My god. This left me speechless, I know NOTHING about CLI tools! Thank you Prime for showing us this!
Everything useful was first a CLI tool. Nothing you can’t do.
@@TheStickofWar It always comes back to command line and config files.
I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter).
The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.
For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo
This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.
I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful.
nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell.
Definitely a good tool for data querying/manipulation though.
This sounds like my next tool time moment. 👍
Update: Awesome it's in the Termux repos 😸
Was about to comment the same. I haven't looked into jq much since I can do the same and more with Nushell without being limited to just json
I am waiting for nushell to mature and stabilize, its awesome but changes too often and scripting just recently became a thing
two other text manipulating commands that are useful are 'awk' and 'sed'
When I use jq, sometimes it's really nice to use the crate "fltn" to pipe json to fzf, and then maybe even pipe my interactive selection back into jq.
I knew and I used jq for quite a long time, but instead I didn't knew about I can pipe a vim buffer to different program xD
I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.
Prime pushed me get more knowledgable at my cli
one of the 1st things after neovim was jq, surprise that you did not know this earlier.
Im actually using to pipe from a curl and into a xargs to perform queries in a row
Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing
In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend
jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!
Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.
After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.
Wait what, this is the first time your hearing about this tool, just wait theres `yq` for yml
The new golang yq is the bomb diggety. Old python yq was whack
There's also xq, but I prefer xmllint --xpath as it's pre-installed on most systems
@@replikvltyoutube3727 It's called xslt, and it's been around for 25 years. xsltproc is the cli tool.
The yq I have installed includes xq and jq. It makes shell scripting suck just a little less.
YQ is a wrapper over JQ 😂
That's the kind of content I started to love you back in the days. Awesome
Been using this bad boy for a week and oh my God I've been missing out badly. More quick tutorials like these and how they work for you!!!
I've been using nvim wrong too that !jq there opened my eyes
Im searching for that CLI thing all my life! Thank you!
i made something similar for web scraping at one point but other than that i have no idea what you need this for so i want to ask what you do with it
It's great to see jq to be maintained again. Great tool that deserves to be maintained.
I use powershell to do this. Can not only read/select/filter json but also xml and csv
That’s cool. How does this work using PowerShell?
@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection.
Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.
@@tom_marsdenpiping to convertfrom-json is the way. It just spits out a native powershell object/hashtable.
Nice one. You could consider making short videos for such cli tools. It's great motivation for using them
it goes very well with jless!
This is why I love the comments section. Now I have a new tool to pair with jq and jet.
“what could this guy possibly so hyped about-oh yeah jq is really cool and i should remember it more often”
jq is life, it makes the AWS cli tool actually usable
Hahah yeah been there before , the aws cli is a bit shit
Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.
Wait until Primeagen discovers "jc", a dark brother of "jq", then it completely changes the interaction with the output of any core linux/bsd utility.
There's nothing more satisfying than a well crafted jq
THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!
I have to use jq so infrequently, I always forget the query syntax and have to look it up again.
That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.
jq is awesome. Yet there's more power if you dive into the nushell. It's like a shell with jq and rust-like typed scripting built-in
Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.
It always warms my heart seeing devs learn more about the myriad of cli tooling available. Just wait till you see parallel
It's been on some vid but just not featured as the main topic. He seemed to appreciate it.
okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious
there's jid, which allows interactive json filtering
than there's jiq, which also allows interactive json filtering, but uses jq as its backend
then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong
we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend
oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql
0:55 that’s a Quagmire pause during flight announcement.
Actual tool is crazy.
i understand and agree with your excitement it's a God sent jewel to make my bash scripts less depressing
This video didnt just teach me more jq than I already have been using, you taught me more about (neo)vim as well
LEGEND
Man finally learns a useful programming language and is blown away.
Well, congratulations Primeagen for being of one of the lucky 10.000s recently
Your wife's idea of a good tool, is different
nice nice nice nice ligma nice nice
what's ligma
ligma balls @@darukutsu
@@wowvain2989 GOTTEM
@@wowvain2989 lmao
jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...
this video is going to change my life in more ways than prime can imagine.
This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!
I just always use jq for parsing out data from AWS secrets manager in a bash script... Used it for years but nice to see it can do way more.
You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).
Might have to check that out. Usually I just paste json into a browser console and use javascript. Handy to see it in a text editor though.
Sick bro squeal for jezson 😂 🎉🎉
That opening was like poetry. jq is a vital tool.
I knew jq for a longtime just never played around with it this much
I use jq too infrequently to remember the syntax despite using it for years.
You can even pipe a curl to a rest api into jq that returns json.. awesome! Great for testing an api from the terminal.. thanks for sharing! If I could add jq to a bash script that’s part of a cron, you could do some interesting data extraction and save it to a new json file to be evaluated or transferred into a folder that an application actually can read or display through a web interface.
9:20 "Tech humor: jq json jokes"
For those who don't know, then there are substitutes for yaml and xml also: yq and xq
Mixing it with fzf makes even more awesome
fz I find for my json digging sessions even more useful.
I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too
knew about jq...but always struggled with the syntax. thx homie
I use jq and yq every so often when I need it. However, because I use it infrequently, I need to go back and learn it again and again
jq is pretty great, between that and jless I can view all my structured data files and manipulate them easily. Nice vid!
btw, there's also yq if you happen to have a whole bunch of yaml that you want to parse through with the same syntax.
Never knew that Jquery had such an awesome command line interface!
have been using this for years. handy tool
I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day
This is a cool tool, I didn't know it existed. I usually just write some basic python code to parse json files. But this seems much faster and easier.
4:43 - 4:58 Poor arch users. Prime went too harsh on them. 😅😅
Jq in terminal is awesome, but jq with vim is incredible!! Installing now.... thanks Prime!!!
~ This video is 101% useful!! ~
I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is.
I actually started using powershell on mac because its support for objects and converting from json is so much better.
This got me thinking about piping files in vim through nu-shell. Sounds nice.
That is handy, but right now I personally prioritize time into learning ksql to analyse data within kafka-topics.
i use yq/jq to flatten out a file into a properties like format and pipe it through fzf for easy querys on the fly
OMG, I wish I knew about this before. Nice tool
4:54, you dont have to attack me like that . That personal bruh...
jq saved me many many times! The ability to select things is amazing!
I wish I had it yesterday when I had to remove some playlists from a massive Spotify backup json file.
JQ plus bash is awesome for api endpoints
Cool, using it often, btw did you know there's also a yq (same thing for yaml)
So this is what it actually feels to see Primeagen talk about something that you know what he's talking about?
Have you seen yq? It allows you to do YAML, JSON, XML, and CSV, but it has less features than JQ.
I personally prefer jaq, which is a RIIR that also fixes a few bugs and footguns of the original
good call getting this off the clips channel, jq needs more love than that
jq+curl has supplanted python for exploring rest apis for me. Also nice to pipe certain fields to sort to get some basic frequency stats
Yes jq is a very handy tool. I don't use it daily but when I need it, it fits the bill.
Honestly very surprised to see that so many devs don't know jq. If you've tried to work with any sort of web API in a shell script, jq is like the tool they teach in 101.
Jq was my best friend last year, last month, last week, yesterday, today as well
*notification for new prime video*
Speaking of getting excited for a tool
I made something like this in PHP years ago but this looks much faster and has more options.
The hair on prime while selected text
Nice 😊! If I wanna look for logs in a humongous log file with a certain timestamp for debugging, this is super super useful! Thanks 😊 prime! NGL this looks super familiar though, I may have used something similar, but this is neat - gonna try it! Yee haw 🤠! - Amy
hmmm also noticed @5:40 a drinking jar...shouting out to Cewpins? :)
I had a similar reaction to learning to use the find command.
JQ? Whoa buddy, I thought this was a programming channel.
lmao
Lol
jq is the best! I used a lot with kubectl output. Especially as the kubectl jsonpath support is abismal still
Love your Easter egg hair color, Prime!
Prime would probably loose his mind if looking at Nushell ❤ JSON, YAML, CSV, TOML, SQL... all at once!