If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
Solid overview. Working with APIs is a good skill set to have for data engineering. Turning JSON formatted data into tabular data for humans to understand is very important!
incredible. i am starting to work with that kind of api you name at the end and it makes me laugh how you describe all the problems i am facing right now. "but whe need the data so that's what we did" 😅 excellent. subscribinnng
You briefly discussed it, but could you talk a little bit more about the config file you discuss for parsing? I just know my code starts to become a little verbose when I have 30 different functions to parse different API calls, especially if there's additional checks that an http status code can't tell you
At my company, we aim to be build infrastructure that's reuseable - why do something 10 times when you can do it once right? For example, we have an agnostic flat file loader that's quiite robust. I'm currently trying to build something similar for APIs, however it's proving quite tricky due to the nature of JSON and semi-structured. So far, I setup a recursive function that turns the nested JSON into one wide table, but it's difficult to then get that into a SQL environment if the table is too wide or to explode out some of the columns in Python. I'm now attempting a different method. I have a function that analyses the deep of the nested JSON, how many levels there are, the deepest path etc. I then want to use the metadata to parse it using json_normaliz, but also tricky due to the nature of JSON. Am I wasting my time trying to make something too dynamic for API extraction?
Thanks for the callout, something happened with editing, I was actually confused by this because i know i film HD, but I didn't realize the resolution was impacted.
yea but its really bored man everyone knows that so superficial when making this with chatgpt at least think like how can i be useful to people rather doing spam like videos, at least add paginated calls, airflow http operators , things like that man. i am really bored .
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
Solid overview. Working with APIs is a good skill set to have for data engineering. Turning JSON formatted data into tabular data for humans to understand is very important!
Enjoying these back to the basics videos! Perfect timing for me too
Yeah, it's been going back through things I take for granted now.
You said there'll be an Extract from Database. I'm still waiting for learning that part 😊
incredible. i am starting to work with that kind of api you name at the end and it makes me laugh how you describe all the problems i am facing right now. "but whe need the data so that's what we did" 😅 excellent. subscribinnng
Love this, perfect level of detail for where i'm at
Great overview! Thanks for sharing!
great explanation.
thank you!
Thanks!!! For this video!
Glad you liked it!
nice overview! thx u
You briefly discussed it, but could you talk a little bit more about the config file you discuss for parsing? I just know my code starts to become a little verbose when I have 30 different functions to parse different API calls, especially if there's additional checks that an http status code can't tell you
At my company, we aim to be build infrastructure that's reuseable - why do something 10 times when you can do it once right? For example, we have an agnostic flat file loader that's quiite robust. I'm currently trying to build something similar for APIs, however it's proving quite tricky due to the nature of JSON and semi-structured. So far, I setup a recursive function that turns the nested JSON into one wide table, but it's difficult to then get that into a SQL environment if the table is too wide or to explode out some of the columns in Python.
I'm now attempting a different method. I have a function that analyses the deep of the nested JSON, how many levels there are, the deepest path etc. I then want to use the metadata to parse it using json_normaliz, but also tricky due to the nature of JSON.
Am I wasting my time trying to make something too dynamic for API extraction?
can you make a video about how to deal with schemas of APIs and how they change over time :) dates and temporal data in particular
Thanks for the video. The quality though is 720p.
I agree with u. a higher resolution is needed
Thanks for the callout, something happened with editing, I was actually confused by this because i know i film HD, but I didn't realize the resolution was impacted.
yea but its really bored man
everyone knows that so superficial
when making this with chatgpt at least think like how can i be useful to people rather doing spam like videos, at least add paginated calls, airflow http operators , things like that man. i am really bored .