How to load data into PostgreSQL using Python | Your first ETL pipeline | Data Engineering Tutorial

What are the basics you should cover when you build an ETL pipeline?
🎬In this episode I am showing you how an ETL process would look like. We are going to load the data from Task 8 and load it into our local PostgreSQL database.
✨If you are new here:
This is a public and weekly online Python Class which you can attend as well. All you have to do is to join our Discord server if you are interested! Links down below :)
==== CLASS STARTS AT 4:30 ====
⏱ Timestamps
0:00 Intro & Agenda
1:00 Recap - How to start web scraping using DEV TOOLS
4:30 Example how my ETL pipeline would look like
9:30 What is a logger?
17:00 Test the database connection
27:25 How to create a database table based on your data
42:00 How to extract the .json data and load into PostgreSQL
========================
If you want to learn how to become a programmer and build automated software, this video is for you. Feel free to drop them in the comments or join our socials!
========================
🙋‍♂️Community Discord:
/ discord
theselfdev.com
blog.theselfdev.com
pomo.theselfdev.com
📲 Instagram:
/ self.devs
📩 Connect with me:
Instagram - / long.integer
Twitch - / longlikeshort
Twitter - / longinteger017
BGM: • Video
Subscriber: 1174
#dataengineering #pythontutorial #dataanalysis
📚The FASTEST way to become a Data Engineer:
• FASTEST Way to Become ...
📚Knowing this about DATA will shock you:
• This SECRET behind NET...
📚FREE Python Programming Class:
• How to start learning ...

[@dogedingo1535]

Also very interesting how powerful try and exception blocks are! Definitely going to implement them more into my code.

[@longinteger]

yea. also look up when you should log errors and when not

[@evaristusokey6825]

Good one sir, however, I didn't get to understand how am supposed to download data straight from a website into the postgres using python

[@sarahcross7895]

this was super helpful, thank you. do you have a tutorial that explains how to load the data to PostgreSQL when the json data is nested?

[@longinteger]

hi Sarah, thank you for your kind words. Unfortunately, I don't have such a video but I can recommend you looking into flattening json objects before loading them into your database.

[@dogedingo1535]

This was written on my vissle_keyboard_keyboard and sent as a comment by clicking 'comment' with my lg_superlight_mouse (@33:40)