Save Pandas Dataframes and CSVs directly to PostgreSQL Database (Must Know Skill in Data Fields)

In this #automation #tutorial you will learn how to save Pandas #dataframes directly to PostgreSQL using one line of code. You will also how to retrieve data from PostgreSQL and loaded directly to a pandas dataframe.
The python packages used are #pandas #psycopg2, and #sqlalchemy .
Blog and Code:
github.com/ami...
Note: I deleted the database I used in this example. The credentials used won't work.
#like & #subscribe
Thanks you For Watching.

[@aravinda6563]

continue posting these kinds of videos 👍👍👍

[@AminBoutarfi]

Thank you, I will!

[@Pasdpawn]

thanks for the effort brother. Trying to land a data engineering job. Developing a portfolio with some ETL pipeline projects. Currently working on spotify api pipeline.

[@veenavipeesh143]

The best explanation. Thanks for the video

[@nataliachernysheva7780]

The best explanation I could find, thank you!

[@Promptcomputing]

Loved this tutorial. Thank you! Followed you on github and youtube.

[@AminBoutarfi]

Thank you! Appreciate the support!

[@boutarfimanel2893]

Great 😃 thank you 😊

[@omarosefau3811]

Fantastic tutorial, thank you Amin!

[@AminBoutarfi]

Thank you for the support 🙏 Subscribe for more!

[@dobrinyanicitich7514]

Thanks, very good!!!

[@AminBoutarfi]

Thank you!

[@alejandrofidececchi3284]

Excellent tutorial! It was very helpful!

[@jamesbrown6591]

Excellent tutorial, thank you sir!

[@user-gh4zy8gu6q]

Thanks you very much, especially for so clear explanations. Have a nice day!

[@AminBoutarfi]

Thank you for the support! Glad you found it useful!

[@useruserrrrrrr]

Great video

[@AminBoutarfi]

Thank you!

[@wheel0513]

gracias!!! great video. super useful and direct

[@juancespedes3141]

Loved this tutorial, really enjoy the way you explain from scratch. But when i try to append the data, i get this error: "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfa in position 192: invalid start byte", I've been trying to automate the proccess of saving CSVs on a postgres database, specifically a Supabase hosted database, but cant manage to get over this error no matter what i do. I was wondering if maybe you went through this error at some point, if so I'd be gratefull if you helped me with this. For context, I am in colombia rn if that affects the encoding somehow

[@mehmetkaya4330]

Great tutorial!! Thank you so much!!

[@AminBoutarfi]

Glad it was helpful!

[@yosepinpakpahan1168]

Thank you sir!

[@boutarfimanel2893]

Thanks 🙏

[@Lina-zn1yq]

thank you 🙏

[@AminBoutarfi]

🙏

[@diaconescutiberiu7535]

What about the dispose of the engine? You got side tracked with the csv. Also how would we get data from sql to dataframe, using pyscopg2 (not sql alchemy). Idea for an extra video: you have a table in your db, with some data. The data in the dataframe changes over time. Task: if record not present in db, create it; if record already in db but some attribute changed, update it; if record no longer in df, remove it also from db

[@AminBoutarfi]

Thank you for the Thoughtful comment! Yes I got side tracked haha. For Psycopg2 and Pandas, I'm not sure, I always use SQLAlchemy. Also thanks for the video Idea, added it to the list!

[@yashodiptakore7599]

what if password contains @ so servername is picked from first @ symbol.

[@adityanagdeve8396]

what if we don't pass the column's name in it? or do we not know the columns?

[@AminBoutarfi]

Not sure what situation you referring to here, but knowing column names in either a Pandas Dataframe or a PostgreSQL table is straightforward. Just Google it!
For example in Pandas you can use: list(df.columns) which returns a list of all column names.
For PostgreSQL you can use the following code:
SELECT *
FROM information_schema.columns
WHERE table_schema = 'your_schema'
AND table_name = 'your_table'
;

[@Nite_coder]

Could you add the code to the description?

[@AminBoutarfi]

I added it! Thanks for reminding me!