How I Use Python as a Data Engineer
ฝัง
- เผยแพร่เมื่อ 27 ก.ค. 2024
- Join Python for Data Engineering Course: datavidhya.com/courses
➡️ Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?pl...
USE CODE: COMBO50 for 50% off
In this video, I have talked about how I use python as a Data Engineer and shared my learnings and best practices I have learned in the last 5 years.
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🎯Twitter - / parmardarshil07
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
Timestamps
0:00 Intro
0:38 Building Data Pipeline using Python
0:51 Data Sources and APIs
1:43 Data Extraction Code
2:14 Data Transformation
2:56 What is Data Pipeline
4:15 Data Storage
4:41 Data Pipeline Deployment
6:44 Course - Python for Data Engineering
7:04 Free Roadmap - Python for Data Engineering
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Hashtags ✨
#dataengineering #python #airflow
![[LIVE] : ONE ลุมพินี 72 วันนี้!! คู่เอก "ก้องศึก vs โจอาคิม"](http://i.ytimg.com/vi/hT1EOSsRPac/mqdefault.jpg)
![มองหน้ากันไม่ติด (Awkward) - โอ๊ต ปราโมทย์ FEAT. MAIYARAP [Official MV]](http://i.ytimg.com/vi/4b02SY_r6Gw/mqdefault.jpg)
Tried new way of editing, let me know if you like this format. Also new project is coming soon on GCP so don't forget to hit LIKE button
Hi. It will be included on your DE course ? or it will be a new one?
Appreciate your good work. Keep doing. Can you please share more details about your editing software. Which software you are using for video editing?
9 Min of Super packed and incredible Video. Thank you so much for sharing the Knowledge.
Lots of great information here - however sometimes you put up these slides and graphics with good information, but you pull away from them really quickly, I have to pause video to ingest all the information I'm seeing, perhaps be mindful not to cut away too fast. This is not a Michael Bay movie! Otherwise excellent video!
@@markslima1557 This is just an overview video, I have indepth video on each topics
Search
1. Twitter Data Pipeline Darshil
2. TH-cam data analytics
3. Uber Data Analytics
You will learn everything in depth
This was really great short and completely on point. Thank you.
Thank you for the clear and crisp summary.
Brother ... learning a lot from you .. Dedication determination and focus ...Thank you so much :)
Exactly what was looking for 👍🏻
Great video!
Great video. Recently I have seen a lot of companies encouraging graphql to query their API.
Thank you Karan
Thank you for good explaination!
I haven’t added any comment on any video on youtube till now, I’m following you and your videos since more than a year, your content for Data Engineering is tooo amazing among all the other data engineers channels.
Keep up the good work buddy!
Awesome! Thank you!
Thank you Darshil🎉
hey darshil, how does python (as you described your process in the video) fit in a cloud platform, such as aws/gcp/azure
I think Im loving the way you lecture. Just followed you on twitter and have subscribed to your channel. I want to verge on Data engineer path. Python doesn't stick to my brain or let me say I am being impatient. Any link to learn python fast and understand better? Atlas I can be doing some data ingestion, Minning and visualisation like you
Amazing video.May I know which editing software are you using?
Crisp and clear ❤
Very valuable information
Good one, covering the overview of DE
Very informative video.
Lots of information covered in simple terms.
This is great , Darshil. This video gives some really wise nugget of knowledge , which is at times hard to get at workplace even and also in any other courses. You have explained it so simply. I am quite experienced but took a long route to understand. You are doing it for free,. Great Job
I am learning AWS Refshift cluster from pluralsight but I don't understand anything. I need to learn redshift clusters in depth for a data engineering job. so please give me some solution.and please reply me 🙏
Hiii.. Can you do the azure data engineering projects end to end for one plssss
I’m interested in signing up your data engineering class. Will pyspark and airflow be learned in the course? Will the projects involve pyspark or airflow?
I enjoy each and every video, learnt something new from you and keep going on your efforts @darshil and Thanks
Thank you Ravi
th-cam.com/play/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg.html
can you please suggest some budget laptops for student studding data engineering.
Bhaiya kya aap bata sakte ho ki data engeneer k liye konsi stream leni sahi hogi aur konsi bachelor's degree leni sahi rahegi ?? Please reply
You are a genius...
What is the best way to make a logic building in python tell me some steps
bro, I want to learn about data engineering for Redshift. But I see that you have course available for snowflake. My work belongs to manage redshift as of now. What should I do? Please help me.
Mast content hai re Bhai 👍🏻
Thank you
Hello bro...Iam having 8+ yrs experience in Sap basis administrator.. Willing to move to Data science will that be good choice? If yes what is best institute you suggest me to train.
Hi Darshil. Is Pandas syntax necessary to remember for interviews? I'm asking because pandas also supports SQL syntax.
Pandas is such a useful and perfect library for working with dataframes in Python, I would definitely make it a priority to have at least the basics down. It isn't difficult, and I think rather fun.
where else you can find all of this!Its all here
Your best video!
Thank you
Please share your Python training video for Spark.
Thanks
Sir you really are my mentor, I followed all you instructions as a Data Engineer intern.
I saw all you Content.
I just want to know, how one can do freelance as a Data Engineer. Are we allow to build ETL tools or the system (ETL+Database design+ APIs) and sell ???? Please make a video this 🙏
th-cam.com/play/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg.html
@darshil parmar Azure users do this python course
And how much use poration of this course
Tnq sir
bhaiya mujhe data science seekhna hai , lekin usse pahle mujhe basic se python advance tak seekhni hai toh kya agar mein aapkaa course purchase karun toh kam se kam beginner to intermediate tak toh pahuch hee jaunga naa🤔
Haa, basic se advance plus project
Please make a data engineering project on Rust using polars library.
th-cam.com/play/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg.html
make video on docker in mac
Hi Darshil, your python course got English subtitles? cheers
No subtitles but content is in english
Nice explanation bro can you please tell me how to extract data from Api using python code . Can you please send me. In our project we have to write this code in function apps. Please share
Check out my video Twitter Data Pipeline with Airflow
I have 2+ years into the DBA, am I eligible to land to Data engineer????
th-cam.com/play/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg.html
Genius
Wow!!
Why you have not discussed about dealing with databases in your course: Python for Data Engineering?
It's a part of SQL DE, in order to deal with database you need basic SQL understanding
Hii darshil pls add the some projects in azure
Coming soon
Can i do it with PYspark
Yes
Please make a course of sql for data engineering as well
It's next in the pipeline
@@DarshilParmar looking forward to it 😁
Yeah! 🎉
th-cam.com/play/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg.html
Make Series on Snowflake with Matillon ETL
Why not azure 🥺
It will come in future
@@DarshilParmar Can you provide a good source to go through for Azure Data Factory not like Udemy.
@@mohammadmauwaz984 www.youtube.com/@WafaStudies
Bhaiya aap hamesha English hee bolte ho kyaa aap thoda saa simple Hindi mein sikhaa dijiye toh jyada better rahega aapka content accha hai lekin aap Hindi bhi bola kijiye sikhate time ya fir Hindi mein ek course launch kar dijiye
As a beginner, just couldn't follow, too fast
There are end-to-end projects that go in-depth
Just search "Darshil Parmar Projects" you'll get hour-long tutorial
brother make it free please people will give you more links and subscribers maybe some people can't afford like me
Bhai thodhi Hindi bolo lo yaar
As a viewer, I've to ask, is that thumbnail makes your content more engaging? I don't understand. For me, it's kind of irritating as everyone opens their mouth...or puts hand on face or smthng
I understand even I don’t like it but when new viewers gets recommended this video reactions make it more clickable, the way TH-cam algo works and I have to do this way
@@DarshilParmar It's okay because your content is good. That's why I see you daily in my feed. No offence on the your thumbnail but I personally think that an standard way of thumbnail (ex. bytebytego, ibm, Hussain Nasser) would work much better in long run and create more value to viewers
Definitely, I’ll look into this
I think you have lost creativity . This Twitter api is just a repetition of one of your earlier video. Put some effort while making content or u can take a break as well to get back the creativity
I knew this might come but think about this there are many people who never watched my content and this introduces them to my channel, not everyone can watch 1 hr long video but this summary can give me path to watch entire video
Right Darshil. This is really helpful for me who has been looking for a video to learn about extracting data using APIs. Thank you.
@@deelovely7565 Do you need Data Engineer+ developer?
@@ajayram7973 I am learning data engineering
@@deelovely7565 All the best 👍
Hi @DarshilParmar I bought the course linked in this video but I’m unable to interact with any videos. Can’t pause, skip, etc.
I have emailed you since then but no response yet. Kindly respond to my email or issue a refund. Thank you
Hey,
Let me check and will get back to you
Thanks so much for the quick resolution! All good now
Hey Johannes....I would like to know your review about this course, before I purchase it. Thanks you.
excellent demo Darshil , Please share your contact number - i would like to get a course from you in Data Engineering