What is AWS EMR | Extract and Transform Redfin data with AWS EMR | EMR Studio | Pyspark Notebook
ฝัง
- เผยแพร่เมื่อ 24 ต.ค. 2023
- #dataengineering #emr #spark #pyspark #jupyterlab #jupyternotebook #aws #emrstudio #etlpipeline #redfin
In this video, I explained what Amazon EMR (Elastic MapReduce) is all about and its benefits in processing big data. I then showed how you can create VPC and then spin up EMR clusters within this VPC. Later, I showed you how to create Amazon EMR studio and Jupyterlab after which I attached the Jupyter notebook to the provisioned cluster. I then showed how to write Pyspark code in the Jupyter notebook attached to the provisioned EMR to extract data from the Redfin data source, process it and load the transformed data as parquet file into an S3 bucket.
Please don’t forget to LIKE, SHARE, COMMENT and SUBSCRIBE to our channel for more AWESOME videos.
*Books I recommend*
1. Grit: The Power of Passion and Perseverance amzn.to/3EZKSgb
2. Think and Grow Rich!: The Original Version, Restored and Revised: amzn.to/3Q2K68s
3. The Book on Rental Property Investing: How to Create Wealth With Intelligent Buy and Hold Real Estate Investing: amzn.to/3LLpXRy
4. How to Invest in Real Estate: The Ultimate Beginner's Guide to Getting Started: amzn.to/48RbuOb
5. Introducing Python: Modern Computing in Simple Packages amzn.to/3Q4driR
6. Python for Data Analysis: Data Wrangling with pandas, NumPy, and Jupyter 3rd Edition: amzn.to/3rGF73G
**************** Commands used in this video ****************
Check out my github Repo
github.com/YemiOla/data_engin...
**************** USEFUL LINKS ****************
1. Redfin Analytics|python ETL pipeline with airflow|Data Engineering Project|Snowpipe|Snowflake|Part 1 • Redfin Analytics|pytho...
2. Redfin Analytics|python ETL pipeline with airflow|Data Engineering Project|Snowpipe|Snowflake|Part 2 • Redfin Analytics|pytho...
3. Zillow Data Analytics (RapidAPI) | End-To-End Python ETL Pipeline | Data Engineering Project |Part 1 • Zillow Data Analytics ...
4. www.redfin.com/news/data-center/
5. docs.aws.amazon.com/emr/lates...
6. PostgreSQL Playlist: • Tutorial 1 - What is D...
7. Apache Airflow Playlist • How to build and autom...
DISCLAIMER: This video and description have affiliate links. This means when you buy through one of these links, we will receive a small commission and this is at no cost to you. This will help support us to continue making awesome and valuable contents for you.
#dataengineering #emr #spark #pyspark #jupyterlab #jupyternotebook #aws #emrstudio #etlpipeline #redfin - วิทยาศาสตร์และเทคโนโลยี
![อาจารย์ยอด : ลุงแว่วเล่าเรื่องผี [ผี]](http://i.ytimg.com/vi/Fd2Qsy4dqpM/mqdefault.jpg)
Your videos really helped me a lot in learning about cloud computing platforms. Please make more tutorials like this!!
Awesome! I'm glad our video is really helping you to learn a lot about cloud computing platforms. I encourage you to go explore my other data engineering project videos where we used other technologies as well in addition to AWS.
Thanks for this sir, I'm always looking forward to your mentorship
My pleasure. Thank you!
Another masterpiece!!! ♥
Thanks for your comment.
Great sessions!
Thanks!!
Good tutorial for my learning, Thank you.
You are welcome! I'm glad it was helpful.
Hi tuplespectra, Great to see this project and learnt a lot. One Suggestion from my side is pls use some small dataset so that cost on AWS will be reduced.
Thanks for your comment. Suggestion noted.
Hello.. i have been following all your vedios and learning to build data pipelines as a beginner. Thanks and your vedios are a blessing for beginners. I request you to do vedio on how to run job on airflow from starting the spin up of emr do the ETL process and terminate the cluster as we do in companies. This would really help
I agree, I wish he can hold a full end to data warehousing course. I really loves how well he breaks the processes. Such facilitators are not that many on this platform
Thanks for your comment. Appreciate it! Your request is exactly what will be released in next 1 or 2 weeks. Work is going on it already. Thanks!
This comment really means a lot to me. It shows that our content is found valuable even to beginners. We hope to start working on creating courses next year, including data warehousing as time permits. Thanks so much.
@@tuplespectra I really can't wait sir, thanks so much for granting my request.
what did it cost you guys in AWS for this project?
Great video. I am really learning a lot from your channel. Can you please make a video using apache Kafka ?
Thanks! I'm glad you are learning a lot from us.
Hi Tuplespectra, great video! Can you let us know how much did AWS charge for this demo project? I plan to follow your video and do one in my personal AWS account
Did you get the cost?
how to update in pyspark?