Part 1 - Project Overview | What is AWS EMR | Extract and Transform Redfin data with AWS EMR

#dataengineering #emr #spark #pyspark #jupyterlab #jupyternotebook #aws #emrstudio #etlpipeline #redfin
In this video, I explained what Amazon EMR (Elastic MapReduce) is all about and its benefits in processing big data. I then showed how you can create VPC and then spin up EMR clusters within this VPC. Later, I showed you how to create Amazon EMR studio and Jupyterlab after which I attached the Jupyter notebook to the provisioned cluster. I then showed how to write Pyspark code in the Jupyter notebook attached to the provisioned EMR to extract data from the Redfin data source, process it and load the transformed data as parquet file into an S3 bucket.
Please don’t forget to LIKE, SHARE, COMMENT and SUBSCRIBE to our channel for more AWESOME videos.
*Books I recommend*
1. Grit: The Power of Passion and Perseverance amzn.to/3EZKSgb
2. Think and Grow Rich!: The Original Version, Restored and Revised: amzn.to/3Q2K68s
3. The Book on Rental Property Investing: How to Create Wealth With Intelligent Buy and Hold Real Estate Investing: amzn.to/3LLpXRy
4. How to Invest in Real Estate: The Ultimate Beginner's Guide to Getting Started: amzn.to/48RbuOb
5. Introducing Python: Modern Computing in Simple Packages amzn.to/3Q4driR
6. Python for Data Analysis: Data Wrangling with pandas, NumPy, and Jupyter 3rd Edition: amzn.to/3rGF73G
**************** Commands used in this video ****************
Check out my github Repo
github.com/YemiOla/data_engin...
**************** USEFUL LINKS ****************
1. Redfin Analytics|python ETL pipeline with airflow|Data Engineering Project|Snowpipe|Snowflake|Part 1 • Redfin Analytics|pytho...
2. Redfin Analytics|python ETL pipeline with airflow|Data Engineering Project|Snowpipe|Snowflake|Part 2 • Redfin Analytics|pytho...
3. Zillow Data Analytics (RapidAPI) | End-To-End Python ETL Pipeline | Data Engineering Project |Part 1 • Zillow Data Analytics ...
4. www.redfin.com/news/data-center/
5. docs.aws.amazon.com/emr/lates...
6. PostgreSQL Playlist: • Tutorial 1 - What is D...
7. Apache Airflow Playlist • How to build and autom...
DISCLAIMER: This video and description have affiliate links. This means when you buy through one of these links, we will receive a small commission and this is at no cost to you. This will help support us to continue making awesome and valuable contents for you.
#dataengineering #emr #spark #pyspark #jupyterlab #jupyternotebook #aws #emrstudio #etlpipeline #redfin

[@vfontefontechavez1226]

Incredible, this is the third of your projects that I replicate. I think it was time to make this comment, you have actually helped me a lot. I hope this channel grows a lot.

[@tuplespectra]

Thanks so much for your comment. I'm glad you were able to replicate the projects and are finding them valuable.

[@daviddean8749]

hey, thanks for the project, helped me a lot to nail an interview :). How is EMR doing compared to aws databricks? Which would you recommend? Would be great if you could make a video on how to process streaming data (kinesis, firehorse, kafka?).

[@tuplespectra]

I'm glad my videos helped you nail an interview. This comment means a lot to me. Thanks for dropping the comment.