Basic Data Engineering Project - End-To-End From Web Scraping to Tableau
ฝัง
- เผยแพร่เมื่อ 30 มิ.ย. 2024
- Building your first data engineering project can be daunting. There are a lot of tools you're supposed to be able to use that range from Python and SQL to the Cloud.
So let's make this project simple and scrape some data, parse it using SQL and loading it into Tableau.
Get A $25 Credit For Bright Data Using The Link Below If You Book a Demo
brightdata.grsm.io/seattledat...
Raw Data
gist.github.com/bAcheron/7a36...
Timeline
0:00 Intro
3:53 Scraping Data
10:00 Reviewing Data
12:11 Working On Snowflake
29:12 Working On Tableau
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you. - บันเทิง
If you like data engineering and you are looking to start you're own data engineering/analytics consulting company, then you should check out my new course here - courses.technicalfreelanceracademy.com/courses/starting-6-7-figure-consulting - and use the coupon code "deconsult" to get 50% off
Thanks Ben! I used something like this to fiqure out Opendoor listings across the Atlanta region and ran it against what they bought at and how much decline they are seeing.
It was fun excersize.
Kind of cool how much free information is out there as long as your willing to scrap the data and make sense of it.
Id also hold on buying a home until 2024... Lot of inventory going to be hitting the market in the next 9-12 months.
👀
Oh yea, its interesting to see even from a high level. Houses are staying on the market at least 2x as long as were only a few months ago
Hello Ben, I just wanted to say that I would love to watch your videos with a better sound quality 😁
Thanks so much for this video.
Glad you found it helpful!
Really good!
thank you!
Thanks Ben. Any resources on swapping out a Python selenium script downloading a csv file for the scraping portion in this vid?
Thanks for making this video. I’ve been contemplating switching to data engineering. This video just reminded me how boring data is to me😂😂😂😂
Glad it was helpful hahahha
I liked the snowflake part of this video.,
Glad you liked it!
Thank you for the vid!! Not sure if this question is obvious or not as i’m VERY much a novice but when you’re working on snowflake (~17:14) where you using SQL queries or coding in a different language? Just not sure what you used to parse everything
It's SQL just likely not all ANSI SQL. A lot of solutions like Snowflake and Databricks (actually postgres has some of these as well) have added in additional functionality. For example, when I create the "Stage" thats not regular SQL. Snowflake lets you query from S3 directly (other tools let you do it too). T
For the Tableau part, can we use Tableau public or should we use Tableau Desktop
SQL parts in snowflake are really confusing and not well explained. Are the scripts available somewhere?
Hello data man
Hello!
Do we need to pay for using the api from bright data?
Great content! Just a suggestion, it might be helpful to slow down and enunciate your words a bit more for better understanding. Keep up the good work!
I'll do my best, yeah I think this video in particular I started mumble rapping
Hi! Recent subscriber here👋 I'm very much a novice on the subject - are the tools you use free? If not, are the free alternatives you can recommend?
There are always free alternatives, you can code similar things.
For projects, do you think it is good to code components like the scraper from scratch like python or is it better to use existing tools? I feel like showing you can code, and know the very granular components of scraping data is great, but showing you use tools that people have heard of is also another argument. The reason being, I wrote a python script that scrapes data using regex, and runs hourly and transforms and loads into postgres. Do you think it would be beneficial to use an existing tool like you used from a hiring manager's perspective?
If you can build it from scratch - do it! Might not be totally necessary, but it shows initiative and is always good practice. The work you do as a student will always be miles different than the work you do as a professional, no matter what technology you choose to use, so whatever increases your skills as a dev or that can be leveraged on your resume - do that!
and how do you connect it to tableau?
Is snowflake free to use to build this project for my portfolio? I only see a trial
You can use it free for like 30days, and you can pretty much keep setting up free instances over and over again.
@@SeattleDataGuy nice thank you! ill try this project and see if i can get a legit DE role. Right now im Senior analyst and not sure if my company will give me opportunities
How long did this take you?
In filming time, probably like 4 hours, i went back a few times when I realized I had been mumbling like SBF for like 5 minutes and had to reshoot.
A little planning too. I had to get used to brightdata, and then there is all the fidgeting around with tableau.
@@SeattleDataGuy I mean total time. Just trying to get a gauge to compare against a newbie like myself :)
Good content.
As a sort of native speaker you are speaking too fast and audience would find it difficult to pace with.
This vidoe isn't beginner-friendly!
Hard to follow what you saying when you are not english speaker because you don't articulate...