ฝัง
- เผยแพร่เมื่อ 15 มิ.ย. 2024
- In this video, I’ll share my step-by-step process on how I would learn Data Engineering if I could start over. Data Engineering is a fast emerging field within the Tech industry; where more and more people from traditional data science/software backgrounds are pivoting towards. We’ll cover the fundamentals of Data Engineering, and talk about some advanced topics you’ll need to learn in order to maximize your chances of success. Hope you enjoy this one!
Timestamps ⏰
0:00 Intro
1:45 Fundamentals of Data Engineering
4:51 Core Data Skills
7:30 Advanced Data Skills
10:07 Mindset
Notes from video 📝
bittersweet-mall-f00.notion.s...
bittersweet-mall-f00.notion.s...
bittersweet-mall-f00.notion.s...
Who am I? 🙋🏻♂️
I'm Jay, I love making videos about travel, self-help and tech. I currently work in New York City as a data engineer, but I grew up in Malaysia and lived in the UK when I was 19. Back then, I had no idea what life was about, moving to so many places, navigating career in Tech. Today, I've learned a lot and wanna share my perspective through filmmaking.
Socials 📱
linkedin: / jayzern
Sub Count: 152
I'm a freelancer and also self-taught programmer who wrote a lot of Data Scraping scripts using Python for my clients. I have a loyal client who requires me to extract data for them every month. I repeated this manual process for about 3 months until I realized the scraped data was getting bigger in size and dimension. I thought it was not a data scraping job anymore, it was a data engineering job. After doing some research online, I figured out some useful tools I can use to manage all data collected by the data scraper. I start by storing all the data in an SQL database and automating the process using Apache Airflow. That was the first time I thought Data Engineering was the right direction for my career path as a developer. Thanks for your advice, Jay.
how long did you train the skills of Data Scraping and Python to make money out of it?
wow, thanks for sharing this, it's really inspiring me to learn more about data engineering
@@ajdinpipo08 You can sharing resource you learn DE ?
A bit of advice.
If you want to work as a data engineer but don't have a computer science background (CS degree or work experience as a software engineer), then there will be a lot of gravity pulling you towards becoming an analyst, i.e. becoming a data user instead of a data builder. If you want to avoid this, you have to be really intentional about educating yourself in the technical aspects of data infrastructure, and pursuing job opportunities where you apply that knowledge.
It's easy to forget that moving data around, not looking at the data itself, is what a data engineer does. But businesses often don't understand that, they just lump all the "data people" together. If you're not careful, you'll find yourself creating Tableau dashboards, wrangling data in Pandas, or maintaining Excel spreadsheets, simply because you're the "data guy".
Also, if you want a successful career as a data engineer, get comfortable with being uncomfortable. New frameworks come out all the time, and you'll be constantly learning. It's not as crazy as web development, but it still moves relatively fast. If you start feeling comfortable at a job, it's time to ask for a transfer or change jobs, or you'll waste away.
Thanks for advice sir. I'm an accounting student but I'm not comfortable with my majority 😂.
Wow!! Great advice!! I really appreciate it! Thank you a lot!
DE since 2016, I agree completely
That’s a great advice.
Well said. In a service-heavy IT market like India, most jobs are skewed towards dashboarding and analysis. Data Engineers have to scour over the internet and build projects credibly prior or while interviewing for relevant Data Engineer roles which are few and far between.
I love the way you split the career to basic and advanced skills. I’ve been keep doing that to become the data engineer. Thanks for sharing!
Beginning my path here, great video with no filler!
I wish I would have seen this sooner! Great and crisp content.
really great and clear explanation ! please keep this way. I would recommend this video to anybody wanting for learn about data engineering.
Best Roadmap ever. Just get started with it.
Thanks a lot!
Loved your video, it surprised me how you started the second section getting us into the book. It felt like already starting :) thank you!
Very useful tips for those who are just starting out. Great content!
I recently signed up for the data engineering course from Datacamp and I have covered pretty much most SQL courses but I'd love to go back on a few of them again, I work as a data analyst but I feel a career in Engineering would be great.
We are in the same boat
@@holmesketrick7mee to
Exactly. SQL can be voluminous to learn. It takes time and practice to let it sink in
Hi, I’ve done the complete python track of data analyst, mostly of the sql courses of DataCamp. I’m from Chile, Latin America, do you have any advice for me to find remote jobs as data analyst in the US?
hi may I ask, does datacamp teach you how to normalize column tables in excel format to tables in sql / other database formats? thankyou
Hands done, the best video I have seen on the subject. It's no nonsense, practical and easy to digest (of course after you put in the work). I am just starting out in my journey to become a Azure Data Engineer so I thank you Jay. Thanks for sharing and being so transparent. I'll keep you posted on my many adventures. For what it's worth, I'm 70 years young and still dedicated to life long learning.
That's amazing! Love the attitude on life long learning
Wow, that is so encouraging! Godspeed!
This is super super SUPER practical and insightful and helpful. Looking forward to your future content and successes!
Thanks for the support Derek
Thanks Jay for posting such an informative and insightful video!!!
Thank you. Very kind of you to put out this valuable easy to digest and to follow content.
What a gem. I am currently pursuing AI/ML. Getting data scientist job as a fresher without experience is almost impossible so had to start with either as data analyst or data engineer. Since now everyone is becoming data analyst so their is demand and supply issue in the job market. Thinking of expanding my domain knowledge to data engineering as well but as you stated “got lost” with the available resources. This video helped me to make a roadmap to pursue data engineering.
Thanks for sharing. I agree it's super difficult to get a DS job fresh out of college. Branching out to DE is a great way to "stand out" from the crowd.
Very good explanation. Love it for the conclusion with the complete motivation.
Amazing video with to the point information. Thank you Jay!
Thanks for the useful tips. Short to the points yet very effective !
Powerful tips! Thanks a lot for lightning the path!
I am transitioning from web development to data. Fortunately for me, I learnt development skills from Learnenough. I have gone through their Commandline course and now on Python and Python for Data Science. I am excited to be doing this. Thank you for this content.
Outstanding video! This video was remarkably well-explained and detailed. It is also structured very well and was easy to follow along with your reasoning. It was a pleasure to watch. I am a Senior iOS Engineer looking to dive deeper into ETL and related topics. This helped me organize what I want to learn. Definitely subscribing.
Great work on explaining the concepts 👍🏽
thanks for delivering a well produced video centered around DE
Thank you for such an informative video, I'm a junior data engineer and your channel is an invaluable resource!
I have been working with python so I think a few very important things 1) pyspark 2) pick AWS and/or Azure to learn their data engineering stack 3) very solid SQL skills.
Holy f, brother this is an amazing and quick overview.
You def got potential in this, keep it up!
Thanks man! So much more to come
nice and well structured information, liking this!
This was really helpful. Thanks for sharing ❤️
Hey Jay thanks for sharing all those valuable information with us. I hope to you see soon in NYC! :D thanks again brother!
Great information. Looking for more videos from you. Keep doing .
Thank you for this video, it created more of a path for me.
Best roadmap for Data Engineering on TH-cam. New sub Here 😊. Greetings from Argentina 🇦🇷
Hello broo, your video was great!
I'm an aspiring Data Engineer and what you said about being overwhelm with many ambigous word on the first year was true, sometimes I feel burn out too. Kinda sad but I know this career is great so I keep it up. This video helps me regain my motivation :))
It was amazing introduction
and guide line.
Thanks
Nice briefing,waiting for more knowledge sharing on data engineering
Love the content bro, hope you keep growing. Its cool to see another Malaysian in the US whose also in the field of Data while making TH-cam videos on the side.
Thanks bro! Awesome to see other Malaysians too
I am a fresh graduate in Information Systems and my internship had me working with the team's data lead using Python, Apache Airflow, and PostgreSQL. I was glad and became more confident from the experience but once I was job hunting, I kept finding data engineering job postings demanding tools like Azure, AWS, Spark, Hadoop, etc. I was discouraged as I only had the three I mentioned. But I was a bit encouraged when you talked about tools as getting more specific on what the task demands, and that foundation through concepts can be better.
i'm a junior data scientist transitioning to data engineering/MLOps. i'm also based in nyc! i'm recently discovering my interest in filmmaking too! looks like i stumbled upon your content at just the right time. just subscribed!!!
No way! data engineering/MLOps and filmmaking is such a great niche
Thanks for this video! I always loved SQL and manipulating data but was put in the software Eng camp. I hated programming so gave it up and moved to IT/admin now I realize data engineering may have been a fun career choice and this video has given me a lot of interest on things to get back into!
Thank you for sharing insights
Amazing and informative!!
Thank you, thank you a lot, Jay!! I recently graduated as a Data Scientist but when I see the options of a career path I think Data Engineer fits better with me than others. This video is helpful!! I'm a new subs!
Hugely useful! Subbed!
i was pretty much nervous and scared how to start, thank you for clearing things up :)
Well developed and presented informative video. Thanks a lot 🙏👍
Very insightful video 👏👏👏
Great job with your career man, mixing technical and artistic paths, great balance.
Thanks for the kind words man!
Excellent information man
Awesome video Jay! I will personally be looking through the resources that you've recommended.
Thanks oliver, u the best
Is an excellent introduction I’m far away from being close to a data engineer. I’m just starting off in learning data analytics, but I enjoyed very much this introduction. I felt like it was well explained, and definitely sets the tone moving forward in my learning.
This is a super high quality video! Cant wait to see this channel grow!
Thanks man!
Eye opening, thanks
Thank you for this great video
Hello bro ,i completed my undergraduate in actuarial but failed miserable and i have been down,filled with regrets and lack motivation ever since but your has given something to believe and get me to regain my confidence . Today begin a new journey,my journey into teaching myself to become a data engineer hope to come back to this post after a uear to share my progress.
you’ll do well, being a DE is thee best. and you’ll easily work worldwide
I am going back to your comment next year to check you. keep up 🫡
How has your journey been so far?
@@Odin.x.He came bck to reality
Thank you so much!
No long ass intro no stupid music I love this video
Useful information thank you
I've been a data analyst for a few years and I am just starting to learn the material to transition into the DE space.
This is hands down the best high level introduction to data engineering I’ve come across. Thank you for making this amazing resource guide as well. I work as a BI analyst now and wanting to expand my skill set so I can truly become a data engineer. I appreciate your time and hard work!
Wow that means a lot! Keep working on expanding your skillsets. Learning is a non-stop journey
@@jayzernI'm a college student , I am currently doing the az900 and then I shall do the data bricks certification, are these enough to guarantee an entry level job in data engineering?
This video helps me learn how to learn ML and process data.
Man, this was an awesome video. I'm currently a data analyst and my company has just offered me a junior data engineer role. I have a lot of learning to do and the path you have provided is helpful!
Amazing man, good luck in your new role!
I started / enrolled in courses on DataCamp on path to be Data Analyst then will take courses for Data Engineering. I’m a super excited to learn about Data and the career opportunities
thank for information and good video
Thank you for a great introduction! I have been working in data science for quite some time, but over the years, I realized that the analysis part is much less interesting for me than the data part.
Starting Data Technologies course on September, for now I don't think about money, I just think how to survive and not become mad during education because of huge amount of information
Hi Jay! Thank you so much for this video. Can you please make a video about DE vs Cloud Engineer. It seems that CE make more money but require more skills like AWS/Google Cloud; but a lot of skills overlap as well.
Thanks for the mindset part! I got a BS in DS, but shifted into DE because I enjoyed the backend of getting data for the analysis more than the analysis itself. Being new in the field, it’s definitely overwhelming with all the tools. However, Ill try to take your advice and start small by reading into a tool and doing instead of learning. I guess the best way to learn is to build and try? I always overcomplicate stuff and end up not following through because I try to tackle all these tools at once but will tackle it in smaller parts. From there, keep adding on to my project. Excited as this will be my first “DE” project. Thanks!
Amazing! I started out with a Bachelors in DS too
I do the same thing. It's a blessing and a curse. Scale back. Focus on one tool. Do a project. Join everything together like one complete Lego set. Blog about it. Reach out to recruiters. Wait for the 🎣 & snag a job
Thankss very helpful..
Great content, keep up the good work!
Thank you!
Great content.. the screen you have online with the contents where are u reading this from? The book or from a website?
Thank you Jayzern
you earned a subscription because of simplicity and being efficient on teaching!
I appreciate this video. At 3:00 he mentions that "the objective is to learn about data concepts, not computer science." I'm literally a total novice (but getting deeper into this), and I always thought that computer science was partly focused on data concepts. Or, maybe he means that data concepts is more specifically focused, in contrast to computer science which is extremely broad.
Do I have opportunity to learn together with you mate? I also a total novice but getting deeper 😂
in the project youve mentioned in the core data skills section. are the resources that are involved free? as far as I know, airflow is open source. I have worked with databricks and the whole azure ecosystem, but it was my dads companies subscription.
Hey, First of all thank you for making this video, Is there any way you could provide the doc you are referring to in the video? That will be so helpful.
About to dive into SQL this weekend. I'm just starting my journey into data analytics though. I don't think I have what it would take for an engineer 😅 but I still like watching videos like this cause I like looking at different things
Amazing! Good luck on your data analytics journey.
@@jayzern thank you! It's a little intimidating, but so far I'm hopeful ❤️. Kind of wish I'd known about this when I was younger though lol
Sis!!! 🗣️YOU HAVE WHAT IT TAKES. Commit to putting in the work and you can become an engineer. Don’t let self-doubt keep you from reaching your potential. You got this!!
How are you doing on this journey?
@@LisaEveShow considering I have ZERO technical background, it's going ok lol. I'm starting to get the basics of SQL and about to start Python. I kind of wish I'd spent more time on the hard skills this whole time
Great information Thanks Jayzern . Could you make a video on HOW I WOULD LEARN SNOWFLAKE DEVELOPER [if i could start over ]
This is awesome, thank you for sharing your knowledge. I'm a Software Engineer studying for my Master of Science in Data Analytics, maybe becoming a Data Engineer is in my future? Right now, focusing on my SWE job and learning SQL and Python through school have been my ways of learning and doing and while gaining experience (as well as coding projects on the side when I can).
Thanks for sharing Aaron. A lot of people are in the same boat and can totally relate. Enjoy the ride and good luck in your Masters degree.
@@jayzern thanks I subbed, will be tuning in for more videos!
I'm currently trying to sharpen my DE skills and make the transition as well, this video was incredibly helpful. I have an MS in Data Science, an undergrad in Data Analytics & Finance, so I'd love to add these skills to my toolbelt. Perhaps I'll look into Analytics Engineering roles!
Hi Jay, is it recommended to have a good overview of certain areas of mathematics before starting to learn the individual stages of data engineering?? Like linear algebra, discrete math, and the like?
Hi, I am looking into this.
I started learning analytics but I want to expand my skillset now to also try for DE jobs.
I have two doubts, would appreciate if you have time
1. Can someone make a career transition to data engineer from a non tech background(not having a STEM or CS degree)? Do companies accept?
2. I browsed through some jobs on LinkedIn and almost every job asked for some kind of experience in data or Software engineering. So, my doubt is if there are jobs for an entry level role like Junior Data Engineer or Associate Data Engineer?
Thank you, its a good video
Thanks for the support!
much appreciated
I'm currently transitioning from Data Analytics to Data Engineering.
I think DE is only going to grow over the next decade as AI continues to blow up. Thanks for the amazing video jayzern!
Why u transitioning from data analysis
To data engineering?
Is there any problem with data analysis?
Designing data intensive applications is another great book by O'Reilly
One of my fav books!
Hi, how do you suggest building and presenting a portfolio?
This is super helpful. More of these please
Thank you can you suggest demanding portfolio or personal project that helps in placing over best mnc
its a very useful video thankyou
Nice Video. May be you can suggest a couple of courses for each section of the video, bcos that is where people have a hard time choosing and taking action.
Exactly
Ooo thanks for the suggestion, will include that in future videos
Hi thank you very much for this explanatory and helpful video, Please I keep trying the link to the notes from notion but it's not working, can you please help sort it out. I'll be very grateful 🙏. Thank again
Could someone please explain how does apache airflow and apache spark differ from Google BigQuery?
Started my journey in learning, I'm currently a systems engineer and it suck lmao, requirements this, requirements that.... anyways, I feel like I found a good solid book to read and some videos on you tube to follow along with for postgreSQL
many thanks, i have confuse 2 things. Learning tools or Building concept . after watch your video. i have more specific look . kee sharing !
Hi, Jay. I've been a senior data analyst for 3 years and it's time to move on to DE. This video is so much useful to my career growth! I use SQL(MS sql server and oracle), python(mainly clean data and upload data to database) and tableau a lot in my job, so I'm wondering what's my next step to start learning the skills you mentioned in the video. Any recommendation would be appreaciated. Thank you.
that is the video I am looking to
Hi Jay, i have a question for data visualization. Do you advise power bi, for data visualization?
thank you
Wow top video! could you give us a list of youtube channels + blogs for Data engineers?
Take care!
Hi, do you think there is a threat of automation (transition to people losing it partially or completely) of this profession in the next 3-5 years?