I'm a working as a Data engineer for 6+years and this is the only video on youtube which gives accurate guidance. Thank you so much It has helped me a lot. I was wondering If you can create a video on Databricks , if someone who has knowledge on Hadoop , hive and spark worked mostly on on premise system wants to get into Databricks world. It will be very helpful. @JashRadia
bro this is like the most to the point data engineering roadmap video on internet. If you could, could you make videos of a complete end to end DE projects using all these steps ; that be BREAKING.👍👍
Thank you, Jash, for creating this amazing data engineering roadmap! Your clear guidance and structured explanation are incredibly helpful . Keep up the great work!
Hi Broo Were you having data engineer background before getting data engineer job ? I am in storage Testing but wanted to switch in Data Engineer domain it would be helpful if you reply. Thanks in advance
I agree with most points, except for the Hadoop/Hive aspect. Any data engineer should have a solid understanding of the underlying architecture before diving into development, and Hadoop provides that foundation from an architectural perspective(not needing to write code in MapReduce). Hive can also be useful in understanding how it acts as an abstraction layer over Hadoop. A strong foundation is essential. Otherwise, everything else in the video is excellent. Cheers for this insightful content!
Hey Jash I'm also working as contingent worker at doordash, currently my role is more of an analyst but i want to move towards DE role so I'm preparing for it.
Thank you very much for this video sir . Please make one detailed video about machine learning engineering or ai engineering role as well. It’s a humble request 🙏🏻
Best place to learn these softwares or any software is the official tutorial and official documentation. Modern software developers maintain very good quality documentation. Some O'Really books are very good. I like GNU documentation , they are hand down the best.
I am a first-year student looking forward to gaining internship opportunities in the near future. In your opinion, which certifications or projects should I pursue to gain the trust of employers? I’d appreciate your advice. I truly hope you will make a video to share about it. Thank you so much!
Hi Jash, im in my third year right now. The data engineering field interests me more than other fields but the internship chances in this field are very low. I am confused about whether to commit to this roadmap or just do something else that can land me some internships. I wanted to know your perspective on my situation. Thank you for the video btw.
I've been wondering on how to practice all the knowledge 'at home'. Reading courses is great but practicing is by far the best way to learn. Since all these techs have only free trials, how do you practice at home without paying anything ?
I'm a data analyst now and feeling saturated just with the skill of sql in my current job. Please guide me how can i jump ahead and make a good portfolio 🙏
I didn't create any 2024 plans. If you were following the one that i created in 2022, then you will find overlap, for such case you can stick to it and see what new topics are added in this one to work on those topics.
Awesome video, my man. I am Data Analyst giving first steps in my career. What's you advice if I want to transition to Data Engineering, not in this god-like level? What are the fundamentals for landing a job?
Sir, how is the data engineering field for freshers after graduation ? I have seen many job descriptions in linkedin of data engineering Most of them are experience required. So what is your point in this. Please reply
If someone is new to Data Engineering and only has experience with the prerequisites, how long would you estimate it would take to go through the full roadmap and be in a position where one is comfortable enough to begin applying for jobs? Thank you for uploading this!
Crazy timing sirji! I recently got placed in fedex as data engineer. So I was looking to learn more abt the role and technologies to study. Thank you so much for this in depth roadmap! I will try to cover as much as possible before my job starts.
So i am a researcher turned data analyst. That too fairly recently (2022). Didnt know much coding whatsoever. But in the last 2 yrs, somehow it has become a hobby (god knows why i didn't go for CS after entrance). So ive been testing the waters for a DE role. Im okish with Python and SQL at this point. Problem is i have a full time job as well. What should be my timeline and chances? Also a suggestion: may be start a discord server where you can build an interactive community?
Hi Jash, I'm currently working as a tester and looking to switch my career to data engineering. I've watched your previous videos and got a clear idea of what steps to take- thank you for that! Could you please share if certifications are necessary for this transition, or would building relevant projects be enough to land a data engineering role? Your guidance would be greatly appreciated!
Thank you! Yes, I'd recommend certifications. Aim to get a couple of certificated in any cloud platform. For example, in AWS, go for sol arch and data specialized certs. This will surely add value. All the best!
"I am a recent master's graduate in Big Data and High-Performance Computing, and I would like to enter the data engineering industry. I have a doubt regarding certifications: is it more credible to get the AWS Cloud Practitioner certification first and then the AWS Certified Data Engineer Associate - Specialty certification, or is getting the AWS Certified Data Engineer Associate more credible?"
Hey Jash, thanks for the informtive video. For a beginner (with basic SQL and basic Python knowledge) what would be a realistic timeline to get through the entire roadmap?
Hi @Jash Radia, Thank you for the video. hey I am 48 yrls old with QA, QA Automation etc roles.. I now want to switch to Data Engineering and I could not decide between "AWS Certified Data Engineer" or the path you mentioned in this video. could you please suggest. I know the concepts of data engineering is the same but want to know which set of the skills are in more demand in USA?
Following this, we can become a great data engineer. But is it really gonna help us land a good job? Because I think this map has too many things to focus on whereas landing a DE job would need us to focus on 4 to 5 skills mainly.
Not really. System design round itself requires you to have many skills like orchestration, real time and batch bug data processing, queue systems, cloud and devops. And that is also a round whwre many people fail. Having said that, this will ensure that you become a good engineer. Becoming good at interview will be a byproduct of this. It's not the only goal of this roadmap
I am working in a data governance company. Will it be a good move to move to data engineer role because I know how the data is being used by them. How can I make with this for a job search ? What are the things I need to work on more to get deep into the data engineering field? Thanks!!
Yes, most of the roles in engineering are better than product roles in terms of pay (generally) and security. So if you're interested in data, this can be a good move. You need to learn the skills first with this roadmap and then do at least 2-3 projects and list it on your github. You might not get data engineering roles directly but look for data analysis kind of role to get in this field and then eventually make a move to be a DE. It's a long road.
Hi @jash what will be usual salary ranges for a data engineer with 1 year experience and how is rhe market for data enginers with 1 yoe or fresher off campus
In general software market is not as great as it was 2 years ago but it's improving. For 1 YOE, the range totoally depends on which company you work for. It can range from 6 to 35 LPA
Hello Jash Sir, I have a finance background and want to transition to data engineering. Your videos are really interesting and are helping me in this transition. However, I am really not confident. I feel like DE is for experienced people, or software engineers have a fair chance as they already have a good grasp of the technical concepts. Do you think aiming for this field as a complete fresher is a good choice?
@@JashRadia Thank you! I will refer to this roadmap and get started. Can you also make a video about good DE projects that are similar to actual tasks done by a Data Engineer?
I'm working in L1 technical support salesforce engineer is it possible to get DE job if I have skills but HR was asking experience in DE but in experience letter mentioned as L1 Support Engineer how to tackle this?????
I am a beginner to intermediate at python, I couldn't find a clear answer anywhere that how much one needs to be proficient in coding to learn the skills after pre-requisits? I think I am stuck in an infinite while loop 😅 Can you please clear my confusion?
I don't understand how can 1 person learn so many tools and languages. Each product in market has so many features and it is nearly impossible for anyone to keep pace with so much constant knowledge and architecture. It will take ages. Not possible in this lifetime
This takes 1 year even if you start from literally 0, not ages.. Also this doesn't even have all tools. These are mostly 1 tool from each category. Fly or die 🫡 Also languages? There is literally only 1 programming language I am talking about.
@@JashRadiaPython, Linux scripting, SQL are all different languages each having their own syntax and complex command and functions, isnt it? Each tool has its own architecture. I was referring in that sense. Even I learn things theoretically, until one gets practical hands-on in real projects, one cannot claim to be an expert. Also when we go to get a new project, everyone demand hands on on the tool.So in a way, it will take lot of years to claim oneself to be an expert. And then by that time there are are 10 new products which become the hot thing to know. I think it is better for anyone to pick 1-2 areas or skills in your list and become a pro instead of knowing everything but being master of none
Hii Jash I was following your previous video of data engineering.Just started 2-3 weeks ago..Have a basic knowledge of python and sql now.Should i focus on this roadmap now or should i continue with the previous one?
Hi Jash, I am a python developer currently having 2 years of exp but finding it hard to get interview calls for a swtich. I have been applying for like last 2 months and barely 1-2 interviews got scheduled. I am confused now whether I should learn data engineering or go with MERN stack with full devops. Any suggestions?
Sir, many people scaring that it is noteasy to do. Sometimes I feel demotivated. Talking about me: I just finioshed my 12th grade and now leraning python and looking forward to learn other tool to become a data engineer. I have a question to you that , From my situation Can i land a internship or job from my situation using resouces from internet? hoping to get positive feedback
@@diyagupta9572 of course there are many off campus opportunities as well man. Also career swtiches. Switching domains in tech has been done by millions of engineers. You just gotta start learning and stop doubting.
Hi jash, what about apache pinot and sql mesh for Data engineering, we are trying to implement it in our system will it be useful for a data engineer. Thank you
Yes Pinot is a good open source tool for clickstream data OLAP. But while following this roadmap, you will learn more OLAP systems during cloud computing courses and data warehouse anyway, didn't want to add pinot separately to reduce redundancy. There are many such tools that are not covered in this video because their alternatives were already covered.
Thank you for watching! We have made a full circle with this video and hope you like it ❤ let's talk!
Thank you for the roadmap.
Can you do roadmap on azure data engineer
I'm a working as a Data engineer for 6+years and this is the only video on youtube which gives accurate guidance. Thank you so much It has helped me a lot. I was wondering If you can create a video on Databricks , if someone who has knowledge on Hadoop , hive and spark worked mostly on on premise system wants to get into Databricks world. It will be very helpful. @JashRadia
Thanks for clearing my doubts & it's a really cool road-map for date engineers.
bro this is like the most to the point data engineering roadmap video on internet.
If you could, could you make videos of a complete end to end DE projects using all these steps ; that be BREAKING.👍👍
web dev krle basis hai woh faang ki
Thank you, Jash, for creating this amazing data engineering roadmap! Your clear guidance and structured explanation are incredibly helpful . Keep up the great work!
the goat is back. thank you for your data engineering content. it help me to get a data engg job.
All the best!
Hi Broo
Were you having data engineer background before getting data engineer job ?
I am in storage Testing but wanted to switch in Data Engineer domain it would be helpful if you reply.
Thanks in advance
Thanks so much! One of the top 3 videos on DE over youtube.
Please make similar roadmap for AI Enginner if possible.
What are the remaining videos
Thanks really good video to the point!
most awaited video 😁 thank you for sharing resources along with 🙂🙂
Thank you :D
THANK OU SO SO MUCH FOR THIS ELABORATE ROAD MAP...GOD BLESS YOU TONNES
I asked for this in your previous roadmap video.. and you put a video.. thanks!!
🫡
Appreciate the content :)
Very informative content as always 👏🏼
Thank you!
I agree with most points, except for the Hadoop/Hive aspect. Any data engineer should have a solid understanding of the underlying architecture before diving into development, and Hadoop provides that foundation from an architectural perspective(not needing to write code in MapReduce). Hive can also be useful in understanding how it acts as an abstraction layer over Hadoop. A strong foundation is essential. Otherwise, everything else in the video is excellent. Cheers for this insightful content!
Thanks bhaiya 😊
This gonna help new commers a lot 😅
Love your videos ,
Simplicity with nice explanation and a handsome man 😅
Top notch! Really loved from scratch to depth! Thanks for the highly informative video❤️
Jash , can you pls start a course for data engineering for product based companies
thank you sir for your data engineering content it helped me
All the best in your journey!
Very helpfull video, Thank you sir.
Thank you
Hi Jash, could you please make a video on open source for data engineers. Love your content ❤️
Sure, many tools I am talking about in this video are open source, but thanks foe suggestion and support!
Hey Jash I'm also working as contingent worker at doordash, currently my role is more of an analyst but i want to move towards DE role so I'm preparing for it.
Thank you very much for this video sir . Please make one detailed video about machine learning engineering or ai engineering role as well. It’s a humble request 🙏🏻
Thabks for watching! This video covers most important parts of AI: th-cam.com/video/3ihrsGN266k/w-d-xo.html
Best place to learn these softwares or any software is the official tutorial and official documentation. Modern software developers maintain very good quality documentation. Some O'Really books are very good. I like GNU documentation , they are hand down the best.
I am a first-year student looking forward to gaining internship opportunities in the near future. In your opinion, which certifications or projects should I pursue to gain the trust of employers? I’d appreciate your advice. I truly hope you will make a video to share about it. Thank you so much!
Just uploaded a video about this.
@@JashRadia thank you so much
It's not even 2025, we've a roadmap for 2025 wah..... thanks jash
Hi Jash, im in my third year right now. The data engineering field interests me more than other fields but the internship chances in this field are very low. I am confused about whether to commit to this roadmap or just do something else that can land me some internships. I wanted to know your perspective on my situation. Thank you for the video btw.
I've been wondering on how to practice all the knowledge 'at home'. Reading courses is great but practicing is by far the best way to learn. Since all these techs have only free trials, how do you practice at home without paying anything ?
I'm a data analyst now and feeling saturated just with the skill of sql in my current job.
Please guide me how can i jump ahead and make a good portfolio 🙏
Hi all. Is this video for complete beginners? Can I become a Data Engineer from completely scratch and with no prior programming knowledge?
I am following 2024 Path and when compared to 2025 the path is completely changed now , which one I have to choose now
I didn't create any 2024 plans. If you were following the one that i created in 2022, then you will find overlap, for such case you can stick to it and see what new topics are added in this one to work on those topics.
@ thank you for the reply actually I am following
2024 ultimate roadmap plan
I'm surprised , but it's actually usefull , thanks!
HI can you make a video on collage 1st year what to do and not to do pls
I am focused on data engineering topics, but sure, it can be a future video!
Hi Jash, Just wanted to know your view on learning Databricks.
It's good to learn if you use it in your job!
Can you please make one by one video for the data engineer
Awesome video, my man. I am Data Analyst giving first steps in my career. What's you advice if I want to transition to Data Engineering, not in this god-like level? What are the fundamentals for landing a job?
Sir, how is the data engineering field for freshers after graduation ? I have seen many job descriptions in linkedin of data engineering Most of them are experience required. So what is your point in this.
Please reply
Apache Aiishberg also
Sir what is the future of data analytics is it good carrier option in 2025 , can Ai replace data analytics?
Not really. Checkout my AI related video to know why: th-cam.com/video/3ihrsGN266k/w-d-xo.html
Data engineer vs backened developer who earn more after five years of experience and which language is best for backened java vs python vs JavaScript
There is no clear winner. Both have similar pay ranges generally.
Hi Jash Bhaiyya, can you please make a video on road map for becoming Cloud Data engineer and how to get selected in Google for same role
Hey, this roadmap is same as cloud data engineer role because it includes cloud as well.
If someone is new to Data Engineering and only has experience with the prerequisites, how long would you estimate it would take to go through the full roadmap and be in a position where one is comfortable enough to begin applying for jobs? Thank you for uploading this!
4-6 months if you spend about 10 hours per week. All the best!
Crazy timing sirji! I recently got placed in fedex as data engineer. So I was looking to learn more abt the role and technologies to study. Thank you so much for this in depth roadmap! I will try to cover as much as possible before my job starts.
hey can you share some points how u get this job ? which skills is required ?
Graduated in 2024 still didn't get full time no openings for freshers mostly for senior 2-5 years.
So what's is best role web developer??
So i am a researcher turned data analyst. That too fairly recently (2022). Didnt know much coding whatsoever. But in the last 2 yrs, somehow it has become a hobby (god knows why i didn't go for CS after entrance). So ive been testing the waters for a DE role. Im okish with Python and SQL at this point. Problem is i have a full time job as well. What should be my timeline and chances?
Also a suggestion: may be start a discord server where you can build an interactive community?
How much time it will take a non tech person to learn this . Eagerly waiting for your reply . Thanks for this detailed video .
is your role in Doordash of data scientist or machine learning or mlops engineer ?
Ya questions i dont understand how to make projects
Hi Jash, I'm currently working as a tester and looking to switch my career to data engineering.
I've watched your previous videos and got a clear idea of what steps to take- thank you for that! Could you please share if certifications are necessary for this transition, or would building relevant projects be enough to land a data engineering role? Your guidance would be greatly appreciated!
Thank you! Yes, I'd recommend certifications. Aim to get a couple of certificated in any cloud platform. For example, in AWS, go for sol arch and data specialized certs. This will surely add value. All the best!
Thanks for the video. How do I showcase these skills to the interviewer if I haven’t worked on any of these in my current job?
Own portfolio projects
Yes, create portfolio of 2-3 projects with these tools
"I am a recent master's graduate in Big Data and High-Performance Computing, and I would like to enter the data engineering industry. I have a doubt regarding certifications: is it more credible to get the AWS Cloud Practitioner certification first and then the AWS Certified Data Engineer Associate - Specialty certification, or is getting the AWS Certified Data Engineer Associate more credible?"
can you make God Tier ML Engineer ( not DS ) roadmap.
Hey Jash, thanks for the informtive video. For a beginner (with basic SQL and basic Python knowledge) what would be a realistic timeline to get through the entire roadmap?
I think anywhere between 4-8 months, depending on how much you put in every day
Hi @Jash Radia, Thank you for the video. hey I am 48 yrls old with QA, QA Automation etc roles.. I now want to switch to Data Engineering and I could not decide between "AWS Certified Data Engineer" or the path you mentioned in this video. could you please suggest. I know the concepts of data engineering is the same but want to know which set of the skills are in more demand in USA?
If I learn data engineer course can I get a job as a fresher
Hi
Following this, we can become a great data engineer. But is it really gonna help us land a good job? Because I think this map has too many things to focus on whereas landing a DE job would need us to focus on 4 to 5 skills mainly.
Not really. System design round itself requires you to have many skills like orchestration, real time and batch bug data processing, queue systems, cloud and devops. And that is also a round whwre many people fail.
Having said that, this will ensure that you become a good engineer. Becoming good at interview will be a byproduct of this. It's not the only goal of this roadmap
@JashRadia Thank you, could you make a video on how to choose companies/job profiles and how to get an interview for them.
I am working in a data governance company. Will it be a good move to move to data engineer role because I know how the data is being used by them. How can I make with this for a job search ? What are the things I need to work on more to get deep into the data engineering field? Thanks!!
What is your current role?
@ product engineer
Yes, most of the roles in engineering are better than product roles in terms of pay (generally) and security. So if you're interested in data, this can be a good move. You need to learn the skills first with this roadmap and then do at least 2-3 projects and list it on your github. You might not get data engineering roles directly but look for data analysis kind of role to get in this field and then eventually make a move to be a DE. It's a long road.
@@JashRadiaThank you so much for sharing!!!
Hi Jash, Want to ask that is datagrowskills bootcamp is good for data engineers it covers most of it and remaining we can do it from your resources?
Hi @jash what will be usual salary ranges for a data engineer with 1 year experience and how is rhe market for data enginers with 1 yoe or fresher off campus
In general software market is not as great as it was 2 years ago but it's improving. For 1 YOE, the range totoally depends on which company you work for. It can range from 6 to 35 LPA
@@JashRadia Thanks
Hello Jash Sir,
I have a finance background and want to transition to data engineering. Your videos are really interesting and are helping me in this transition. However, I am really not confident. I feel like DE is for experienced people, or software engineers have a fair chance as they already have a good grasp of the technical concepts.
Do you think aiming for this field as a complete fresher is a good choice?
It's not easy, but it's definitely possible - there are tons of people in data engineering who didn't start as software engineers.
@@JashRadia Thank you! I will refer to this roadmap and get started.
Can you also make a video about good DE projects that are similar to actual tasks done by a Data Engineer?
I'm working in L1 technical support salesforce engineer is it possible to get DE job if I have skills but HR was asking experience in DE but in experience letter mentioned as L1 Support Engineer how to tackle this?????
I am a beginner to intermediate at python, I couldn't find a clear answer anywhere that how much one needs to be proficient in coding to learn the skills after pre-requisits?
I think I am stuck in an infinite while loop 😅
Can you please clear my confusion?
Can I switch from an oracle DBA to data engineering roles?
Yes
I don't understand how can 1 person learn so many tools and languages. Each product in market has so many features and it is nearly impossible for anyone to keep pace with so much constant knowledge and architecture. It will take ages. Not possible in this lifetime
This takes 1 year even if you start from literally 0, not ages.. Also this doesn't even have all tools. These are mostly 1 tool from each category. Fly or die 🫡
Also languages? There is literally only 1 programming language I am talking about.
@@JashRadiaPython, Linux scripting, SQL are all different languages each having their own syntax and complex command and functions, isnt it? Each tool has its own architecture. I was referring in that sense. Even I learn things theoretically, until one gets practical hands-on in real projects, one cannot claim to be an expert. Also when we go to get a new project, everyone demand hands on on the tool.So in a way, it will take lot of years to claim oneself to be an expert. And then by that time there are are 10 new products which become the hot thing to know. I think it is better for anyone to pick 1-2 areas or skills in your list and become a pro instead of knowing everything but being master of none
How to get calls for Data Engineering job without any job experience.
Is it possible to learn complete data engineering roadmap skills except devops in 3 months??
No, you need at least 6 months if you're a complete beginner
Are this roadmap require having a knowledge of java
Hii Jash
I was following your previous video of data engineering.Just started 2-3 weeks ago..Have a basic knowledge of python and sql now.Should i focus on this roadmap now or should i continue with the previous one?
Basics are the same in this and that one as you'd have noticed. Pick the rest from this one.
I iwant to buy m2 air 8gb base varient is it enough for data engineering and data science
Im an SQL Developer with 1.3 yoe, will i be able to apply for experienced position? Can anyone clarify?
Hi Jash, I am a python developer currently having 2 years of exp but finding it hard to get interview calls for a swtich. I have been applying for like last 2 months and barely 1-2 interviews got scheduled. I am confused now whether I should learn data engineering or go with MERN stack with full devops. Any suggestions?
Data engineering is a good switch for a Python developer, especially if you're already familiar with data concepts.
Sir, many people scaring that it is noteasy to do. Sometimes I feel demotivated. Talking about me: I just finioshed my 12th grade and now leraning python and looking forward to learn other tool to become a data engineer. I have a question to you that , From my situation Can i land a internship or job from my situation using resouces from internet? hoping to get positive feedback
Learn some basics of gen ai with that, you can definitely land a job.
Should freshers aim for dtat engieer role?
Of course there are many off campus opportunities as well man. Also career switches.
is,it relevent to a Fresher ! anyone please.
Is it good to switch from .NET developer to Data Engineering? Having 2+ years of experience in .NET ,SQL
It's upto your interest but definitely possible.
is fresher get a job directly as a data engineering role ?
I did through campus placement
@@JashRadia yah, only campus placement; is there any chance like through referral or off campus?
Can we just switch into the data engineer field after becoming experienced in any tech field/domain?
@@diyagupta9572 of course there are many off campus opportunities as well man. Also career swtiches. Switching domains in tech has been done by millions of engineers. You just gotta start learning and stop doubting.
@@JashRadia ok great, I'll start learning. Thankyou so much for replying
make more content videos bhyaa
Yes sir, trying 😅
Please upload the same kind of roadmap for experienced data engineers
Doordash office Tour karo bhaiya
Hi jash, what about apache pinot and sql mesh for Data engineering, we are trying to implement it in our system will it be useful for a data engineer.
Thank you
Yes Pinot is a good open source tool for clickstream data OLAP. But while following this roadmap, you will learn more OLAP systems during cloud computing courses and data warehouse anyway, didn't want to add pinot separately to reduce redundancy.
There are many such tools that are not covered in this video because their alternatives were already covered.