Im a data engineer with 4 years experience. Harsh reality of being a data engineer is: - 5% building new pipelines - 15% automating the pipelines - 20% adding data quality checks to the pipelines And 98% of the time fixing bugs in OLD pipelines. For people who like building new products, this is not the right role. Because most of the time you're only firefighting. And since you work in backend, you don't get much recognition.
Can you please guide me with a roadmap for getting into data engineering if I am from a college with good on campus placements, and whether I should learn data engineering in college instead of other conventional paths like say web development.
@@baybars3138 i have been both a software engineer and a data engineer. I find software engineering to be much easier. Ofcourse fixing bugs is on both side. But in data engineering, i have observed it much frequent. Example data source may change, data quality issues, errors due to upstream systems, errors due to missing information etc.
Well, everything new requires you to figure out your path. I studied to be a mathematician but needed to find a job. Salaries in programming supported a more affluent lifestyle, so just a few classes in CS helped get my foot in the door. I think that the guy who wrote a violent race car game had a history Ph. D. I was reading about a philosopher who worked as a glass lens grinder, who died from cumulatively inhaling glass fibers in his 30s. Like everything else, you might need to change careers or job titles. Being a Data Engineer and knowing programming will get you further.
@@MitchMcCall-bd4ul100% these TH-camrs enjoy fear mongering for clicks, as well as inflating their salaries to get you to buy their products. It’s actually insane
There are two types of data engineers: Software Engineers who are also data engineers and ETL engineers. If you are an ETL engineer (building pipelines using python and SQL) then you are just considered a glorified data analyst and your pay always be less than SE. My advice for them is to take the next step and become a cloud engineer (snowflake, Azure, AWS) and see your pay increase significantly.
Always pursue what brings you true fulfillment. Don’t let the opinions of others, especially from TH-cam or social media, discourage you. While it's easy to get caught up in what others say, relying solely on their views can hold you back. Invest time in doing your own research and trust your journey. Remember, just because many are choosing one path doesn’t mean you can’t succeed in another. Your purpose is unique embrace it with confidence.
I have been working as a Data Engineer for 3 years and I think it depends on companies and how they define their data engineers. I always find myself building data pipelines, API design, Web applications, Data Encryption and Database Security
Hi please can you guide me how to get initial internship for this role as well have just started to prepare what projects and techstacks would it be needing
stressing on the point of a software engineer being able to do what a data engineer does is absolutely correct. I work as a data analyst and in one of my previous roles, the software engineer was pretty much the data engineer completing all the data engineering task.
But I think Data Analyst jobs can get saturated more easily than DE, considering the entry barrier is lower and there are a lot more people applying for DA, and then dashboard creation, SQL and less-advanced ML skills can be done by AI nowadays, don't you think so?
The roles of a software engineer and data engineer are totally different, even though they might use related tech stacks. Also a data scientist can't take the role of a data engineer because, apart from being technical, a data engineer is expected to communicate with stakeholders and build solutions that turn data into insigts/revenue; data scientists suck at this. So a good data engineer should be able to do the work of a data analysts effectively. According to a Gartner report, 80% of data science projects fail. The number reason is: Lack of understanding of business context and user needs.
Hello Abiola, I'm working to becoming a data engineer and would appreciate a mentorship as it's quite a large field and would appreciate any help you can give. I promise to not disturb you much. Thanks as I look forward..
absolutely disagree that SE can easily do DE role, working with data is different than coding, I have seen so many SE struggle with structured data let alone distributed computing and unstructured data, understanding data modeling, performance, ETL, understanding multiple storage and DBs. This is the high ground that SE need to get off, this is the reason why leetcode hiring gets you a really bad data engineer who is good at coding vs getting a really good data engineer who is average at coding.
Sounds familiar! My career path has been somewhat similar. I started as a PHP developer building websites, which was a great foundation. Then, I got the opportunity to work on a similar project, but this time in Java. It felt like a step up in terms of how much more interesting the projects were. From there, I transitioned to Scala, which again elevated the complexity and excitement of the work I was doing. Eventually, I decided to explore data engineering, working with Scala and Python. Learning Python was relatively easy, but the ecosystem itself took more time to grasp. Now, I can’t imagine doing anything else - data engineering offers some of the most rewarding and intellectually stimulating projects I've ever worked on. While data engineering might not be the highest-paying field in the US compared to some other tech roles, it's globally rewarding and certainly pays the bills. Plus, the work itself makes it worthwhile!
The aspect you might not have covered is that as most Data Engineers progress, they don't remain DEs but more so SREs and System Software Engineers, or systems architect as since they have a clear idea of how the information (data) is flowing, they have a clear idea of the system flow.
i was confused about that thank you i will focus on data science then i will convert to data engineer because i don't see a particular road map for data engineer
Im agree with all except number 2 software engineer can make easily DE work?, i think no, many DE were DBAs in the past. DBA is complete different skill to software engineering. 3. i think there is growing for DE you can skill up into machine learning engineer more easily than other roles.
Skill set is the key I assume. DEs also can take the jobs of DSs/DAs. Besides, I belive the business sector is also crucial. Yes SEs can take the jobs of DEs but , i.e., in the context of telecommunication sector, industry experience and knowledge will also be important for the role. I believe employees will navigate to projects in a general manner and focus more on project management by passing the automated tasks to AI agents. A SE (or other data employees) without the industry expertise may lack behind, if they are working in a specific industry, not a solid Tech firm/dept.
Thank you - there are couple interesting points. Most software engineers who I know miss the skill of data modeling. Data engineers speaks more with data users and they are closer to data analysts (at least this is what I do). SDE need to know more technologies and dependencies but DE need to know more about data she/he plays with. Point about career growth has something, however I meet some Data Engineering Managers. But probably that is it. Then maybe DE switch to DS job but it is horizontal move and not a promotion. One last point is about logic - if SDE earns more then they will not do DE job for less. If they would be forced to do DE but paid the same it would still mean for them losing skill and time for something not related to SDE. It would be harder for them to change a job. That is why there is probably still demand for DE.
Hi Sundas, i m writing this comment to share with you that i got fresher job as a Data Engineer in one of the startups. I believe if u have the knowledge of tech stack required for the role, u can get hired. In btw, i have worked as data science intern for 3 months before(as it is considered hot topic in the field of tech and i went for it initially but coudnt find good pay jobs for it).
Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!
Hi I'm a high school senior right now and I want to pursue a career in data science job family..... specifically AIML.....it would be wonderful if you could do a video on AIMl because I feel like there isn't much on the internet about the same
I am interested in this field i come from mobile developpement (coding) , i still dont understand , you are dealing with data and SQL ? And little of python , ? If anyone know a course to buy or path to follow please help me
Instead of making this useless video why don't you try to make something useful like the ROad map or path to become a data engineer and courses to learn
Hi Sundas, I am a bit confused. You are initially saying that SEs can do DEs role and also, with advancements in AI and possible taking over core SE's role, even they would do DE's role. Then, at the end, you are saying that DE's role will still be in demand?
Not everybody agrees with this (especially juniors) but Data Engineering is a subset of Software Engineering. There are two types of DE: - software oriented DE - BI oriented DE DE is SWE. Be very good in software engineering with a focus on Big Data. You will able to switch roles easily.
Thanks for watching! You can download the Python ebook (free) 👉🏼 clickhubspot.com/bbd
Im a data engineer with 4 years experience. Harsh reality of being a data engineer is:
- 5% building new pipelines
- 15% automating the pipelines
- 20% adding data quality checks to the pipelines
And 98% of the time fixing bugs in OLD pipelines.
For people who like building new products, this is not the right role. Because most of the time you're only firefighting. And since you work in backend, you don't get much recognition.
Exactly agreed
Can you please guide me with a roadmap for getting into data engineering if I am from a college with good on campus placements, and whether I should learn data engineering in college instead of other conventional paths like say web development.
Software development is samething, most of the times you are just fixing bugs. No?
@@baybars3138 i have been both a software engineer and a data engineer. I find software engineering to be much easier. Ofcourse fixing bugs is on both side. But in data engineering, i have observed it much frequent. Example data source may change, data quality issues, errors due to upstream systems, errors due to missing information etc.
@@AchuVlogs which one you found most interesting? Sounds like Data Engineer always has a job when Software Engineer doesn't.
Well, everything new requires you to figure out your path. I studied to be a mathematician but needed to find a job. Salaries in programming supported a more affluent lifestyle, so just a few classes in CS helped get my foot in the door. I think that the guy who wrote a violent race car game had a history Ph. D. I was reading about a philosopher who worked as a glass lens grinder, who died from cumulatively inhaling glass fibers in his 30s. Like everything else, you might need to change careers or job titles. Being a Data Engineer and knowing programming will get you further.
yah these videos are clickbait. Along with the fear mongering over being automated out of a job which is barely happening.
@@MitchMcCall-bd4ul100% these TH-camrs enjoy fear mongering for clicks, as well as inflating their salaries to get you to buy their products. It’s actually insane
There are two types of data engineers: Software Engineers who are also data engineers and ETL engineers. If you are an ETL engineer (building pipelines using python and SQL) then you are just considered a glorified data analyst and your pay always be less than SE. My advice for them is to take the next step and become a cloud engineer (snowflake, Azure, AWS) and see your pay increase significantly.
Sir, are you advising DE with ETL engineer to take cloud engineer course or take Software Engineer bootcamp? Which one is better?
Insightfull
Hello sir. I don't want to code , programm like dev team. But I can work in cloud. Where should I go. DE or devops one. Please help
Yes! There are technical DE and BI DE.
Two different types of DE. Totally different culture and minset.
The real question is how to get the education without wracking up stupid provider bills
Always pursue what brings you true fulfillment. Don’t let the opinions of others, especially from TH-cam or social media, discourage you. While it's easy to get caught up in what others say, relying solely on their views can hold you back. Invest time in doing your own research and trust your journey. Remember, just because many are choosing one path doesn’t mean you can’t succeed in another. Your purpose is unique embrace it with confidence.
ya, got it.
I have been working as a Data Engineer for 3 years and I think it depends on companies and how they define their data engineers. I always find myself building data pipelines, API design, Web applications, Data Encryption and Database Security
Do you code ? Or program from scratch?
Hey, where do u work?
we like that! well done
Thank you for putting this out, appreciate it! 🙌🏼
just got an internship as a data engineer(while at college) for the city of ottawa(only 1 spot btw)...wont have done it without you. Appreciate you..
Hi please can you guide me how to get initial internship for this role as well have just started to prepare what projects and techstacks would it be needing
stressing on the point of a software engineer being able to do what a data engineer does is absolutely correct. I work as a data analyst and in one of my previous roles, the software engineer was pretty much the data engineer completing all the data engineering task.
I'm sure DE's can switch to SWE too 🙂
But I think Data Analyst jobs can get saturated more easily than DE, considering the entry barrier is lower and there are a lot more people applying for DA, and then dashboard creation, SQL and less-advanced ML skills can be done by AI nowadays, don't you think so?
The roles of a software engineer and data engineer are totally different, even though they might use related tech stacks. Also a data scientist can't take the role of a data engineer because, apart from being technical, a data engineer is expected to communicate with stakeholders and build solutions that turn data into insigts/revenue; data scientists suck at this. So a good data engineer should be able to do the work of a data analysts effectively. According to a Gartner report, 80% of data science projects fail. The number reason is: Lack of understanding of business context and user needs.
Hello Abiola, I'm working to becoming a data engineer and would appreciate a mentorship as it's quite a large field and would appreciate any help you can give.
I promise to not disturb you much. Thanks as I look forward..
absolutely disagree that SE can easily do DE role, working with data is different than coding, I have seen so many SE struggle with structured data let alone distributed computing and unstructured data, understanding data modeling, performance, ETL, understanding multiple storage and DBs. This is the high ground that SE need to get off, this is the reason why leetcode hiring gets you a really bad data engineer who is good at coding vs getting a really good data engineer who is average at coding.
Great review, but i cant agree career growth. From Data Engineer you become a solutions architect. Or bridge being a DE to the next sensible path.
Sounds familiar! My career path has been somewhat similar. I started as a PHP developer building websites, which was a great foundation. Then, I got the opportunity to work on a similar project, but this time in Java. It felt like a step up in terms of how much more interesting the projects were. From there, I transitioned to Scala, which again elevated the complexity and excitement of the work I was doing.
Eventually, I decided to explore data engineering, working with Scala and Python. Learning Python was relatively easy, but the ecosystem itself took more time to grasp. Now, I can’t imagine doing anything else - data engineering offers some of the most rewarding and intellectually stimulating projects I've ever worked on.
While data engineering might not be the highest-paying field in the US compared to some other tech roles, it's globally rewarding and certainly pays the bills. Plus, the work itself makes it worthwhile!
Just started Data Analysist course few weeks ago.
How is it going so far?
The aspect you might not have covered is that as most Data Engineers progress, they don't remain DEs but more so SREs and System Software Engineers, or systems architect as since they have a clear idea of how the information (data) is flowing, they have a clear idea of the system flow.
i was confused about that thank you i will focus on data science then i will convert to data engineer because i don't see a particular road map for data engineer
Even I make less than a developer, it's still a helluva lot better than my customer service job!
Im agree with all except number 2 software engineer can make easily DE work?, i think no, many DE were DBAs in the past. DBA is complete different skill to software engineering. 3. i think there is growing for DE you can skill up into machine learning engineer more easily than other roles.
Skill set is the key I assume. DEs also can take the jobs of DSs/DAs.
Besides, I belive the business sector is also crucial. Yes SEs can take the jobs of DEs but , i.e., in the context of telecommunication sector, industry experience and knowledge will also be important for the role. I believe employees will navigate to projects in a general manner and focus more on project management by passing the automated tasks to AI agents. A SE (or other data employees) without the industry expertise may lack behind, if they are working in a specific industry, not a solid Tech firm/dept.
Thank you - there are couple interesting points.
Most software engineers who I know miss the skill of data modeling. Data engineers speaks more with data users and they are closer to data analysts (at least this is what I do). SDE need to know more technologies and dependencies but DE need to know more about data she/he plays with. Point about career growth has something, however I meet some Data Engineering Managers. But probably that is it. Then maybe DE switch to DS job but it is horizontal move and not a promotion.
One last point is about logic - if SDE earns more then they will not do DE job for less. If they would be forced to do DE but paid the same it would still mean for them losing skill and time for something not related to SDE. It would be harder for them to change a job. That is why there is probably still demand for DE.
I like the logic part in your comment, good take and that could be a good possibility.
Hi Sundas, i m writing this comment to share with you that i got fresher job as a Data Engineer in one of the startups. I believe if u have the knowledge of tech stack required for the role, u can get hired. In btw, i have worked as data science intern for 3 months before(as it is considered hot topic in the field of tech and i went for it initially but coudnt find good pay jobs for it).
Hello, thats is nice to hear :)
Do you happen to have any advice for applying to data internships?
Data science or data enginering which is better based on opportunites as a freshers?
Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!
AWS beta certification for data engineer , Please talk about it.
Yes I also want to know about this
Nice, I came to know that data engineers paid lesser than the average software engineer
Felt sad 😔
Hi I'm a high school senior right now and I want to pursue a career in data science job family..... specifically AIML.....it would be wonderful if you could do a video on AIMl because I feel like there isn't much on the internet about the same
Ma'am I m confused between data engineer or data scientist, which one should I choose , kindly suggest me
I am interested in this field i come from mobile developpement (coding) , i still dont understand , you are dealing with data and SQL ? And little of python , ? If anyone know a course to buy or path to follow please help me
seems you went with your default thumbnail :D
sadly yes 😭
Sundas name is very unique for me 💕
Thx!!.....some feedback though - your hand gesturing is a bit distracting
thank you very much 👍👏
Through the whole video I was really concerned she will accidentally cut me with her fingers...............
Lol my gosh there are so many harsh realities why would anyone wanna pursue this.
I’m a Database Administrator/ Network Administrator/ Full Stack Developer making 60k…. I would love 155k damn!
you are all that in one job... amazing
What is with your hands, its like your clawing
Instead of making this useless video why don't you try to make something useful like the ROad map or path to become a data engineer and courses to learn
good question
Please make sundas khaild Urdu/hindi channel for south asia audience specially Pakistani audience
Hi Sundas, I am a bit confused. You are initially saying that SEs can do DEs role and also, with advancements in AI and possible taking over core SE's role, even they would do DE's role. Then, at the end, you are saying that DE's role will still be in demand?
Not everybody agrees with this (especially juniors) but Data Engineering is a subset of Software Engineering.
There are two types of DE:
- software oriented DE
- BI oriented DE
DE is SWE. Be very good in software engineering with a focus on Big Data. You will able to switch roles easily.
You've no idea how fabulous you look!! 😇🫠
Ma'am I m confused between data engineer or data scientist, which one should I choose , kindly suggest me
Do your research.