@@malcorub Yeah, I get to face the clients and business vendors and we need to understand the same business problems that Data Scientists, Insight Analysts and Project managers need to understand but our job is to provide them the correct data, make Data Pipelines and refresh the datamart for the dashboards etc. But I feel like a data tech support for the rest of the project team. Also involved in working with legacy system in Teradata and modernizing it with the latest Big Data tools. But I am slowly learning the Data Analyst / Scientist work as well. That is where the "sexy" part of the work revolves around. Data Engineering is not a sexy job but it is one of the most important positions in a company.
@@sanjogh777 As long as you like working with data there is hope. I am a data analyst/report developer, I rely on my D.E's (who get paid more than I do) heavily to keep the data infrastructure going, for data pipelines and building complex data models and without you guys, my job would be so much harder or dare I say, IMPOSSIBLE. Like Jona said at the end of the video and as you mentioned in your comment, don't limit yourself. Dabble in the data analyst, PM and data scientists side. It's much more satisfying when you understand and can derive meaning from the data you are working with which many times, at least in my world, the DE's don't or choose not too.
I've tried Data Analyst in the past and it wasn't my thing. You have to have that "analytical thinking" to translate data to words. Soon after, I switched to become a Data Engineer and have been loving it so far because as the title says for itself, it's more of an engineering role rather than analytical. There comes a task where you need to get some data in a specific format and that's it! You don't need to waste time shift through data to understand anything and that's what I love about being a Data Engineer. Data Scientists however, is a role that I have not tried yet as it requires a certain knowledge of mathematics and analytics which to be honest I have no intention to be mastered at but imho DS is an upgraded version of DA with more complexity of machine learning models and mathematics. Therefore I would say it depends on people's mindset to see which one is for them. If you think more analytical, then go with DA or DS but if you prefer less analytical and more engineering, then DE is your chance!
dude u make no sense DS is no upgrade of DE its not even the same job nor the same skills , a data engineer is a software engineer while the data scientist is not an engineer im sure u re not a data engineer cause i ve never seen someone move from data analytics to data engineering
I am a data analyst/report developer, we do not have any data scientists at my place of work. ... truth be told, the data engineers at my place of work get paid more than me and my peers and they have alot less face time with the business units and costumer... but because they are so behind the scenes, the analysts are seen in a more positive light by the customers, business units and leadership than the data engineers so we get more accolades and bonuses by leadership.
> Data science analytics is super rewarding > Leaves his data analyst job at Facebook to become a software engineer because he's tired of doing bitch work
I felt this is because you are in a big company like Facebook. If you were in a startup or medium size company, data engineers have chance to do a lot of exciting things including algorithm, data pipeline, or even machine learning model. Nevertheless, thanks for sharing the relations between these three roles in big companies! Hope to see more video like this!
Wow this is the exact opposite of my experience. In NYC the market is saturated with math / physics grads willing to work as data scientists, but who are less familiar with coding. As such wages are suppressed and it pays around $90k - $110k here. On the other hand, data engineering sits between devops and software engineering, and pays between $140k and $180k, although I have acquaintances making north of $400k as data engineers. As a data scientist you mostly write SQL queries and make Tableau dashboards on top of the ETL pipeline that data engineers built out. It's less challenging and honestly kind of mundane. Given a question, you usually know which SQL query to write to answer it. The few data scientists where I work are trying to shift into data engineering by learning more about kubernetes / spark / airflow / clojure.
From my little experience I agree with you. I always considered DE > DS. Especially since they are the ones building a platform that all of the analysis/ML is done on. Meaning that company pay big bucks for very good DEs.
@@sohum6326 Yup, Without the DE the DS can't do their job. Luckily I'm kinda a hybrid of DE/SE/PM and a little DS but that just comes from years of experience.
Right! Same is true here in Italy. I think Joma said it best when he pointed out that these colossal SaaS companies usually have all their main processes already mapped out and not much challenging/stimulating work is left... thus the job market shift
Interesting perspective. My experience is that data science is "sexier". Anyway, I'm a data engineer, I actually have a video about it on my channel! :)
I'm a data engineer , I use a variety of tools to do my job including Informatica, SQL Server, Oracle etc. As part of my role I am also learning about data governance and will soon go on training courses how to use Governance and Data Quality tools. A good data engineer would also have good working knowledge of the corporate data. I have to disagree with the notion that DE is not intellectually stimulating... but it might be different over at Facebook where you used to work.
Hey man, I'm currently doing my current job as an ETL Developer. But sometimes I feels like "am I a Data Engineer or an ETL Developer?" I uses IBM DataStage ,Pentaho, Bash, Oracle, Postgresql, and Mysql. So Am I a Data Engineer or ETL Developer? Sometimes I'm confused.
Great advice on not restricting yourself! I landed a Data Engineer internship for the summer - will definitely keep this in mind and try to be as proactive as I can. Thank you Joma :)
Data scientist is a just an illusion of a cool title for what is really a data analyst or just simply working in analytics. It's all just names and semantics. Data engineer can be limited in scope but it depends on what you're doing like how you're developing and maintaining a framework or pipeline. A good infrastructure is always needed and requires having a good knowledge all roundz that is, develops for building and deploying, sound testing techniques for validating the data, framework for expanding how data is added and so on. A data scientist typically don't have that background. Like i said most of who you call data scientist are really data analyst. Nowadays machine Learning engineers are the real data scientist but they also have a very solid data Engineer background for understanding algorithms and setting up platforms.
This comment put into words exactly what I feel about the data science space. An machine learning engineer is just an advanced software engineer with more knowledge about data science.
Thank you for the truth about modern day glorified data analysts lol. This is partly why I rather have a data engineering background before embarking to a role more involved in machine learning etc.
Agree 100%. Most often the Data engineer has to know the data better than the Scientist. This is like being a report developer and you get an ad hoc request from a dept manager asking for everything. A good report developer will know to ask 'what are you really looking for? I'm not just dumping a whole dataset on you for you to slice/dice in excel'. Same applies to DE/DS. But Data Scientist is sexy term and front facing. If you have good mgmt they know the DEs provide the data.
“Technical part is the easiest part of the job at a large company” I used to work at a relatively large company in Tokyo, still it felt exactly the same!!! Thank you for uploading new video!!
like your video as always! I will give my two cents as a data engineer. Pay really depends on region and company. In Toronto, data engineers typically get paid higher than data scientists and software engineers. But data engineers in Toronto typically work on hadoop/spark/distributed system problem, which is different from large tech companies, like facebook.
DE is akin to being a construction worker. And honestly, I love it. Versus being a home architect=DS, interior designer=front end dev etc. It's a technical way to be a builder. Yes, less creative freedom and intellectually less challenging. But if you like semi mundane & repetitive tasks that still make a difference than DE is the way to go! It's more of a personality fit, I'd say.
In a smaller market this has been the opposite of my experience: when you have to build stuff from scratch the order of priority and value comes from data engineers then data scientists. Sometimes data scientists do both though because of a shortage of money and/or talent. Facebook is the opposite probably because as he says most of the work has already been done and so they call typical BI analysts Data Scientists and DevOps people Data Engineers and the really hard stuff (from a technical standpoint) is left to people who are way more senior than he was or have PhDs.
Can you do software engineer vs data scientist? About their pay, working hours, and which is better. Since you have experience in bothof these fields, you must be knowing a lot.
As a data engineer, it was the toughest part is constructing data quality because there are so much stress to fix things under a short period of time. The requirements constantly change so you are supposed to be flexible.
@Marc B. Requirements are based on the clients needs. Sometimes there are errors found and you have to address it to the clients to see how they want the dashboards to look or how the data is structured. Requirements are based on business logic needs. Some business have tables they don’t even use anymore but that is communicated last min sometimes
Joma makes a lot of good points here, and I just want to rant a little as a DE myself. DE vs DS is not a good comparison, because one is an engineering role while the other is an analytical/quantitative role. Very different career paths and skills. I think it's better to compare data engineering with software engineering. The difference is that software engineers build the tools, while data engineers use the tools. For example, software engineers build Spark, while data engineers use Spark. Joma is right here about data engineering not being as intellectual challenging. Think about it, would you rather be the one that build the distributed systems underlying Spark, or do you just want to import pyspark and use APIs? However, for most engineers, they would never get the opportunity to build cool distributed systems like Spark. Most engineers work at product driven companies that just need to build a good product with good UI/UX. If you work at such a company, I think DE is a bit more intellectually challenging than frontend web development. If you want to build cool distributed systems, then you gotta go work of companies like Google and Databricks. It might be better to compare DE to backend engineering. I actually think a backend engineer with solid data modelling skills and database knowledge would be able to transition to a DE role very easily. However, a DE who only works with pyspark and cloud technologies would find it a bit more difficult to transfer to backend engineer role. In conclusion, if an "engineer" wants to stay relevant and a "beast" like Joma, then they should build the tools, rather than use them.
Great vid! I’m an ex PM of a bluechip, turned into a ‘everything’ guy in a data analytics startup. Company culture plays a big part in how you gain value out of your role. I’m in a much happier place because I am now learning new skills again, and feel like I’m adding tangible value. Sometimes you need to drop the roles, forget about pay for a second and just follow the pathway that interests you / gets you out of bed in the morning.
"Take the opportunity to invest in yourself so that you can be a beast at everything." Well said, thank you Joma. I agree that too many people limit themselves and miss out on opportunities for growth and career advancement, and even just for personal pleasure.
This hits hard. As a data science analyst turned PM, I miss the technical stuff but am highly aware that going back into a technical role is equal to destroying my career.
Scorpion in our company, technical roles are well paid but are limited in perspective. For instance the maximum a data scientist gets paid is around 6.200 euro. For other roles like (product) manager the maximum is above 7.800 euro.
Data Engineer in Silicon Valley: SQL, Python, R, Java, C#, Scala, AWS/Azure/GCP, PySpark, Databricks, Tableau, Azure Data Factory... works along with Data Scientists and ML Engineers. Data Engineers from the rest of the world: SQL, SSRS, SSAS, SSIS, Qlikview, Cognos, Crystal Reports, advanced Excel skills creating macros and VBA... works alongside BI Developers.
I think you can transition into ML from DS/DE because a large portion of ML is cleaning, normalizing and legitimizing the incoming data plus you work with a lot of graphs?
This defenetely made me want to get a job as a data scientist, I hace knowledge of it already, and the part about thinking about the product, its future and supporting It with conclussions from data totally defines what I like. And so does when you talk about a data scientist with PM knowledge. Thanks a lot now I am more determined about my future.
yeah Data Analyst is basically business job.. like BS Data Science is a mixture of Business Management and Computer Science. And a Data Analyst/Scientist can work in both Tech & Business.
Thank you for this video. I heard there are different types of data engineer, like data engineer analitycs and architect etc. So I think in this video you told of the first.
I work as a Database Engineer in a small team and I get to fully create cloud-based data solutions by doing DE, DS, Architecture, and DevOps Management plus I get paid more than Project Managers. That said, I have to still deal with Project Managers, and sometimes it's an absolute pain in my ass.
@@jenniferbyrne2980 It is, but if they're unfamiliar with the jobs they are managing, the expectations of their timeline can become extremely unrealistic. Additionally, if you are more forward-thinking, some can't keep up.
Sometimes, I am so confused about how to distinguish DE vs DS and how can i involve to either. You clarified clearly what I curious about and much appreciated! Thank you. I cant leave without comment. :)))
In term of number of job openings, data engineer outweigh data science. According to indeed there are 6-70,000 jobs available for data engineers, while 10,000 jobs for data scientists in 2020. What do you all think on this?
@@t-bone9239 it depends.. If data engineer work is just sql based ETL Pipeline, then it is yes.. if data engineer is writing real time streaming pipeline and complex batch pipeline using Spark, Kafka, Flink etc, then it is not possible.. I would say DE with some ML knowledge can do DS work..
I am a DataNinja. I work in a tangential field as a Data Engineer, Scientist and PM. There are 700 of us at Syniti who do not allow ourselves to be boxed in. While the bifurcation here is natural and aligns to our consulting levels, we are still all hybrids. Is this crazy to this group?
I am landing a job as DE. At the beginning I did image that I would work about distributed system such as data, task and solve these relational problem. But in short time I realized that this is belong to your company size ( Joma has video talk about this), and it like Joma said, DE just use tools and build some kind of pipelines, monitor pipelines from raw to usable data. Moreover, the tools that I use that is developed by another team with stuffs back-end services. For now, I install Hadoop/ Spark on local machine and learn by self. So I really appreciate Joma because you help me to understand more behind the since :D .
fucking genious, thats a nice end out conclution. fucking love your channel, i'm your fucking biggest fan in Argentina, and also i'm fucking love meditating... HAHAHA
Except based on what sort of data asks he's referring to, most of those would be done by an analyst...not a data scientist. In my experience, data scientists are brought in to work on very specific features or upgrades (and are then let go once done), while data engineers are kept around to make sure it all keeps working.
Hi Joma, I'm currently struggling to choose between those 2 (also thinking about going into masters degree for ML, dunno). I was at first very convinced to go for Data Science, but my main difficulty is on the analysis and cases of the interviews. I don't know if this is something of a natural "talent" I lack, or something I can work and practice to learn. I find no difficulty in digging into large data and organizing it or creating a dashboard, but I struggle to actually create something meaningful with it and finally generating the "insight" I'm looking for. I don't know if you can understand what I mean, but could you give me some advice?
As a DS I can tell you it’s not a lack of talent - without an understanding of what questions need to be answered it can be very overwhelming to pore through a large dataset with tons of variables to try to find a needle in a haystack that you know may not even exist
I'm kind of transitioning myself from data science to data engineering, I technically already have lol I have a hard time believing that you dislike data engineering given you switched to software engineering. Data Engineering is software engineering with big data technologies.
i'm breaking my back learning stuff, taking online courses and learning in school a lot of tthe technical part and it worried me a lot when in 5:55 he said it is the easy part
Im sorry to say but he is right.I am a data engineer and the most important aspect for me is the ability to convey concepts to non-tech people as well as the ability to persuade the team to do things in more effective/productive ways.The learning is the easy part. Once you've learnt something its in your noggin forever.
Currently taking on a data scientist role in my grad capstone project, and while I like it, I prefer aspects from DE more. Very jealous of my colleague who is taking on the data engineering role with the project I'm on haha.
Create a course for the basic start of DS/DE/DA, for those who can't pay like me but can be a master by learning from an extraordinary educator as you.
Do companies typically look for Data Analysts that come from a math background? I did 3 years of my undergrad in Finance and I switched to business analytics and a minor in data science. I did my first internship as a DS consultant for a startup over the summer but now I'm having trouble landing my second internship. Most of the listings I am seeing on my school's job board are looking for students with a degree in Software Engineering or CS, I apply anyways but because my major is Commerce I have a feeling my application is just being ignored. Feeling pretty discouraged.
learn ML and you'll be able to do data mining and create neural networks. In my opinion ML is the best choise if you need to have a balance between engineering and analytics
@@hopelesssuprem1867 thanks. But sometimes when I fill an online form, and they ask me which job title I'm interested in, I see DS & DE but I never see AI. This makes me confused
I don’t think it’s necessary true that data engineers makes less than software engineers. Of course a lot of it varies for company, but imo de make the same if not more than swe. A DE can be anything from writing a lot SQL, to managing dbs, to apis, doing some DS things, to etl.
I am a Data Engineer. I hate it.
:( I’m sorry
Do you get much face time with the customer or are you behind the scenes?
Why do u hate it?
@@malcorub Yeah, I get to face the clients and business vendors and we need to understand the same business problems that Data Scientists, Insight Analysts and Project managers need to understand but our job is to provide them the correct data, make Data Pipelines and refresh the datamart for the dashboards etc. But I feel like a data tech support for the rest of the project team. Also involved in working with legacy system in Teradata and modernizing it with the latest Big Data tools. But I am slowly learning the Data Analyst / Scientist work as well. That is where the "sexy" part of the work revolves around. Data Engineering is not a sexy job but it is one of the most important positions in a company.
@@sanjogh777 As long as you like working with data there is hope. I am a data analyst/report developer, I rely on my D.E's (who get paid more than I do) heavily to keep the data infrastructure going, for data pipelines and building complex data models and without you guys, my job would be so much harder or dare I say, IMPOSSIBLE. Like Jona said at the end of the video and as you mentioned in your comment, don't limit yourself. Dabble in the data analyst, PM and data scientists side. It's much more satisfying when you understand and can derive meaning from the data you are working with which many times, at least in my world, the DE's don't or choose not too.
As a poet who wants a data scientist/engineer job to just make money temporarily, "not intellectually challenging" is exactly what I'm looking for.
Hit me up for more details ☝️☝️
Are you me?
@roshankr3938 bhai (i assume) can you update your email in settings so i might be able to reach out to you?
I've tried Data Analyst in the past and it wasn't my thing. You have to have that "analytical thinking" to translate data to words. Soon after, I switched to become a Data Engineer and have been loving it so far because as the title says for itself, it's more of an engineering role rather than analytical. There comes a task where you need to get some data in a specific format and that's it! You don't need to waste time shift through data to understand anything and that's what I love about being a Data Engineer. Data Scientists however, is a role that I have not tried yet as it requires a certain knowledge of mathematics and analytics which to be honest I have no intention to be mastered at but imho DS is an upgraded version of DA with more complexity of machine learning models and mathematics.
Therefore I would say it depends on people's mindset to see which one is for them. If you think more analytical, then go with DA or DS but if you prefer less analytical and more engineering, then DE is your chance!
dude u make no sense DS is no upgrade of DE its not even the same job nor the same skills , a data engineer is a software engineer while the data scientist is not an engineer im sure u re not a data engineer cause i ve never seen someone move from data analytics to data engineering
@@abdessamadbaahmed8930 Lol okay, well I am a DE and I did it! And I never said DS is an upgrade of DE, idk where did you infer that from.
@@faridakhavan1996 but imho DS is an upgraded version of DA with more complexity of machine learning models and mathematics.
@@abdessamadbaahmed8930 I said DA not DE! Data Analyst not Engineer!
@@faridakhavan1996 ooops my bad man
I am a data analyst/report developer, we do not have any data scientists at my place of work. ... truth be told, the data engineers at my place of work get paid more than me and my peers and they have alot less face time with the business units and costumer... but because they are so behind the scenes, the analysts are seen in a more positive light by the customers, business units and leadership than the data engineers so we get more accolades and bonuses by leadership.
> Data science analytics is super rewarding
> Leaves his data analyst job at Facebook to become a software engineer because he's tired of doing bitch work
I felt this is because you are in a big company like Facebook. If you were in a startup or medium size company, data engineers have chance to do a lot of exciting things including algorithm, data pipeline, or even machine learning model. Nevertheless, thanks for sharing the relations between these three roles in big companies! Hope to see more video like this!
This guy looks like Yagami Light from Death Note. He needs a shinigami.
He has a millionaire one
@@PsyberSenpai LOL
Wow this is the exact opposite of my experience. In NYC the market is saturated with math / physics grads willing to work as data scientists, but who are less familiar with coding. As such wages are suppressed and it pays around $90k - $110k here. On the other hand, data engineering sits between devops and software engineering, and pays between $140k and $180k, although I have acquaintances making north of $400k as data engineers. As a data scientist you mostly write SQL queries and make Tableau dashboards on top of the ETL pipeline that data engineers built out. It's less challenging and honestly kind of mundane. Given a question, you usually know which SQL query to write to answer it. The few data scientists where I work are trying to shift into data engineering by learning more about kubernetes / spark / airflow / clojure.
That's interesting. In what field is this?
From my little experience I agree with you. I always considered DE > DS. Especially since they are the ones building a platform that all of the analysis/ML is done on. Meaning that company pay big bucks for very good DEs.
@@sohum6326 Yup, Without the DE the DS can't do their job. Luckily I'm kinda a hybrid of DE/SE/PM and a little DS but that just comes from years of experience.
Right! Same is true here in Italy. I think Joma said it best when he pointed out that these colossal SaaS companies usually have all their main processes already mapped out and not much challenging/stimulating work is left... thus the job market shift
Interesting perspective. My experience is that data science is "sexier". Anyway, I'm a data engineer, I actually have a video about it on my channel! :)
I'm a data engineer , I use a variety of tools to do my job including Informatica, SQL Server, Oracle etc. As part of my role I am also learning about data governance and will soon go on training courses how to use Governance and Data Quality tools.
A good data engineer would also have good working knowledge of the corporate data. I have to disagree with the notion that DE is not intellectually stimulating... but it might be different over at Facebook where you used to work.
Hey man, I'm currently doing my current job as an ETL Developer. But sometimes I feels like "am I a Data Engineer or an ETL Developer?"
I uses IBM DataStage ,Pentaho, Bash, Oracle, Postgresql, and Mysql.
So Am I a Data Engineer or ETL Developer? Sometimes I'm confused.
Great advice on not restricting yourself! I landed a Data Engineer internship for the summer - will definitely keep this in mind and try to be as proactive as I can. Thank you Joma :)
Data scientist is a just an illusion of a cool title for what is really a data analyst or just simply working in analytics. It's all just names and semantics. Data engineer can be limited in scope but it depends on what you're doing like how you're developing and maintaining a framework or pipeline. A good infrastructure is always needed and requires having a good knowledge all roundz that is, develops for building and deploying, sound testing techniques for validating the data, framework for expanding how data is added and so on. A data scientist typically don't have that background. Like i said most of who you call data scientist are really data analyst. Nowadays machine Learning engineers are the real data scientist but they also have a very solid data Engineer background for understanding algorithms and setting up platforms.
This comment put into words exactly what I feel about the data science space. An machine learning engineer is just an advanced software engineer with more knowledge about data science.
Thank you for the truth about modern day glorified data analysts lol. This is partly why I rather have a data engineering background before embarking to a role more involved in machine learning etc.
yeaah i feel u, idk why they are called scientists but then again, most ppl who hold these positions are either master or phd grads in math
Agree 100%. Most often the Data engineer has to know the data better than the Scientist. This is like being a report developer and you get an ad hoc request from a dept manager asking for everything. A good report developer will know to ask 'what are you really looking for? I'm not just dumping a whole dataset on you for you to slice/dice in excel'. Same applies to DE/DS. But Data Scientist is sexy term and front facing. If you have good mgmt they know the DEs provide the data.
“Technical part is the easiest part of the job at a large company”
I used to work at a relatively large company in Tokyo, still it felt exactly the same!!! Thank you for uploading new video!!
like your video as always!
I will give my two cents as a data engineer. Pay really depends on region and company. In Toronto, data engineers typically get paid higher than data scientists and software engineers. But data engineers in Toronto typically work on hadoop/spark/distributed system problem, which is different from large tech companies, like facebook.
"Take the opportunity so that you can invest in yourself so that you can be a best AT EVERYTHING!" 인생의 교훈.
DE is akin to being a construction worker. And honestly, I love it. Versus being a home architect=DS, interior designer=front end dev etc. It's a technical way to be a builder. Yes, less creative freedom and intellectually less challenging. But if you like semi mundane & repetitive tasks that still make a difference than DE is the way to go! It's more of a personality fit, I'd say.
Security room outside your entrance that's where cybersecurity comes into the picture
@@garimagupta7227 Lol or the guesthouse. Not sure which analogy fits best. Haha
@@ajtam05 In you Astral Chart, in one of them ( Sun, Mercury or Mars ), might have either Capricorn or Taurus.
In a smaller market this has been the opposite of my experience: when you have to build stuff from scratch the order of priority and value comes from data engineers then data scientists. Sometimes data scientists do both though because of a shortage of money and/or talent. Facebook is the opposite probably because as he says most of the work has already been done and so they call typical BI analysts Data Scientists and DevOps people Data Engineers and the really hard stuff (from a technical standpoint) is left to people who are way more senior than he was or have PhDs.
Can we get an update on how the SWE job is going and how your life is now as a SWE compared to a DS?
Oh yea, that’s a good idea
lol next is devops vs ios engineer vs sysadmins vs embed systems engineer .... ooops infinite loop
please !
Oh yup, this sounds interesting!
Can you do software engineer vs data scientist? About their pay, working hours, and which is better. Since you have experience in bothof these fields, you must be knowing a lot.
As a data engineer, it was the toughest part is constructing data quality because there are so much stress to fix things under a short period of time. The requirements constantly change so you are supposed to be flexible.
What do you mean by “requirements” ?
@Marc B. Requirements are based on the clients needs. Sometimes there are errors found and you have to address it to the clients to see how they want the dashboards to look or how the data is structured. Requirements are based on business logic needs. Some business have tables they don’t even use anymore but that is communicated last min sometimes
@crystal12w Do you enjoy your job as a data engineer?
@@88andreaw I do like it but I also need creative outlet because I like photography and videography as well. It’s best to have hobbies too.
Agree. And what’s tough for DEs is that the moment a project starts, immediately we are the dependency, the pressure is always on us 😅
wearing patagonia... seems like someone watched the latest techlead video :D
Joma makes a lot of good points here, and I just want to rant a little as a DE myself.
DE vs DS is not a good comparison, because one is an engineering role while the other is an analytical/quantitative role. Very different career paths and skills.
I think it's better to compare data engineering with software engineering. The difference is that software engineers build the tools, while data engineers use the tools. For example, software engineers build Spark, while data engineers use Spark. Joma is right here about data engineering not being as intellectual challenging. Think about it, would you rather be the one that build the distributed systems underlying Spark, or do you just want to import pyspark and use APIs? However, for most engineers, they would never get the opportunity to build cool distributed systems like Spark. Most engineers work at product driven companies that just need to build a good product with good UI/UX. If you work at such a company, I think DE is a bit more intellectually challenging than frontend web development. If you want to build cool distributed systems, then you gotta go work of companies like Google and Databricks.
It might be better to compare DE to backend engineering. I actually think a backend engineer with solid data modelling skills and database knowledge would be able to transition to a DE role very easily. However, a DE who only works with pyspark and cloud technologies would find it a bit more difficult to transfer to backend engineer role.
In conclusion, if an "engineer" wants to stay relevant and a "beast" like Joma, then they should build the tools, rather than use them.
👏👏👏❤️
Great insights
Love the ending. Thank you for the advice. 👍
Be a beast at everything. My motto for sure. Great info. Thanks
Great vid! I’m an ex PM of a bluechip, turned into a ‘everything’ guy in a data analytics startup. Company culture plays a big part in how you gain value out of your role. I’m in a much happier place because I am now learning new skills again, and feel like I’m adding tangible value. Sometimes you need to drop the roles, forget about pay for a second and just follow the pathway that interests you / gets you out of bed in the morning.
why did you turn from PM role to data analytics field?
"Take the opportunity to invest in yourself so that you can be a beast at everything."
Well said, thank you Joma. I agree that too many people limit themselves and miss out on opportunities for growth and career advancement, and even just for personal pleasure.
Machine Learning Engineer vs Software Engineer?
This hits hard. As a data science analyst turned PM, I miss the technical stuff but am highly aware that going back into a technical role is equal to destroying my career.
What's a PM?
product manager -> product owner deluxe
"going back into a technical role is equal to destroying my career"
WUT
Scorpion in our company, technical roles are well paid but are limited in perspective. For instance the maximum a data scientist gets paid is around 6.200 euro. For other roles like (product) manager the maximum is above 7.800 euro.
Astarte Rap, I can’t fully disclose. Large financial services company with strong focus on data (and more and more on data science).
Data Engineer in Silicon Valley: SQL, Python, R, Java, C#, Scala, AWS/Azure/GCP, PySpark, Databricks, Tableau, Azure Data Factory... works along with Data Scientists and ML Engineers.
Data Engineers from the rest of the world: SQL, SSRS, SSAS, SSIS, Qlikview, Cognos, Crystal Reports, advanced Excel skills creating macros and VBA... works alongside BI Developers.
We need more of joma startup...
absolutely love your brutal honesty!
I think you can transition into ML from DS/DE because a large portion of ML is cleaning, normalizing and legitimizing the incoming data plus you work with a lot of graphs?
Boy, thanks for this video, helped me a ton!
"Data Science Analytics" just sounds like a fresh name for business intelligence work, not actual science.
You may hear it as a "Data Analyst" role too.
This defenetely made me want to get a job as a data scientist, I hace knowledge of it already, and the part about thinking about the product, its future and supporting It with conclussions from data totally defines what I like. And so does when you talk about a data scientist with PM knowledge. Thanks a lot now I am more determined about my future.
You should rename the title to Data Analyst vs Junior Data Engineer
yeah Data Analyst is basically business job.. like BS Data Science is a mixture of Business Management and Computer Science. And a Data Analyst/Scientist can work in both Tech & Business.
Finally, Joma make good contet, yeaah
Wow. This video really hits. Blunt honesty. Currently a Data science masters student at Pace university
Good to see you are still on TH-cam
Thank you for this video. I heard there are different types of data engineer, like data engineer analitycs and architect etc. So I think in this video you told of the first.
Thanks for the insight! Totes new to this area but defs keen to learn more! Hello from Australia
Hit me up for more details ☝️☝️
I work as a Database Engineer in a small team and I get to fully create cloud-based data solutions by doing DE, DS, Architecture, and DevOps Management plus I get paid more than Project Managers. That said, I have to still deal with Project Managers, and sometimes it's an absolute pain in my ass.
Well, that's their purpose, right? :D
@@jenniferbyrne2980 It is, but if they're unfamiliar with the jobs they are managing, the expectations of their timeline can become extremely unrealistic. Additionally, if you are more forward-thinking, some can't keep up.
@@ChronicSurfer you are so right! I've experienced both
Great video Joma.
Congrats!
Sometimes, I am so confused about how to distinguish DE vs DS and how can i involve to either. You clarified clearly what I curious about and much appreciated! Thank you.
I cant leave without comment. :)))
Joma rox! Thank you for all pointers :)
Just wanna say bro, love your videos. Keep up the great work!
In term of number of job openings, data engineer outweigh data science. According to indeed there are 6-70,000 jobs available for data engineers, while 10,000 jobs for data scientists in 2020.
What do you all think on this?
Data Scientists could still work as Data Engineers but not vice versa
@@t-bone9239 it depends.. If data engineer work is just sql based ETL Pipeline, then it is yes.. if data engineer is writing real time streaming pipeline and complex batch pipeline using Spark, Kafka, Flink etc, then it is not possible.. I would say DE with some ML knowledge can do DS work..
@@sundarkris1320 agreed.
I was exactly searching for this video. What about entering Data Engineering as a bridge to switch later to Data Science (as a software engineer) ?
I’ve seen a few do that at old company. For people who enjoys more analytics
Really awesome video man. Keep up the good work!
Great career tips man :)
I am a DataNinja. I work in a tangential field as a Data Engineer, Scientist and PM. There are 700 of us at Syniti who do not allow ourselves to be boxed in. While the bifurcation here is natural and aligns to our consulting levels, we are still all hybrids. Is this crazy to this group?
I am landing a job as DE. At the beginning I did image that I would work about distributed system such as data, task and solve these relational problem. But in short time I realized that this is belong to your company size ( Joma has video talk about this), and it like Joma said, DE just use tools and build some kind of pipelines, monitor pipelines from raw to usable data. Moreover, the tools that I use that is developed by another team with stuffs back-end services. For now, I install Hadoop/ Spark on local machine and learn by self. So I really appreciate Joma because you help me to understand more behind the since :D .
So that you can be a beast at everything. Amazing advice.
thanks for this video Joma! Gave me a new perspective on the DE aspect of things.
that last bit was inspiring. Thanks for that
That last statement... 🔥🔥🔥🔥🔥🔥🔥🔥
Make a video about Research or Practical path in Ai/ML !!!
fucking genious, thats a nice end out conclution. fucking love your channel, i'm your fucking biggest fan in Argentina, and also i'm fucking love meditating... HAHAHA
I watch this video again, I do have a bit confusion that why you decided to be a SE ? and how DS skill support you as SE ?
Yeah man, I'm a Data Engineer!
Do you have a CS degree?
I have to say, your words are so true about data stuff = =
Good to see you man
Zein El Mokhtar thanks man u 2
Except based on what sort of data asks he's referring to, most of those would be done by an analyst...not a data scientist. In my experience, data scientists are brought in to work on very specific features or upgrades (and are then let go once done), while data engineers are kept around to make sure it all keeps working.
Honestly DE sounds awesome. I am working on my Master degree but I don't mind the mundane work.
Hi Joma, I'm currently struggling to choose between those 2 (also thinking about going into masters degree for ML, dunno). I was at first very convinced to go for Data Science, but my main difficulty is on the analysis and cases of the interviews. I don't know if this is something of a natural "talent" I lack, or something I can work and practice to learn. I find no difficulty in digging into large data and organizing it or creating a dashboard, but I struggle to actually create something meaningful with it and finally generating the "insight" I'm looking for. I don't know if you can understand what I mean, but could you give me some advice?
As a DS I can tell you it’s not a lack of talent - without an understanding of what questions need to be answered it can be very overwhelming to pore through a large dataset with tons of variables to try to find a needle in a haystack that you know may not even exist
More videos like this please!
Great video! I watch you from MX, YUC.
I have the same Patagonia jacket Joma oppa 😍
Everyone in tech does!
Well said Joma
I'm kind of transitioning myself from data science to data engineering, I technically already have lol
I have a hard time believing that you dislike data engineering given you switched to software engineering. Data Engineering is software engineering with big data technologies.
DE job is not innovative and challenging? You did not work in right team, then 🤣😂
i'm breaking my back learning stuff, taking online courses and learning in school a lot of tthe technical part and it worried me a lot when in 5:55 he said it is the easy part
Im sorry to say but he is right.I am a data engineer and the most important aspect for me is the ability to convey concepts to non-tech people as well as the ability to persuade the team to do things in more effective/productive ways.The learning is the easy part. Once you've learnt something its in your noggin forever.
Nice channel...nicee space work
thank you is all I can say
Your channel is getting better, I'm resubscribing :)
Loved it ❣️
i love this video a lot
9:09 bro Awesome 🔥🔥🔥
redbull, into workable software!
Nice patagucci fleece, I have the same one!
Currently taking on a data scientist role in my grad capstone project, and while I like it, I prefer aspects from DE more. Very jealous of my colleague who is taking on the data engineering role with the project I'm on haha.
Got hired as a data engineer out of uni 3 months ago at a good company it's great
Congratulations !
What do you major in?
@@nomadicgoddess3648 masters in computer science
That self-slap at 2:17 though
alright people! get back to work
Lol 😆😂
Thanks for the insight. What about data architects? Is this position still alive and relevant? Or has it merged in one of DS, DAnalyst, DEng?
Create a course for the basic start of DS/DE/DA, for those who can't pay like me but can be a master by learning from an extraordinary educator as you.
7:00 I knew it. Sounds like an "intrapreneur" with an entrepreneurial mindset.
it would be awesome if you make a podcast about information technology job..
Do companies typically look for Data Analysts that come from a math background? I did 3 years of my undergrad in Finance and I switched to business analytics and a minor in data science. I did my first internship as a DS consultant for a startup over the summer but now I'm having trouble landing my second internship. Most of the listings I am seeing on my school's job board are looking for students with a degree in Software Engineering or CS, I apply anyways but because my major is Commerce I have a feeling my application is just being ignored. Feeling pretty discouraged.
Try Facebook, your background is pretty common for DS analytics there
They way I see it, what will happen to DS, is what happens (still happens) to BI. That is, "most BI projects have and will fail".
Great video Joma but can you reverse a linked list with your eyes closed?
When is the startup series coming out, Jonathan ?
Great video
This reminds me of Software Engineer VS Developer. Yes, there is a difference.
its a dumb statement cause actually u need far more skills to be a data engineer than being a data scientist
Sung Kim Then tell me difference between comp sci and comp eng
@@abdessamadbaahmed8930 I'd say you need far more practical skills
i love that screen thanks bro
Yo joma please make videos like they were back then - Funny !
please don't be greedy and i missing ur video editing skills .
I want to get into filmmaking now
th-cam.com/video/uHt01D6rOLI/w-d-xo.html
Joma.... I want your advice on having a data scientist girlfriend
Brother the yellow shirt is on point.
learn ML and you'll be able to do data mining and create neural networks. In my opinion ML is the best choise if you need to have a balance between engineering and analytics
I've been learning ML & DL. But I'm still confused, what category does ML fall under? Is it a data scientist job or a data engineer job?
@@mizile1486 ds is just analytics, de is data bases, ml is AI
@@hopelesssuprem1867 thanks. But sometimes when I fill an online form, and they ask me which job title I'm interested in, I see DS & DE but I never see AI. This makes me confused
Hi Joma, I am a SWE going my masters in Big Data Analytics
Beast is coming. :D
I don’t think it’s necessary true that data engineers makes less than software engineers. Of course a lot of it varies for company, but imo de make the same if not more than swe. A DE can be anything from writing a lot SQL, to managing dbs, to apis, doing some DS things, to etl.