Being A Data Engineer: Expectations vs Reality
ฝัง
- เผยแพร่เมื่อ 31 ก.ค. 2024
- What is a data engineer?
The goals of a data engineer are much more big-picture and development focused. Data engineers build automated systems and model data structures to allow data to be efficiently processed.
This means the goal of a data engineer is to create and develop tables and data pipelines to support analytical dashboards and other data customers (like data scientists, analysts, and other engineers).
It's similar to most engineers. There is a lot of design, assumptions, limitations, and development that occurs to be able to create some sort of final robust system.
Learn more about being a DE for free with googles DE certificate
coursera.pxf.io/rn1rGv
0:00 Intro
0:48 Hadoop And Spark?
3:04 Isn't it All About Data Pipelines
5:02 All The Focus Is Not On Data Engineers
6:15 What to expect for a data engineering salary?
If you need data consulting help, then reach out to our team here:
www.theseattledataguy.com/
Also, if you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Check out my Medium here:
/ seattledataguy
What Skills Do Data Engineers Need?
• What Skills Do Data En... - ตลก
If you enjoyed this video, then consider subscribing today! th-cam.com/channels/mLGJ3VYBcfRaWbP6JLJcpA.html
I've been a Business Intelligence Engineer for 7 years now. You got it, one other piece that you are missing is auditing and accounting. Ensuring your numbers are correct to the source system.
I totally agree and can't believe I forgot this. Actually the first team I worked on was a finance team so I really was spending time on accounting data making sure costs lined up between reports, the data warehouse and the source system.
It would be great if you could give some examples of what causes a discrepancy in reporting. I recently discussed this with a teacher of data analytics and visualization and received push back on messy data and discrepancy in reporting. I have specific examples that I use to help people understand different problems and I need some good examples for differences in reporting.
@@StoneyVintson same source different teams manipulating/cleaning data differently. One was a web ui using MERN, another was vertica tableau links (not that logic differs). Reporting is something if you’re cross Platform bound to be done by various teams/stacks. No communication btw these silos/cross validation led to a huge P0.. once upon a time
Loved the crisp to the point no bs video. Thank you!
I am really glad you enjoyed my video! Are you a data engineer as well?
I'm not one to usually comment but you are incredibly good at explaining and layering information, you taught more in 7 minutes than others with 30 minute videos, thank you for taking the time to share your experiences.
Liked and subscribed btw will definetly keep an eye out for new videos.
You're too kind! I appreciate your comment more than you know.
Very insightful and I can definitely relate as to how DE is a "behind the scenes" type of role. Great video 👍
Yes, its an interesting place to be. Sometimes its great, other times you wish people would realize all the work you put in. Pros and cons
I've been working 4 years as a Data Engineer with the BI Developer title. All you said is on point, and SSIS brought back memories of working at the beginning in a Microsoft only environment.
Yeah, SSIS and Azure data factory is where a lot of us start
How did you guys make your company transition into using linux in the mix as well. I’ve just started my first job and my job is automating stuff and I’d prefer to be doing it in linux but they’re exclusively in the MS ecosystem rn and I’m curious how and when you decided it was best to do it in linux rather than windows
It's a year and a half I started as Data Engineer and every word you said it's truth. Thanks for the video.
I am glad this resonated with you. I am planning to make many more videos that should connect with data engineers, data scientists, and tech consultants.
Thanks man! Appreciate your sharing and honesty! Good luck and best wishes.
First off, I am glad to provide insights and I really appreciate the comment!
P.S. To quote graham stephan, all I ask for in return is that you take a moment and smash that like button!
Thanks for the information, I learn a lot about Data Engineer expectations in this video
Thank you, I am glad you enjoyed it
Newt week I'm starting a position as DE I'm really excited!
Thanks for the video. Appreciate it!
Thank you for the comment and the appreciation!
Thanks for this! I've been studying to be a data analyst and have realized that I'm far more interested in a data engineering role. I've been a bit put off by the Spark, Hadoop, Kafka, etc. neverending list of "must have" skills for it though. Interested to see what else I can learn from you!
I wouldn't be put off by those skills right away. I honestly haven't had to use any of those in my full-time job yet. A few for consulting projects but most of the time, even if you are working on hadoop, you might just have to interact with it at a SQL layer or a managed service layer. All of which make these tools slightly easier to work with. I recall the first time i had to spin up a hadoop instance on a server...I hated it. I just wanted to work on the data...Thanks for the comment and the Substack subscription
Essentially you need to learn programming first, then apply programming to data using mathematics and data principles. The part of learning programming first is a put-off for many
Can't disagree more about Data Engineers being the middle / behind the scene man... great video :)
Yeah, but maybe this will be our year! * Laughs awkwardly *
I appreciate the tips I am new to the term ETL but I have done many data migrations and relational table design. I at least have a little bit of background on the role.
Nice video, I wasn’t expecting to hear SSIS but it was nice to see you mention it. I’ve used in school and in my own side projects. It’s like a love hate relationship at first because when it works I have to issues with it, but when there’s an error sometimes it’s hard to figure what the error code really means and what’s going on with the rows being affected. It seems like SSIS is dying from what I’ve been told and I’m not sure how easy it is to use Hadoop or spark as I’ve only used those programs once in lecture.
Thank you for the comment! Yeah, generally most low-code style tools pose some challenges in terms of error messaging. Most tools, even the modern ones, have limits and its never fun trying to spend a day all to figure out there is one configuration field is wrong.
SSIS is both dying and not dying all at once. For example, there are plenty of companies that still use SSIS. I have worked with several clients that are building new pipelines with SSIS. On the other side some are modernizing to use Azure data factory or some other low-code solutions but there are still others that don't plan to change. I would say it is worth learning new tools just in case.
Besides spark and hadoop, do you have any plans to try other tools?
@@SeattleDataGuy not currently, right now I’m try to build on what I know with Python, the whole sql suite (SSRS, SSIS, SSAS, and SSMS), excel, and PowerBi. I mainly use excel at work and a few times for Python but the other tools I learnt them in and school and don’t want to forget how to use them so I just use all of those tools in two of my side projects so when the time comes I could confidently explain on my resume what I did on those projects and apply to a wider range of jobs
@@SeattleDataGuy what are some examples of low code tools?
I'm so glad I got to skip over the Hadoop era... can't imagine how complex it would be to try and manage all of that (especially on prem)
Yeah, luckily a lot of this is abstracted away now. You can now layer presto or some other sql engine on top. Thanks for the comment!
Love your Channel!
Thank you!
I've been seeing something like that. Since I started as a data engineer as just code a bit more with SQL ( even with spark). Besides it I just use interfaces to manage jobs (jobs written in SQL + shell script but a very basic shell script demands.)
This was a very good video, I am now trying to switch to a career in Data Engineer
Awesome! Good luck on your data engineering journey
Thank you for a great video!
Thank you! good luck on your data engineering journey.
I am trying to switch career from data analyst to data engineer. This video really motivated me.
I am glad it did! Have you watched my video on switching from an analyst to a DE? th-cam.com/video/lGzh-QendJc/w-d-xo.html
Thank you, very useful information.
I am glad you enjoyed this video about data engineering videos
Very helpful, thank you for explaining!
Glad you enjoyed it!
@@SeattleDataGuy definitely!
Recently I got selected as a Jr. Data engineer in a company...I watched your video that helped me understand the concepts of data engineering. Thank you...
I am glad this. video helped you on your data engineering journey. Let me know if you have any more questions
@Sawon Bhowmik. are you a fresher for that job or experienced
@@c.giriyathmary9863 I am fresher..
Hi! Really informative video. Currently on a 3 month data engineering course fresh out of uni. I studied maths and data analytics and I’m really loving the skills and tools I’ve learnt so far! The course has covered 5 topics, sql (data warehouses and other data structures), python (classes, functions, api’s and sql), NoSQL (MongoDB), Big data (Hdfs, sparksql, pyspark, hive) and AWS (ec2, emr and s3).
Out of these skills which would you recommend I priorities and commit more time to mastering before my first entry level role?
I think generally you will need to know SQL, Python and data warehouses and data pipelines. The other stuff are company specific so it will depend where you want to work.
I'm going to apply for this position next year. Thanks for the video.
Thanks for the comment! Let me know if you have any questions about data engineering?
Not sure if you deleted your second comment or if TH-cam did. However, I would say all engineering work eventually becomes the same. Data engineering work for the most part is the same project every 2-3 years. But there are always niches and new technologies you can learn and explore. I personally also have a consulting company that helps keep things fresh and new.
@@SeattleDataGuy I don't know why it disappeared. But thanks for the advice you gave.
For people who are interested in what I had asked, it was about consideration on backend developer or data engineering as I wanted variety in my daily work.
@@__thytran Thanks for filling in the question! Any type of specific work that you think you would enjoy as a backend developer?
@@SeattleDataGuy Personally I prefer how I can interact with the whole system and have a view on the overall infrastructure of something and work towards enhancing it. But I also enjoy generating functions and that’s the specific work I think I would enjoy the most as a backend engineer.
Thank you!
-New subscriber
You're welcome!
Hey Ben, I'm working on a Data Pipeline Project ( portfolio project) that begins with web scraping. I'm using Docker, Airflow, Snowflake, AWS, and Google Data Studio. I've been facing challenges trying to run my web scraping process inside a Docker Container, particularly in headless mode. I've spent a week on this without much success. Would it be acceptable for the pipeline to start with the web scraping process running locally to collect the raw data, and then continue the rest of the processing within a Docker Container (specifically, running Airflow in Docker) for data processing and visualization?
Thank you for your video.
You are welcome
Very good video :)
Glad you liked it!
I appreciate this
And I appreciate you!
Thanks for the video. I am currently an intern learning about DevOps (mainly aws). I do not know what I like/enjoy. I am currently looking at data engineer / machine learning engineer. Any suggestions or wise words for someone as inexperienced and lost as I am?
Assuming you're still getting out of college, I would say try to get an other few internships if possible. You don't know what you don't know.
So the only way to really figure out what type of jobs you will enjoy is taking on new jobs/internships. So I would say try to take on as many roles as you can. Luckily tech pays well so you aren't even risking much in terms of trying out different jobs.
Where are you in your college journey?
Thank you :)
No problem 😊
Thanks!
You're too kind! thank you
Do you recommend Designing data driven applications in order to do well for an interview? Or is that overkill/what would you recommend instead to focus?
I think you should focus on SQL, coding, data warehouse design and etl design.
I enjoy programming and solving complex and abstract problems. I don't think I would enjoy a job where I use excel-type tools on a regular basis. Would you recommend data engineering for someone like me? or does it depend on the company I work at?
Hmm, why not just be a software engineer?
Thanks
No, thank you!
Would you mind sharing a reference site where I could understand further SSIS? Since it is not popular now. Appreciate anyone reply. Thanks
Its a little old, but I used wiseowl th-cam.com/video/3cPq9FXk-RA/w-d-xo.html
Thanks for sharing. Same feeling that DE is behind the scenes. I am curious on how to develop analytics mindset which is sth lack of comparing to data scientists. Wondering if you have any clips talking about it?
That would be a fun video. I think it would be a good idea to get a data set, like some of the free ones big query offers and play around with it. Perhaps right an article about it. Like Felipe does here. hoffa.medium.com/400-000-github-repositories-1-billion-files-14-terabytes-of-code-spaces-or-tabs-7cfe0b5dd7fd
It doesn’t matter which tool you use. You end up doing the same thing. Loading a data model.
Do you have a preference of tools?
Amazing content and really detailed information..I'm trying to start my career in DE and would be really thankful if you could share some tips to stand out? I am planning to take the Azure Data Engineer certification to boost my resume..
Thank you so much! That's exciting, I think a certification can help. However, overall, having tangible projects and work you can point back to probably stand out more. Certifications, in my opinion help provide more context to an experienced user. The same way an MBA is probably more valuable to someone who has been working for a while because then you have events in your past you can relate to.
@@SeattleDataGuy Thanks a lot for the context..I will certainly start building some DE projects before committing to a certification..Your content is amazing and thanks again.
@@rohit-pr I am happy to help! Whether it be by content, comment, etc. I appreciate all your feedback and time spent watching my videos. I will keep working to create valuable content!
I am applied scientist but some of my works are related to DE (such as etl) i fell a loop of my job now so i want to change my career path in DE. Do you have any advices/pet project for me to become a fresher/junior DE.
I have an article for this Check this one out->www.theseattledataguy.com/5-data-engineering-projects-to-add-to-your-resume/
Thanks for the video. What are the skills required to become data engineer. How can a data analyst transition to a role of data engineer ?
I will likely make a video discussing the skills a data engineer needs. But overall you will need some programming language like python, you will need to understand ETLs/ELTs, data lakes/data warehouses and data architecture. Most importantly SQL.
That will be a good baseline for you. I also talk about this more in an article about how do i become a data engineer here.
betterprogramming.pub/how-do-i-become-a-data-engineer-42b74c1e6094?source=friends_link&sk=33cff0a41bed23f83fb5242e05dab4f0
I love design databases, buy you didint mencionate in your video, have design o do you see design on your years as a data engenieer?
Amazing video btw
If I didn't cover it, it's still important! A lot of interviews will ask for it too.
Have you reviewed azure data engineer certification?? I am planning to switch my domain to data engineering with dp203 exam certification
I have a whole list of data engineering certificates I would like to get to. Its going to be a while but hopefully soon.
I just started my IT career this year with my CompTIA A+ certification. How much should I prioritize certifications? I already work with SQL every day and would love to expand my knowledge and get some more experience under my belt. Any tips and recommendations anyone would have are greatly appreciated!
I would get some solid python skills under your belt and add in data ware housing and etl work. Have you seen my data roadmap video?
Hi i'm currently in college taking up the course BS in Business AD and i'm starting to get interested to become Data Engineer. Can you tell me what are the process and qualifications to become a Data Engr.
You will need to work on your database and programming skills. In addition you will need to learn about data warehouses, data lakes and data pipelines. I have a video where I break down the skills you will need as well as another one about a data engineering roadmap. Check them out! th-cam.com/video/SpaFPPByOhM/w-d-xo.html
Is an “Enterprise Data-Warehouse (EDW)” the same as an Data-Warehouse?
Truthfully, the difference between an EDW and a DW is honestly semantics. You could...arguably use the term data warehouse even when referring to a EDW and only the strictest of data specialist might complain.
A smaller data warehouse may be specific to a business department or line of business (like a data mart).
In contrast, an EDW is intended to be a single repository for all of an organization’s data.
And I wanted to be really sure about this so I took it from snowflakes definition.
Nice video and very insightful! Btw is that a package from AWS as a community builder?
Thanks for the comment! Oh, I see you are also a man of taste. Yes it is XP.
@@SeattleDataGuy My man!
Keep up the great content 💪 (liked and subbed)!
@@XShollaj I appreciate it! About to do the same for you. Maybe we will need to collab some day.
@@SeattleDataGuy Sure thing, it would be an honor and pleasure! Im doing a Msc right now, and I have little to no time - but as soon as I come back I will keep doing tutorials on AWS and Analytics (by then I believe your channel will explode in views and engagement - data engineering will be the new sexy job in no time)
@@XShollaj Awesome! Well Good luck with finishing your Msc and then I look forward to looping back and making a few videos. It looks like we both do consulting in the BI space. So I am sure we will run into each other again one way or another!
Great content I am also planning to do a AZURE Data Engineer Certification as Well as Data Analyst Certification. My technical expertise is SAP BO, HANA and Tableau any tips is greatly appreciated.
That is awesome. I still need to review the Azure data engineer certification
@@SeattleDataGuy If you listen to the Firebolt data warehouse podcast episodes (also on youtube), you would hear them say that they are focusing at delivering their solution on AWS first for Snowflake users that have some queries that are beyond the scale of Snowflake.
thx
You're welcome!
You nailed the point man most of the people think if you are working as DE you are working on fancy tools like spark,hadoop and all but in reality you work on something else.😅
Yes, I think spark is starting to get implemented more often as I am seeing it in a lot of tech companies and even non-tech companies. But Hadoop seems to have been abstracted away.
Hi, I'm from South America and I don't speak English, I'm using a translator, I'm currently studying computer engineering and I'm interested in learning about data engineering, do you think that by learning English and preparing myself in this area I can work for a US company. Or is it very difficult?
I am sure there is a chance. I am not sure the process to get hired by a US country. However, I have noticed that Upwork data engineers and analysts can actually charge a good amount. So even if you don't get hired directly you can still make really good money.
who is that lady in the wimdy server room at 1:01
Honestly, I am trying to figure out how to keep y'all engaged. I have decided, after watching this video like 3x that randomly splicing some clip in makes no sense. Thanks for joining me on the process of figuring out how to make these videos more compelling!
Many companies are asking for AWs and databricks now , as a junior how I could learn those skills ?
You can get a cert for aws. I wouldn't worry too much about databricks until you need to work on the product
@@SeattleDataGuy thank you so much 😊
Hello, is it possible to contact you via some social media for questions ?
You talked about SQL use, how much Python do you use as a data engineer?
I use a decent amount of python. Of course, I mostly use it as a wrapper for my SQL. SQL tends to manage a lot of the logic and python tends to be more of the orchestrating tool. That being said, there are some reasons you may want to use python to do more of the logic. You also occasionally need to create a custom data connector to APIs. So I usually tell people to have a baseline of python, you should understand classes, functions, connecting to APIs, building your own API with flask, loops, and some data structures and algorithms(especially if you want to pass an interview).
Would a degree in information systems be acceptable for data engineering jobs? My university offers IS with a Data Engineering, web-development, or cybersecurity emphasis but the degree I get would just called Information Systems.
Well, I have an IS degree and I work at FB as a data engineer :). So I think the answer is yes! It will also depend on your experience. Some people go the business analyst route instead.
@@SeattleDataGuy Thank you so much! I want to work in the Seattle area. Do you know if there is a big need for data engineers around there? And would going for a masters degree before working be worth it?
@@parkerbutterfield5967 You shouldn't need a masters degree to be a data engineer. You might be able to find a jr. data engineer position at either a start-up or a larger big tech company(They are always looking for DEs. They also have a decent amount of DE internships. At least FB does). The other option is to work in an analyst position and try to laterally move into a Data engineering position.
Where are you in terms of Sr. Jr. Sophomore status in college?
@@SeattleDataGuy I am currently halfway through my sophomore year. I’ve taken a few intro CS courses as well since I’m still not 100% sure if I’m going for an IS or CS degree. I don’t want to program/code all day but I’ve heard CS degrees are more likely to land me a job. I might go for a minor in one or the other.
@@parkerbutterfield5967 CS degrees probably will be more likely to get you a SWE or data engineering job. I would say, if you can go for a CS degree and intern at a few companies if you can. You can switch into a job with less programming easily if you dislike it with a CS background but it will be harder to go the other way. I have seen plenty of sales engineers, solution architects, project managers, business analysts and so on have CS degrees.
Maybe a front end/back end engineer get paid a lot more than a data engineer in the same company at the same level(L4 for instance)?
It really depends on the company you work at. I was talking to a friend at lyft and they said data engineers get paid just as much as software engineers. Whereas, when I was looking in another video comparing data engineering vs software salaries, it seemed like software engineers made a decent amount more.
@@SeattleDataGuy it is totally true. And I think SWE make more in most companies and the gap is quite large. I wonder do you think a DE can switch to SWE backend? What backend framework/tech stack will be easier for DE to pick up? And how much time do you think DE needs to switch to SWE(assuming working day job as DE and study at night). Thanks a lot.
@@leecharlie2513 I see a lot of people switch into SWE roles after being data engineers. I think it all depends on the type of SWE you want to be.
It sounds like you want to work as a back-end SWE. So as long as you have Data structures and algorithms to pass the interview and some system design, then you're probably good in terms of getting your foot in the door. From there it will all depend on the language of the company you are working for.
Is there a specific language you currently code in?
@@SeattleDataGuy Yes, I am familiar with python. I have also done a little bit javascript as well. But my python is a lot better than javascript. In this case, which backend stack should I learn? I learn python while doing spark and leetcode. I really just want to learn some very high-demand backend stack that many companies will hire for.
Thanks so much.
Data analyst or DBA first before Data eng?
Hmmm, well if you can be a DBA I can see that being more closely aligned with data engineering. Only because you will be working on the infrastructure more and maybe even coding occasionally.
Yo do you think its okay for me to like start a career as a Data engineer? Im actually in a totaly different world right now as a nurse but i want to become one you think thats possible? Cheers!
Well if you got to ask? Nah just kidding. It just depends how long you're willing to wait to get to being a data engineer. I think most people start in some other role like data analyst or software engineer and switch into data engineer. So if you're willing to wait for the time it takes to learn all the different skills to become a data engineer. Then the answer is yes.
What's your plan?
Yea i can endure, i'll probably do it on my spare time like study about it on my non working days. I just came across something our government in philippines is offering thats why i got interested you can check it at sparta.dap.edu.ph.
I think i might start with become an analyts first. Thank you for your answer sir!
I’m a BIE with extensive sql and reporting experience. I’m looking to pivot into DE roles. Any advice on how and where to get started?
do you have experience building data warehouses?
@@SeattleDataGuy No not extensive background building it but mostly using it and creating informal data marts and tables and views useful for analysis.
Would it be a false expectation to hope for a 6 figure salary right after college as a data engineer?
Yes. That goes for literally any field
Yeah, I think in general you will likely start closer to the 60-80k range if you live in a high - mid level cost of living area in the US. Unless you get hired by big tech.
I think in data engineering the work/pay ratio is too high
Do you mean we are paid too little?
Omg Drake from Drake and Josh!!!
You know, in my most recent video I made a reference to me looking like ace venture but drake might be more accurate hahahaha.
Being a Data Scientist-Expectation versus Reality:
Expectation $ $ $
Reality $ $ $ $ $ $ $ $ $
How much you making 😯
Wait so I won’t do coding?
Did I say that? Hahaha, you will likely do coding. But I have interviewed plenty of people with 10 years of experience that did very little coding because they used drag and drop tools.
I always break down data pipelines into custom code, code libraries like airflow and python, and low code options like SSIS and Fivetran.
So in many ways, you could work as a data engineer who rarely codes, but I think you should know some.
@@SeattleDataGuy I think I’ll stick to being a full stack python developer
@@opcon3155 Yeah! do you you build specific types of applications? Or just an all-around full stack python developer?
@@SeattleDataGuy my stack is :
Django REST framework
React
AWS
I know pandas quite well + plotting
A bit of ML
@@opcon3155 Thats a great combo. I pretty much have switched to using Flask and Django for everything...after spending so much time learning spring and ASP.NET.
Are you maybe looking to go into ML then? Do some MLOps?
So just to make sure I understood things,
1) Data engineers gather the data and create ETL pipelines and data lakes
2) Data analysts create analytics & dashboards to explain the data for business decisions
3) Machine learning engineers do some statistics and build machine learning models.
And these are the three pillars that make up Data Science as a whole correct ?
Hmmm, I think data engineers and analysts is about on point. Although we occasionally build dashboards to.
Of course there are probably some other things we take on too depending on the size of the company and how heavy the company is into data.
Machine learning engineers depends. Usually I find that data scientists will often do the research and ML engineers may implement it. Of course, I have also seen ML engineers do the research.
@@SeattleDataGuy Right I've of course over simplified it alot, there is always a lot of overlap of responsibilities on job postings, I even saw a Data Science role on indeed where they were expecting you to do everything (I think it may have been a startup)