The Harsh Reality of Being a Data Engineer
ฝัง
- เผยแพร่เมื่อ 13 มิ.ย. 2024
- Download Python ebook (free) 👉🏼 clickhubspot.com/bbd
In this video, we are discussing the harsh reality of being a data engineer, the good, the bad and the ugly. Wait till the end of the video and share your thoughts in comments. Anything surprising to you?
Learn more about AWS Startups GenAI Day - Watch the GenAI Webinar on-demand 👉🏼 rmbrnd.com/AWS-GenAI
Virtual product placement by Rembrand 👉🏼 www.rembrand.com
WATCH THESE VIDEOS NEXT
My Self-Taught Data Science Journey: • My Data Science Journe...
How I Would Learn Data Science in 2023: • How I Would Learn Data...
Harsh Reality of Being a Data Scientist? • The Harsh Reality of B...
Try AI for coding & data analysis:
🏅 Julius AI for data analysis 👉🏼 julius.ai/?via=Sundas
~~~ CERTIFICATES & COURSES ~~~
Data Scientist
📊 Data Science Certificate 👉🏼 imp.i384100.net/DataScienceCert
🏅 Data Science Bootcamp 👉🏼 bit.ly/data-scientist-bootcamp
🌍 Data Scientist Bootcamp 👉🏼 bit.ly/data-scientist-bootcamp
Data Analyst
🏅 Google Data Analytics Certificate 👉🏼 imp.i384100.net/GoogleDataCert
🏅🏅 Google Advanced Data Analytics Certificate 👉🏼 imp.i384100.net/advanced-data...
⚡ IBM Data Analytics Certificate 👉🏼 imp.i384100.net/ibm-data-analyst
📊 Microsoft BI Data Analyst Certificate 👉🏼 imp.i384100.net/msft-dataanalyst
📑 Cyber Security Analyst 👉🏼 imp.i384100.net/cybersecurity
🐍 Meta Marketing Analytics Certificate 👉🏼 imp.i384100.net/meta-analytics
Data Engineer
📊 Meta Data Engineer Certificate 👉🏼 imp.i384100.net/meta-dataengineer
⚡ IBM Data Engineer 👉🏼 imp.i384100.net/data-engineer
Python
🐍 Python for Everybody 👉🏼 imp.i384100.net/PythonSpec
⚡ Python for Data Science 👉🏼 bit.ly/python-data-science-ml
📑 Python for Data Visualization 👉🏼 bit.ly/python-data-viz
🏅 Python Pandas 👉🏼 bit.ly/python-pandas-data-ana...
SQL
⚡ SQL Basics for Data Science 👉🏼 imp.i384100.net/SQLforDS
🏅 SQL Bootcamp 👉🏼 bit.ly/sql-bootcamps
Machine Learning
🏅 Machine Learning Bootcamp 👉🏼 bit.ly/machine-learning-atoz
⚙️ Machine Learning Specialization 👉🏼 imp.i384100.net/ML
⚡ Deep Learning 👉🏼 imp.i384100.net/deeplearning
⚙️ AI Engineer Certificate imp.i384100.net/AI
Statistics
⚙️ Statistics for Data Analysis 👉🏼 bit.ly/statistics-data-scienc
📑 Intro to Statistics 👉🏼 imp.i384100.net/Stats
EXCEL
🎨 Excel for Data Analysis 👉🏼 imp.i384100.net/Excel
✍️ IBM Data Analysis in Excel & R Certificate 👉🏼 bit.ly/data-scientist-bootcamp
🌍 ALL COURSES 👉🏼 kit.co/SundasKhalid
🏅 Practice Coding Interviews 👉🏼 www.stratascratch.com/?via=su...
~~~~~~~~~~~~~~~~~~~~~~~~~~~
MY FAVORITE TOOLS
📚 Read Books FAST on Shortform 👉🏼 www.shortform.com/sundas
✍️ Coursera Plus 7 day free trial 👉🏼 imp.i384100.net/CourseraPlus
🏅 Practice Coding Interviews 👉🏼 www.stratascratch.com/?via=su...
🏅 Julius AI for data analysis 👉🏼 julius.ai/?via=Sundas
✍️ Get 20% HubSpot Discount 👉🏼 hubspot.sjv.io/c/3311887/1863...
SUBSCRIBE FOR MORE VIDEOS
th-cam.com/users/sundaskhalid?...
BINGE THESE PLAYLISTS NEXT
Starting in Data Science: • Starting in Data Science
Learn SQL: • Learn SQL
The Tech Lounge: • The Tech Lounge
ABOUT ME
I'm Sundas. I'm a self-taught data scientist from a non-tech background, currently at a FAANG company. I have been in the industry for +10 years spread across two big tech companies. On this channel, I share tips for people interested in entering data science with the goal to democratize knowledge and make complicated topics digestible for everyone. Learn more at sundaskhalid.com All opinions are mine!
PS. Some of the links included are affiliate links and help me keep this channel going. Thanks for the support.
Business inquiries: info@careerpy.com - วิทยาศาสตร์และเทคโนโลยี
Thanks for watching! You can download the Python ebook (free) 👉🏼 clickhubspot.com/bbd
Well, everything new requires you to figure out your path. I studied to be a mathematician but needed to find a job. Salaries in programming supported a more affluent lifestyle, so just a few classes in CS helped get my foot in the door. I think that the guy who wrote a violent race car game had a history Ph. D. I was reading about a philosopher who worked as a glass lens grinder, who died from cumulatively inhaling glass fibers in his 30s. Like everything else, you might need to change careers or job titles. Being a Data Engineer and knowing programming will get you further.
Im a data engineer with 4 years experience. Harsh reality of being a data engineer is:
- 5% building new pipelines
- 15% automating the pipelines
- 20% adding data quality checks to the pipelines
And 98% of the time fixing bugs in OLD pipelines.
For people who like building new products, this is not the right role. Because most of the time you're only firefighting. And since you work in backend, you don't get much recognition.
Exactly agreed
Thank you for putting this out, appreciate it! 🙌🏼
The roles of a software engineer and data engineer are totally different, even though they might use related tech stacks. Also a data scientist can't take the role of a data engineer because, apart from being technical, a data engineer is expected to communicate with stakeholders and build solutions that turn data into insigts/revenue; data scientists suck at this. So a good data engineer should be able to do the work of a data analysts effectively. According to a Gartner report, 80% of data science projects fail. The number reason is: Lack of understanding of business context and user needs.
There are two types of data engineers: Software Engineers who are also data engineers and ETL engineers. If you are an ETL engineer (building pipelines using python and SQL) then you are just considered a glorified data analyst and your pay always be less than SE. My advice for them is to take the next step and become a cloud engineer (snowflake, Azure, AWS) and see your pay increase significantly.
Sir, are you advising DE with ETL engineer to take cloud engineer course or take Software Engineer bootcamp? Which one is better?
Insightfull
Hello sir. I don't want to code , programm like dev team. But I can work in cloud. Where should I go. DE or devops one. Please help
Yes! There are technical DE and BI DE.
Two different types of DE. Totally different culture and minset.
The real question is how to get the education without wracking up stupid provider bills
Hi Sundas, I am a bit confused. You are initially saying that SEs can do DEs role and also, with advancements in AI and possible taking over core SE's role, even they would do DE's role. Then, at the end, you are saying that DE's role will still be in demand?
Not everybody agrees with this (especially juniors) but Data Engineering is a subset of Software Engineering.
There are two types of DE:
- software oriented DE
- BI oriented DE
DE is SWE. Be very good in software engineering with a focus on Big Data. You will able to switch roles easily.
just got an internship as a data engineer(while at college) for the city of ottawa(only 1 spot btw)...wont have done it without you. Appreciate you..
@max1232 brother I'm doing data engineering course. Is there any internship for Data Engineering from the same place where you're working
I have been working as a Data Engineer for 3 years and I think it depends on companies and how they define their data engineers. I always find myself building data pipelines, API design, Web applications, Data Encryption and Database Security
Do you code ? Or program from scratch?
Hey, where do u work?
Hi I'm a high school senior right now and I want to pursue a career in data science job family..... specifically AIML.....it would be wonderful if you could do a video on AIMl because I feel like there isn't much on the internet about the same
i was confused about that thank you i will focus on data science then i will convert to data engineer because i don't see a particular road map for data engineer
stressing on the point of a software engineer being able to do what a data engineer does is absolutely correct. I work as a data analyst and in one of my previous roles, the software engineer was pretty much the data engineer completing all the data engineering task.
Data science or data enginering which is better based on opportunites as a freshers?
Ma'am I m confused between data engineer or data scientist, which one should I choose , kindly suggest me
Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!
thank you very much 👍👏
Hi Sundas, i m writing this comment to share with you that i got fresher job as a Data Engineer in one of the startups. I believe if u have the knowledge of tech stack required for the role, u can get hired. In btw, i have worked as data science intern for 3 months before(as it is considered hot topic in the field of tech and i went for it initially but coudnt find good pay jobs for it).
Great review, but i cant agree career growth. From Data Engineer you become a solutions architect. Or bridge being a DE to the next sensible path.
Nice, I came to know that data engineers paid lesser than the average software engineer
Felt sad 😔
Just started Data Analysist course few weeks ago.
How is it going so far?
Im agree with all except number 2 software engineer can make easily DE work?, i think no, many DE were DBAs in the past. DBA is complete different skill to software engineering. 3. i think there is growing for DE you can skill up into machine learning engineer more easily than other roles.
AWS beta certification for data engineer , Please talk about it.
Yes I also want to know about this
seems you went with your default thumbnail :D
sadly yes 😭
Sundas name is very unique for me 💕
Through the whole video I was really concerned she will accidentally cut me with her fingers...............
I’m a Database Administrator/ Network Administrator/ Full Stack Developer making 60k…. I would love 155k damn!
Even I make less than a developer, it's still a helluva lot better than my customer service job!
Lol my gosh there are so many harsh realities why would anyone wanna pursue this.
Please make sundas khaild Urdu/hindi channel for south asia audience specially Pakistani audience
You've no idea how fabulous you look!! 😇🫠
Ma'am I m confused between data engineer or data scientist, which one should I choose , kindly suggest me
Do your research.