If this intro doesn't convince you that Data Engineers are going to be one of the top most needed jobs for the foreseeable future... I don't know what will! Great content, Ben!
@@SeattleDataGuy It's funny you say that, because so many companies hiring DEs these days, put on this act as though their data stack is the holy grail, when in reality they have just evolved from vendor lock in to vendor pathing. It's good to be humble and take a step back to actually realize the on slaught of too many options or ways to do the same thing, it is a good thing (competitive pricing, anti-vendor lock in etc...) but also bad thing (focus, overwhelmed on where to start, endless learning different vendor paths). I think your 3 layers is a great start for the base without getting overwhelmed in vendor marketing BS. Hats off to trying the hold the raging bull by its horns. Looking forward to more material.
Really looking forward to this series. Would love if you would discuss an end to end data Infrastructure design and walk us through the thought process while selecting the tools for each of them with diff use cases. Thank you again for guiding through your videos. They are a big help!☺️
Hello again, brother. I want to brag again. After successfully snagging a DE job inspired by you . Now I obtained the DP-203 Microsoft Certified DE. Thanks again, brother for helping me make the big step a year and a half ago.
Great video, Ben! It is nice to hear this in simple words, so it helps me transmit such ideas to my team and the stakeholders, hopefully in a way the latter will understand.
@@SeattleDataGuy I'm especially interested in all of this as a startup founder. We're just starting out, but already feel the need for a structured approach to our data
That makes sense. I am also kind of going through a similar walk through in my newsletter and it actually has links to the images seattledataguy.substack.com/p/the-baseline-datastack-going-beyond. It might be helpful
Thanks a lot for the video, the series has a lot of potential. May I request if you could talk about cost effective tools? Maybe a sample stack for companies of different sizes or maturity (like the Analysts chart that you showed).
Yeah, I love this. I do find that there is the open source data stack for example. If I hear postgres, metabase and airflow. I know the company is trying to keep costs low or like engineering.
Looking forward to this series! We're implementing a MDS in my org right now and I think it will greatly improve our efficiency and ability to make data-driven decisions.
On a trip this week and I actually have 2 other videos in front of this getting edited...but I will be back next week and filming the next part. You can also check out my newsletter where I am going over similar topics seattledataguy.substack.com/
Hahaha,, the project series I need to restart. However, the next video in this series is completed I will be posting it towards the end of next week (most likely).
It would be great to hear more about the testing part of the process. I feel like this gets talked about a lot for more core software engineering but I think data engineering presents unique challenges that make adapting something akin to Test Driven Development far from straightforward (many external dependencies, rapidly changing state, etc).
Very nice! This series will help me about I've wrote in the previous video! thanks! I don`t know if it is the purpose of series, but if you can show tools for big files, high (distributed and parallel) processing, professional environments I appreciate! I see a lot youtube channels that give just simple (educational) examples but when you try to apply in the pipelines, in hard life (lol), nothing works!!
Yeah, its always hard to actually create situations where you are processing lots of files. Mostly because it starts becoming expensive to store that much data. Overall, its just always easier to show a, here is a hello world vs here is a difficult configuration/scaling issue.
Your videos are always very informative and Valuable for Data Lovers. Can you please tell me what you thing have better future , Jobs Modern Data Stack or Cloud Data Stack.
Something of that nature. I generally mean the most granular form of your production tables. So essentially, what everyone else builds their KPIs, reporting, etc off of.
Given that the Analytics Engineering role is mostly based on the advent of DBT--do you see that role continuing to be viable? Or in other words, is DBT and the like just a fad? or should it be considered in the long-term.
I am interested in this question as well. I have heard that big companies that already have solid data infrastructure, the data engineers are actually analytics engineers. Whereas companies that don't have a solid data infrastructure, really need data engineers.
Hi! I usually just watch your videos and not really comment (LOVE ur contents btw). Not really related to this video I just wanted to know your opinion about Data Engineering Bootcamps when transitioning from DA to DE. Do you consider that an option for the transition? Thanks!
It's hard to say. Any specific bootcamps? They can help. But also, if as a data analyst, you can take data engineering projects. That can also bee a great way of shifting careers.
@@SeattleDataGuy Thanks for the reply Ben! ☺️I found LearningFuze’s Data Science Bootcamp to be appealing since I live in Orange County and could work in person. Im a recent grad from UCSB with math major and I am trying to break into data field as a data analyst and hopefully be a data engineer later in my career. Do you think the bootcamp program would help me break into the field? Thank you for your time and thoughts!! I found your videos to be so inspiring and helpful guiding me through my tough time!!
Would it not be true that a business just starting out (low maturity) could just get sources like Fivetran and a cloud data warehouse managed by an analytics engineer and if needed an analyst?
There are a whole bunch of ways people can set up their data infrastructure depending on budget, head count, priorities, etc. I do plan to talk through some of these in my next video. I was going to do EL. But I think it would be interesting to discuss different stack based on priorities, preferences, etc. The example I often give is the "open source stack". This is usually some combo like Postgres, Airflow, Metabase, re_data or datahub...etc vs the analytics engineer data stack. This tends to be Fivetran, Snowflake and Looker. But there are so many other different versions.
@@SeattleDataGuy Thanks for the reply! If you wouldn't mind one more question? :) Given that the Analytics Engineering role is mostly based on the advent of DBT--do you see that role continuing to be viable? Or in other words, is DBT and the like just a fad? or should it be considered in the long-term. Thanks again!
I think there has always been a natural split for data engineers. Some data engineers tend to be more technical and software focused where others are more analytical. So even if the actual title of analytical engineer goes away..I think the role itself will always exist.
@@SeattleDataGuy Right yeah, I guess a lot of the job descriptions I see are software focused, but I am definitely more analytical. I want to work with the data using technical skills, but I will always prefer to be closer to the data and not so much down the software engineer side.
If this intro doesn't convince you that Data Engineers are going to be one of the top most needed jobs for the foreseeable future... I don't know what will!
Great content, Ben!
It's a wild world right now. So many tools, heres to hoping I can make some sense of it all.
True.
Exactly and this channel is the place to be for learning about this exciting area 😆
@@DataProfessor You're too kind!
@@SeattleDataGuy It's funny you say that, because so many companies hiring DEs these days, put on this act as though their data stack is the holy grail, when in reality they have just evolved from vendor lock in to vendor pathing. It's good to be humble and take a step back to actually realize the on slaught of too many options or ways to do the same thing, it is a good thing (competitive pricing, anti-vendor lock in etc...) but also bad thing (focus, overwhelmed on where to start, endless learning different vendor paths). I think your 3 layers is a great start for the base without getting overwhelmed in vendor marketing BS. Hats off to trying the hold the raging bull by its horns. Looking forward to more material.
Happy to finally find a channel that talks about data 360 degrees. Thank you for your effort to bring this to the audience.
thank you! hopefully you're finding it helpful
Really looking forward to this series.
Would love if you would discuss an end to end data Infrastructure design and walk us through the thought process while selecting the tools for each of them with diff use cases.
Thank you again for guiding through your videos. They are a big help!☺️
Hello again, brother. I want to brag again. After successfully snagging a DE job inspired by you . Now I obtained the DP-203 Microsoft Certified DE. Thanks again, brother for helping me make the big step a year and a half ago.
Congrats. That's so exciting. I am glad you're continuing to grow. How is the new DE job going?
Great video! Can't wait to see where this series goes.
Yeah, so many tools to go over. VCs got to chill
Perfect choice for a series! Looking forward to it
Me too! Thanks for your support.
So useful video! I’m so excited to see what’s next on this series of videos! Thanks for your work
Yup, I am very excited for this series
Another fantastic video. Thanks in advance for all your hard work!
Thank you!
finally!!!! been looking forward to this series to have a more concrete and contextual understanding of data infrastructure.
Yeah, I am very excited! I actually think the next video might be something like "x types of data stacks" and then the EL video. We shall see.
Great info! Thanks for the steady stream of super useful videos!!
I am doing my best to keep a steady stream going
Insightful video. Looking forward to the rest of this series!
Thanks, I look forward to it!
Amazing video! Excited for the next part!
Me too!
Great video, Ben! It is nice to hear this in simple words, so it helps me transmit such ideas to my team and the stakeholders, hopefully in a way the latter will understand.
I am glad you found it helpful! I hope your stakeholders can also understand the value of building a reliable data stack.
This is amazing sir. Thank you ahead for the series!
Definitely help all of us to keep up with the changing landscape.
Watch half the videos I make be out of date 3 weeks after I put them out.😅
Excited for this. Thanks!
Me too! I love this high level stuff.
@@SeattleDataGuy I'm especially interested in all of this as a startup founder. We're just starting out, but already feel the need for a structured approach to our data
That makes sense. I am also kind of going through a similar walk through in my newsletter and it actually has links to the images seattledataguy.substack.com/p/the-baseline-datastack-going-beyond. It might be helpful
@@SeattleDataGuy Sounds interesting, thanks! I’ll give it a read
Thanks a lot for the video, the series has a lot of potential. May I request if you could talk about cost effective tools? Maybe a sample stack for companies of different sizes or maturity (like the Analysts chart that you showed).
Yeah, I love this. I do find that there is the open source data stack for example. If I hear postgres, metabase and airflow. I know the company is trying to keep costs low or like engineering.
Super userful chart in describing the datastack!!
Glad you found it useful!
Looking forward to this series! We're implementing a MDS in my org right now and I think it will greatly improve our efficiency and ability to make data-driven decisions.
That's exciting! What tools are you using?
@@SeattleDataGuy Our main tools are snowflake, fivefran, dbt, and looker!
Solid stack. Where are you guys at thus far?
More Videos like this!!
On a trip this week and I actually have 2 other videos in front of this getting edited...but I will be back next week and filming the next part. You can also check out my newsletter where I am going over similar topics seattledataguy.substack.com/
I Love this.. Ben please consider making videos how you are monitoring the data applications & Best practices to ensure Data Quality.
Yup! I have a few tools I like in that space.
Great video. Looking forward to you finishing this series…and hopefully you finish the DE project series too 🙃
Hahaha,, the project series I need to restart. However, the next video in this series is completed I will be posting it towards the end of next week (most likely).
Thank you very much, very insightful.
glad you found it helpful!
It would be great to hear more about the testing part of the process. I feel like this gets talked about a lot for more core software engineering but I think data engineering presents unique challenges that make adapting something akin to Test Driven Development far from straightforward (many external dependencies, rapidly changing state, etc).
Hmm, I might need to think about this for sure!
Love it!
Thank you!
If you could cover something on data
observability , & how big techs implement data quality Audit Frameworks.
Its on the list
Very nice! This series will help me about I've wrote in the previous video! thanks! I don`t know if it is the purpose of series, but if you can show tools for big files, high (distributed and parallel) processing, professional environments I appreciate! I see a lot youtube channels that give just simple (educational) examples but when you try to apply in the pipelines, in hard life (lol), nothing works!!
Yeah, its always hard to actually create situations where you are processing lots of files. Mostly because it starts becoming expensive to store that much data. Overall, its just always easier to show a, here is a hello world vs here is a difficult configuration/scaling issue.
Your videos are always very informative and Valuable for Data Lovers. Can you please tell me what you thing have better future , Jobs Modern Data Stack or Cloud Data Stack.
I think both will have their place. A lot is shifting currently due to funding. It will take 2-3 years for it all to shake out.
great video! what type of data infrastructure do you recommend to use for image data?
Image data is generally stored in like S3 or a similar solution. Then you store the url in the database with metadata.
Very valuable video. Do you have plans to make part 2 or continuing the series ?
There are parts 2 and 3 th-cam.com/video/lSoEI8jia3Q/w-d-xo.html
Do any of your videos cover managing expectations and timelines as a data engineering consultant? Would love to check that out if it exists!
Not currently, but it might have to go on the list
What kind of data observability tools (ideally open source) would you recommend to monitor the data pipelines?
I should probably make a video on that!
where do services like infra-as-code, configuration management, IAM, SSO live in this system?
Hopefully with a different team...cries in reality
What does "Core Data" mean? Is it a central data system like EDW?
Something of that nature. I generally mean the most granular form of your production tables. So essentially, what everyone else builds their KPIs, reporting, etc off of.
Given that the Analytics Engineering role is mostly based on the advent of DBT--do you see that role continuing to be viable? Or in other words, is DBT and the like just a fad? or should it be considered in the long-term.
I am interested in this question as well. I have heard that big companies that already have solid data infrastructure, the data engineers are actually analytics engineers. Whereas companies that don't have a solid data infrastructure, really need data engineers.
I actually plan to respond to a post I read on the data engineering subreddit on some of this.
Hi! I usually just watch your videos and not really comment (LOVE ur contents btw). Not really related to this video I just wanted to know your opinion about Data Engineering Bootcamps when transitioning from DA to DE. Do you consider that an option for the transition? Thanks!
It's hard to say. Any specific bootcamps? They can help. But also, if as a data analyst, you can take data engineering projects. That can also bee a great way of shifting careers.
@@SeattleDataGuy Thanks for the reply Ben! ☺️I found LearningFuze’s Data Science Bootcamp to be appealing since I live in Orange County and could work in person. Im a recent grad from UCSB with math major and I am trying to break into data field as a data analyst and hopefully be a data engineer later in my career. Do you think the bootcamp program would help me break into the field? Thank you for your time and thoughts!! I found your videos to be so inspiring and helpful guiding me through my tough time!!
Would it not be true that a business just starting out (low maturity) could just get sources like Fivetran and a cloud data warehouse managed by an analytics engineer and if needed an analyst?
There are a whole bunch of ways people can set up their data infrastructure depending on budget, head count, priorities, etc. I do plan to talk through some of these in my next video. I was going to do EL. But I think it would be interesting to discuss different stack based on priorities, preferences, etc. The example I often give is the "open source stack". This is usually some combo like Postgres, Airflow, Metabase, re_data or datahub...etc vs the analytics engineer data stack. This tends to be Fivetran, Snowflake and Looker. But there are so many other different versions.
@@SeattleDataGuy Thanks for the reply! If you wouldn't mind one more question? :)
Given that the Analytics Engineering role is mostly based on the advent of DBT--do you see that role continuing to be viable? Or in other words, is DBT and the like just a fad? or should it be considered in the long-term.
Thanks again!
I think there has always been a natural split for data engineers. Some data engineers tend to be more technical and software focused where others are more analytical. So even if the actual title of analytical engineer goes away..I think the role itself will always exist.
@@SeattleDataGuy Right yeah, I guess a lot of the job descriptions I see are software focused, but I am definitely more analytical. I want to work with the data using technical skills, but I will always prefer to be closer to the data and not so much down the software engineer side.
is that a pink quartz crystal.... its huge!!!!
Do you have any resources for data lineage and governance?
Do you mean like, which tools are solid? Also for data governance, there is the lightsondata channel.
@@SeattleDataGuy Yeah what tools or any sort of books, articles. I'm going to check out lightsondata now
Is the series out yet?
Here is part 2 th-cam.com/video/lSoEI8jia3Q/w-d-xo.html
I haven't seen a single comment about the Naruto reference, so here it is!
Also, along with the new guys, Informatica went IPO with around $10B valuation
Yeah! For real. And I haven't even ever worked with it. There is so much money going into the space.