Hey, really appreciate this video. If I could summarize, it sounds like: - (transactional) databases are generally closer to the data generation source and tend to be closer to operations - data warehouses are further downstream of the transactional databases and have usually gone through some pre-processing to make it more accessible for downstream usage (ie: analytics, machine learning, etc.) - data lakes are kind of a catch all storage method for your data that may require a little more technical knowledge and effort to access
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
Great video to compare the differences among the 3 types and their general use cases; it is very helpful to help me identify which type I'm dealing with on my job. Their definitions have always been debatable because their use cases vary a lot by how companies define them for their projects.
Nice video! One thing that I noticed is none of the content creators (relates to data science) have been talking about technologies like Druid or Clickhouse. Im a telecom engineer and radio access network data is massive, we use Clickhouse to save performance counters and Presto+S3 for taking network configurations snapshots. Teams for other countries use druid, really nice tools not so mentioned here on youtube
I always thought 'database' was just an umbrella term for referring to any storage thing which stores data, whether its a relational, non-relational, object, etc. type database.
data warehouses represent a centralized location for storing data assets from various other sources where the centralization allows data experts to answer business and analytics questions with a 360 view of data that the company has. Often the underlying format of the data is based on the analytical engine of the warehouse chosen. Whether your warehouse is row-based or columnar or just files is decision made by the engine responsible for handling load/insert/query operations. You can have a warehouse that doesn't leverage star schema or snowflake design and still call it a warehouse albeit probably not one that is efficient to analyze.
Very interesting guide... Was stuck on a decision earlier on what approach to take but I guess my uncertainty was a result of the evolving use cases and requirements.... Awesome explanation here💯
@@SeattleDataGuy yes, thx , I'm trying to understand how Knowledge graph/Vector DB's will integrate into this too, is it safe to assume both will be essential pieces of the enterprise ai layer/stack now being invested in heavily, or do you see one being more relevant in next 2-5 yrs?
What is the advantage of snapshots in a data warehouse instead of just saving a copy of the database each period? Also, you can use these separate copies for analytics without interfering with the transaction DB version.
Thank you for this great content. How to reach out if I have other questions? I just got certified data warehouse engineer, so, I'm new to this but I have a good knowledge of the whole concept.
So if you have a lot of document journals that you need to like archived but accessible for read access. Would you recommend a wear house instead of a lake?
9:06 a well designed star schema aka dimensional model is quite easy to add new facts or dimensions. Opposite of rigid, if designed with shared dimensions in mind. See Kimball.
Hey Ben! when you say row oriented data warehouse, it caught my attention and I tried to look it up on google but did not get any satisfactory results. Could you elaborate on this term? what are the use cases these address? Why do they exist in the first place?
Can you tells how you switch from data analyst to data engineering in your 2 years of being a data analyst, what did you expose your self first into, is it going to be mastering python and SQL then etl? Thank you
THanks for the comment. By chance have you watched my video about this topic? th-cam.com/video/lGzh-QendJc/w-d-xo.html If this wasn't helpful happy to answer more questions
If your company needs to process transactions quickly, but you also need reporting, then wouldn’t you have BOTH OLAP and OLTP databases?? Instead of picking one or the other??
operational systems (crm, erp) are usually transactional databases. to make reporting more efficient and not bringing down the operational system with reporting a data warehouse is usually created. the data warehouse could be a olap database but i have seen oltp databases in use for that too
![[LIVE] BABYMONSTER(베이비몬스터) - DRIP | 두시탈출 컬투쇼](http://i.ytimg.com/vi/UG05pxLGwzc/mqdefault.jpg)
Hey, really appreciate this video. If I could summarize, it sounds like:
- (transactional) databases are generally closer to the data generation source and tend to be closer to operations
- data warehouses are further downstream of the transactional databases and have usually gone through some pre-processing to make it more accessible for downstream usage (ie: analytics, machine learning, etc.)
- data lakes are kind of a catch all storage method for your data that may require a little more technical knowledge and effort to access
Glad you liked it, hopefully people find your summary helpful
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
7:05 might be time to mention Dr. Ralph Kimball’s contributions to dimensional data warehouse design.
Great video to compare the differences among the 3 types and their general use cases; it is very helpful to help me identify which type I'm dealing with on my job. Their definitions have always been debatable because their use cases vary a lot by how companies define them for their projects.
Awesome video. I am prepping for an interview for my dream job and this helped me so much. Thank you!
glad it helped!
Nice video! One thing that I noticed is none of the content creators (relates to data science) have been talking about technologies like Druid or Clickhouse.
Im a telecom engineer and radio access network data is massive, we use Clickhouse to save performance counters and Presto+S3 for taking network configurations snapshots. Teams for other countries use druid, really nice tools not so mentioned here on youtube
Boy I love the way you say Seattle data guy
oh hahaah
I always thought 'database' was just an umbrella term for referring to any storage thing which stores data, whether its a relational, non-relational, object, etc. type database.
I should have said transactional databases
data warehouses represent a centralized location for storing data assets from various other sources where the centralization allows data experts to answer business and analytics questions with a 360 view of data that the company has. Often the underlying format of the data is based on the analytical engine of the warehouse chosen. Whether your warehouse is row-based or columnar or just files is decision made by the engine responsible for handling load/insert/query operations. You can have a warehouse that doesn't leverage star schema or snowflake design and still call it a warehouse albeit probably not one that is efficient to analyze.
Nice video, might be useful to show examples of each at the end.
Very interesting guide... Was stuck on a decision earlier on what approach to take but I guess my uncertainty was a result of the evolving use cases and requirements.... Awesome explanation here💯
Glad it was helpful!
You made it super easy, thanks heaps!
You're welcome!
thx for breaking it down
glad you found it helper!
@@SeattleDataGuy yes, thx , I'm trying to understand how Knowledge graph/Vector DB's will integrate into this too, is it safe to assume both will be essential pieces of the enterprise ai layer/stack now being invested in heavily, or do you see one being more relevant in next 2-5 yrs?
what's your opinion on Databricks?
In terms of data lakehouses?
@@SeattleDataGuy not op but that would be great!
What is the advantage of snapshots in a data warehouse instead of just saving a copy of the database each period?
Also, you can use these separate copies for analytics without interfering with the transaction DB version.
Thank you for this great content.
How to reach out if I have other questions?
I just got certified data warehouse engineer, so, I'm new to this but I have a good knowledge of the whole concept.
So if you have a lot of document journals that you need to like archived but accessible for read access. Would you recommend a wear house instead of a lake?
Great review.
Glad you found it helpful!
9:06 a well designed star schema aka dimensional model is quite easy to add new facts or dimensions. Opposite of rigid, if designed with shared dimensions in mind. See Kimball.
i would appreciate it if u talk in much slower rate to be able to catch these valuable information, I tried to put the video sppeed on 0.75
Hey Ben! when you say row oriented data warehouse, it caught my attention and I tried to look it up on google but did not get any satisfactory results. Could you elaborate on this term? what are the use cases these address? Why do they exist in the first place?
At my company they treat data stores as the new shiny mirror. Nobody really knows what are the limits and the use cases for the different options
Its always the next shiny thing
Can you tells how you switch from data analyst to data engineering in your 2 years of being a data analyst, what did you expose your self first into, is it going to be mastering python and SQL then etl?
Thank you
THanks for the comment. By chance have you watched my video about this topic? th-cam.com/video/lGzh-QendJc/w-d-xo.html If this wasn't helpful happy to answer more questions
Thanks, can you become a Data Warehouse engineer without learning programming? I just want to learn SQL
Great video.
Glad you liked it!
Amamzing thank u
you're welcome!
Great and informative video, what about datalakehouses? Thanks!!
Great suggestion! I should do that one next
Listening to the data lake explanation all i could think about was the old AS400 XD
If your company needs to process transactions quickly, but you also need reporting, then wouldn’t you have BOTH OLAP and OLTP databases?? Instead of picking one or the other??
operational systems (crm, erp) are usually transactional databases. to make reporting more efficient and not bringing down the operational system with reporting a data warehouse is usually created. the data warehouse could be a olap database but i have seen oltp databases in use for that too
@12:00 data lakes
Thanks for the timestamp
Hey Gary, is data engineer for introvert people, do u have to communicate a lot to stakeholders ?
Who is Gary o.O. I am Ben :). Yes you do still have to do that from time to time
the data warehouses i worked with were all not columnar
yeah I have seen a lot of SQL servers used for DWs
#data #$$$ #analytics
#$$$ data
What is this code😶
It’s like you’re speaking a different language 😅 I have no idea about whatever this is.
what would help it be clearer? Let me know!
Wtf is this?
What a shit definition of "warehouse" by Bill Inmon!