Hello guys. Thank you for this blackboard session. I have a suggestion: Theory is nice and it is the first step to get into the specific IT topic. But as they say: seeing is believing... could you please make a video with practical use case, how data observability practices are being utilized to solve data pipeline issues? In other words: How data observability looks and works in real life on real tools. That would be awesome to see. And if you could merge this topic with Data Governance that would be totally awesome! I am loving these blackboard explanations! Thank you very much!
What I understand from this video is that data observability is just a grouping of "data quality", "data lineage", "ETL Logging" etc. into one umbrella. These are common concepts within data warehousing/engineering. Some teams solve them with frameworks/vendor tools, some through custom development and some use both approaches. Data observability is just a new name for it. good presentation though.
Great illustration to the concept of Data Observability - thanks for making it simple enough to understand. PS... I'm very impressed with your "backwards writing skills"!!!!
Absolutely great overview. Appreciate the train analogy. I got a good understanding of the premise of Data Observability, plus the bonus of a concise understanding of data engineering and pipelines. Need this gentleman explaining more concepts!
Hello, thanks for this explanation. I would like to see more videos on NLP domain. To be more specific using pre-trained tranformers for text analytics ( Transfer Learning )
Thanks much for the really cool explanation. Wondering whether there is any tool available that provides the capability to really ‘Observe’ any changes/incident in the data pipeline?
Understood in theory..Thanks! Several years ago, one issue we used to run into was some of our Enterprise customers were missing sending data on defined schedule into our SaaS env. Our ETL data ingestion job (sort of data pipeline tool) won't help since it won't be kicked off. So we wrote something outside of the ETL job to ensure we can catch such data misses by the customer...could/should this be part of data observability?
I think it would have been a better presentation if he had stuck to concrete examples more; like with the betting company. The train was cute, and I did get it, but he explained something abstract by making it even more abstract. Heading in the other direction and injecting some hard reality would have been more instructive. At the end of the day, this is not an academic exercise, it's there to solve real world problems.
I think that Data Observability is more of a technical implementation to identify/remediate issues in the data pipeline that is managed by Data Engineers and others potentially involved in the process due to cross-functional responsibilities. Whereas DataOps is a methodology to foster a culture of effective communication and collaboration between data stewards, data users, and executives/business process owners. So, Data Observability helps to ensures quality data so that DataOps can glean business value from data presented. In essence, Data Observability integrally folds into DataOps.
SoooOoooooo we've been moving data for years with ETL tools. Tools that generally don't require a super coder. Some of these tools have some level of monitoring built in, and some have data quality modules. BUT NOW..... let's go back to hand-writing code to move data. Hhhhhhmmm seems like a bad idea.
Hello guys. Thank you for this blackboard session. I have a suggestion: Theory is nice and it is the first step to get into the specific IT topic. But as they say: seeing is believing... could you please make a video with practical use case, how data observability practices are being utilized to solve data pipeline issues? In other words: How data observability looks and works in real life on real tools. That would be awesome to see. And if you could merge this topic with Data Governance that would be totally awesome! I am loving these blackboard explanations! Thank you very much!
What I understand from this video is that data observability is just a grouping of "data quality", "data lineage", "ETL Logging" etc. into one umbrella. These are common concepts within data warehousing/engineering. Some teams solve them with frameworks/vendor tools, some through custom development and some use both approaches. Data observability is just a new name for it.
good presentation though.
Thanks for breaking this down clearly. It makes it easy my product designer brain to understand the concepts
Very insightful, learned a very useful concept on a high level. Keep bringing such videos!
Great illustration to the concept of Data Observability - thanks for making it simple enough to understand.
PS... I'm very impressed with your "backwards writing skills"!!!!
Really great overview. Thank you
Absolutely great overview. Appreciate the train analogy. I got a good understanding of the premise of Data Observability, plus the bonus of a concise understanding of data engineering and pipelines. Need this gentleman explaining more concepts!
Thank you. Great session. Could I ask what technology you use to put this together?
I really like this idea to shift thinking paradigm from been software engineer to data engineer:)
Hello, thanks for this explanation. I would like to see more videos on NLP domain. To be more specific using pre-trained tranformers for text analytics ( Transfer Learning )
Great explanation, nice one
Thanks much for the really cool explanation. Wondering whether there is any tool available that provides the capability to really ‘Observe’ any changes/incident in the data pipeline?
Great job!
Great overview and love the train analogy
Understood in theory..Thanks! Several years ago, one issue we used to run into was some of our Enterprise customers were missing sending data on defined schedule into our SaaS env. Our ETL data ingestion job (sort of data pipeline tool) won't help since it won't be kicked off. So we wrote something outside of the ETL job to ensure we can catch such data misses by the customer...could/should this be part of data observability?
Good!!
I think it would have been a better presentation if he had stuck to concrete examples more; like with the betting company.
The train was cute, and I did get it, but he explained something abstract by making it even more abstract. Heading in the other direction and injecting some hard reality would have been more instructive.
At the end of the day, this is not an academic exercise, it's there to solve real world problems.
Is Data Observability same as DataOps ?
I think that Data Observability is more of a technical implementation to identify/remediate issues in the data pipeline that is managed by Data Engineers and others potentially involved in the process due to cross-functional responsibilities. Whereas DataOps is a methodology to foster a culture of effective communication and collaboration between data stewards, data users, and executives/business process owners. So, Data Observability helps to ensures quality data so that DataOps can glean business value from data presented. In essence, Data Observability integrally folds into DataOps.
SoooOoooooo we've been moving data for years with ETL tools. Tools that generally don't require a super coder. Some of these tools have some level of monitoring built in, and some have data quality modules. BUT NOW..... let's go back to hand-writing code to move data. Hhhhhhmmm seems like a bad idea.