It would have been better if demo included how to move data pipelines between different environments like dev/qa/production...most of the videos are just focused on showing sample product demo but in real life ...there are lot of things to consider like CI/CD between environments, managing data pipelines, set up different environments, project structure and folder structure etc.
agree. so many data talks are essentially Hello, Word! examples which don't fully display the actual problems solved. And 18 minutes is too short for anything more than that, sadly.
A nice tools to start with, but there are lot of modifications that needs to be done 1. changing of date column to string data type at the destination end. 2. Handling of updated column 3. Naming each connections
In reality, over the last 20 years, ETL wasn't really ETL anymore. This is because many data & BI team were extracting their data into a transactional ODS first, and subsequently agregating & transforming that into datamarts. So in effect, many 'traditional' data pipelines in the last 20 years or so, were already ELT, whereby the biggest chunk of the transformations were performed downstream from the ODS, not upstream to that (in situations where the ODS was a physical representation of the source - which is the easiest method to implement). Granted, with cloud based datawarehouses, and the need to integrate on-premise data with cloud based SAAS application data, decoupling the extractions from the transformations, has become inevitable due to the explosion of data and the fact it is often based in different data centres and sometimes regions. However pragmatic design practices which followed an ODS-first designed pattern, were often already advocating that.
hello, I try to replicate the example by done, but I can't. I use S3 as a datasource and S3 as a destination, but it only lets me replicate the data and not transform and normalize the same data. Can you tell me the reason for that? thank you
How is extracting data into a bunch of JSON blob objects actually helping anyone? Especially when you have to "normalise" it on top of that to be able to use more SQL native reporting tools? Or even transform it further with DBT? I am not seeing any "paradigm" shift here, compared to traditional ETL tools, and the use of a "staging" area in most datawarehouses, when the raw extracted data is stored, prior to normalisation into an ODS layer. Not saying the Airbyte technology doesn't have its merit - it is open source and looks very cloud-aware. Just pointing out that the so-called paradigm shift between ELT and ETL, doesn't stand under scrutiny. Most datawarehouses over the last 20 years were actually already built using the same ELT design pattern that is described here, and most were built with....ETL tools. The tool doesn't force you to follow a certain pattern, you can do ELT with....an ETL tool. "ETL" is just an acryonym, and a label -> those traditional ETL tool are capable of anything really (which is often their pitfall - too many ways to skin the data-cat).
You really don’t want us to use custom transformations on AirByte, you put DBT to video’s title, you put it into the slide as a seperate page but you just use one little sentence about it in whole video. There’s nothing about what did you transform? How did you transform? Interesting. Btw, the video might be 2 years old but I have feelings quite new.
10:25 any chance we can get a link to that github repo? I'd like to test out this stack with DBT. Thanks in advance ;)
It would have been better if demo included how to move data pipelines between different environments like dev/qa/production...most of the videos are just focused on showing sample product demo but in real life ...there are lot of things to consider like CI/CD between environments, managing data pipelines, set up different environments, project structure and folder structure etc.
agree. so many data talks are essentially Hello, Word! examples which don't fully display the actual problems solved. And 18 minutes is too short for anything more than that, sadly.
Very cool! Can't wait to try out on a project
A nice tools to start with, but there are lot of modifications that needs to be done
1. changing of date column to string data type at the destination end.
2. Handling of updated column
3. Naming each connections
Loving this already!!
Link to github for airbyte?
the "behind the scenes" Airflow configuration would have been the most interesting part from the talk. But thanks for sharing..
Bravo Michel
In reality, over the last 20 years, ETL wasn't really ETL anymore.
This is because many data & BI team were extracting their data into a transactional ODS first, and subsequently agregating & transforming that into datamarts.
So in effect, many 'traditional' data pipelines in the last 20 years or so, were already ELT, whereby the biggest chunk of the transformations were performed downstream from the ODS, not upstream to that (in situations where the ODS was a physical representation of the source - which is the easiest method to implement).
Granted, with cloud based datawarehouses, and the need to integrate on-premise data with cloud based SAAS application data, decoupling the extractions from the transformations, has become inevitable due to the explosion of data and the fact it is often based in different data centres and sometimes regions.
However pragmatic design practices which followed an ODS-first designed pattern, were often already advocating that.
Why do we need Airflow in your demo ? when scheduling can be done airbyte itself ?
hello, I try to replicate the example by done, but I can't.
I use S3 as a datasource and S3 as a destination, but it only lets me replicate the data and not transform and normalize the same data.
Can you tell me the reason for that?
thank you
How is extracting data into a bunch of JSON blob objects actually helping anyone? Especially when you have to "normalise" it on top of that to be able to use more SQL native reporting tools? Or even transform it further with DBT?
I am not seeing any "paradigm" shift here, compared to traditional ETL tools, and the use of a "staging" area in most datawarehouses, when the raw extracted data is stored, prior to normalisation into an ODS layer.
Not saying the Airbyte technology doesn't have its merit - it is open source and looks very cloud-aware.
Just pointing out that the so-called paradigm shift between ELT and ETL, doesn't stand under scrutiny.
Most datawarehouses over the last 20 years were actually already built using the same ELT design pattern that is described here, and most were built with....ETL tools.
The tool doesn't force you to follow a certain pattern, you can do ELT with....an ETL tool.
"ETL" is just an acryonym, and a label -> those traditional ETL tool are capable of anything really (which is often their pitfall - too many ways to skin the data-cat).
You really don’t want us to use custom transformations on AirByte, you put DBT to video’s title, you put it into the slide as a seperate page but you just use one little sentence about it in whole video.
There’s nothing about what did you transform? How did you transform? Interesting.
Btw, the video might be 2 years old but I have feelings quite new.