I know you mentioned that this has been a "thing" for quite some time. But, it also has had a name. It was always called harmonization in the past. After data has been merged, cleaned, and enhanced it was common to harmonize it back to the source. Beside giving the source access to this enhanced data it would also allow for more efficient ETL as it would not need to be transformed over and over again. A lot of times these harmonization engines were tied to master data management and data stewardship. So, it was often a manual process to approve the harmonization. But, the process itself was automatic - once approved. I know that I am old but it is a bit shocking how much of modern data engineering is just reinventing the wheel.
What has changed over the past 5-10 years is the analysts having more control over their data stacks and being less reliant on data engineering to manage ETLs. With more modern tooling, the implementation of technology gets closer to the user, and will limit the investments needed to bring data closer to user actions with more impact.
I was so confused when I first heard this term, only to discover I'd been doing this with various products for decades 🤣🤣🤣 honestly this is quite simply marketing genius, its a way for newer smaller products to "ok boomer" the mature products like Informatica that have done this for decades. A lot of the sales teams will not know what this term means and hence will get caught out in front of a customer when they ask "can you do reverse etl?". Its a bit like DBT used to push this story they were a no code when in fact dbt is all about code!
Hi Ben, one question, do I want to specialize in a cloud. Are there AWS, GCP, Azure... which one do you recommend to specialize as a Data Engineer? Which one has better demand, tools, also in which one do you recommend me to get certified as a beginner.
Ok, so would an example of an R-ETL tool be something that takes in raw data, processes data with BI logic transformations, and then output something like a chart? so all you would see in the UI is basically you drop in data, and you can click a button that says "generate chart" ?
Not so much a chart. Thats more standard etl to data warehouse to reporting layer to dashboard. Instead, its more like, from data warehouse back to salesforce and add in something that makes it clear to target customer.
Great video as always. Noticed you sounded just every so slightly less energetic than usual, wondering if you were tired or just had a long day the same day of recording?
@@SeattleDataGuy I really admire your width to cover so much in such diverse DATA-STACK environment. With that huge combinations of choices for different scenarios, I can see why you change to be a "Full Stack" DATA SOLUTION CONSULTANT. There is definitely a huge need from a spectrum of different organizations in different size, legacy background & their position in their data-legacy journey to be a fully data-driven organization.
Very, very informative. Also love these types of videos Ben.
Glad you enjoyed it!
I know you mentioned that this has been a "thing" for quite some time. But, it also has had a name. It was always called harmonization in the past. After data has been merged, cleaned, and enhanced it was common to harmonize it back to the source. Beside giving the source access to this enhanced data it would also allow for more efficient ETL as it would not need to be transformed over and over again. A lot of times these harmonization engines were tied to master data management and data stewardship. So, it was often a manual process to approve the harmonization. But, the process itself was automatic - once approved. I know that I am old but it is a bit shocking how much of modern data engineering is just reinventing the wheel.
Thanks a lot for this comments. As a Junior Data Engineer I found it quite useful to hear about "Harmonization".
Thanks, very beneficial 👍
Glad it was helpful!
What has changed over the past 5-10 years is the analysts having more control over their data stacks and being less reliant on data engineering to manage ETLs. With more modern tooling, the implementation of technology gets closer to the user, and will limit the investments needed to bring data closer to user actions with more impact.
I was so confused when I first heard this term, only to discover I'd been doing this with various products for decades 🤣🤣🤣 honestly this is quite simply marketing genius, its a way for newer smaller products to "ok boomer" the mature products like Informatica that have done this for decades. A lot of the sales teams will not know what this term means and hence will get caught out in front of a customer when they ask "can you do reverse etl?". Its a bit like DBT used to push this story they were a no code when in fact dbt is all about code!
Hi Ben, one question, do I want to specialize in a cloud. Are there AWS, GCP, Azure... which one do you recommend to specialize as a Data Engineer?
Which one has better demand, tools, also in which one do you recommend me to get certified as a beginner.
AWS and Azure have the largest market share. I would say just pick one of the big three and you should be good
Ok, so would an example of an R-ETL tool be something that takes in raw data, processes data with BI logic transformations, and then output something like a chart? so all you would see in the UI is basically you drop in data, and you can click a button that says "generate chart" ?
Not so much a chart. Thats more standard etl to data warehouse to reporting layer to dashboard. Instead, its more like, from data warehouse back to salesforce and add in something that makes it clear to target customer.
@@SeattleDataGuy any ETL tool could work in both directions
Great video as always. Noticed you sounded just every so slightly less energetic than usual, wondering if you were tired or just had a long day the same day of recording?
Maybe a little tired. I can't fully recall. I did film it the same day I did looker vs tableau
Instead of calling it “Reverse ETL” , WE should call it CRYSTALLIZATION
This just makes me think of crystal reports...
@@SeattleDataGuy I really admire your width to cover so much in such diverse DATA-STACK environment.
With that huge combinations of choices for different scenarios, I can see why you change to be a "Full Stack" DATA SOLUTION CONSULTANT. There is definitely a huge need from a spectrum of different organizations in different size, legacy background & their position in their data-legacy journey to be a fully data-driven organization.
Next time please don't use an analogy from a movie or TV series. Not all of us watch those
You must be fun at parties