Simplify ETL pipelines on the Databricks Lakehouse
ฝัง
- เผยแพร่เมื่อ 30 มี.ค. 2023
- In this excerpt from The Best Data Engineering Platform is a Lakehouse, you’ll learn why the lakehouse is the best place to build and run modern data pipelines, and learn Delta Live Tables best practices. Watch the full webinar: dbricks.co/3nCP3Jm
Watch the full webinar to learn how to:
-Leverage lakehouse architecture for a simple, unified approach to data engineering
-Meet your needs for real-time data streaming
-Improve the latency, throughput, accuracy and cost-effectiveness of your data pipelines
-Serve your entire data team with one solution
You can watch the full webinar here: dbricks.co/3nCP3Jm - วิทยาศาสตร์และเทคโนโลยี
Michael, thank you very much for creating Delta Live Tables, which is one of the most innovative features.
Thank You Databricks. ❤
Best presentation! Love databricks team ❤
So does databricks provide ingestion process like ADF. ADF has 90+ data connectors to pull data from data sources
What is the difference between live table vs streaming live table ?
All good but, the one thing I don’t like is limitations. 100 concurrent pipelines updates per workspace is very low. I have many more than that. It’s never go work out with such limit
after all this ads, could apply change be applied more than one time in a single one way data pipeline?
Michael can we parameterize input paths for Delta Live Tables ,it appears Delta Live Tables enforce a static path ? how can DLTs work against data partitioned over a file structure for e.g. yyyy-mm-dd/hh/*.csv , all the examples I have seen in Documentation assume a fixed input path, what if the input path changes over time e.g. hourly
given data partitioned over day hours on storage , can a trigger DLT pipeline path update through parameterization ?
❤
❤