Databricks LakeFlow: A Unified, Intelligent Solution for Data Engineering. Presented by Bilal Aslam

Speaker:
Bilal Aslam, Sr. Director of Product Management, Databricks
Bilal explains that everything starts with good data and outlines the three steps to good data including, ingesting, transforming and orchestrating your data. Then Bilal announces Databricks LakeFlow - a unified solution for data engineering. With LakeFlow you can ingest data from databases, enterprise apps and cloud sources, transform it in batch and real-time streaming, and confidently deploy and operate in production. Includes a live demo of Databricks LakeFlow.
To learn more about Databricks LakeFlow, see the announcement blog post: www.databricks...

[@anandahs6078]

This is exactly what I was looking for long time, tired of learning multiple tools. Thank you databricks

[@rbharath89]

Databricks should have a built in SFTP capability…that writes to some cloud storage or volume…only missing link in Databricks..

[@rhambo5554]

If you use Azure, a workaround is SFTP endpoint on ADLS and then you can run cloudfiles over a storage queue from Databricks.

[@plartoo]

:D Show us how to do more complex data transformations than just a simple join you demo-ed and what the actual limitations are (because that's where the reality meets the demo). While you are at it, tell us how to automate (schedule) this pipeline and set up notifications and data quality checks. Next, let us know how to QA that dashboard you let GenAI created (to make sure it's not hallucinating and spitting out bullshit while destroying our firm's reputation), and how to surface it to customers via URL in a secure way (without paying you through our noses). Finally, tell us how much it costs to process GBs of data per month.
This is the unbearably condescending demo that assumes the attendees are stupid and don't know what entails in serious, real-world data wrangling. And I know a couple of my clients who are leaving Databricks because they are freaking expensive.

[@ser1ification]

Exactly. I’m tired of these hype machines. Everything is in beta. Customers are the beta testers. Only thing these guys did good is the Unity Catalog. Of course Spark and Delta as well.

[@supermodern]

Excellent..

[@kuto1]

this is crazy

[@AleksandarKrumov-pm4tk]

wow

[@Milhouse77BS]

"building a dashboard" I'd rather build a "semantic model".