Real Time Streaming with Azure Databricks and Event Hubs
ฝัง
- เผยแพร่เมื่อ 8 ก.ค. 2024
- 🚀Join me in this tutorial as we build a real-time analytics streaming solution using Azure Databricks and Event Hubs.
⌚Timestamps:
Project Overview: 00:00:00
Solution Architecture: 00:00:44
Event Hubs Overview and Implementation: 00:04:11
Databricks Cluster and Library Set Up: 00:12:32
Importing the Project Notebook: 00:16:35
Initiating the Stream and Processing the Bronze Layer: 00:18:20
Processing the Silver Layer: 00:33:08
Processing the Gold Layer: 00:43:51
Near Real-Time Power BI Report: 00:54:43
Project Debrief: 00:59:41
🔥Project Overview:
We'll start by creating real time data in Azure Event Hubs and then store and process the data in our Databricks Lakehouse, implementing the Bronze, Silver, and Gold layers of the Medallion Architecture. The final step involves creating a near real-time report on Power BI, showcasing the power of streaming analytics.
Links and Resources
🔗 GitHub: github.com/malvik01/Real-Time...
🔗 Spark Structured Streaming API: spark.apache.org/docs/latest/...
🔗 Watermarking: www.databricks.com/blog/featu...
🔗 Check Out My Udemy Courses: www.pathfinder-analytics.net/...
📺 Don't forget to hit the like button, subscribe, and turn on notifications to stay updated with our latest tutorials and tech insights!
📬 For more information, queries, or feedback, feel free to drop a comment below.
🔖 Tags
azure data engineering, end-to-end azure data engineering project, azure data engineering project free
Hashtags
#azure #dataengineering #databricks #DataEngineeringProject #RealTimeDataEngineering #EventHubsTutorial #BigDataSolutions #CloudDataEngineering #DataScienceProject #AzureStreaming #AdvancedDataEngineering - วิทยาศาสตร์และเทคโนโลยี
![ตีสิบเดย์ [FULL] | ครอบครัวสะท้านทุ่ง "นางบี & อ้ายสติ๊ก"](http://i.ytimg.com/vi/A1rhYcxxNtA/mqdefault.jpg)
Precise , Detailed and to the point . Thanks
Excellent presentation. Thank You.
Lovely! thanks a ton. You've earned a subscriber.
You make perfect videos and this is really quality content !!!
Great Video !!! Could you please help us to understand on Tumbling window and Watermark concept while reading and writing real time data. If possible.
Awesome video Bhai :)
Dear , you have used the the protocol AQMP, should it will be AMQP? Kindly clarify on that.
Very useful .Can you also explain how we can compress and send data to event hub and read in spark ?
its gem of a knowledge. I was searching for it. finally found one. just one question how to do similar thing if we dont have unity catalog
Thank you! Without Unity Catalog you will need to use the Hive Metastore and Service Principals / Mount Points
how is this done for this project? as also don't have unity catalog under a student account.@@pathfinder-analytics
Is eventhub necessary ? I think we can use autoloader directly to connect to Kafka . Please let me know if there are any limitations that would warrant use of event-hub in between
does eventhub has to be in the same resource group as storage account?
Hi, while creating the cluster I'm unable to add unity catalog? Please help
How to capture batchid as newcolumn and write it to adls. Plz help me
Did u use Autoloader here ...
How to trigger my pipeline when any new message reached to eventhub
can you please do similar video with Synpase spark cluster instead of databricks
How to stop the job automatically
Hi,
i mounted my container correctly, but the df.writestream give the following error for checkpointlocation, what should I check?
nalysisException: [RequestId=0b4e8597-1d71-483d-9447-ab4e6c3d9470 ErrorClass=INVALID_STATE.UC_CLOUD_STORAGE_ACCESS_FAILURE] Failed to access cloud storage: [AbfsRestOperationException] error code: UNKNOWN, status code: -1 exceptionTraceId=de24bf5b-e34c-41a8-8855-710ae71aabd2
thank you.