วีดีโอ

Very well explained. I was frustrated with only theory of the unity catalog

getting below error for silver layer processing file AnalysisException Traceback (most recent call last) Cell In[14], line 2 1 # df now is a Spark DataFrame containing JSON data ----> 2 df = spark.read.option("multiline", "true").json(f"Files/{start_date}_earthquake_data.json")

Does external table created in UC using parquet get auto refreshed when additional parquet files are added?

Amazing I will definitely try this project in Microsoft Fabric step by step... Thanks for sharing very useful..😊😊..Keep sharing..🤟🤟.

This is way better than official Databricks Academy courses. Thanks for sharing

Hi thanks for your excellent training video. I followed step by step but when I wanted to read json file in Bronze layer I faced this error" Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column (named _corrupt_record by default). For example: spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count() and spark.read.schema(schema).csv(file).select("_corrupt_record").show(). Instead, you can cache or save the parsed results and then send the same query. For example, val df = spark.read.schema(schema).csv(file).cache() and then df.filter($"_corrupt_record".isNotNull).count()." thanks for your help in advance

Great content. Thanks a lot.

Amazing overview!😁 I could think of one drawback of unity here: say user1 has access to catalog1, workspace1 and workspace2 has access to catalog1 as well. That would mean that user1 now has access to workspace1 and workspace2. is that correct? If yes then is there anyway to prevent this?

Thanks a lot from sharing your knowledge

But in my laptop when i try to create new connection i able to see databases of another tables

Insightful. Thanks a lot for the work and the sharing!

thanks for the demo, it was really helpfull

Anyone else receive this error when trying to use a Fabric tool? I followed these steps, and got access to a Fabric environment, but can only use PowerBI related tools. "To create a Dataflow Gen2, you need a free Microsoft Fabric trial capacity. Unfortunately, your Microsoft tenant has reached the maximum number of available free trial capacities..."

Workspace admins do not have access to administrative portal. So I'm wondering how they can add users and groups to a workspace?

thank you

Legend. Thanks a ton....worked like a charm !!

i have not used fabric yet I just want to know whether they charge or i can do this on free trail

Thanks!

bro please make a video for data analyst and how can we use this in data analyst

Best preparation about unity catalog I've ever seen!

Excellent. Highly appreciated

It did work. Thank you from the Job Seeker community 🙏

That's for a really clear and straight forward tutorial. The one question that nagged me though (and you have may have touched on it, but I missed it), is why we are creating an additional storage account and account connector when one already appears to exist in the managed resource group that is generated when you created the initial premium workspace. I even saw in your tutorial when I re-watched it, the drop-down menus referenced the ones (storage account and access connector) that were already available in addition to the ones you created. Is there anything wrong with using the ones the Storage Account and Access Connector that were already created on your behalf as opposed to re-creating your own in order to enable UC? Or is it a best practice to create additional separate resources for UC enablement rather than using what exists?

Precise , Detailed and to the point . Thanks

Unity catalog can't be enabled for the hive_metastore, that's there by default?

Can i create delta sharing on hive_metastore tables?

Thank you so much...

How to do this oauth way?

This work is worth more than rubies and gold. Keep it up, bro! Thank you.

Great overview Malvik!!

Fabulous video Malvik, this clears all my doubts

Nice Presentation. It is well organized and precise. Thank you for your work

Legend.

Great content, explained very well. Thanks for sharing. Is there any chargeback to the Snowflake connection in Databricks?

Thanks for the demo. Btw if the table type is External, can we also see the lineage?

TYSM for putting this out. I was struggling to stitch it all together by reading the documentation, but so many details were still unclear. Documentation is still written as if it were 1995, prioritizing exhaustiveness of option descriptions over clarity. Step-by-step tutorials like these are priceless and should be included by editors because it's a real pain in the butt to go through Russian doll documentation pages, get lost, and waste 6 hours of trial and error just to set up an option correctly.

Thanks for the video.

Is there a way to access Azure SQLDW through MSI using the same connector

You've just heightened my desire to adopt MS Fabric as the primary tool/platform on my new journey to Data Engineering. This tutorial was as smooth, unambiguous and interesting as anything should be. Thank you.

How to trigger my pipeline when any new message reached to eventhub

Very useful .Can you also explain how we can compress and send data to event hub and read in spark ?

helped me a lot. thanks

Thanks, exactly what I needed right now.

does eventhub has to be in the same resource group as storage account?

Hi, i mounted my container correctly, but the df.writestream give the following error for checkpointlocation, what should I check? nalysisException: [RequestId=0b4e8597-1d71-483d-9447-ab4e6c3d9470 ErrorClass=INVALID_STATE.UC_CLOUD_STORAGE_ACCESS_FAILURE] Failed to access cloud storage: [AbfsRestOperationException] error code: UNKNOWN, status code: -1 exceptionTraceId=de24bf5b-e34c-41a8-8855-710ae71aabd2 thank you.

Could you please tell me what are a few pre-requisites to know before watching this video ? Thanks.

Awesome. Thanks for end- to end project. Need more like this. It realy widen my knowledge horizon. thank you very much. Looking for more like these.