Just wondering. If I want to do an incremental load by using the merge statement, would that read the whole source table and match that to the destination table? And would that therefor have a big performance impact? Let's say I have a source table of Terabytes in size, would you still use this mechanism? Or would you do it in a different way?
Thanks for video. Helps a lot. I like format a lot: short videos with reference to other related to comprise end-to-end scenario vs long end-to-end video. Suggestions for videos: 1. Can you expand on versioning when loading the data into bronze, silver and then Power BI query? Example: mistake was made - customer name was incorrect and needs fixing, so we need to generate new version for data that came 2 weeks ago (coming on daily basis). Don't get me wrong, I do not want you to do my job, but it is such usual scenario and I find it difficult to find examples how versions are generated and then used. Especially if I have big dataset in standardized zone, but I version just part of raw files. And I assume that this kind of video will be useful for many people. 2. PowerQuery vs Spark: PowerQuery is fantastic when you need to handle conformance - that is load from landing zone raw files to conformant files according to cloud-scale analytics landing zones for Azure. This is also related to building landing zones.
Great video. Didn't realise that each 'lakehouse' tableset is actually the equivalent of a 'database' in 'hive metatore' speak. On another note, when will you guys rename the channel to Microsoft Fabric? 😉
hi Stijn, thanks for the video. I added a shortcut to the lakehouse but any attempt to read a file/folder results in a "java.nio.file.AccessDeniedException: Operation failed: "Forbidden", 403" error. I am using an access key so I don't think it is a permission thing. I can browse through the landing zone storage account in the lakehouse explorer without issues. Any thoughts? Thanks
pls. why do you prefer to use merge already in the bronze layer? I prefer to have bronze reflected all source data as it comes. Also, I have not seen schema validation of landing zone files in your pipeline. But you use specific collumns to get data to bronze layer. The load will fail if schema accidently change. Again, I prefer to load everything to bronze as is. So to merge schema here and validate everything before loading to silver. What is your opinion?. Thanks.
Yes, this is only a sample video without the schema validation, the same rules apply when you want to tackle schema validation. Indeed, there are different ways of tackling this, I have seen customers go for bronze/raw with delta, I have seen without. One of the main reasons I hear for delta in bronze is to ensure any latter action will be an incremental one. But as always it depends on the needs of your organization
Unfortunately, the audio is very bad, and also the screen resolution of these recordings makes it very difficult to read - the occasional zooming in just makes it confusing. It's a pity as the content is pretty good - perhaps include the example files so people can try this themselves. This series could really use an Editor to make sure the content is uniformly presented.
Super useful video.
Interesting way of incremental load with Spark.
Can't wait to test it out on our workload
These are all good so far. I'm a little behind on Databricks Delta, but this helps.
Just wondering. If I want to do an incremental load by using the merge statement, would that read the whole source table and match that to the destination table? And would that therefor have a big performance impact? Let's say I have a source table of Terabytes in size, would you still use this mechanism? Or would you do it in a different way?
Thanks for video. Helps a lot. I like format a lot: short videos with reference to other related to comprise end-to-end scenario vs long end-to-end video.
Suggestions for videos:
1. Can you expand on versioning when loading the data into bronze, silver and then Power BI query? Example: mistake was made - customer name was incorrect and needs fixing, so we need to generate new version for data that came 2 weeks ago (coming on daily basis). Don't get me wrong, I do not want you to do my job, but it is such usual scenario and I find it difficult to find examples how versions are generated and then used. Especially if I have big dataset in standardized zone, but I version just part of raw files. And I assume that this kind of video will be useful for many people.
2. PowerQuery vs Spark: PowerQuery is fantastic when you need to handle conformance - that is load from landing zone raw files to conformant files according to cloud-scale analytics landing zones for Azure. This is also related to building landing zones.
voice is muffled .. love the fabric learning series please improve voice equipment .. keep up the great work you are doing for the community
Great video. Didn't realise that each 'lakehouse' tableset is actually the equivalent of a 'database' in 'hive metatore' speak.
On another note, when will you guys rename the channel to Microsoft Fabric? 😉
could you share the notebooks?
hi Stijn, thanks for the video. I added a shortcut to the lakehouse but any attempt to read a file/folder results in a "java.nio.file.AccessDeniedException: Operation failed: "Forbidden", 403" error. I am using an access key so I don't think it is a permission thing. I can browse through the landing zone storage account in the lakehouse explorer without issues.
Any thoughts? Thanks
Great video. Agree with the other comments, voice is badly muffled. please have a listen to the video and fix the mic. many thx.
thanks.. can we have E2E medallion architecture on Fabric?
pls. why do you prefer to use merge already in the bronze layer?
I prefer to have bronze reflected all source data as it comes.
Also, I have not seen schema validation of landing zone files in your pipeline. But you use specific collumns to get data to bronze layer.
The load will fail if schema accidently change. Again, I prefer to load everything to bronze as is. So to merge schema here and validate everything before loading to silver.
What is your opinion?.
Thanks.
Yes, this is only a sample video without the schema validation, the same rules apply when you want to tackle schema validation.
Indeed, there are different ways of tackling this, I have seen customers go for bronze/raw with delta, I have seen without. One of the main reasons I hear for delta in bronze is to ensure any latter action will be an incremental one.
But as always it depends on the needs of your organization
I also have bronze for raw and merge into silver. I would be interested to see what @stijnwynants7307 gold layer looks like
Unfortunately, the audio is very bad, and also the screen resolution of these recordings makes it very difficult to read - the occasional zooming in just makes it confusing. It's a pity as the content is pretty good - perhaps include the example files so people can try this themselves.
This series could really use an Editor to make sure the content is uniformly presented.
voice is really really bad. Please fix it, content is good.