I simplified the demo during this video so you see only one lakehouse, but a better option is to create separate Lakehouses per data layer: Bronze, Silver, and Gold - 3 lakehouses.
Great video! Thank you for your time and effort. In the traditional warehouse, we use separate bronze, silver and gold schemas. Can't we do the same in Fabric?
Thanks. Medallion architecture (AKA data design pattern) is rather applicable to Lakehouses. Traditional warehouses have/had layers like Staging and main DWH, but nothing should stop you to use B/S/G layers if you have justification for them. However, it doesn't make bigger sense, as you won't store unstructured data (e.g. files) in "bronze" layer in Warehouse. So, it's more about naming each stage (layer) and describe precisely their role in the whole data platform.
Beautifully explained and demonstrated. I had a question, If we use plain SQL by selecting %SQL in the Notebook, can we do all the sql operations like Merge, add new columns which you did in video Or are there any limitations if we use SQL?
It's worth to be aware of two separate endpoints available in Microsoft Fabric: one (presented in this video) is SPARK SQL in notebook - read/write capabilities to the Lakehouse. The second one is SQL Analytics endpoint (aka Warehouse) which is T-SQL compatible (with limitation) and allows read-only operations. More: learn.microsoft.com/en-us/fabric/data-warehouse/tsql-surface-area#limitations
Very informative video. I wanted to know if I Power BI can source data from Bronze layer? Also, do I need to run every time the SQL code manually to transform data from Bronze to Sliver or is there an autoloader?
Thanks. Answering your questions: 1) Technically you can read any data layer from Power BI, however reading Bronze (RAW) layer is not a good practise. This layer is generally reserved for ETL/ELT process. 2) No, of course, you don't have to. This video is a demo only. In Production workload your pipelines or notebooks would be run by scheduled trigger or job. Autoloader is not available in Fabric, but you can always read the data incrementally from Bronze to Silver using ID/timestamp column.
If we create powerBI report in fabric and we published it and send it to other person with you only access and if that person access it will it cost to that person and if yes how much it will cost. And kindly share the link Please let me know
It all depends on where the report is hosted: whether it is in a workspace backed by Power BI Premium capacity or not. If so, users with a free license can view the report without additional cost. If the report is not in a Premium capacity workspace, the viewer will need a Power BI Pro license, which costs approximately $9.99 per user per month. You can find out more here: www.microsoft.com/en-us/power-platform/products/power-bi/pricing & here: learn.microsoft.com/en-us/power-bi/consumer/end-user-features
excellent explanation , shall we create folders like silver, gold under 'Tables' , so that we can see the different layer separately
I simplified the demo during this video so you see only one lakehouse, but a better option is to create separate Lakehouses per data layer: Bronze, Silver, and Gold - 3 lakehouses.
Really high quality material!
Thx, very much appreciate the support! Stay tuned.
Great Video! Thanks. You have a great knowledge in coding.
Thanks! I like to think coding is my second language, right after emoji 😉
Great video! Thank you for your time and effort. In the traditional warehouse, we use separate bronze, silver and gold schemas. Can't we do the same in Fabric?
Thanks. Medallion architecture (AKA data design pattern) is rather applicable to Lakehouses. Traditional warehouses have/had layers like Staging and main DWH, but nothing should stop you to use B/S/G layers if you have justification for them. However, it doesn't make bigger sense, as you won't store unstructured data (e.g. files) in "bronze" layer in Warehouse. So, it's more about naming each stage (layer) and describe precisely their role in the whole data platform.
Beautifully explained and demonstrated. I had a question, If we use plain SQL by selecting %SQL in the Notebook, can we do all the sql operations like Merge, add new columns which you did in video Or are there any limitations if we use SQL?
It's worth to be aware of two separate endpoints available in Microsoft Fabric: one (presented in this video) is SPARK SQL in notebook - read/write capabilities to the Lakehouse. The second one is SQL Analytics endpoint (aka Warehouse) which is T-SQL compatible (with limitation) and allows read-only operations. More: learn.microsoft.com/en-us/fabric/data-warehouse/tsql-surface-area#limitations
@@KamilNowinski Got it. Thank you :)
Very informative video. I wanted to know if I Power BI can source data from Bronze layer? Also, do I need to run every time the SQL code manually to transform data from Bronze to Sliver or is there an autoloader?
Thanks. Answering your questions: 1) Technically you can read any data layer from Power BI, however reading Bronze (RAW) layer is not a good practise. This layer is generally reserved for ETL/ELT process. 2) No, of course, you don't have to. This video is a demo only. In Production workload your pipelines or notebooks would be run by scheduled trigger or job. Autoloader is not available in Fabric, but you can always read the data incrementally from Bronze to Silver using ID/timestamp column.
If we create powerBI report in fabric and we published it and send it to other person with you only access and if that person access it will it cost to that person and if yes how much it will cost. And kindly share the link
Please let me know
It all depends on where the report is hosted: whether it is in a workspace backed by Power BI Premium capacity or not. If so, users with a free license can view the report without additional cost. If the report is not in a Premium capacity workspace, the viewer will need a Power BI Pro license, which costs approximately $9.99 per user per month. You can find out more here: www.microsoft.com/en-us/power-platform/products/power-bi/pricing & here: learn.microsoft.com/en-us/power-bi/consumer/end-user-features