Microsoft Fabric: Load data in Warehouse- Dataflow Gen2| Note- Warehouse Destination available now
ฝัง
- เผยแพร่เมื่อ 30 ก.ค. 2024
- Microsoft Fabric: How to load data in Warehouse using Dataflow Gen2 and analyze using SQL, and Power BI Dataset. Load and Transform the data using Dataflow Gen2 into Microsoft Fabric Lakehouse
Watch the video for the new Warehouse Destination
Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data
• Microsoft Fabric: Powe...
Microsoft has Microsoft Fabric, the next-generation Analytic Platform. Check out how to enable it on Power BI Service. How to start a Fabric (Preview) trial
The Microsoft Fabric Platform provides customers with a SaaS-ified, open, lake-centric(One Lake), full-featured data, analytics, and AI platform that meets all their data estate needs. Power BI, Synapse Data Warehouse, Data Factory, Spark, and Notebook all under one platform
00:00 Create Warehouse
03:00 GIT file to be loaded in Warehouse
04:30 Load Data in Lakehouse using Dataflow Gen2
21:00 Checking data using SQL
31:00 Create Model
33:30 Create Measure and Test Data on Power Bi Service Report
36:30 Test Data in Power BI Desktop
38:30 Analyze data in Excel
GIT File link - github.com/amitchandakpbi/pow...
All files: github.com/amitchandakpbi/
Power BI - Azure Free Tier, Create Azure Data Lake Gen 2 and connect that in Power BI: • Power BI: Connect with...
Get Azure SQL DB and Configure it on Power BI: • Abstract Thesis 93: Po...
Official Document: learn.microsoft.com/en-us/fab...
Video from GuyinaCube: • Microsoft Fabric Launc...
Buy a Microsoft Fabric subscription: learn.microsoft.com/en-us/fab...
SKU: learn.microsoft.com/en-us/fab...
Licenses: learn.microsoft.com/en-us/fab...
Enable Fabric: learn.microsoft.com/en-us/fab...
🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.
With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.
How to enable Fabric, find details - learn.microsoft.com/en-us/fab...
Microsoft Fabric licenses details -learn.microsoft.com/en-us/fab...
Don't forget to like, comment, and subscribe for more great content!
▶️Please Subscribe: / @amitchandak
▶️Data:
• Covid 19: covid19.who.int/WHO-COVID-19-...
• Get The data and Pbix files on Github: github.com/amitchandakpbi/pow...
• DropBox files: www.dropbox.com/sh/90ycq7hpmn...
---------
▶️Follow me on:
• Medium: / amitchandak
• Twitter: / amitchandak78
• LinkedIn: / amitchandak78
---------
▶️My Other Videos:
• Mastering Power BI: • Power BI for Beginners...
• Expertise Power BI: • Expertise Power BI- Ho...
~-~~-~~~-~~-~
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
• Microsoft Power BI Tut...
~-~~-~~~-~~-~ - วิทยาศาสตร์และเทคโนโลยี
![[ไฮไลต์] แบดมินตันหญิง "เมย์ รัชนก" ไทย vs เบลเยี่ยม รอบแบ่งกลุ่ม | โอลิมปิก 2024](http://i.ytimg.com/vi/qfMROCoiKMY/mqdefault.jpg)
Watch the video for the new Warehouse Destination
Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data
th-cam.com/video/U6fsHSiNpz4/w-d-xo.html
Thank you very much Sir
Amazing video with a detailed explanation. I have covered each piece of information from this video.
Looking forward to learning more from you.
You are most welcome
Wonderful overview! Thanks so much!
I'm glad you found the overview helpful!
Thanks 🙏
Thanks 🙏. Hope you will like other videos of series too.
Nice. quick question, why we are using Warehouse and also in another video we are using Lakehouse. in which use cases we have to decide on this? Also is this warehouse dedicated SQL pool like we jhave on synapse or is it just a lakehouse platform where warehouse is created?
Microsoft's Fabric architecture offers several data management options that cater to different needs and scenarios. Here's a breakdown to help you decide between using a Data Warehouse and a Lakehouse:
Lakehouse Architecture:
Scenario: When you need a unified platform for both big data analytics and machine learning.
Data Stages: Typically divided into Bronze (raw data), Silver (cleaned data), and Gold (business-ready data).
Tools: You can use Spark SQL for SQL queries or Spark Python for ETL processes.
Data Warehouse:
Scenario: Best for scenarios where you require traditional SQL databases for transactional processing or when specific SQL performance optimizations are needed.
Tools: SQL (like in DB SQL) is used.
Data Flow: For instance, you might move from Lakehouse (Bronze) to a hybrid Lakehouse/Warehouse (Silver) and purely Warehouse (Gold) for optimized querying and performance.
Choosing Between Lakehouse and Warehouse:
Full Lakehouse: Ideal for complete big data and machine learning pipelines, utilizing tools like PySpark for transformation.
Warehouse: Preferable if your focus is on leveraging SQL capabilities for complex queries and transactional data processing.
Tools and Integration:
Dataflow Gen2: This can be used for data transformation in both Lakehouse and Warehouse setups.
Notebooks/Spark: Currently, these are more aligned with Lakehouse setups and do not directly interact with traditional Warehouses.
In summary, the choice between Lakehouse and Warehouse depends on your specific data processing needs, the scale of data, and the analytical tools you prefer. Lakehouses are versatile for handling large-scale data and machine learning workflows, whereas Warehouses are more suited for intensive SQL querying and transaction-based operations.
Hey Amit, I am trying to use Dataflow Gen2 to update a Warehouse (not LakeHouse). I'm following exactly what you did, so using the SQL Connection string with target Azure Synapse (SQL DW). However I get this weird error:
"An exception occurred: Microsoft SQL: This function doesn't support the query option 'EnableCrossDatabaseFolding' with value 'true'. "
I cannot find anyone who had such an error...I tried different options, no luck. Did you get that by any chance?
I tried again and able to save the Dataflow Gen2. Hope you are getting error before that.
Make sure you have taken correct URL and opted for Organization Authentication. Also make sure a warehouse is created; add a table using create statement and check it is working .
If everything is fine, connect to me on LinkedIn and share screenshot. I would check the issue
www.linkedin.com/in/amitchandak78/
Hi,
Thank you for your efforts. Can you please create some more videos on loading in to Lake house and warehouse other than Dataflow Gen2.
Please refer to full series. Most of the options are explored
Mastering Microsoft Fabric 31 videos: th-cam.com/video/p-v0I5S-ybs/w-d-xo.html
It looks like the Azure Synapse Analytics (SQL DW) option as a data destination for a Dataflow Gen2 has been replaced with Warehouse, making the process more straight forward as we only need to choose the Warehouse name now.
Yes, Agree. New one is much better. Discussed in the recent video - th-cam.com/video/U6fsHSiNpz4/w-d-xo.html
Hope you will like the full series
Microsoft Fabric: A Brief Introduction-
th-cam.com/video/p-v0I5S-ybs/w-d-xo.html
How to enable Microsoft Fabric; on Power BI Service| Start the Fabric (Preview) trial
th-cam.com/video/8kIMHmkQ9m4/w-d-xo.html
Microsoft Fabric-What it is, Why Use, Who Should, When and How to use
th-cam.com/video/sbWNJMaspeA/w-d-xo.html
Microsoft Fabric: Various Components| Microsoft Fabric Create Workspace, Lakehouse, Warehouse
th-cam.com/video/pVvvb-MzxKw/w-d-xo.html
Microsoft Fabric part 5: How to load data in Lakehouse using Dataflow Gen2
th-cam.com/video/aPI6NT3lfsE/w-d-xo.html
Microsoft Fabric Part 6: How to load data in Warehouse using Dataflow Gen2
th-cam.com/video/Mn1GB8rSeoQ/w-d-xo.html
Microsoft Fabric Part 7: How to load data in Warehouse using Data Pipeline | End-to-End Scenario
th-cam.com/video/idBrer4NrnY/w-d-xo.html
Let's have a spark in fabric- Microsoft Fabric Part 8: How to load data in Lakehouse using Spark; Python/Notebook
th-cam.com/video/jWDFcpK_k9I/w-d-xo.html
Microsoft Fabric Part 9: Integrated Machine Learning- k-Means clustering: th-cam.com/video/lsnvQ4UoTwo/w-d-xo.html
Microsoft Fabric Part 10: Create a Date table in Dataflow Gen2, and use in Lakehouse, and Warehouse| Time Intelligence with Power BI th-cam.com/video/g3BEVXQdk-g/w-d-xo.html
Microsoft Fabric Part 11: How to load data in Lakehouse using Data Pipeline | End to End Scenario:
th-cam.com/video/jn_D2-b5YpY/w-d-xo.html
Microsoft Fabric Part 12: What are the options to load local files in Lakehouse:
th-cam.com/video/Pi5IvkHxJBs/w-d-xo.html
There is no shortcut to success. However, in MS Fabric, you can shortcut to Azure DataLake
Microsoft Fabric Part 13: Shortcut- within Fabric/ Azure Data Lake: th-cam.com/video/KIbt9GzAdWk/w-d-xo.html
Microsoft Fabric Part 14: Import Azure SQL Data to Warehouse | Multiple tables using Pipeline-
th-cam.com/video/4wj4b3EvFpI/w-d-xo.html
Microsoft Fabric Part 15: Incremental Data Load(ETL) for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline: th-cam.com/video/mpFRnZBXsvQ/w-d-xo.html
How about incremental without Dataflow, Only using pipeline and SQL procedure for Microsoft Fabric Warehouse
Microsoft Fabric Part 16: th-cam.com/video/qsOIfTzjCSQ/w-d-xo.html
Microsoft Fabric Part 17: Load local SQL server data in Lakehouse using on-premise Gateway and Dataflow Gen2:
th-cam.com/video/oEF-jHVmvdo/w-d-xo.html
Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data
th-cam.com/video/U6fsHSiNpz4/w-d-xo.html
Microsoft Fabric Video 19: Load Azure SQL Data to Lakehouse & Warehouse simultaneously with Dataflow Gen2:
th-cam.com/video/40KK-dquvLE/w-d-xo.html
Blogs
Microsoft Fabric: New Age Analytics
medium.com/microsoft-power-bi/microsoft-fabric-new-age-analytics-ba90b894633f
Microsoft Fabric- What, Why, Who, When, How
medium.com/microsoft-power-bi/microsoft-fabric-what-why-who-when-how-638b391df4f1
Hello Amit... When you created a table using SELECT INTO or created a view in Warehouse by copying query from Lakehouse.. I believe it is not actually pointing to Lakehouse. You have the tables with same name present in Warehouse and it actually sourced data from there... Not sure but you can try appending Lakehouse before schema name and see if it works... Though it doesn't work on Azure Synapse
Yes, seem like tables are there at both places. But with the same workspace, you can query tables across LakeHouse and the warehouse, for example th-cam.com/video/KIbt9GzAdWk/w-d-xo.html
So what is the difference between a data warehouse and a data lake? You basically loaded the same data for the 2 places
Warehouses are like databases, where data is stored in a structured format and can be accessed and modified using SQL queries. Lakehouses are like data lakes, where data can be stored in a variety of formats, including structured, semi-structured, and unstructured. Structured data in a Lakehouse can be accessed using SQL queries(Read Only) .
I am loading the same data to show case various features.
Refer first and third video of series
th-cam.com/video/p-v0I5S-ybs/w-d-xo.html
Could you please upload higher quality video, not able to see details clearly as it is a 360p video
Hi Venkat, HD Processing was ongoing and finished just a few mins back. Please check now. Appreciate your support for the channel. 🙏
@@AmitChandak thank you
Is Power BI Copilot will be killed a lot of jobs?
Power BI developers/designers will have to master Copilot to increase their productivity. That will save Job.