You've just heightened my desire to adopt MS Fabric as the primary tool/platform on my new journey to Data Engineering. This tutorial was as smooth, unambiguous and interesting as anything should be. Thank you.
Hi malvik, I was searching for a clear, informative tutorial for my team, and your video was exactly what we needed. It’s incredibly well-made, with rich content, clear step-by-step guidance, and excellent language. Thank you for your valuable contribution to the community and for helping so many of us!
This is so helpful as an introduction to a fabric data pipeline. Great use case, helpful example of the functionality, nothing complicated or fiddly. Really smart. I appreciated; - process layering - parameterising notebooks - configuring custom spark env Thanks!
I am very grateful for such a video, where you described the entire process in general. There are a lot of educational materials, not to say that they are bad, but they provide a lot of information in detail. Because of this overload, there is no general understanding of the process of automating analytics itself. And it seems that you do not understand anything, do not understand the sequence of actions. You made a cool video that clarifies this sequence. Thanks again.
Thank you. Many projects/videos tell us to schedule data pipeline for latest data but not show. You did it. I'm glad. This is really useful for me. Also you explain things really well.
This is great, thank you so much. One question though, if I have multiple data sources coming from different locations, do I create seperate medallian architecture and notebooks for each of them
Great content, congratulations! For future videos on Microsoft Fabric, it would be interesting to see a logical update process instead of append and also the formation of a star schema between layers.
Hi, thanks for this tutorial. You use a notebook to fetch data via API. Could I use a dataflow created via blank query? in addition, APIs almost always require authorization and authentication to be used to retrieve data. How can I handle this situation? If I use the notebook, do I need to first create a function that retrieves the access token and then the function that, using this token, retrieves the requested information? Thank you
Thanks for this amazing explanation. In the example the URL goes from 2024-04-9 to 2025-04-15. However max date on the first .json is 2024-04-14. That’s because end date is only less than the end date?. The url had =end_date (2024-04-15).
Thank you for the great explanation, but I am still stuck. at 38:02, your header is well-set but mine has the incorrect column names. Basically, I want my 3rd row to be set up as a column header, could you please help me how to do it?
Just amazing but have one query how to integrate azure repo to fabric because here we downloaded raw file then uploaded directly into fabric but is there any option to integrate repository to fabric ?
very good content, you earned a new subscriber. Could you please just explain the part where you set a filter for start date before appending to the gold layer, you say it's for avoiding duplication but I'm not sure I understand what you mean by this.
Can you please let me know licensing cost for fabric PowerBI for below users for users who will create the report using copilot. for users who will just do prompts and derive insights using copilot...
Hi thanks for your excellent training video. I followed step by step but when I wanted to read json file in Bronze layer I faced this error" Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column (named _corrupt_record by default). For example: spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count() and spark.read.schema(schema).csv(file).select("_corrupt_record").show(). Instead, you can cache or save the parsed results and then send the same query. For example, val df = spark.read.schema(schema).csv(file).cache() and then df.filter($"_corrupt_record".isNotNull).count()." thanks for your help in advance
This probably means you’re not an admin in your Fabric workspace, if you’re using your organisational account you’ll need to speak to your team to assign you the relevant permissions
@@pathfinder-analytics I've utilized the free trial of Fabric based on your previous video- Thank you! I am still not able to access the map visuals setting, though. Do you know if it is possible on the Fabric free trial? Thanks.
You've just heightened my desire to adopt MS Fabric as the primary tool/platform on my new journey to Data Engineering. This tutorial was as smooth, unambiguous and interesting as anything should be. Thank you.
Hi malvik, I was searching for a clear, informative tutorial for my team, and your video was exactly what we needed. It’s incredibly well-made, with rich content, clear step-by-step guidance, and excellent language. Thank you for your valuable contribution to the community and for helping so many of us!
@@parry1454 thanks so much for your kind words. I’m glad I was able to help your team
Thanks!
Very Helpful tutorial! thank you for sharing. Love how smooth the whole end to end process was.
This is so helpful as an introduction to a fabric data pipeline. Great use case, helpful example of the functionality, nothing complicated or fiddly. Really smart. I appreciated;
- process layering
- parameterising notebooks
- configuring custom spark env
Thanks!
I am very grateful for such a video, where you described the entire process in general. There are a lot of educational materials, not to say that they are bad, but they provide a lot of information in detail. Because of this overload, there is no general understanding of the process of automating analytics itself. And it seems that you do not understand anything, do not understand the sequence of actions. You made a cool video that clarifies this sequence. Thanks again.
Excellent explanation with clear examples how to implement end to end data flow from source thru Pipeline to the PBI Visual. THANK YOU
This work is worth more than rubies and gold. Keep it up, bro! Thank you.
@peterodedeyi3366
i have not used fabric yet
I just want to know whether they charge or i can do this on free trail
Very helpful video. Now I have actual knowledge of what my team are putting in-place.
Thank you. Many projects/videos tell us to schedule data pipeline for latest data but not show. You did it. I'm glad. This is really useful for me. Also you explain things really well.
Thank you for the kind words
Q
i work on cloud.....my code not running 23.49 times ...can i get your linkdin id ? i need your help
Thank you for your time on such a great project!
This video is great! Clear explanations, and very simple to follow. Thank you!
Amazing I will definitely try this project in Microsoft Fabric step by step... Thanks for sharing very useful..😊😊..Keep sharing..🤟🤟.
Awesome. Thanks for end- to end project. Need more like this. It realy widen my knowledge horizon. thank you very much. Looking for more like these.
I'm glad you found it helpful 👍
This is great, thank you so much. One question though, if I have multiple data sources coming from different locations, do I create seperate medallian architecture and notebooks for each of them
thanks a lot for such knowledge, can you also share if we want to update and insert both on same pipeline.
Great content, thanks a lot for this video, I am now a new subscriber. You made the fabric platform very easy to navigate
Great content, congratulations!
For future videos on Microsoft Fabric, it would be interesting to see a logical update process instead of append and also the formation of a star schema between layers.
Great video. Thanks ❤
Hi, thanks for this tutorial. You use a notebook to fetch data via API. Could I use a dataflow created via blank query? in addition, APIs almost always require authorization and authentication to be used to retrieve data. How can I handle this situation?
If I use the notebook, do I need to first create a function that retrieves the access token and then the function that, using this token, retrieves the requested information?
Thank you
Insightful. Thanks a lot for the work and the sharing!
Thank you so much for the great demo
Thanks for this amazing explanation. In the example the URL goes from 2024-04-9 to 2025-04-15. However max date on the first .json is 2024-04-14. That’s because end date is only less than the end date?. The url had =end_date (2024-04-15).
Thank you for the great explanation, but I am still stuck. at 38:02, your header is well-set but mine has the incorrect column names. Basically, I want my 3rd row to be set up as a column header, could you please help me how to do it?
Just amazing but have one query how to integrate azure repo to fabric because here we downloaded raw file then uploaded directly into fabric but is there any option to integrate repository to fabric ?
neat and clear explanation!!!!
so clear!!
i have not used fabric yet
I just want to know whether they charge or i can do this on free trail
very good content, you earned a new subscriber. Could you please just explain the part where you set a filter for start date before appending to the gold layer, you say it's for avoiding duplication but I'm not sure I understand what you mean by this.
Excellent. Highly appreciated
Superr video, useful as always...😊
Thank you!
Can you please let me know licensing cost for fabric PowerBI for below users
for users who will create the report using copilot.
for users who will just do prompts and derive insights using copilot...
Great video. I am subscribing. Thank you. However, I am not seeing the tables when I go to SQL. Any suggestions?
Hi, I think this is currently a bug with the schema enabled lakehouses. I understand that this is currently an open ticket.
Fantastic intro video, thx!!!
thankyou 👌👌
Hi thanks for your excellent training video. I followed step by step but when I wanted to read json file in Bronze layer I faced this error" Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the
referenced columns only include the internal corrupt record column
(named _corrupt_record by default). For example:
spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count()
and spark.read.schema(schema).csv(file).select("_corrupt_record").show().
Instead, you can cache or save the parsed results and then send the same query.
For example, val df = spark.read.schema(schema).csv(file).cache() and then
df.filter($"_corrupt_record".isNotNull).count()." thanks for your help in advance
Could you please tell me what are a few pre-requisites to know before watching this video ? Thanks.
I discuss the pre-requisites at 2:16
@@pathfinder-analytics Thanks
Amazing..can i get your linkdin id bro ?
i dont see tenant setting in admin portal.. i see only 3 options in admin portal ----1.capacity settings 2.refresh summary 3.help+support
This probably means you’re not an admin in your Fabric workspace, if you’re using your organisational account you’ll need to speak to your team to assign you the relevant permissions
@@pathfinder-analytics I've utilized the free trial of Fabric based on your previous video- Thank you! I am still not able to access the map visuals setting, though. Do you know if it is possible on the Fabric free trial? Thanks.
same for me and I am also using free trail account
Excellent. Highly appreciated