Microsoft Fabric Briefing - after 6 months of use on the private preview.

Microsoft Fabric is a third generation data and analytics platform, building on strong foundations that have been established by Azure Synapse Analytics and Power BI. It is a SaaS solution that further lowers barriers to adoption. Endjin has been on the Private Preview for Microsoft Fabric since late 2022, and have put the platform through its paces using real-world data and scenarios.
In this 20 minute chat, Microsoft MVP Ian Griffiths interviews Barry Smart, Director of Data & AI, and Ed Freeman, Senior Data Engineer, about their experiences of this new unified data platform.
- 0:50 What is Microsoft Fabric?
- 2:12 How does that compare with other offerings like Databricks or Snowflake?
- 3:43 What does the transition from PaaS to SaaS mean?
- 5:41 Does Fabric represent a shift in maturity and how would that look on a Wardley map?
- 7:41 What have you been doing with Fabric?
- 8:30 Do you have a favourite feature?
- 9:40 Is Fabric ready for prime time?
- 12:19 Could you sketch out how you might approach building a spike to explore Fabric?
- 15:20 Ian's final thoughts
- 16:02 Barry's final thoughts
- 17:23 Ed's final thoughts
If you enjoyed this talk, watch our Fabric End to End Demo series:
📺 Part 1 - Lakehouse & Medallion Architecture - • Microsoft Fabric: Lake...
📺 Part 2 - Planning and Architecting a Data Project - • Microsoft Fabric: Insp...
📺 Part 3 - Ingest Data - • Microsoft Fabric: Inge...
📺 Part 4 - Creating a shortcut to ADLS Gen2 in Fabric - • Microsoft Fabric: Crea...
📺 Part 5 - Navigating OneLake data locally - • Microsoft Fabric: Loca...
Read all our content about Microsoft Fabric:
endjin.com/microsoft-partner/...
Read our Azure Radar "Assess" recommendation for Microsoft Fabric:
endjin.com/microsoft-partner/...
Read our Microsoft Fabric Blog posts:
endjin.com/what-we-think/edit...
#MicrosoftFabric #Data #analytics #lakehouse #dataengineering #Microsoft #PowerBI #ai #MicrosoftBuild #MicrosoftBuild2023

[@jacobgoerz1953]

Excellent talk! Thank you for sharing!

[@MatthewRoche]

What a fantastic video - thank you!

[@Yavin4]

Great video. Thank you all very much for this. Would you agree that at a bare minimum, Fabric gives businesses a more specific framework to structure their overall business data objectives and define data roles?

[@endjin]

I'm not sure about "a more specific framework", but the whole platform has been designed with some very precise personas in mind, and their use cases have been optimised. There's a lot less "infrastructure engineering" required, as this has moved towards being more of a SaaS product, which means as a business is should significantly reduce your "time to value". A lot of the focus has been on making it far easier for the consumers of the data (from the Power BI layer upwards) to get what they need with minimal effort, and for data engineering personas to provide everything that's needed to move their orgs closer to some form of "self-service". It's really all about reducing friction and complexity... and I guess that could be called "democratising data". But WRT the notion of a framework... it enabled recent concepts like a lakhouse architecture or datamesh - in terms of creating data products for your different organisational units.... whether that's what your organisation needs is for you to establish. We advocate focusing on an insight discovery process - i.e. working from the business requirements back to the data (rather than the traditional "start with the data" approach). We have a good talk about it here endjin.com/what-we-think/talks/how-to-define-business-requirements-for-a-successful-cloud-data-and-analytics-project and a blog series here: endjin.com/blog/2022/09/insight-discovery-01-why-do-data-projects-fail

[@Yavin4]

@@endjin Wow. Thank you again.

[@hahuyhung5248]

I am using the trial Fabric account, but I wonder whether I set auto refresh dataflow in data pipeline, would this refresh the dataset?
Please show me how

[@bhaskaraggarwal8971]

Great discussion. Thank you!

[@endjin]

We're glad you enjoyed it! If you've not seen Ed's 10 minute tour around Fabric - take a look: th-cam.com/video/Qu6my9b2FWg/w-d-xo.html

[@damiancz2339]

What about configuration Microsoft Fabric with Azure sql databases behind firewall i see it nearly impossible without enable public communication between azure services, and private endpoints are not enabled for Fabric, any other option to comunicate between azure and fabric?

[@endjin]

Security (both ACLs and network security) is something that's evolving over the public preview. We would expect it to behave just like other Azure services - and that you'd be able to use Private Link or the like for this type of connection.

[@johnfromireland7551]

I had to pause and laugh when I heard the phrase "more opinionated view". I suppose data warehouses can hold opinions?! Anything's possible, now, in the Modern-AI age, right? Looking forward to your future videos.

[@scharlesworth93]

Cool, other videos I have seen on this have been mostly hype

[@mnhworkdev1652]

What does that mean for job roles if there are no centralised Data Teams? This sounds like its a similar Epoch to when dedicated network teams moved to the cloud.

[@endjin]

It's worth reading up on Data Mesh - this is the approach that moves from a centralised team to a much more distributed, per domain, data product type structure. Fabric has support for these concepts.

[@mnhworkdev1652]

@@endjin That's much appreciated and interesting reading, So would you say that data teams will be less embedded in a business in the future and contracted out to dedicated cloud analytics companies?
I suppose there is a certain amount of domain knowledge that requires a dedicated team within the business for the time being. Interesting to see if this moves though.

[@endjin]

"So would you say that data teams will be less embedded in a business in the future and contracted out to dedicated cloud analytics companies?"
​
Quite the opposite - data mesh as a principle is about democratizing analytics within an organization, and as analytical tools get more and more intelligent and business user-friendly, the data maturity of organizations will steadily increase. This will likely mean less outsourcing, but naturally there will always be a balance!
​
"I suppose there is a certain amount of domain knowledge that requires a dedicated team within the business for the time being. "
​
Absolutely, but instead of thinking of it as a "centralized data team", start thinking of it as a "Centre of Enablement" team (or, in Microsoft parlance - "Centre of Excellence" - but we prefer the former), where the goal is to build a network of expertise and put the team in charge of diffusing that expertise across different domains within the business.

[@mnhworkdev1652]

@@endjin Sorry to clarify, does that mean technically trained analysts using SQL will end up in a non-technical role because most of that technical side is more easily available? If so they technical people traditionally in IT roles will end up moving away from business wont they?

[@endjin]

@@mnhworkdev1652 This topic actually gets covered in a video we've just posted: th-cam.com/video/9kd_cekeBbM/w-d-xo.html

[@ranjanroy1623]

If F64 is compared to Premium P1 capacity within power BI which is available for 4995 USD but here in above chart i see the monthly charge of F64 to be 8409 , pls help me understand the diff .
Is it only the power is same i.e. CPU and QPU ? but not the price

[@dezcraft_dev]

It’s the $8.4k amount with the pay as you go model. Microsoft announced there will be an option to do 1 year reserved instances in which the pricing for F64 should be very compatible to P1

[@liamboyd4676]

This is a quite biased discussion… databricks had all these features implemented years ago and MS obviously copying lots of concepts and formats directly from them. And the fact that these folks are aware of databricks yet didn’t mention any of that or offered specific comparisons tells me (1) they are biased or obligated to say good things (2) if not comparing = good for MS that might mean fabric is not as good (or not even close)

[@endjin]

I wouldn't say it's biased. We've been delivering solutions based on Databricks since 2017 and know the platform inside out, and we're also Snowflake partners. This briefing was focused on comparisons to Azure Synapse, because we know orgs (especially our customers) who have adopted that platform want to know what Microsoft Fabric means for their strategic investment. We're certainly not obligated to say good things; if you follow the link to our Azure Radar entry about Fabric we do list pros and cons. Our biggest concern is the move away from serverless pricing (pay as you query) as we say that as a true USP that SME's really loved. There will be future videos doing a side by side comparison to Databricks and Snowflake. I don't accept that Databricks "had all these features implemented years ago"; Vertiparquet would be a good example of innovation. The great thing about a modern data platform arms race between Databricks, Snowflake & Microsoft is that we all benefit.

[@endjin]

And what I would say is that we haven't gone into any depth regarding feature-by-feature comparison in this discussion. We briefly broached a comparison, but that was high-level and talking about general perception, where we're just reflecting what we hear from lots of our customers.
We are certainly not anti-Databricks. As you say, they've been leader of the pack in the Lakehouse space for years now. Their implementation of Delta Lake is still the best out there (generally since Delta Lake is still not 100% OSS (despite their announcement last year!)). Delta Live tables are a great feature, which hasn't been replicated in Fabric. However, certain features that we see in Fabric aren't (and probably won't ever) be a feature in Databricks, like Data Factory pipelines - general purpose orchestration/integration pipelines with a wide-range of activities. Databricks also isn't a semantic modeling/data viz tool, whereas Fabric is (since it engulfs Power BI).
So that's where our "broader vision" comment comes from - not because Fabric is inherently superior to Databricks in the areas that can/should be compared, rather that Fabric in its nature is more of an E2E platform (from data integration through to data viz).
Do stay tuned, because we have videos on the way which dive a bit deeper into the specific comparisons.