@@bluedreamerssso I can understand that Scala/python issue, but is cost really a thing? I think you could make it with a reasonable cost in Azure using Data Factory pipelines that call (Azure) Databricks' notebooks and do all the ELT and data modeling you would need that way (getting data flow model) without using Synapse. -> Hard to believe that would be more expensive solution than using Synapse, but of course needs Scala/Python expertise.
Both Synapse and Databricks use Spark. Synapse uses an open source version of Spark with built in support for .NET, while Databricks uses an optimized version of Spark that improves performance and allows for GPU-enabled clusters with higher data concurrency which improve processing performance. Synapse may be easier for a BI team to pick up, and Power BI can be used inside Synapse Studio. At this time, Databricks outperforms Synapse for ML. Microsoft is pumping R&D into Synapse, so it is worth watching.
@@benjamincarter6095thanks for nice clarification. What do you think now, is it same thing in your opinion? MS putting a lot of effort on Fabric side, and on the other hand Databricks is constantly evolving...
need more of this, moving from AWS and Databricks to Azure Synapse Studio
Can u tell me the benefits of moving away from Databricks go Synapse ?
@@ryanbowns1517 cost, unfamiliarity with Scala/python, data flow model.
@@bluedreamerssso I can understand that Scala/python issue, but is cost really a thing? I think you could make it with a reasonable cost in Azure using Data Factory pipelines that call (Azure) Databricks' notebooks and do all the ELT and data modeling you would need that way (getting data flow model) without using Synapse. -> Hard to believe that would be more expensive solution than using Synapse, but of course needs Scala/Python expertise.
Concerning Spark Pool settings: I think they mixed up "auto scale" and "dynamic allocation"; these are two different things
Very good and easy explained thanks
Started using this today, but noted that there wasn't so much in term of documentation. Can you recommend a good source of further reading?
Can you give some guidance on choosing optimal sizing of spark pools
and understanding DAG
Is spark is good for DW? How this differ from sql dedicate ?
Great episode!
Loved it!
Thanks for the video.
was that pyspark?
Python+ spark = Pyspark... The language that used in Spark cluster for data processing, transformation etc
Note sure how this is better than using Databricks in Azure. Can anyone shed some light here.
Both Synapse and Databricks use Spark. Synapse uses an open source version of Spark with built in support for .NET, while Databricks uses an optimized version of Spark that improves performance and allows for GPU-enabled clusters with higher data concurrency which improve processing performance. Synapse may be easier for a BI team to pick up, and Power BI can be used inside Synapse Studio. At this time, Databricks outperforms Synapse for ML. Microsoft is pumping R&D into Synapse, so it is worth watching.
@@benjamincarter6095thanks for nice clarification. What do you think now, is it same thing in your opinion? MS putting a lot of effort on Fabric side, and on the other hand Databricks is constantly evolving...
Im 314 th person to view this