It was nice content. I had an interview question on performance issue of using Merge into query in Databricks and what is the best way to load the data using Spark SQL or PySpark. Could you please make a video on this?
* for upserts use MERGE * MERGE INTO myTable USING myData * Loading via autoloader, you need to read the stream first * val df = spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .load("/input/path") df.writeStream.trigger(Trigger.Once) .format("delta") .start("/output/path")
Can you make a video on Performance optimization in pyspark with real time example, which will be helpful
It was nice content. I had an interview question on performance issue of using Merge into query in Databricks and what is the best way to load the data using Spark SQL or PySpark.
Could you please make a video on this?
* for upserts use MERGE *
MERGE INTO myTable USING myData
* Loading via autoloader, you need to read the stream first *
val df = spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/input/path")
df.writeStream.trigger(Trigger.Once)
.format("delta")
.start("/output/path")
Great...thanks
is there any future demand for Azure Synapse as Fabric is its successor? should we learn fabric instead as a Data engineer?
You can learn the entire Microsoft Fabric in 3 hrs and become a Fabric expert after watching Mr. K's video.