21. Databricks| Spark Streaming

#DatabricksStreaming, #SparkStreaming, #Streaming,
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@gulsahtanay2341]

Raja makes my databricks journey easy with his series. Thanks a lot.

[@rajasdataengineering7585]

Glad to hear that! Thanks for watching

[@ShubhamFarande-pi1bf]

while writing stream I can see writestream path and check point path is given but there is no readstream path given then how is it able to understand from where to read ? I also noticed you cancelled the readstream query after its demo for writing I think it was in cancelled state.

[@patrickbateman7665]

Explained in very Simple Way. Thanks for such a great video Raja. Don't know how to thank you. 👏👏

[@rajasdataengineering7585]

Thank you Dileep, for your kind words

[@arthireddyannadi8121]

Hi Raja,
I am doing the series and and its worth watching. I have a question from the video and hope you answer it.
At the end of the video , you read the file in parquet form and displayed the result which appeared in tabular form. In previous video when you opened the parquet, it is not human readable but when you read in data bricks notebook, it appeared in tabular form, could you please explain?

[@rajasdataengineering7585]

Thanks Arthi for your comment!
Yes parquet file is not human readable. But when we create a dataframe (out of any file format csv, parquet, json etc), data is copied from native format to spark environment. It is not anymore parquet format once created dataframe. So when we display dataframe, it is in tabular format

[@SqlMastery-fq8rq]

Very well explained Sir. Thank you.

[@rajasdataengineering7585]

Glad you liked it! Thank you

[@ETLMasters]

Built my first pipeline from this video. Thanks.

[@rajasdataengineering7585]

Fantastic! Glad to hear 👍🏻

[@sureshkoduru8810]

Thanks raja good explanation

[@rajasdataengineering7585]

Glad it helps, keep watching! Thanks

[@vydudraksharam5960]

Raja, you connect dots that i am missing from my real time experience. Expected more on checkpoint and how to handle it. Thank you very much.

[@rajasdataengineering7585]

Glad it was helpful!

[@khalilahmad6279]

The best tutorial I've come across. Thank you.

[@rajasdataengineering7585]

Glad it was helpful! Thanks for your comment

[@kartikmudgal2127]

@@rajasdataengineering7585 sir please help with the excel u r using

[@kiranachanta6631]

Awesome content!! One question though :)
I have built a streaming pipeline.
Now let's assume, events are getting generated every 3 hrs in my source.
How will the data bricks cluster & notebook be invoked every 3 hrs to process the new events? does the cluster should be up and running all the time?

[@rajasdataengineering7585]

In the streaming, there is an option of trigger. Using trigger, we can specify whether it should be live and batch processing. In this case you can specify trigger interval of 3 hours so that cluster does not need to be up and running all the time

[@kiranachanta6631]

@@rajasdataengineering7585 - Awesome

[@rajunaik8803]

@@kiranachanta6631 Trigger with processing time as 3 hrs will not keep your cluster idle, in streaming cluster is always up and running. In your case, you will need to go with normal batch process like scheduling notebook

[@sumanmondal8836]

Thanks Raja...Just one question before writing the file DO I need to run that readstream API ...simultaneously both readstream and writeStream will run?

[@rajasdataengineering7585]

Yes you can execute readstream first

[@chandandutta2007]

@@rajasdataengineering7585 Thanks Raja. Good presentation. extending Suman's question, will it work with just the writeStream and not running the readStream like you have shown in the presentation?

[@rajasdataengineering7585]

Yes it will work

[@akshaygupta013]

While writing the streaming data how come the files were read as the read part was not running the op of amazon was 300 during entire write of 5 files

[@captainlevi5519]

nice tutorial , you have explained in very easy languauge

[@prashantmehta2832]

Hello sir, to be a data engineer do we have learn Kafka and nosql or any data ingest tool?

[@rajasdataengineering7585]

Yes. Kafka is not mandatory but good to have

[@prashantmehta2832]

@@rajasdataengineering7585 Thank you so much sir for the information... ♥️

[@sujitunim]

Good content one suggestion recording should be compatible with mobile full view. It's hard to go through these videos on mobile. Intial 2-3 video was very nice in this series in terms of mobile view

[@rajasdataengineering7585]

Thanks Sujit for valuable suggestion. Will make sure it is compatible with mobile view

[@saipoojithakondapally4136]

Great explanation in a simple way sir.. Thank a lot

[@rajasdataengineering7585]

Thanks!

[@parameshgosula5510]

It's explained well but only concern is the font size very small..

[@rajasdataengineering7585]

Thank you for suggestion. Will take care of font size next video onwards

[@naveennagar507]

Excellent . Very simple and crisp explanation.

[@rajasdataengineering7585]

Thanks Naveen

[@tanushreenagar3116]

Perfect 👌 explanation sir

[@rajasdataengineering7585]

Thank you!

[@dineshdeshpande6197]

How can we connect to Kafka or any other streaming application, what parameters we need to have to authenticate the connections with DataBricks

[@lakshayagarwal4953]

What will happen if i will upload the same file again. Will it be replaced or checkpoint will neglect it because it is already processed before??

[@sagnikmukherjee5108]

Thanks for the tutorial, buddy. I am able to make my first streaming pipeline :)

[@rajasdataengineering7585]

Glad I could help!

[@rishadm1771]

Great explanation sir, thank you!

[@rajasdataengineering7585]

Thank you

[@Rafian1924]

Excellent explanation. However, one example with some real world data and end to end data engineering will help a lot.. like using adls, blob storage and then in synapse .. power bi

[@rajasdataengineering7585]

Sure Sandesh, will try to create a video with more complex example

[@Rafian1924]

@@rajasdataengineering7585 Thanks for replying Raja. Eagerly awaiting that video.

[@ankitsahay8499]

Can we do realtime streaming directly from adls gen 2?
Suppose if we have a folder in adls and it keeps getting updated at every 4 hrs with new files.

[@rajasdataengineering7585]

Yes we can do. In this example, I have used DBFS file system. Instead of this file system, we can use ADLS also

[@ankitsahay7650]

@@rajasdataengineering7585 we don't need to refresh the mounting point then right?

[@Poori1810]

how long cluster runs? redo the calculation when the new file is posted?

[@rajasdataengineering7585]

There is an option of trigger in spark streaming. We can choose once or any interval so the cluster will turn on for that particular time. If continuous process is needed in your requirement, cluster will up and running all the times and will incur huge cost. In that mode, files will be processed as soon as it arrives

[@ezdevops101]

Sir can we use kafka as a source for the same streaming?

[@rajasdataengineering7585]

Kafka can be used as well. Kafka is one of the most commonly used source for databricks streaming solution

[@sumitambatkar3903]

@@rajasdataengineering7585 and also nifi is mostly used to handle streaming data and its widely used nowday also

[@SanchitVashisth]

Can we put trigger interval in the reading stream if not then why ?

[@rajasdataengineering7585]

Trigger means initiating the execution. It can be one time execution or continuous execution or based on interval. In microbatch trigger mode we can specify time interval as well

[@prabhatgupta6415]

u hv stopped the read_stream then how come it is writing before reading the files?

[@saishahsankreddy920]

sir, How come a dataframe ( df1) be mutable for streaming data

[@sharanyas1220]

can have checkpoint for reading data?

[@rajasdataengineering7585]

No, checkpoint can't be set only for reading, it is mainly for processing data

[@cloudquery]

I did not get how this process will pick up, new files automatically, you have not shown that i guess,

[@rajasdataengineering7585]

When we use readstream api, it will pick them automatically

[@cloudquery]

Thanks so readstream api is nothing but the statement we have used as readstream right?

[@itzmekallam7277]

can you share all this code git hub links ?

[@shadabsiddiqui28]

thank you so much raja

[@rajasdataengineering7585]

You are most welcome

[@YashTalks_YT]

Finally a good video

[@rajasdataengineering7585]

Glad you liked it!

[@trilokinathji31]

Could you please share your 5 input files ?

[@rajasdataengineering7585]

Share your email id..I will send it

[@trilokinathji31]

Thank you very much sir for this efforts

[@trilokinathji31]

@@rajasdataengineering7585 : I am writing email id however it is not posted here. I think there is some issue with PII data here.

[@trilokinathji31]

@@rajasdataengineering7585 nngoyal

[@pankajmehar8916]

sir if it possible plz share files for practicals

[@pankajjagdale2005]

Thanks

[@rajasdataengineering7585]

Welcome

[@vickyrai2799]

please share me the 5 input files.

[@rahullagad2455]

Please share that files..

[@sumitambatkar3903]

yes sir , please try to share that files it more helpful for practicing

[@thourayasboui376]

Hi and thanks for the video ! I am trying spark streaming on databricks "lines1=ssc.socketTextStream(hostname="localhost", port=9999)" I am entering data via terminal but there is no results and there is no errors !! how to streaming (using rdd) on databricks? thanks !