Sir Your videos are really very helpful, nobody has given such information in simple language in their video. Keep up the good work of spreading the knowledge❤
Sir your videos are really good and very understandable in a very simple language. I'm stuck as my cluster is not running. it is saying - Azure Quota Exceeded Exception. I'd be grateful if you could help solve this.
I don't have such option as you have shown in community edition related to cluster, it is mentioned as your compute will automatically terminate after an idle period of one or two hours.
I am confused about the terminate after _ minute option what will happen after a specified time period is that cluster will terminate permanently or will that cluster restart again while executing the next query
It will permanently shut down after x minutes of idle time, if this option is enabled. So when we need to execute the next query, need to turn on the cluster manually
Hi Raja u r superb... 🌸 Thank you so much for your excellent work 💐💐.. Raja I am stuck I an situation... actually, I have a oltp postgres server and datalake as olap ... currently, all the customer service on my oltp server .... due to which we have indexes on oltp ... which is making huge latency on inserting ... COULD you pls suggest how to add a staging server .. which will get change feeds from my main oltp ... Please, help .. 🙏 thank u :)
Thanks Arup. If index is problem while inserting, you can think of partitin switch/exchange which is most commonly available in most of the databases. Also you can think of option where we can disable index while bulk loading and enable right after loading
@@rajasdataengineering7585 Thank you so much Raja .🍁.for replying ... Actuall, data coming as as streaming so bulk data load and applying index won't be a good option ...but I will do POC on partition switch.. thank u once again...
Sir Your videos are really very helpful, nobody has given such information in simple language in their video. Keep up the good work of spreading the knowledge❤
Thank you, Ashutosh!
Great content. Smooth explaination. Keep up the good work!
Glad you liked it!
Hi Raja, that's a great lecture series on pyspark. Is there any possibility of getting the slides for reference?
Great content, thank you!
Glad you liked it! My pleasure!
Sir your videos are really good and very understandable in a very simple language. I'm stuck as my cluster is not running. it is saying - Azure Quota Exceeded Exception.
I'd be grateful if you could help solve this.
Superb Raja
Helpful and knowledgeable session
Thank you Kamal
I don't have such option as you have shown
in community edition related to cluster, it is mentioned as your compute will automatically terminate after an idle period of one or two hours.
Databricks community edition has many limitations and this is one of that limitation
@@rajasdataengineering7585 ok can you share the data sample it would be helpful for practice
Hi sir I don't understand use of spot instance?
whats the difference between standard and single node what woukd be the driver /worker configuration for standard cluster
Standard have both worker and driver single node will have only driver this driver only have to do execution
I am confused about the terminate after _ minute option
what will happen after a specified time period
is that cluster will terminate permanently
or will that cluster restart again while executing the next query
It will permanently shut down after x minutes of idle time, if this option is enabled. So when we need to execute the next query, need to turn on the cluster manually
Hi Raja Can you please provide Remaining Transformations also
Hi Satish, sure I can provide remaining transformations
thank you sir
Most welcome!
Raja the great 👍
Thank you
done
Hi Raja u r superb... 🌸
Thank you so much for your excellent work 💐💐..
Raja I am stuck I an situation... actually, I have a oltp postgres server and datalake as olap ... currently, all the customer service on my oltp server .... due to which we have indexes on oltp ...
which is making huge latency on inserting ... COULD you pls suggest how to add a staging server .. which will get change feeds from my main oltp ...
Please, help .. 🙏 thank u :)
Thanks Arup.
If index is problem while inserting, you can think of partitin switch/exchange which is most commonly available in most of the databases. Also you can think of option where we can disable index while bulk loading and enable right after loading
@@rajasdataengineering7585 Thank you so much Raja .🍁.for replying ...
Actuall, data coming as as streaming so bulk data load and applying index won't be a good option ...but I will do POC on partition switch.. thank u once again...
Hi raja nice video, I have farced the one of the interview question what is cluster size in your project, how to configure cluster size
Hi are you working related to data engineering
Hi sir is there any python script to create the cluster automatically
Hi, yes it is possible through cluster APIs provided by databricks
@@rajasdataengineering7585 can you please provide any link for reference here.