I am facing challenges when i run pyspark code in mu databricks notebook. I want to read 700 million records and ingest those to API which has throttling limit of 130k per second (6 million per minute) i want to meet that threshold but since i have 700 million records in my databricks table, my cluster can not handle and there is lots of spill happening and job stays in hanging state. Please give me solution
God level content ❤
Thank you, please make sure to share with your network over LinkedIn 👍
Waiting for more such informative videos!!
❤ Thanks for making these videos!!
Pls complete this playlist fast and also add end projects.
Planning to finish this series by December. There is a lot to cover. Dont want this series to be like the one that are already available.
I am facing challenges when i run pyspark code in mu databricks notebook. I want to read 700 million records and ingest those to API which has throttling limit of 130k per second (6 million per minute) i want to meet that threshold but since i have 700 million records in my databricks table, my cluster can not handle and there is lots of spill happening and job stays in hanging state. Please give me solution
Do you foresee this replacing adf anytime soon
Yes