Data Caching in Apache Spark | Optimizing performance using Caching | When and when not to cache

Learn Certified Data Engineering. Fill out the inquiry form, and we will get back to you with a detailed curriculum and course information.
shorturl.at/klvOZ
Master Data Engineering using Spark, Databricks, and Kafka. Prepare for cracking Job interviews and perform extremely well in your current job/projects. Beginner to advanced level training and certifications on multiple technologies.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning.
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
==============================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================

[@gurumoorthysivakolunthu9878]

Very detailed... best ever explanation of a topic, Sir... This is amazing... Thank you, Sir....

[@williamhaque6183]

Wonderful. Cleared a lot of doubt.

[@machisri]

sir, Could you make a video on Generative AI in Databricks ( LLM, LongChain, DBRX, HuggingFace, MLFlow)

[@jayaananthjayaram9228]

can you post how to use iceberg in emr or using pyspark

[@sandeepnarwal8782]

Best Video on TH-cam

[@srinubathina7191]

Wow super content
Thank You Sir

[@rajat_ComedyCorner]

Great job, Sir

[@soumikdas7709]

Nice explanation

[@jsnode7696]

I took your udemy course, its great. I have a doubt
I have a hive table, having parquet files with different schema(2 columns varying in data type)
when reading the data as dataframe, and writing it to another table, I am getting error:Parquet files cannot be converted
How to handle with schema data type mismatch ?

[@andre__luiz__]

the best teacher!!!!!

[@balaji348]

Sir, please share writing spark streaming from Kafka topic and with consumer record and again sending that record to another topic

[@nagabadsha]

Well explained, Thanks

[@jay_rana]

where is the next part of the video, can you drop the link ?

[@debojitpaul5779]

Where I acn find the whole video series?

[@omkarm7865]

great explanation

[@nareshdulam58]

Are you going to answer rest of question on what happens to cache if table/view data modified?

[@ScholarNest]

It is automatically refreshed

[@nareshdulam58]

Thank you @@ScholarNest .

[@Ramakrishna410]

How it is automatically refreshed..can you make an video on modified cache

[@Ramakrishna410]

It will bring only from memory after cache, but how spark onows if new data in source table when we are not reading the table..

[@prasadpatil5397]

When actual data changes, the resulted cache data is immediately invalidated.
Any query after that onwords have used, the cache results set will query the database again and re populate the cache. So this way cache data remain synchronised with source dataframe.

[@gautam0086]

Very informative, thx

[@karthikeyanr1171]

Although the content is good, Too lengthy video to explain this concept
This whole concept could be covered shortly