DAG and Lazy Evaluation in spark
ฝัง
- เผยแพร่เมื่อ 17 เม.ย. 2023
- In this video I have talked about dag and lazy evaluation in spark in great detail. please follow video entirely and ask doubt in comment section below.
Directly connect with me on:- topmate.io/manish_kumar25
For more queries reach out to me on my below social media handle.
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj
![ความรักไม่ได้น่ากลัวขนาดนั้น (TRUST ME) - LYKN [ OFFICIAL MV ]](http://i.ytimg.com/vi/9np-lAtCRpQ/mqdefault.jpg)
Faadu explanation! Finally someone explained why Lazy evaluation actually works betters for Big Data processing. Amazing examples, very nice code! Loved the way you explained each line and each job step by step
I think spark considers 'sum' as action. hence 4 jobs. Greatt series !!!
Wow! Thank you for such great explanation. ❤
Awesome bhaiyaji!!!!!
He has mastered and crunched the spark.
Thank you Thank you Thank you Manish for this video✨✨✨
Thank you so much for clarifying my doubts 🙏
wow! very nicely explained. Thank you! :)
great video Manish bro
Truly impressed Manish bhai. Great explanation as you mentioned already "Itna detail mein kahin nhi milega"
amazing content
very well explained...thanks Manish
@wow...!..v.nice bro
bohot sahi explanation !
Hi Manish
Thanks for explaining the concept in detail by running the code.
I have understood the concept of dag ,lazy evaluation and optimization.
Awesome lecture...thanks a lot!
Very good Bhaiya. Aaj bhaut kuch isme indepth sikhne ko Mila jo phele pata hi nahi tha. Keep it up 👍
very well explained Sir..Thank You.
thank you ! great teaching style
Great explanation sir, really helpful!
Maza aa gya lekin video dekh ke... Wahhh❤
You are an excellent teacher, you make lectures so interesting! ye answer dekar to interview ko sikha denge :D
"Job kaise create hota hai?" ye question pehle kabi clear nai hua tha but thanks Manish for clearing this out and add on was lazy evaluation understanding. 👍
NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark
please I stuck here
one Action one job is true for rdd api only. one action in dataframe or dataset can lead to multiple actions being generated internally. or sue to adaptive query executions as well multiple jobs are created in databricks which you can see using describe method
Detailed Explanation. Better than paid lectures.
Very well explained.
Thanks for the session and Please make video on Spark Lineage
Sir ur teaching is good
bhai gajabe padate ho yar maja hi a gya yar
Awesome. By the way, do we know why its creating 4 Spark Jobs instead of 3 ?
I have confusion? read and inferSchema are typically used with Spark's DataFrame API, which is part of Spark SQL. They are not transformations or actions ,part of the logical and physical planning phase of Spark, which occurs before any actions are executed
I too got confused as to why sometimes number of jobs as more or less than Actions. Try clearing the state using menu option run ->clear state and option and then run the cell again which has code from reading of file till all the things you want to perform . I think Data bricks intelligently stores state of system and later when you run same read command the Jobs count might not match
I tried this and it seems to be working
Jobs count is right it is 4 only because sum() function is an action, which I guess Manish missed by mistake. Btw @Manish thank you so much for the amazing course.
@@jatinyadav6158 If 'sum' is an action then why it didn't create a job before adding 'show' codeline ?
@deepanshuaggarwal7042 yes sum is an action, I am not sure why it didn't show a job earlier
One thing, spark.read is a transformation not action. Is it an action because we included any options? Pls clarify
Bro Plese add summery at the end.
Is there any cheat sheet to know what all are transformations and actions, like read is a action whereas filter is a transformation?
Sir Read an inferschema to Transformations hai na and not actions? to inki alag jobs kyu bani hai?
waiting for ur next vidio...
@@rampal4570 is it true bro
Aaj aa jayega
@@AmitSharma-ow8wm NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark
please I stuck here
Sir please ye series thodi jaldi finish karo taki ham interview de sake mene apki puri channel dekhli hai
Thank you
Do you have theory notes for spark?
Databricks community ko kaise use karege, Spark kaise setup karege databricks ke sath. Please ye bata dijiye so that code write kr sake
Because of group by jobs is created
Nope
where is the practical session link ?
is lazy evaluation consider the partition also
like after we have applied orderby on salary col and now we want to show only first two highest salary
so will lazy evaluation also works here spark will process only that partition which has these two salary records or it will process all partitions and then extract first two highest salary record for us
Yes until you write .head(2) for 2 highest record your process will not start although in backend it will create DAG.
any idea now, where the 4th job came from?
Where he explained about fligh data download
Hello Manish.
When you mentioned the sql query gets optimized by spark. Just wanted to know will it help improve the execution time if we use the optimized query in our code itself so that spark need not do it ?
Spark optimization is very limited. So as a developer we should write optimized code to run our process faster
Sir apko samjh aya ki ek action hit karne par 2 jobs kaise create hui?
Hi Sir..jab main yeh code run kar raha hoon I am getting error as--name 'col' is not defined
isko kaise solve kare?
I am trying to find , why 4 jobs are generating here although we have provided only 3 actions
Aj apne interview questions bataya Nehi.
Basic questions would be there. Like what is DAG and what is edges and vertices in it.
ek code likha hai using spark .usse dekhkar kese bataya ja sakta hai ki wo code 'action' hai ya 'transformation' hai.
Aapko google karke pata karna chahiye ki kon kon se actions hai. Rest are transformation
where to find the files ? I just have started the course
well explained.....Sir could you please clarify, 3 actions and 4 jobs created?
Aapke 3 actions me 4 jobs create hue hai?
Aapne show use kara hai? And aap apna code v paste kar dijiye comment section me
@@manish_kumar_1 Aapne jo code likha hai usme bhi 4 jobs create hue hain. Can you explain that?
Below is my code and same thing is happening. 4 Jobs are getting created. Stage is getting skipped but why do we have an extra job with 4 diff Job Ids?
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
spark = SparkSession.builder.master('local[5]') \
.appName("Lazy Evaluation internal working") \
.getOrCreate()
flight_data = spark.read.format("csv")\
.option("header","true")\
.option("inferSchema","true")\
.load("D:\\Spark\\flight_data.csv")
flight_data_repartition = flight_data.repartition(3)
us_flight_data = flight_data.filter(col("DEST_COUNTRY_NAME")=='United States')
us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India') | (col("ORIGIN_COUNTRY_NAME")=='Singapore'))
total_flight_ind_sing = us_india_data.groupby("DEST_COUNTRY_NAME").sum("count")
total_flight_ind_sing.show()
input("Enter to terminate")
Why do we have 4 jobs when we are using only 3 actions in the application??
Hi Manish Bhaiya, in the code snippet you told there are three actions in this applications(read, infer schema and show) but in spark ui there are 4 jobs created ,can you please explain this.
1 job skip hua hoga. Agar data Kam hai to explain karke dekhiye 3 aana chahiye
Bhai iske phle wala video dedo, mujhe mil nhi rha hai..
In my case same code has created 5 jobs?I have import col so it will also be treated as action and new job will be created is it so?
Can you please paste your code in comment section
@@manish_kumar_1 from pyspark.sql.functions import col
flight_data=spark.read.format("csv")\
.option("header","true")\
.option("inferSchema","true")\
.load("dbfs:/FileStore/tables/flight_data.csv")
flight_data_reparition=flight_data.repartition(3)
us_flight_data=flight_data_reparition.filter("DEST_COUNTRY_NAME='United States'")
us_india_data=us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India')|(col("ORIGIN_COUNTRY_NAME")=='Singapore'))
total_flight_ind_sing=us_india_data.groupby("DEST_COUNTRY_NAME").sum("count")
total_flight_ind_sing.show()
(5) Spark Jobs
Job 22 View(Stages: 1/1)
Job 23 View(Stages: 1/1)
Job 24 View(Stages: 1/1)
Job 25 View(Stages: 1/1, 1 skipped)
Job 26 View(Stages: 1/1, 2 skipped)
flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field]
flight_data_reparition:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field]
us_flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field]
us_india_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field]
total_flight_ind_sing:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, sum(count): long]
+-----------------+----------+
|DEST_COUNTRY_NAME|sum(count)|
+-----------------+----------+
| United States| 100|
+-----------------+----------+
Why it is called acyclic?
Because it doesn't make cycle. If it's get into cycle or you can consider it as a circle then it will run endlessly
Bro, you are explaining superbly but why don't you explain in English. Everyone doesn't know Hindi. I don't know Hindi but watching your videos to understand the concepts but not getting it fully because it was explained in Hindi.
please make a video in Windows function
Sure
Sir also considers spark 5s problems.spill, shuffle, storage, etc
Where I get CSV file
How many more videos to come on Apache spark??
Around 20-30. It's just beginning of spark
could you provide notebook link?. It will good for hands on
I don't provide notebook or pdf. Take notes and type every line of code by yourself. This will give you confidence
@@manish_kumar_1 this is indeed really great point! However, if possible, do share your own reference material for our benefit! Thanks! This series is really helpful, I've 4+ YoE in DE but never tried to go into spark internals, now while interviewing for switch, I'm definitely going to utilize all this! Keep 'em coming!! 🙌🏻👏🏻
bhai iski file de do iss lecture ki
NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark
please I stuck here
Seems like your df is not defined
@@manish_kumar_1 ok working now thanks
Sir csv read karne pr two jobs kaise create ho raha hai, read only one action call kr rahe hain.
If possible please clearify
Aur inferschema v use kiye honge. Isliye aa rha hoga
@@manish_kumar_1 thank you sir, can you please share the action list how many action hai spark me, if possible please share sir
Lec 7 Tak view kr liya Jo aap code dika rahe hai usko kise databricks and pyspark
Aapko practical and fundamentals sath me dekhne hai. First video me hi bataya tha
Hi manish Bhai, Could u provide the flight data file.
Kisi ek video me bataya tha. Please follow all videos in sequence
bhai please vo extra job kese create hua ye doubt clear krdo
Aap sql tab me jaake dekho. Kitne jobs skip hue hai. And share me your code and screenshot of the sql tab on LinkedIn or Instagram.
@@manish_kumar_1 abe chutiya tere video ka bol Raha he wo 😂
Where can I get the flight_data.csv file? Please help.
Got the file, thanks
us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME") == 'India') |
(col("ORIGIN_COUNTRY_NAME") == 'Singapore'))
Ismein error bata raha hai : NameError: name 'col' is not defined
Isko kaise define kare?
Col ko import karna padenga voh ek function hai toh import functions karo I think so
Correct "from pyspark.sql.functtions import *"
@@AliKhanLuckky NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark
please I stuck here
@@manish_kumar_1 NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark
please I stuck here
Guru ibutton khi nhi aaya na idhr na udhr😂😂😂
Bhaari mistake ho gaya 😂
csv kaha milegi sir ?
2 playlist hai. Parallely dekhiye. Practical wale me data milega description me usko copy karke save Kar lijiye as csv
Directly connect with me on:- topmate.io/manish_kumar25
Hi sir.. In which video series u have explained about schema @manish_kumar_1