schema in spark | Lec-4
ฝัง
- เผยแพร่เมื่อ 14 ต.ค. 2024
- In this video I have talked about reading schema in spark.
Directly connect with me on:- topmate.io/man...
Spark Document:- spark.apache.o...
For more queries reach out to me on my below social media handle.
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
My Total Gear:-
Rode Mic:- amzn.to/3vzsohT
Boya M1 Mic- amzn.to/3g1NduJ
Tripod -- amzn.to/3CEfvW1
camera:- amzn.to/3UMDmut
Mic:- amzn.to/3PjIPrp
Mobile:- amzn.to/3kvRX0U ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3jQjyWw
Mouse -- amzn.to/3jUAfQu
Monitor-- amzn.to/3sfCGkM
iPad Pencil:- amzn.to/3fZLtFz
iPad 9th Generation:- amzn.to/3EnRgya My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj
![[DRIP] ‘Love, Maybe’ PREVIEW](http://i.ytimg.com/vi/U7hwHp95m1k/mqdefault.jpg)
Bhai humne to umeed chhod di thi iss video se pehle... Bas banate jaiye video. Gazab samjha rhe.
Brother how are you drawing very easily like pen but using mouse?
Thanks bhai for continuing this series... Please upload videos more frequently.
I have recently started watching your videos (3rd day today). I cant thank you enough for these videos. Top quality content bhai. please keep going on.
Discovered this channel recently, and there's so much more to explore. Wishing you all the best, and a huge thank you for these invaluable lessons!🫶
thanks wonderful way of teaching !
Thanks for the simplification !!
Very nicely explaining 🎉🎉🎉byou are reallly abvery good teacher
Nice Explanation, you really have the very good understanding of the concepts. Appreciate your efforts.
Dude, you are best, I gave up on databricks before 3 days. I watched your total 7 videos till now, learning new things. Keep going brother. ❤❤❤
thank you sir very useful and easy to understand everything
Thanks Manish for this series.
Thank you for posting these videos . Very useful 👍👍
Brother, please provide "Spark Documentation link" that you were referring at the end of the video.
Thanks
thanks bro keep posting such informatic content❤
mza aa gael bhaia! bahute bahute dhanyawaad🙏
Thanks bro 👍 keep guiding
Bro can we import all methods and classes like: from pyspark.sql.types import *
from pyspark.sql.functions import *. Is this considered a wrong practice in DE community?
love your teaching bro .manish bhai apke practicals lecture khatam ho gaye kya
Almost
Hi Manish, you missed talking about the 2nd way
Oh
Schema = 'id int, name string, dates timestamp"
This is the second way to create schema
Hello Manish, for skipping the rows, I was trying to achieve the same but it was not skipping rows. Could you please tell me why that is happening. Also I was running spark in vs code
Thanks manish.
Could you please make a video on dev , test and prod environment for data engineering projects
thanks Manish bro....for consistent videos
bro bahut badiya
done
agar hum .option("header","true")\ de last me to bhi to me jo null record hai wo delete ho jayenge
how you are able to write like pen or pencil very comfortable? please explain. are. you using a mouse or something else?
Pentab Wacom medium size
@@manish_kumar_1 thank you for reply bro
Hi Manish,
Please share the spark docu. link
Please put the link of the spark documentation for the reference
Bhaiya I'm first year cse student. Bhaiya will this playlist be covered from basic to advanced for entry level data engineer and if yes then what's the duration of this playlist.. please reply bhaiya
It will take around 4 month. If you seriously Follow the course then in 4 month you will be job ready at least from spark perspective
hello sir, i am getting this continuously even when cluster is green :
"Exception when creating execution context: java.util.concurrent.TimeoutException: Timed out after 15 seconds"
i have tried making my account with different mail id but still same error.
Thanks
when to use structType and when to use DDl to create schema?
DDL gets converted into struct type internally. So you can use either of this based on your preference.
In structfield schema , we have set nullable true , so why mode failfast is throwing error.Despite it should follow schema which is stating that null values are allowed.
Because it wasn't throwing an error for a null value, instead for "count" which is neither null nor integer, but a string!
Thanks, man
How to allow/reject null values using ddl schema?
Why it failed on failfast mode?
kya yeh seekhne se phle python seekhna imp hai?
document ka link ni mill raha h...please provide the same
Thanks, bhai. If possible, upload videos daily.
I will try my best
Thanks bhai
After adding .schema(my_schema)\ -> I am getting an error that, "unexpected character after line continuation character" what could be the solution for this ?
There is a space after \
Hi Sir, skip rows is not working for me
my code:
flight_df = spark.read.format('csv') \
.option("header", "true") \
.option('skipRows', 1)\
.option("inferSchema", "false") \
.option("mode", "PERMISSIVE") \
.schema(my_schema)\
.load("/FileStore/tables/2010_summary.csv")
flight_df.show(2)
Its returning NULL
.option("skipRows",1), will remove extra header or any column you want
I believe it skips the first '1' columns!
in my case I am not able to see the schema of tables while defining my own schema
flight_df_header = spark.read.format("csv")\
.option("header","true")\
.option("inferSchema","true")\
.option("skipRows",0)\
.schema(my_schema)\
.option("mode","FAILFAST")\
.load("/FileStore/tables/Flight_data-1.csv")
use StructType and StructField command to defines schema
from pyspark.sql.types import StructField,StructType,IntegerType,StringType
my_schema = StructType(
[
StructField("DEST_COUNTRY_NAME", StringType(), True),
StructField("ORIGIN_COUNTRY_NAME", StringType(), True),
StructField("count", IntegerType(), True)
])
Bhai you haven't given link for the documentation
Oh sorry
why cant we use always inferschema, what is the use of defining the schema by your own.
my opinion - Inferschema may detect wrong datatype or you want to enforce a datatype to a column
please share your views
Correct
Also apart from this inferring schema takes twice the processing than explicitly defining it. Because it first has to read the data to infer schema and next time to show the data.
Hello bhai, in my case :
%fs
ls /FileStore/tables/
give me syntax error
ChatGPT debugs:
The error indicates that the %fs magic command is not recognized, possibly due to running in a non-Databricks environment or the notebook not being configured for Databricks-specific commands. Here are alternative methods to access files in a compatible Python environment:
files = dbutils.fs.ls("/FileStore/tables/")
display(files)
It worked for me!
Bhai kya aap online training de sakte ho ?
Not possible
Directly connect with me on:- topmate.io/manish_kumar25
Hi I'm getting the data in every field as NULL though I made skipRows for only 3 rows. Can someone help please. Thanks in advance.
check the schema creation command and verify it according to manish videos
Change the mode to PERMISSIVE from FAILFAST and check
why we put "true" at last?
Already bataya hai video me. Dhyan se dekha and suna kijiye😀. Last wala true Hota hai ki aapke us column ke data me null rah sakta hai ya nhi
Hi Brother,Upon running below code, I am getting null values through all the columns. Could you please tell me the reason?
# Creating DataFrame
flight_data = spark.read.format("csv")\
.option("header","true")\
.option("skipRows",1)\
.schema(flight_data_schema)\
.option("mode","FAILFAST")\
.load('/FileStore/tables/flight_data.csv')
# Show dataframe
flight_data.show(5,truncate=False)
Output:
+-----------------+-------------------+-----+
|Dest_Country_Name|Origin_Country_Name|Count|
+-----------------+-------------------+-----+
|null |null |null |
|null |null |null |
|null |null |null |
|null |null |null |
|null |null |null |
+-----------------+-------------------+-----+
chnage option("mode","FAILFAST")\ to .option("mode","PERMISSIVE")\ first then try
I m not able to skip 1st row i have passed skip row but its still same
BELOW IS MY CODE AND I AM GETTING unexpected character after line continuation character AGAIN AND AGAIN,,,, PLZ SOMEONE HELP
flight_df = spark.read.format("csv")\
.option("header", "false")\
.option("skipRows", 1)\
.option("inferschema", "false")\
.schema(my_schema)\
.option("mode", "PERMISSIVE")\
.load("/FileStore/tables/2010_summary.csv")
flight_df.show(5)
Because in load you have given path in 2 line keep it in one line
@@manish_kumar_1 I resolved it, actually while pressing enter after a \ , it was creating some space after \ ,
i guess i have to manually remove the space everytime i press enter now :(
same error I am getting