Capgemini Data Engineer Interview Question - Round 1 | Save Multiple Columns in the DataFrame |
ฝัง
- เผยแพร่เมื่อ 5 ก.ย. 2024
- Input
data = [
(1, "Sagar", 23, "Male", 68.0),
(2, "Kim", 35, "Female", 90.2),
(3, "Alex", 40, "Male", 79.1),
]
schema = "Id int,Name string,Age int,Gender string,Marks float"
df = spark.createDataFrame(data, schema)
Solution:
from pyspark.sql.functions import col
set_of_dtypes=set(i[1] for i in df.dtypes)
for i in set_of_dtypes:
cols=[]
for j in df.dtypes:
if(i==j[1]):
cols.append(j[0])
df.select(cols).write.mode('overwrite').save(f'/FileStore/tables/output_capegmini/{i}')
Combo course package : www.geekcoders...
I have prepared many courses on Azure Data Engineering
1. Build Azure End to. End Project
www.geekcoders...
2. Build Delta Lake project
www.geekcoders...
3. Master in Azure Data Factory with ETL Project and PowerBi
www.geekcoders...
4. Master in Python
www.geekcoders...
Check out my courses on Azure Data Engineering
www.geekcoders...
hastags
tags
#dataengineer #interviewquestions #spark
#hashtags #hastag #tags
very good you are posting real interview questions many of them simply explain concer defentitiins
@@kunuturuaravindreddy5879 thanks
# creating a dict of columns as to avoid checking multiple datatypes
d={}
for col in df.dtypes:
if col[1] not in d:
d[col[1]] = [col[0]]
else:d[col[1]].append(col[0])
for key,val in d.items():
df.select(val).show()
# write df to the location
int_cols = [col for col, dtype in df.dtypes if dtype == 'int']
string_cols = [col for col, dtype in df.dtypes if dtype == 'string']
float_cols = [col for col, dtype in df.dtypes if dtype == 'float']
Creating DataFrames for each data type
int_df = df.select(int_cols)
string_df = df.select(string_cols)
float_df = df.select(float_cols)
# creating a dict of columns to avoid checking multiple datatypes
d={}
for col in df.dtypes:
if col[1] not in d:
d[col[1]] = [col[0]]
else:d[col[1]].append(col[0])
print(d)
for key,val in d.items():
df.select(val).show()
# write df to the location
# df.write.mode('overwrite').save(f'temp_loc/{key}')
Good problem to solve. Thanks for posting sagar!
Thank you
Thank you for posting this video. But, can you please post pyspark interview questions for freshers. Thank you!
My Way Sir
intType = []
stringType = []
floatType = []
for i in df.dtypes:
if i[1] == 'int':
intType.append(i[0])
elif i[1] == 'string':
stringType.append(i[0])
elif i[1] == 'float':
floatType.append(i[0])
dfInt = df.select(*intType)
dfString = df.select(*stringType)
dfFloat = df.select(*floatType)
Nice
Great problem sagar
Thanks a lot Sir
Thank you
Completed 👏
cool question
My solution is as follows:
string = df
integer = df
float = df
for i in df.dtypes:
if i[1]!='string' and i[1]=='int':
string = string.drop(i[0])
float = float.drop(i[0])
elif i[1]!='string' and i[1]=='float':
string = string.drop(i[0])
integer = integer.drop(i[0])
elif i[1]!='int' and i[1]=='string':
integer = integer.drop(i[0])
float = float.drop(i[0])
elif i[1]!='int' and i[1]=='float':
integer = integer.drop(i[0])
string = string.drop(i[0])
elif i[1]!='float' and i[1]=='string':
float = float.drop(i[0])
integer = integer.drop(i[0])
else:
float = float.drop(i[0])
string = string.drop(i[0])
print(string)
print(integer)
print(float)
Shouldn’t you use append instead of overwrite
Hi Sagar
this Capgemini Data Engineer Interview Question - Round 1 | Save Multiple Columns in the DataFrame
what was the experience the candidate has ?
4 years
okay, is this internal functionality of conversion to parq format
yes
my solution:
dict={}
for i in df.dtypes:
if i[1] in dict.keys():
l=dict.get(i[1])
l.append(i[0])
dict.update({i[1]:l})
else:
l=[]
l.append(i[0])
dict.update({i[1]:l})
for i in dict.keys():
df_s=df.select(dict.get(i))
df_s.show()
##did show instead of writing
Were u asked for any imocha test ?
No
@@GeekCoders okk...