Capgemini Data Engineer Interview Question - Round 1 | Save Multiple Columns in the DataFrame |

Input
data = [
(1, "Sagar", 23, "Male", 68.0),
(2, "Kim", 35, "Female", 90.2),
(3, "Alex", 40, "Male", 79.1),
]
schema = "Id int,Name string,Age int,Gender string,Marks float"
df = spark.createDataFrame(data, schema)
Solution:
from pyspark.sql.functions import col
set_of_dtypes=set(i[1] for i in df.dtypes)
for i in set_of_dtypes:
cols=[]
for j in df.dtypes:
if(i==j[1]):
cols.append(j[0])
df.select(cols).write.mode('overwrite').save(f'/FileStore/tables/output_capegmini/{i}')
Combo course package : www.geekcoders...
I have prepared many courses on Azure Data Engineering
1. Build Azure End to. End Project
www.geekcoders...
2. Build Delta Lake project
www.geekcoders...
3. Master in Azure Data Factory with ETL Project and PowerBi
www.geekcoders...
4. Master in Python
www.geekcoders...
Check out my courses on Azure Data Engineering
www.geekcoders...
hastags
tags
#dataengineer #interviewquestions #spark
#hashtags #hastag #tags

[@kunuturuaravindreddy5879]

very good you are posting real interview questions many of them simply explain concer defentitiins

[@GeekCoders]

@@kunuturuaravindreddy5879 thanks

[@sourav_sarkar_2000]

# creating a dict of columns as to avoid checking multiple datatypes
d={}
for col in df.dtypes:
if col[1] not in d:
d[col[1]] = [col[0]]
else:d[col[1]].append(col[0])
for key,val in d.items():
df.select(val).show()
# write df to the location

[@Offical_PicturePerfect]

int_cols = [col for col, dtype in df.dtypes if dtype == 'int']
string_cols = [col for col, dtype in df.dtypes if dtype == 'string']
float_cols = [col for col, dtype in df.dtypes if dtype == 'float']
Creating DataFrames for each data type
int_df = df.select(int_cols)
string_df = df.select(string_cols)
float_df = df.select(float_cols)

[@sourav_sarkar_2000]

# creating a dict of columns to avoid checking multiple datatypes
d={}
for col in df.dtypes:
if col[1] not in d:
d[col[1]] = [col[0]]
else:d[col[1]].append(col[0])
print(d)
for key,val in d.items():
df.select(val).show()
# write df to the location
# df.write.mode('overwrite').save(f'temp_loc/{key}')

[@myl1566]

Good problem to solve. Thanks for posting sagar!

[@GeekCoders]

Thank you

[@aamirmansuri69]

Thank you for posting this video. But, can you please post pyspark interview questions for freshers. Thank you!

[@rawat7203]

My Way Sir
intType = []
stringType = []
floatType = []
for i in df.dtypes:
if i[1] == 'int':
intType.append(i[0])
elif i[1] == 'string':
stringType.append(i[0])
elif i[1] == 'float':
floatType.append(i[0])
dfInt = df.select(*intType)
dfString = df.select(*stringType)
dfFloat = df.select(*floatType)

[@GeekCoders]

Nice

[@Dataengineeringlearninghub]

Great problem sagar

[@rawat7203]

Thanks a lot Sir

[@GeekCoders]

Thank you

[@vutv5742]

Completed 👏

[@pradishpranam6175]

cool question

[@2412_Sujoy_Das]

My solution is as follows:
string = df
integer = df
float = df
for i in df.dtypes:
if i[1]!='string' and i[1]=='int':
string = string.drop(i[0])
float = float.drop(i[0])
elif i[1]!='string' and i[1]=='float':
string = string.drop(i[0])
integer = integer.drop(i[0])
elif i[1]!='int' and i[1]=='string':
integer = integer.drop(i[0])
float = float.drop(i[0])
elif i[1]!='int' and i[1]=='float':
integer = integer.drop(i[0])
string = string.drop(i[0])
elif i[1]!='float' and i[1]=='string':
float = float.drop(i[0])
integer = integer.drop(i[0])
else:
float = float.drop(i[0])
string = string.drop(i[0])
print(string)
print(integer)
print(float)

[@Nextgentrick]

Shouldn’t you use append instead of overwrite

[@pratyushkumar8567]

Hi Sagar
this Capgemini Data Engineer Interview Question - Round 1 | Save Multiple Columns in the DataFrame
what was the experience the candidate has ?

[@GeekCoders]

4 years

[@bhumikalalchandani321]

okay, is this internal functionality of conversion to parq format

[@rawat7203]

yes

[@souvikchattopadhyay4760]

my solution:
dict={}
for i in df.dtypes:
if i[1] in dict.keys():
l=dict.get(i[1])
l.append(i[0])
dict.update({i[1]:l})
else:
l=[]
l.append(i[0])
dict.update({i[1]:l})

for i in dict.keys():
df_s=df.select(dict.get(i))
df_s.show()
##did show instead of writing

[@ug1880]

Were u asked for any imocha test ?

[@GeekCoders]

No

[@ug1880]

@@GeekCoders okk...