Hi...How to save header of text file in hdfs? If I am trying to do df.write.option("header","true").format("text").save("/user/cloudera/output") , Header is not displaying in the hdfs output
I fail to accept that the best practice way of reading files from a local file system is by reading files first using pandas and converting to spark type. Looks more like a hack, not an actual method. Are there any other ways of doing that?
So using pandas, we still created a dataframe on gateway node. How did we pass it to the cluster (worker nodes)?
can we copy the files from local to hdfs using hadoop command instead of using pandas in exam ?
Sir if we are using Cluster mode, is not the driver code will run in any of the worker node
That is right.
hey durga any chance of lecture on pyspark with kafka ?
Hi...How to save header of text file in hdfs? If I am trying to do df.write.option("header","true").format("text").save("/user/cloudera/output") , Header is not displaying in the hdfs output
I think it should work
When you ll upload lastest course of pyspark in udemy
How will signup and login slack channel . please do kindly help me
I fail to accept that the best practice way of reading files from a local file system is by reading files first using pandas and converting to spark type. Looks more like a hack, not an actual method. Are there any other ways of doing that?
This is one of the way to access the local files. With Cloud it need not be relevant any more.