Reading Semi-Structured data | Spark with Scala | Realtime scenario

[@mayur4155]

Thank you, Sravana for sharing the solution 👏👏
I have learned a new Function foldleft

[@sravanalakshmipisupati6533]

Thanks a lot, Mayur. Many other foldLeft() scenarios can be found at - github.com/sravanapisupati/SampleDataSet/blob/main/TH-cam_videos_list

[@mayur4155]

​@@sravanalakshmipisupati6533 Can you help me with this one
input List
to_remove = ['%', '-', '&', ":"]
removeFrom = ['XYZ','12%', 'MALE', 'ABC','1-7KG', '12KG', 'MALE&FEMELE', '1:1',"script"]
output=['MALE','ABC','XYZ','script']
We need to remove element from removeFrom which is contained to_remove char element
how to do In scala efficient way

[@sravankumar1767]

Nice explanation sravana 👌 👍 👏

[@sravanalakshmipisupati6533]

Thanks a lot, Sravan

[@sushmamc8904]

Really good video to understand foldleft function 🙏 thank you

[@DileepKumar-ko6cy]

Good Efforts ,very clear explanation 👍🎉

[@sravanalakshmipisupati6533]

Thanks a lot, Dileep.

[@ramanareddy4037]

good explanation .. you have a sound knowledge on spark and scala

[@subhadarshisethy1680]

Mam pls scala notes provide kijiye...pls attach file link to description...

[@sravanalakshmipisupati6533]

github.com/sravanapisupati/SampleDataSet/blob/main/ReadingUnstructuredData.scala

[@subhadarshisethy1680]

@@sravanalakshmipisupati6533 tysm...🙏

[@suprakashkhuntia7345]

The element_at function is not getting resolved. I have imported all the requisites. What could be the issue ?
Error:(20, 52) not found: value element_at
val df1 = df.withColumn("DateTimeStamp", rtrim(element_at(split(df("TimeStamp"), " "),1)))

[@sravanalakshmipisupati6533]

Please check the spark and scala versions.

[@zakeer1830]

Could you please explain Repartition concept in spark

[@sravanalakshmipisupati6533]

Sure Zakeer, I will get back to you soon.