3. Preserve RDBMS table's metadata when overwriting table from Spark using TRUNCATE | Spark🌟Tips 💡

[@prabhakaran8965]

I loaded data from MySQL to dataframe using spark.read.format , after that I did some transformation and created final dataframe to load in to same table in MySQL.....so I did truncate and used overwrite using write.mode() but I am not able to see data in MySQL but if I load to different table it works but not on same table .....please provide your suggestions.

[@angad2364]

please share github

[@techisland1520]

import findspark
findspark.init('D:\spark\spark-2.3.1-bin-hadoop2.7')
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.master("local")
.appName("test")
.config("spark.jars", 'mysql-connector-java-5.1.49-bin.jar')
.getOrCreate())


customer_df = (spark.read
.option("header",True)
.option("inferSchema",True)
.csv(r"customer_data.csv"))
(customer_df.write.format("jdbc")
.option("url", f"jdbc:mysql://localhost:3306/{database_name}")
.option("driver", "com.mysql.jdbc.Driver")
.option("dbtable", "CUSTOMER_DATA")
.option("user", db_user)
.option("password", db_password)
.option("truncate", True) # default is False
.mode("overwrite")
.save())