Another fix is to add the following two lines at the beginning of your code: os.environ['PYSPARK_PYTHON'] = sys.executable os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable The error in question often arises from having multiple conflicting Python versions. By adding these lines, you ensure that your Python script, the driver process, and the worker processes all use the same Python version.
Another fix is to add the following two lines at the beginning of your code:
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
The error in question often arises from having multiple conflicting Python versions. By adding these lines, you ensure that your Python script, the driver process, and the worker processes all use the same Python version.
Thanks!