SPRKPY1050

pyspark.conf.SparkConf.set

Message: pyspark.conf.SparkConf.set has a workaround

Category: Warning.

Description

This issue appears when the tool detects the usage of pyspark.conf.SparkConf.set which has a workaround.

Scenario

Input

Below is an example that sets a variable using conf.set.

conf = SparkConf().setAppName('my_app')

conf.set("spark.storage.memoryFraction", "0.5")

Output

The tool adds the EWI SPRKPY1050 indicating that a workaround can be implemented.

conf = SparkConf().setAppName('my_app')

#EWI: SPRKPY1050 => pyspark.conf.SparkConf.set has a workaround, see documentation for more info
conf.set("spark.storage.memoryFraction", "0.5")

Recommended fix

SparkConf.set is used to set a configuration setting only used by Pyspark and doesn't apply to Snowpark. You can remove or comment on the code

#conf.set("spark.storage.memoryFraction", "0.5")

Additional recommendations

• For more support, you can email us at [email protected] or post an issue in the SMA.