SPRKPY1050

pyspark.conf.SparkConf.set has workaround

Description

This issue appears when the tool detects the usage of pyspark.conf.SparkConf.set which has a workaround.

Input code:

conf.set("spark.storage.memoryFraction", "0.5")

Output code:

#EWI: SPRKPY1050 => pyspark.conf.SparkConf.set has a workaround, see documentation for more info
conf.set("spark.storage.memoryFraction", "0.5")

Scenario

SparkConf.set(key: str, value: str) → pyspark.conf.SparkConf

Action: SparkConf.set is used to set a configuration setting only used by Pyspark and doesn't apply to Snowpark. You can remove or comment the code

Manual Adjustment:

#conf.set("spark.storage.memoryFraction", "0.5")

Recommendation

• For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.