SPRKPY1005
pyspark.conf.SparkConf
This issue code has been deprecated since Spark Conversion Core Version 4.8.0
Message: pyspark.conf.SparkConf is not required
Category: Warning.
Description
This issue appears when the tool detects the usage of pyspark.conf.SparkConf which is not required.
Scenario
Input
SparkConf could be called without parameters or with loadDefaults.
from pyspark import SparkConf
my_conf = SparkConf(loadDefaults=True)
Output
For both cases (with or without parameters) SMA creates a Snowpark Session.builder object:
#EWI: SPRKPY1005 => pyspark.conf.SparkConf is not required
#from pyspark import SparkConf
pass
#EWI: SPRKPY1005 => pyspark.conf.SparkConf is not required
my_conf = Session.builder.configs({"user" : "my_user", "password" : "my_password", "account" : "my_account", "role" : "my_role", "warehouse" : "my_warehouse", "database" : "my_database", "schema" : "my_schema"}).create()
Additional recommendations
This is an unnecessary parameter being removed with a warning comment being inserted. There should be no additional action from the user.
For more support, you can email us at sma-support@snowflake.com or post an issue in the SMA.
Last updated