Links

SPRKPY1048

pyspark.sql.session.SparkSession.conf has workaround

Description

This issue appears when the tool detects the usage of pyspark.sql.session.SparkSession.conf which has a workaround. Note: This workaround is deprecated because a later tool version supports the conf snowflake function as direct.
Input code:
spark.conf.set("spark.sql.crossJoin.enabled", "true")
Output code:
#EWI: SPRKPY1048 => pyspark.sql.session.SparkSession.conf has a workaround, see documentation for more info
spark.conf.set("spark.sql.crossJoin.enabled", "true")

Scenario

property SparkSession.conf
Action: SparkSession.conf is used to pass some specific settings only used by Pyspark and doesn't apply to Snowpark. You can remove or comment the code
#spark.conf.set("spark.sql.crossJoin.enabled", "true")

Recommendation

  • For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.