SPRKPY1006

pyspark.context.SparkContext

This issue code has been deprecated since Spark Conversion Core Version 4.8.0

Message: pyspark.context.SparkContext is not required

Category: Warning.

Description

This issue appears when the tool detects the usage of pyspark.context.SparkContext, which is not required in Snowflake.

Scenario

Input

In this example there are two context to create a connections to an Spark Cluster

from pyspark import SparkContext

sql_context1 = SparkContext(my_sc1)
sql_context2 = SparkContext(sparkContext=my_sc2)

Output

Because there are no clusters on Snowflake the Context is not required, note that the variables my_sc1 and my_sc2 that contains Spark properties may be not required or it will to be adapted to fix the code.

from snowflake.snowpark import Session
#EWI: SPRKPY1006 => pyspark.sql.context.SparkContext is not required
sql_context1 = my_sc1
#EWI: SPRKPY1006 => pyspark.sql.context.SparkContext is not required

sql_context2 = my_sc2

Additional recommendations

• This is an unnecessary parameter being removed with a warning comment being inserted. There should be no action from the user.

• For more support, you can email us at [email protected] or post an issue in the SMA.