SPRKPY1066

The Spark element does not apply.

Message: The Spark element does not apply since snowflake use micro-partitioning mechanism are created automatically.

Category: Warning

Description

This issue appears when the tool detects the usage of elements related to partitions:

• pyspark.sql.catalog.Catalog.recoverPartitions

• pyspark.sql.dataframe.DataFrame.foreachPartition

• pyspark.sql.dataframe.DataFrame.sortWithinPartitions

• pyspark.sql.functions.spark_partition_id

Those elements do not apply due the use of micro-partitions of Snowflake.

Input code

In this example sortWithinPartitions it's used to create a partition in a DataFrame sorted by the specified column.

df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])
df.sortWithinPartitions("age", ascending=False)

Output code

The SMA adds an EWI message indicating that Spark element is not required.

df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])
#EWI: SPRKPY1066 => The element does not apply since snowflake use micro-partitioning mechanism are created automatically.
df.sortWithinPartitions("age", ascending=False)

Recommended fix

Remove the usage of the element.

df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])

Additional recommendations

• For more support, you can email us at [email protected] or post an issue in the SMA.