SPRKPY1071
pyspark.rdd.RDD.getNumPartitions
Message: The function pyspark.rdd.RDD.getNumPartitions is not required in Snowpark. So, you should remove all references.
Category: Warning
Description
This issue appears when the tool finds the use of the pyspark.rdd.RDD.getNumPartitions function. Snowflake uses micro-partitioning mechanism, so the use of this function is not required.
Scenario
Input
The getNumPartitions returns the quantity of partitions on a RDD.
df = spark.createDataFrame([('2015-04-08',), ('5',), [Row(a=1, b="b")]], ['dt', 'num', 'row'])
print(df.getNumPartitions())
Output
The tool adds this EWI to let you know that getNumPartitions is not required.
df = spark.createDataFrame([('2015-04-08',), ('5',), [Row(a=1, b="b")]], ['dt', 'num', 'row'])
#EWI: SPRKPY1071 => The getNumPartitions are not required in Snowpark. So, you should remove all references.
print(df.getNumPartitions())
Recommended fix
Remove all uses of this function.
df = spark.createDataFrame([('2015-04-08',), ('5',), [Row(a=1, b="b")]], ['dt', 'num', 'row'])
Additional recommendations
For more support, you can email us at sma-support@snowflake.com or post an issue in the SMA.`
Last updated