SPRKPY1018

pyspark.sql.functions.date_sub

This issue code has been deprecated since Spark Conversion Core Version 4.8.0

Message: pyspark.sql.functions.date_sub has a workaround

Category: Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.functions.date_sub which has a workaround.

Scenario

Input

In this example we use date_add to calculate the date 5 days before the current date for the dataframe df.

col = df.select(date_sub(df.colName, 5))

Output

SMA returns the EWI SPRKPY1018 over the line where date_sub is used, so you can use to identify where to fix.

#EWI: SPRKPY1018 => pyspark.sql.functions.date_sub has a workaround, see documentation for more info
col = df.select(date_sub(df.colName, 5))

Recommended fix

Import snowflake.snowpark.functions, which contains an implementation for date_sub function.

from pyspark.sql.functions import date_sub
df.withColumn("date", date_sub(df.colName, 5))

Additional recommendation

• For more support, you can email us at [email protected] or post an issue in the SMA.