SPRKPY1021

pyspark.sql.functions.last has a workaround

Category

Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.functions.last which has a workaround.

Input code:

df.withColumn('LastEmp', F.last('EmployeeName').over(windowSpec))

Output code:

#EWI: SPRKPY1021 => pyspark.sql.functions.last has a workaround, see documentation for more info
df.withColumn('LastEmp', F.last('EmployeeName').over(windowSpec))

Scenario:

last(col: ColumnOrName, ignorenulls: bool = ...) Action: There is not implementation for "last" function, but there is a builtin function "LAST_VALUE" that can be used with call_builtin and using col for the column name.

df.withColumn('LastEmp',F.call_builtin('last_value',col('EmployeeName')).over(windowSpec))

Recommendation

• For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.