SPRKPY1022
pyspark.sql.functions.log10
Message: pyspark.sql.functions.log10 has a workaround, see documentation for more info
Category: Warning
Description
This issue appears when the SMA detects a use of the pyspark.sql.functions.log10 function, which has a workaround.
Scenario
Input
Below is an example of a use of the pyspark.sql.functions.log10
function that generates this EWI. In this example, the log10
function is used to calculate the base-10 logarithm of the value column.
df = spark.createDataFrame([(1,), (10,), (100,), (1000,), (10000,)], ["value"])
df_with_log10 = df.withColumn("log10_value", log10(df["value"]))
Output
The SMA adds the EWI SPRKPY1022
to the output code to let you know that this function is not directly supported by Snowpark, but it has a workaround.
df = spark.createDataFrame([(1,), (10,), (100,), (1000,), (10000,)], ["value"])
#EWI: SPRKPY1022 => pyspark.sql.functions.log10 has a workaround, see documentation for more info
df_with_log10 = df.withColumn("log10_value", log10(df["value"]))
Recommended fix
As a workaround, you can use the snowflake.snowpark.functions.log function by passing the literal value 10
as the base.
df = spark.createDataFrame([(1,), (10,), (100,), (1000,), (10000,)], ["value"])
df_with_log10 = df.withColumn("log10_value", log(10, df["value"]))
Additional recommendations
For more support, you can email us at sma-support@snowflake.com or post an issue in the SMA.
Last updated