SPRKSCL1117

org.apache.spark.sql.functions.translate

This issue code is deprecated since Spark Conversion Core 2.40.1

Message: org.apache.spark.sql.functions.translate has a workaround, see documentation for more info

Category: Warning

Description

This issue appears when the SMA detects a use of the org.apache.spark.sql.functions.translate function, which has a workaround.

Scenario

Input

Below is an example of the org.apache.spark.sql.functions.translate function that generates this EWI. In this example, the translate function is used to replace the characters 'a', 'e' and 'o' in each word with '1', '2' and '3', respectively.

val df = Seq("hello", "world", "scala").toDF("word")
val result = df.withColumn("translated_word", translate(col("word"), "aeo", "123"))

Output

The SMA adds the EWI SPRKSCL1117 to the output code to let you know that this function is not fully supported by Snowpark, but it has a workaround.

val df = Seq("hello", "world", "scala").toDF("word")
/*EWI: SPRKSCL1117 => org.apache.spark.sql.functions.translate has a workaround, see documentation for more info*/
val result = df.withColumn("translated_word", translate(col("word"), "aeo", "123"))

Recommended fix

As a workaround, you can convert the second and third argument into a column object using the com.snowflake.snowpark.functions.lit function.

val df = Seq("hello", "world", "scala").toDF("word")
val result = df.withColumn("translated_word", translate(col("word"), lit("aeo"), lit("123")))

Additional recommendations

• For more support, you can email us at [e-mail] or post an issue in the SMA.