SPRKPY1044

pyspark.sql.functions.split

This issue code has been deprecated since Spark Conversion Core Version 2.4.0

Message: pyspark.sql.functions.split has a workaround

Category: Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.functions.split which has a workaround.

Scenarios

There are a couple of scenarios depending on the amount of parameters passed to the method.

Scenario 1

Input

Below is an example when the function split has just the str and pattern parameters

F.split('col', '\\|')

Output

The tool shows the EWI SPRKPY1044 indicating there is a workaround.

#EWI: SPRKPY1044 => pyspark.sql.functions.split has a workaround, see the documentation for more info
F.split('col', '\\|')

Recommended fix

As a workaround, you can call the function snowflake.snowpark.functions.lit with the pattern parameter and send it into the split.

F.split('col', lit('\\|'))
# the result of lit will be sent to the split function

Scenario 2

Input

Below is another example when the function split has the str, pattern, and limit parameters.

F.split('col', '\\|', 2)

Output

The tool shows the EWI SPRKPY1044 indicating there is a workaround.

#EWI: SPRKPY1044 => pyspark.sql.functions.split has a workaround, see the documentation for more info
F.split('col', '\\|', 2)

Recommended fix

This specific scenario is not supported.

Additional recommendations

• For more support, you can email us at [email protected] or post an issue in the SMA.