SPRKPY1045

pyspark.sql.functions.map_values has workaround

Description

This issue appears when the tool detects the usage of pyspark.sql.functions.map_values which has a workaround.

Input code:

df.select(map_values(col)) 

Output code:

#EWI: SPRKPY1045 => pyspark.sql.functions.map_values has a workaround, see documentation for more info
df.select(map_values(col))

Scenarios

map_values(col: ColumnOrName) -> pyspark.sql.column.Column

This function is used for column that contains a map/dictionary (keys/values), to extract the list of values. Action: you can create a udf to get the values for a column. Example:

# example of a proposed udf
from snowflake.snowpark import functions as F
from snowflake.snowpark.types import ArrayType, MapType
map_values_udf=None
def map_values(map):
    global map_values_udf
    if not map_values_udf:
        def _map_values(map: dict)->list:
            return list(map.values())
        map_values_udf = F.udf(_map_values,return_type=ArrayType(),input_types=[MapType()],name="map_values",is_permanent=False,replace=True)
    return map_values_udf(map)

F.map_values = map_values

// using map_values defined previously
df.select(map_values(colDict))

Recommendation

• For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.