SPRKPY1045
pyspark.sql.functions.map_values has workaround
This issue appears when the tool detects the usage of pyspark.sql.functions.map_values which has a workaround.
Input code:
df.select(map_values(col))
Output code:
#EWI: SPRKPY1045 => pyspark.sql.functions.map_values has a workaround, see documentation for more info
df.select(map_values(col))
map_values(col: ColumnOrName) -> pyspark.sql.column.Column
This function is used for column that contains a map/dictionary (keys/values), to extract the list of values.
Action: you can create a udf to get the values for a column. Example:
# example of a proposed udf
from snowflake.snowpark import functions as F
from snowflake.snowpark.types import ArrayType, MapType
map_values_udf=None
def map_values(map):
global map_values_udf
if not map_values_udf:
def _map_values(map: dict)->list:
return list(map.values())
map_values_udf = F.udf(_map_values,return_type=ArrayType(),input_types=[MapType()],name="map_values",is_permanent=False,replace=True)
return map_values_udf(map)
F.map_values = map_values
// using map_values defined previously
df.select(map_values(colDict))
- For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.
Last modified 21d ago