SPRKPY1068

toPandas contains columns of type ArrayType that is not supported and has a workaround.

Category

Warning.

Description

toPandas doesn't work properly If there are columns of type ArrayType. The workaround for these cases is converting those columns into a Python Dictionary by using json.loads method.

Example:

df = dfOriginal.toPandas()​

# check/convert all resulting fields from calling toPandas when they are of 
# type ArrayType,
# they will be reasigned by converting them into a Python Dictionary 
# using json.loads method​

for field in df.schema.fields:
    if isinstance(field.datatype, ArrayType):
        df[field.name] = df[field.name].apply(lambda x: json.loads(x) if x is not None else x)

Recommendations

• For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.