SPRKPY1016

pyspark.sql.functions.collect_set has a workaround

Category

Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.functions.collect_set which has a workaround.

Input code:

col = collect_set(colName)

Output code:

#EWI: SPRKPY1016 => pyspark.sql.functions.collect_set has a workaround, see documentation for more info
col = collect_set(colName)

Scenario:

collect_set(col: ColumnOrName) Action: Use function array_agg, and add a second argument with the value True.

col = array_agg(col, True)

Recommendation

• For more support, you can email us at [e-mail]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.