Issue Analysis Issue Codes by Source Python Every Error, Warning, and Issue (EWI) in PySpark
Issues Codes
All of the warnings, parsing errors, and conversion exceptions caused generated by the SMA when taking in Python as a source will appear below. If you have any concerns or see something that's not right, please reach out to the SMA support team at sma-support@snowflake.com .
Code
Description
Category
Deprecated since
Not supported spark version
An error occurred when loading the symbol table
The symbol table could not be loaded
pyspark.conf.SparkConf is not required
pyspark.context.SparkContext is not required
pyspark.sql.context.SQLContext is not required
pyspark.sql.context.HiveContext is not required
pyspark.sql.dataframe.DataFrame.approxQuantile has a workaround
pyspark.sql.dataframe.DataFrame.checkpoint has a workaround
pyspark.sql.dataframe.DataFrameStatFunctions.approxQuantile has a workaround
pyspark.sql.dataframe.DataFrameStatFunctions.writeTo has a workaround
pyspark.sql.functions.acosh has a workaround
pyspark.sql.functions.asinh has a workaround
pyspark.sql.functions.atanh has a workaround
pyspark.sql.functions.collect_set has a workaround
pyspark.sql.functions.date_add has a workaround
pyspark.sql.functions.date_sub has a workaround
pyspark.sql.functions.datediff has a workaround
pyspark.sql.functions.instr has a workaround
pyspark.sql.functions.last has a workaround
pyspark.sql.functions.log10 has a workaround
pyspark.sql.functions.log1p has a workaround
pyspark.sql.functions.log2 has a workaround
pyspark.sql.functions.ntile has a workaround
pyspark.sql.readwriter.DataFrameReader.csv has a workaround
pyspark.sql.readwriter.DataFrameReader.json has a workaround
pyspark.sql.readwriter.DataFrameReader.orc has a workaround
pyspark.sql.readwriter.DataFrameReader.parquet has a workaround
pyspark.sql.session.SparkSession.Builder.appName has a workaround
pyspark.sql.column.Column.contains
pyspark.sql.functions.asc has a workaround
pyspark.sql.functions.desc has a workaround
pyspark.sql.functions.reverse has a workaround
pyspark.sql.column.Column.getField has a workaround
pyspark.sql.functions.sort_array has a workaround
Element is not recognized
pyspark.sql.column.Column.getItem has a workaround
pyspark.sql.functions.explode has workaround
pyspark.sql.functions.explode_outer has workaround
pyspark.sql.functions.posexplode has workaround
pyspark.sql.functions.posexplode_outer has workaround
pyspark.sql.functions.split has workaround
pyspark.sql.functions.map_values has workaround
pyspark.sql.functions.monotonically_increasing_id has workaround
pyspark.context.SparkContext.setLogLevel has workaround
pyspark.sql.session.SparkSession.conf has workaround
pyspark.sql.session.SparkSession.sparkContext
pyspark.conf.SparkConf.set has workaround
pyspark.sql.session.SparkSession.Builder.master has a workaround
pyspark.sql.session.SparkSession.Builder.enableHiveSupport has workaround
An error occurred when extracting the dbc files
Spark element with a given argument is not supported
Spark element with a given key - value argument is not supported
Option argument contains a value that is not a literal, therefore cannot be evaluated
Method is not supported with a Platform specific key
pyspark.sql.group.GroupedData.pivot without 'values' parameter is not supported
pyspark.sql.pandas.functions.pandas_udf has a workaround
to_pandas contains columns of type ArrayType that is not supported and has a workaround.
pyspark.rdd.RDD.getNumPartitions is not required
pyspark.storagelevel.StorageLevel is not required
pyspark.sql.functions.udf without parameters or return type parameter are not supported
File has mixed indentation (spaces and tabs).
The expected result may be different
Parameters in pyspark.sql.readwriter.DataFrameReader methods are not supported
The value of SparkContext is replaced with 'session' variable.
pyspark.sql.readwriter.DataFrameWriter.partitionBy is not supported, but has a workaround.
pyspark.sql.readwriter.DataFrameReader.load is not supported, but it has a workaround.
pyspark.sql.readwriter.DataFrameWriter.save is not supported, but it has a workaround.
pyspark.sql.readwriter.DataFrameWriter.option is not supported.
pyspark.ml.feature.VectorAssembler is not supported.
pyspark.ml.linalg.VectorUDT is not supported.
pyspark.sql.dataframe.DataFrame.writeTo is not supported, but it has a workaround.
pyspark.sql.readwriter.DataFrameWriter.option
pyspark.sql.readwriter.DataFrameWriter.options
This code section has recovery from parsing errors
