Python

Every Error, Warning, and Issue (EWI) in PySpark

Issues Codes

All of the warnings, parsing errors, and conversion exceptions caused generated by the SMA when taking in Python as a source will appear below. If you have any concerns or see something that's not right, please reach out to the SMA support team at [e-mail].

Code	Description	Category	Deprecated since
SPRKPY1000	Not supported spark version	Warning	-
SPRKPY1001	File with parsing errors	Parsing error	-
SPRKPY1002	Element is not supported	Conversion error	-
SPRKPY1003	An error occurred when loading the symbol table	Conversion error	-
SPRKPY1004	The symbol table could not be loaded	Parsing error	-
SPRKPY1005	pyspark.conf.SparkConf is not required	Warning	-
SPRKPY1006	pyspark.context.SparkContext is not required	Warning	-
SPRKPY1007	pyspark.sql.context.SQLContext is not required	Warning	-
SPRKPY1008	pyspark.sql.context.HiveContext is not required	Warning	-
SPRKPY1009	pyspark.sql.dataframe.DataFrame.approxQuantile has a workaround	Warning	-
SPRKPY1010	pyspark.sql.dataframe.DataFrame.checkpoint has a workaround	Warning	-
SPRKPY1011	pyspark.sql.dataframe.DataFrameStatFunctions.approxQuantile has a workaround	Warning	-
SPRKPY1012	pyspark.sql.dataframe.DataFrameStatFunctions.writeTo has a workaround	Warning	-
SPRKPY1013	pyspark.sql.functions.acosh has a workaround	Warning	-
SPRKPY1014	pyspark.sql.functions.asinh has a workaround	Warning	-
SPRKPY1015	pyspark.sql.functions.atanh has a workaround	Warning	-
SPRKPY1016	pyspark.sql.functions.collect_set has a workaround	Warning	-
SPRKPY1017	pyspark.sql.functions.date_add has a workaround	Warning	-
SPRKPY1018	pyspark.sql.functions.date_sub has a workaround	Warning	-
SPRKPY1019	pyspark.sql.functions.datediff has a workaround	Warning	-
SPRKPY1020	pyspark.sql.functions.instr has a workaround	Warning	-
SPRKPY1021	pyspark.sql.functions.last has a workaround	Warning	-
SPRKPY1022	pyspark.sql.functions.log10 has a workaround	Warning	-
SPRKPY1023	pyspark.sql.functions.log1p has a workaround	Warning	-
SPRKPY1024	pyspark.sql.functions.log2 has a workaround	Warning	-
SPRKPY1025	pyspark.sql.functions.ntile has a workaround	Warning	-
SPRKPY1026	pyspark.sql.readwriter.DataFrameReader.csv has a workaround	Warning	-
SPRKPY1027	pyspark.sql.readwriter.DataFrameReader.json has a workaround	Warning	-
SPRKPY1028	pyspark.sql.readwriter.DataFrameReader.orc has a workaround	Warning	-
SPRKPY1029	pyspark.sql.readwriter.DataFrameReader.parquet has a workaround	Warning	-
SPRKPY1030	pyspark.sql.session.SparkSession.Builder.appName has a workaround	Warning	-
SPRKPY1031	pyspark.sql.column.Column.contains	Warning	-
SPRKPY1032	Element is not defined	Conversion error	-
SPRKPY1033	pyspark.sql.functions.asc has a workaround	Warning	-
SPRKPY1034	pyspark.sql.functions.desc has a workaround	Warning	-
SPRKPY1035	pyspark.sql.functions.reverse has a workaround	Warning	-
SPRKPY1036	pyspark.sql.column.Column.getField has a workaround	Warning	-
SPRKPY1037	pyspark.sql.functions.sort_array has a workaround	Warning	-
SPRKPY1038	Element is not recognized	Conversion error	-
SPRKPY1039	pyspark.sql.column.Column.getItem has a workaround	Warning	2.3.0
SPRKPY1040	pyspark.sql.functions.explode has workaround	Warning	-
SPRKPY1041	pyspark.sql.functions.explode_outer has workaround	Warning	-
SPRKPY1042	pyspark.sql.functions.posexplode has workaround	Warning	-
SPRKPY1043	pyspark.sql.functions.posexplode_outer has workaround	Warning	-
SPRKPY1044	pyspark.sql.functions.split has workaround	Warning	-
SPRKPY1045	pyspark.sql.functions.map_values has workaround	Warning	-
SPRKPY1046	pyspark.sql.functions.monotonically_increasing_id has workaround	Warning	-
SPRKPY1047	pyspark.context.SparkContext.setLogLevel has workaround	Warning	-
SPRKPY1048	pyspark.sql.session.SparkSession.conf has workaround	Warning	-
SPRKPY1049	pyspark.sql.session.SparkSession.sparkContext	Warning	-
SPRKPY1050	pyspark.conf.SparkConf.set has workaround	Warning	-
SPRKPY1051	pyspark.sql.session.SparkSession.Builder.master has a workaround	Warning	2.4.0
SPRKPY1052	pyspark.sql.session.SparkSession.Builder.enableHiveSupport has workaround	Warning	-
SPRKPY1053	An error occurred when extracting the dbc files	Warning	-
SPRKPY1054	Spark element with a given argument is not supported	Warning	-
SPRKPY1055	Spark element with a given key - value argument is not supported	Warning	-
SPRKPY1056	Option argument contains a value that is not a literal, therefore cannot be evaluated	Warning	-
SPRKPY1062	pyspark.sql.group.GroupedData.pivot without 'values' parameter is not supported	Warning
SPRKPY1063	pyspark.sql.pandas.functions.pandas_udf has a workaround	Warning
SPRKPY1068	to_pandas contains columns of type ArrayType that is not supported and has a workaround.	Warning	-
SPRKPY1071	pyspark.rdd.RDD.getNumPartitions is not required	Warning	-
SPRKPY1072	pyspark.storagelevel.StorageLevel is not required	Warning	-
SPRKPY1073	pyspark.sql.functions.udf without parameters or return type parameter are not supported	Warning
SPRKPY1074	File has mixed indentation (spaces and tabs).	Parsing error
SPRKPY1075	The expected result may be different	Warning
SPRKPY1076	Parameters in pyspark.sql.readwriter.DataFrameReader methods are not supported	Warning
SPRKPY1080	The value of SparkContext is replaced with 'session' variable.	Warning
SPRKPY1081	pyspark.sql.readwriter.DataFrameWriter.partitionBy is not supported, but has a workaround.	Warning
SPRKPY1082	pyspark.sql.readwriter.DataFrameReader.load is not supported, but it has a workaround.	Warning
SPRKPY1083	pyspark.sql.readwriter.DataFrameWriter.save is not supported, but it has a workaround.	Warning
SPRKPY1084	pyspark.sql.readwriter.DataFrameWriter.option is not supported.	Warning
SPRKPY1085	pyspark.ml.feature.VectorAssembler is not supported.	Warning
SPRKPY1086	pyspark.ml.linalg.VectorUDT is not supported.	Warning
SPRKPY1087	pyspark.sql.dataframe.DataFrame.writeTo is not supported, but it has a workaround.	Warning
SPRKPY1101	This code section has recovery from parsing errors	Conversion error