Python

Every Error, Warning, and Issue (EWI) in PySpark

Issues Codes

All of the warnings, parsing errors, and conversion exceptions caused generated by the SMA when taking in Python as a source will appear below. If you have any concerns or see something that's not right, please reach out to the SMA support team at [e-mail].

Code	Description	Category	Deprecated since
SPRKPY1000	Not supported spark version	Warning	-
SPRKPY1001	File with parsing errors	Parsing error	-
SPRKPY1002	Element is not supported	Conversion error	-
SPRKPY1003	An error occurred when loading the symbol table	Conversion error	-
SPRKPY1004	The symbol table could not be loaded	Parsing error	-
SPRKPY1005	pyspark.conf.SparkConf is not required	Warning	-
SPRKPY1006	pyspark.context.SparkContext is not required	Warning	-
SPRKPY1007	pyspark.sql.context.SQLContext is not required	Warning	-
SPRKPY1008	pyspark.sql.context.HiveContext is not required	Warning	-
SPRKPY1009	pyspark.sql.dataframe.DataFrame.approxQuantile has a workaround	Warning	-
SPRKPY1010	pyspark.sql.dataframe.DataFrame.checkpoint has a workaround	Warning	-
SPRKPY1011	pyspark.sql.dataframe.DataFrameStatFunctions.approxQuantile has a workaround	Warning	-
SPRKPY1012	pyspark.sql.dataframe.DataFrameStatFunctions.writeTo has a workaround	Warning	-
SPRKPY1013	pyspark.sql.functions.acosh has a workaround	Warning	-
SPRKPY1014	pyspark.sql.functions.asinh has a workaround	Warning	-
SPRKPY1015	pyspark.sql.functions.atanh has a workaround	Warning	-
SPRKPY1016	pyspark.sql.functions.collect_set has a workaround	Warning	-
SPRKPY1017	pyspark.sql.functions.date_add has a workaround	Warning	-
SPRKPY1018	pyspark.sql.functions.date_sub has a workaround	Warning	-
SPRKPY1019	pyspark.sql.functions.datediff has a workaround	Warning	-
SPRKPY1020	pyspark.sql.functions.instr has a workaround	Warning	-
SPRKPY1021	pyspark.sql.functions.last has a workaround	Warning	-
SPRKPY1022	pyspark.sql.functions.log10 has a workaround	Warning	-
SPRKPY1023	pyspark.sql.functions.log1p has a workaround	Warning	-
SPRKPY1024	pyspark.sql.functions.log2 has a workaround	Warning	-
SPRKPY1025	pyspark.sql.functions.ntile has a workaround	Warning	-
SPRKPY1026	pyspark.sql.readwriter.DataFrameReader.csv has a workaround	Warning	-
SPRKPY1027	pyspark.sql.readwriter.DataFrameReader.json has a workaround	Warning	-
SPRKPY1028	pyspark.sql.readwriter.DataFrameReader.orc has a workaround	Warning	-
SPRKPY1029	pyspark.sql.readwriter.DataFrameReader.parquet has a workaround	Warning	-
SPRKPY1030	pyspark.sql.session.SparkSession.Builder.appName has a workaround	Warning	-
SPRKPY1031	pyspark.sql.column.Column.contains	Warning	-
SPRKPY1032	Element is not defined	Conversion error	-
SPRKPY1033	pyspark.sql.functions.asc has a workaround	Warning	-
SPRKPY1034	pyspark.sql.functions.desc has a workaround	Warning	-
SPRKPY1035	pyspark.sql.functions.reverse has a workaround	Warning	-
SPRKPY1036	pyspark.sql.column.Column.getField has a workaround	Warning	-
SPRKPY1037	pyspark.sql.functions.sort_array has a workaround	Warning	-
SPRKPY1038	Element is not recognized	Conversion error	-
SPRKPY1039	pyspark.sql.column.Column.getItem has a workaround	Warning	2.3.0
SPRKPY1040	pyspark.sql.functions.explode has workaround	Warning	-
SPRKPY1041	pyspark.sql.functions.explode_outer has workaround	Warning	-
SPRKPY1042	pyspark.sql.functions.posexplode has workaround	Warning	-
SPRKPY1043	pyspark.sql.functions.posexplode_outer has workaround	Warning	-
SPRKPY1044	pyspark.sql.functions.split has workaround	Warning	-
SPRKPY1045	pyspark.sql.functions.map_values has workaround	Warning	-
SPRKPY1046	pyspark.sql.functions.monotonically_increasing_id has workaround	Warning	-
SPRKPY1047	pyspark.context.SparkContext.setLogLevel has workaround	Warning	-
SPRKPY1048	pyspark.sql.session.SparkSession.conf has workaround	Warning	-
SPRKPY1049	pyspark.sql.session.SparkSession.sparkContext	Warning	-
SPRKPY1050	pyspark.conf.SparkConf.set has workaround	Warning	-
SPRKPY1051	pyspark.sql.session.SparkSession.Builder.master has a workaround	Warning	2.4.0
SPRKPY1052	pyspark.sql.session.SparkSession.Builder.enableHiveSupport has workaround	Warning	-
SPRKPY1053	An error occurred when extracting the dbc files	Warning	-
SPRKPY1054	Spark element with a given argument is not supported	Warning	-
SPRKPY1055	Spark element with a given key - value argument is not supported	Warning	-
SPRKPY1056	Option argument contains a value that is not a literal, therefore cannot be evaluated	Warning	-
SPRKPY1068	to_pandas contains columns of type ArrayType that is not supported and has a workaround.	Warning	-
SPRKPY1071	pyspark.rdd.RDD.getNumPartitions is not required	Warning	-
SPRKPY1072	pyspark.storagelevel.StorageLevel is not required	Warning	-
SPRKPY1073	pyspark.sql.functions.udf without parameters or return type parameter are not supported	Warning
SPRKPY1074	File has mixed indentation (spaces and tabs).	Parsing error