SPRKPY1026

pyspark.sql.readwriter.DataFrameReader.csv

This issue code has been deprecated since

Message: pyspark.sql.readwriter.DataFrameReader.csv has a workaround, see documentation for more info

Category: Warning

Description

This issue appears when the SMA detects a use of the function, which has a workaround.

Scenario

Input

Below is an example of a use of the pyspark.sql.readwriter.DataFrameReader.csv function that generates this EWI. In this example, the csv function is used to read multiple .csv files with a given schema and uses some extra options such as encoding, header and sep to fine-tune the behavior of reading the files.

file_paths = [
  "path/to/your/file1.csv",
  "path/to/your/file2.csv",
  "path/to/your/file3.csv",
]

df = session.read.csv(
  file_paths,
  schema=my_schema,
  encoding="UTF-8",
  header=True,
  sep=","
)

Output

The SMA adds the EWI SPRKPY1026 to the output code to let you know that this function is not directly supported by Snowpark, but it has a workaround.

file_paths = [
  "path/to/your/file1.csv",
  "path/to/your/file2.csv",
  "path/to/your/file3.csv",
]

#EWI: SPRKPY1026 => pyspark.sql.readwriter.DataFrameReader.csv has a workaround, see documentation for more info
df = session.read.csv(
  file_paths,
  schema=my_schema,
  encoding="UTF-8",
  header=True,
  sep=","
)

Recommended fix

In this section, we explain how to configure the path parameter, the schema parameter and some options to make them work in Snowpark.

1. path parameter

Snowpark requires the path parameter to be a stage location so, as a workaround, you can create a temporary stage and add each .csv file to that stage using the prefix file://.

2. schema parameter

3. options parameters

The following options are not supported by Snowpark:

columnNameOfCorruptRecord
emptyValue
enforceSchema
header
ignoreLeadingWhiteSpace
ignoreTrailingWhiteSpace
inferSchema
locale
maxCharsPerColumn
maxColumns
mode
multiLine
nanValue
negativeInf
nullValue
positiveInf
quoteAll
samplingRatio
timestampNTZFormat
unescapedQuoteHandling

Below is the full example of how the input code should look like after applying the suggestions mentioned above to make it work in Snowpark:

stage = f'{session.get_fully_qualified_current_schema()}.{_generate_prefix("TEMP_STAGE")}'
session.sql(f'CREATE TEMPORARY STAGE IF NOT EXISTS {stage}')

session.file.put(f"file:///path/to/your/file1.csv", f"@{stage}")
session.file.put(f"file:///path/to/your/file2.csv", f"@{stage}")
session.file.put(f"file:///path/to/your/file3.csv", f"@{stage}")

df = session.read.schema(my_schema).option("encoding", "UTF-8").option("sep", ",").csv(stage)

Additional recommendations

PreviousSPRKPY1025 NextSPRKPY1027

Last updated 5 months ago

file_paths = [ "path/to/your/file1.csv", "path/to/your/file2.csv", "path/to/your/file3.csv", ] df = session.read.csv( file_paths, schema=my_schema, encoding="UTF-8", header=True, sep="," )

file_paths = [ "path/to/your/file1.csv", "path/to/your/file2.csv", "path/to/your/file3.csv", ] #EWI: SPRKPY1026 => pyspark.sql.readwriter.DataFrameReader.csv has a workaround, see documentation for more info df = session.read.csv( file_paths, schema=my_schema, encoding="UTF-8", header=True, sep="," )

stage = f'{session.get_fully_qualified_current_schema()}.{_generate_prefix("TEMP_STAGE")}' session.sql(f'CREATE TEMPORARY STAGE IF NOT EXISTS {stage}') session.file.put(f"file:///path/to/your/file1.csv", f"@{stage}") session.file.put(f"file:///path/to/your/file2.csv", f"@{stage}") session.file.put(f"file:///path/to/your/file3.csv", f"@{stage}") df = session.read.schema(my_schema).option("encoding", "UTF-8").option("sep", ",").csv(stage)