SPRKPY1010

pyspark.sql.dataframe.DataFrame.checkpoint has a workaround

Category

Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.dataframe.DataFrame.checkpoint which has a workaround.

Input code:

checkpoint(False)

Output code:

#EWI: SPRKPY1010 => pyspark.sql.dataframe.DataFrame.checkpoint has a workaround, see documentation for more info
checkpoint(False)

Scenarios:

Checkpoint has two scenarios, but both of them have the same solution. checkpoint() and checkpoint(eager: bool = True) Action: The Time Travel feature could be used and the data can be saved into a snowflake table. Time travel can be used to get the date at the moment of the checkpoint (Time travel the travel feature must be enabled: https://docs.snowflake.com/en/user-guide/data-time-travel.html#enabling-and-disabling-time-travel).

timeStampValue = 'Fri, 01 May 2015 16:20:00 - 0700'
select * from table at(timestamp => timeStampValue::timestamp_tz)

Recommendation

• For more support, you can email us at [email protected]. If you have a contract for support with Snowflake, reach out to your sales engineer and they can direct your support needs.