SPRKPY1008

pyspark.sql.context.HiveContext

Message: pyspark.sql.context.HiveContext is not required

Category: Warning.

Description

This issue appears when the tool detects the usage of pyspark.sql.context.HiveContext, which is not required.

Scenario

Input

In this example an example to create a connection to an Hive store.

from pyspark.sql import HiveContext
hive_context = HiveContext(sc)
df = hive_context.table("myTable")
df.show()

Output

In Snowflake there are not Hive stores, so the Hive Context is not required, You can still use parquet files on Snowflake please check this tutorial to learn how.

#EWI: SPRKPY1008 => pyspark.sql.context.HiveContext is not required
hive_context = sc
df = hive_context.table("myTable")
df.show()

the sc variable refers to a Snow Park Session Object

Recommended fix

For the output code in the example you should add the Snow Park Session Object similar to this code:

# Here manually we can add the Snowpark Session object via a json config file called connection.json
import json
from snowflake.snowpark import Session
jsonFile = open("connection.json")
connection_parameter = json.load(jsonFile)
jsonFile.close()
sc = Session.builder.configs(connection_parameter).getOrCreate()

hive_context = sc
df = hive_context.table("myTable")
df.show()

Additional recommendations

• For more support, you can email us at [email protected] or post an issue in the SMA.