SPRKPY1008
pyspark.sql.context.HiveContext
Message: pyspark.sql.context.HiveContext is not required
Category: Warning.
Description
This issue appears when the tool detects the usage of pyspark.sql.context.HiveContext, which is not required.
Scenario
Input
In this example an example to create a connection to an Hive store.
from pyspark.sql import HiveContext
hive_context = HiveContext(sc)
df = hive_context.table("myTable")
df.show()
Output
In Snowflake there are not Hive stores, so the Hive Context is not required, You can still use parquet files on Snowflake please check this tutorial to learn how.
#EWI: SPRKPY1008 => pyspark.sql.context.HiveContext is not required
hive_context = sc
df = hive_context.table("myTable")
df.show()
the sc variable refers to a Snow Park Session Object
Recommended fix
For the output code in the example you should add the Snow Park Session Object similar to this code:
# Here manually we can add the Snowpark Session object via a json config file called connection.json
import json
from snowflake.snowpark import Session
jsonFile = open("connection.json")
connection_parameter = json.load(jsonFile)
jsonFile.close()
sc = Session.builder.configs(connection_parameter).getOrCreate()
hive_context = sc
df = hive_context.table("myTable")
df.show()
Additional recommendations
For more support, you can email us at sma-support@snowflake.com or post an issue in the SMA.
Last updated