SPRKSCL1111
org.apache.spark.sql.types.DataTypes.CreateDecimalType
Message: CreateDecimalType is not supported.
Category: Conversion error.
Description
This issue appears when the SMA detects a usage org.apache.spark.sql.types.DataTypes.CreateDecimalType function.
Scenario
Input
Below is an example of usage of org.apache.spark.sql.types.DataTypes.CreateDecimalType function.
var result = DataTypes.createDecimalType(18, 8)
Output
The SMA adds the EWI SPRKSCL1111
to the output code to let you know that CreateDecimalType function is not supported by Snowpark.
/*EWI: SPRKSCL1111 => CreateDecimalType is not supported*/
var result = createDecimalType(18, 8)
Recommended fix
There is not a recommended fix yet.
Message: Spark Session builder option is not supported.
Category: Conversion Error.
Description
This issue appears when the SMA detects a use of the org.apache.spark.sql.SparkSession.Builder.config function, which is setting an option of the Spark Session and it is not supported by Snowpark.
Scenario
Input
Below is an example of the org.apache.spark.sql.SparkSession.Builder.config
function used to set an option in the Spark Session.
val spark = SparkSession.builder()
.master("local")
.appName("testApp")
.config("spark.sql.broadcastTimeout", "3600")
.getOrCreate()
Output
The SMA adds the EWI SPRKSCL1104
to the output code to let you know config method is not supported by Snowpark. Then, it is not possible to set options in the Spark Session via config function and it might affects the migration of the Spark Session statement.
val spark = Session.builder.configFile("connection.properties")
/*EWI: SPRKSCL1104 => SparkBuilder Option is not supported .config("spark.sql.broadcastTimeout", "3600")*/
.create()
Recommended fix
To create the session is require to add the proper Snowflake Snowpark configuration.
In this example a configs variable is used.
val configs = Map (
"URL" -> "https://<myAccount>.snowflakecomputing.com:<port>",
"USER" -> <myUserName>,
"PASSWORD" -> <myPassword>,
"ROLE" -> <myRole>,
"WAREHOUSE" -> <myWarehouse>,
"DB" -> <myDatabase>,
"SCHEMA" -> <mySchema>
)
val session = Session.builder.configs(configs).create
Also is recommended the use of a configFile (profile.properties) with the connection information:
# profile.properties file (a text file)
URL = https://<account_identifier>.snowflakecomputing.com
USER = <username>
PRIVATEKEY = <unencrypted_private_key_from_the_private_key_file>
ROLE = <role_name>
WAREHOUSE = <warehouse_name>
DB = <database_name>
SCHEMA = <schema_name>
And with the Session.builder.configFile
the session can be created:
val session = Session.builder.configFile("/path/to/properties/file").create
Additional recommendations
For more support, you can email us at sma-support@snowflake.com or post an issue in the SMA.
Last updated