SPRKPY1015

pyspark.sql.functions.atanh

Message: pyspark.sql.functions.atanh has a workaround

Category: Warning.

Description

This issue appears when the tool detects the usage of which has a workaround.

Scenario

Input

On this example pyspark calculates the atanh for a dataframe by using .

from pyspark.sql import SparkSession
from pyspark.sql.functions import atanh
spark = SparkSession.builder.getOrCreate()
data = [['V1', 0.14],
        ['V2', 0.32],
        ['V3', 0.4],
        ['V4', -0.36]]

columns = ['Paremeter', 'value']
df = spark.createDataFrame(data, columns)
df_result = df.withColumn("atanh_value", atanh(df["value"]))

Output

SMA returns the EWI SPRKPY1015 over the line where atanh is used, so you can use to identify where to fix.

from snowflake.snowpark import Session

spark = Session.builder.getOrCreate()
spark.update_query_tag({"origin":"sf_sit","name":"sma","version":{"major":0,"minor":0,"patch":0},"attributes":{"language":"Python"}})
data = [['V1', 0.14],
        ['V2', 0.32],
        ['V3', 0.4],
        ['V4', -0.36]]

columns = ['Paremeter', 'value']
df = spark.createDataFrame(data, columns)
#EWI: SPRKPY1015 => pyspark.sql.functions.atanh has a workaround, see documentation for more info
df_result = df.withColumn("atanh_value", atanh(df["value"]))

Recommended fix

import snowflake.snowpark as snowpark
from snowflake.snowpark import Session
from snowflake.snowpark.functions import call_function, col

spark = Session.builder.getOrCreate()
spark.update_query_tag({"origin":"sf_sit","name":"sma","version":{"major":0,"minor":0,"patch":0},"attributes":{"language":"Python"}})
data = [['V1', 0.14],
        ['V2', 0.32],
        ['V3', 0.4],
        ['V4', -0.36]]

columns = ['Paremeter', 'value']
df = spark.createDataFrame(data, columns)
df_result = df.select(call_function('atanh', col('value')))

Additional recommendations

PreviousSPRKPY1014 NextSPRKPY1016

Last updated 6 months ago

from pyspark.sql import SparkSession from pyspark.sql.functions import atanh spark = SparkSession.builder.getOrCreate() data = [['V1', 0.14], ['V2', 0.32], ['V3', 0.4], ['V4', -0.36]] columns = ['Paremeter', 'value'] df = spark.createDataFrame(data, columns) df_result = df.withColumn("atanh_value", atanh(df["value"]))

from snowflake.snowpark import Session spark = Session.builder.getOrCreate() spark.update_query_tag({"origin":"sf_sit","name":"sma","version":{"major":0,"minor":0,"patch":0},"attributes":{"language":"Python"}}) data = [['V1', 0.14], ['V2', 0.32], ['V3', 0.4], ['V4', -0.36]] columns = ['Paremeter', 'value'] df = spark.createDataFrame(data, columns) #EWI: SPRKPY1015 => pyspark.sql.functions.atanh has a workaround, see documentation for more info df_result = df.withColumn("atanh_value", atanh(df["value"]))

import snowflake.snowpark as snowpark from snowflake.snowpark import Session from snowflake.snowpark.functions import call_function, col spark = Session.builder.getOrCreate() spark.update_query_tag({"origin":"sf_sit","name":"sma","version":{"major":0,"minor":0,"patch":0},"attributes":{"language":"Python"}}) data = [['V1', 0.14], ['V2', 0.32], ['V3', 0.4], ['V4', -0.36]] columns = ['Paremeter', 'value'] df = spark.createDataFrame(data, columns) df_result = df.select(call_function('atanh', col('value')))