Release Notes

Updates on each released version of Snowflake SnowConvert for Spark

Release 1.0.306 - February 14, 2023

Scala 0.2.13
SparkSnowConvert Core 1.1.27
New Features
- Jupyter notebooks (.ipynb) processing
- EWI generation when a dependency couldn't be added to the project config file
- Lambda scopes opening and closing
Bug Fixes
- Bug 680497: The remaning to full qualified for functions is not working fine
- Bug 681704: Unable to generate final report

Release 1.0.273 - February 2, 2023

Scala 0.2.4
SparkSnowConvert Core
- API endpoints update

Release 1.0.263 - January 31, 2023

Scala 0.2.4
SparkSnowConvert Core
  • .NET Core 6 Upgrade
  • ElementPackage column added to imports inventory
  • Sizing table added to assessment reports
  • Add conversion percentage in the reports synced with BDS
  • Add issues.csv file in the output
  • Generate SummaryReport.html and DetailedReport.html (mirror docx html) locally on Reports folder
  • Add ConversionStatus keywords to GenericScanner
  • Support full name conversion
  • org.apache.spark.mllib mappings added to the core reference table
  • [UI] Fix wording when cancelling the execution
  • [UI] Change UI phase titles
  • Group issues by EWI code
  • Update TOOL_VERSION column value format on Execution info table
  • Simplified the Issue summary table so it is not too big
Bug Fixes
  • Resolved Issue with backslash
  • Resolved BreakLine Issue
  • Resolved Lambda blocks corner case
  • Remove AssessmentReport.html generation (local html report)

Release 1.0.191 - December 27, 2022

Scala 0.1.493
SparkSnowConvert Core
  • Uploading packages inventory to cloud telemetry
  • Detailed report
    • Minor visual improvements
    • Sorting issue table by:
      • Instances
      • Code
      • Description

Release 1.0.166 - December 21, 2022

Scala 0.1.492
SparkSnowConvert Core
  • Added a margin of error description in the detailed report
  • Improved sorting of issues table in the detailed report
  • Improved display of percentages in the detailed report
Bug Fixes
  • <#> character is showing issues
  • Compose is not recognized as a keyword
  • Parser is not working on 'join' argument
  • Scala code processor throwing critical error

Release 1.0.132 - December 13, 2022

Scala 0.1.487
SparkSnowConvert Core 1.0.88
  • Customer information added to the detailed assessment report
  • Transformation logging messages
Bug fixes
  • An issue with expressions like (a, b) =>val c
  • compose not being recognized as a keyword

Release 1.0.107 - December 7, 2022

Scala 0.1.484
SparkSnowConvert Core 1.0.77
  • Snowpark mappings update to 1.6.2 version
  • Functions without parentheses collection improvements on assessment
  • Maven project (pom.xml) file processing
  • ClassName column renamed to 'alias' on SparkUsagesInventory.pam and ImportUsagesInventory.pam
  • Added margin of error to the readiness score
  • Snowpark Python and Scala posted version update
  • Issue with a new line after the name of functions

Release 1.0.59 - November 29, 2022

Scala 0.1.478
SparkSnowConvert Core 1.0.60
  • Basic companion object support
  • org.apache.spark.sql.Column mappings update
  • org.apache.spark.sql.Expression mappings update
  • org.apache.spark.sql.functions mappings update
  • Reference extensions dependency from project config file (SBT)
  • Reference extensions dependency from project config file (Gradle)
  • "Script" code is not supported

Release 1.0.17 - November 23, 2022

Scala 0.1.472
SparkSnowConvert Core 1.0.44
  • Spark mappings update
  • Trim "FileId" column value on all .pam files
  • ConversionStatus and scala_spark_mappings_core.csv unification

Release 1.0.1 - November 17, 2022

Scala 0.1.472
SparkSnowConvert Core 1.0.37
  • SparkSession, DataFrameReader, and DataFrameWriter mappings update
  • EWI Generation for unary and binary expressions
  • Writer replacer supports csv, parquet, json, and options
  • Reader replacer is not supporting functions without parentheses
  • Writer replacer is not supporting functions without parentheses
  • Currently, the transformation of InsertInto is not a valid code.
  • Writer replacer is not including all functions.

Release 0.1.873 - November 11, 2022

Scala 0.1.468
SparkSnowConvert Core 1.0.23
  • Symbol resolution for function calls without parentheses
  • Scopes opening/closing exceptions handling(at Replacers)
  • EWI generation for not supported imports (complex cases)
  • EWI generation for not defined imports
  • SparkSession transformation improvements
  • DataFrame reader/writer transformation improvements
  • "Spark Usages by Support Category", "Scala Import Call Summary" sections added to Detailed report
  • RDD mappings update
  • Stack Overflow, output files were not generated
  • Expression without parentheses on Spark Session replacer transformation

Release 0.1.770 - October 21, 2022

Scala 0.1.458
SparkSnowConvert Core 0.1.530
  • Updated helper/extension .jar to latest version
  • Updated assessment .docx report template
  • Import usages inventory generation
  • Generating EWIs for not supported imports (simple case)
  • Indeterminism issue on SymblTable
  • Error when sorting spark usages inventory files
  • SclSingleExprPath must not contain null members
  • The collection was modified; the enumeration operation may not execute
  • Parsing does not finish when there are multiple closing multi-line in a row
  • Issue with expression
  • Error FileNotGenerated

Release 0.1.705 - October 04, 2022

Scala 0.1.442
SparkSnowConvert Core 0.1.499
  • The setting button is not refreshing when the license is changed.

Release 0.1.702 - September 28, 2022

Scala 0.1.442
SparkSnowConvert Core 0.1.498
  • Symbol table built ins loading improvements
  • Adding robustness to symbol table loaders
  • Error in the total of Scala files in the AssessmentReport
  • Symbol resolving for generic functions using an asterisk
  • Comments inside comments and id prefix and interpolation parsing error
  • The comma after identifier parsing error
  • Parsing error of the expression when the first statement is taking the pattern of the second statement
  • "and", "::","++" and "or" operators parsing errors

Release 0.1.687 - September 20, 2022

Scala 0.1.430
SparkSnowConvert Core 0.1.491.0
  • Symbol loading/resolving - Add support for generic methods with asterisk params.
  • Symbol loading/resolving - Add type inference for type defs.
  • Symbol loading/resolving general improvements
Issue related to the import usages not being stored if there are no Spark references.

Release 0.1.677 - September 15, 2022

Scala 0.1.427
SparkSnowConvert Core 0.1.486.0
  • Cloud telemetry and sending email mechanism now available in Conversion Mode
  • Update contact information in the email template

Release 0.1.653 - September 06, 2022

Scala 0.1.426
SparkSnowConvert Core 0.1.476.0
  • 'SnowConvert Version' and 'Snowpark version' columns to SparkUsagesInventory
  • Improvements to speed analysis

Release 0.1.624 - August 31st, 2022

Scala 0.1.422
SparkSnowConvert Core 0.1.454.0
- Automated and Status columns added to SparkReferenceInventory.csv
- Summary and detailed html report uploading to Snowflake
- Mappings update
- Summary and detailed report wordings fixes
- Email template wording fixes.

Release 0.1.579 - August 23th, 2022

Scala 0.1.421 Spark
SnowConvert Core 0.1.414
Added - Email template update
Adding "Version information" section to Summary report
Adding "Resources" section to Detailed report
Final screen UI changes
Report missing spark functions on sparkUsagesInventory.pam
Detailed report logos update
Percentage values precision on summary and detailed assessment reports

Release 0.1.595 - August 17th, 2022

Scala 0.1.421
SparkSnowConvert Core 0.1.396
Added Spark read and write transformations improvement
Adding session id column to spark usages inventory

Release 0.1.479 - June 30th, 2022

Scala 0.1.411
SparkSnowConvert Core 0.1.279
  • Spark read and write transformations
  • Spark trim, rtrim and ltrim function transformations
  • String interpolation parsing
  • Increasing sql extraction match patterns

Release 0.1.447 - June 14th, 2022

Scala 0.1.402
SparkSnowConvert Core 0.1.274
  • File operations robustness
  • Output folders reorganization
  • SparkSession builder transformation
  • Adding "Scala files with embedded sql" count in assessment reports
  • Cyclic dependencies issue on Symbol Table
  • Empty case clause parsing
  • Multiple statements on lambda block parsing
  • Case clause pattern parsing

Release 0.1.380 - June 1st, 2022

Scala 0.1.391
SparkSnowConvert Core 0.1.229
  • Parsing robustness
  • .sbt configuration files processing
  • Issues breakdown section added in assessment html report
  • Look and feel improvements in assessment html report
  • Using RapidScanner inventories to calculate the spark usages assessment
  • macOS CLI & UI support
  • Improvements in import statements mappings

Release 0.1.7 - May 17th, 2022

Scala 0.1.380
  • Scala Parser
    • Double exclamation mark support
  • Conversion tool
    • Sql extraction
    • object_struct function transformation
    • avg function transformation
    • Snowpark extensions .jar update
    • Lines of code report
    • Import mappings
    • Docx and html assessment reports
    • RapidScan integration
    • Linux OS support
  • Binary expressions special cases parsing

Release 0.1.3 - March 18th, 2022

Scala 0.1.358
  • Scala Parser
    • Support underscore followed by newline when parsing expressions
    • Improve parsing errors handling
  • Symbols
    • Improve support of Unresolved Symbols
    • Improve creation of Generic Symbols to reuse existing ones
    • Support Loading and Resolution of Lambda Expressions
  • Mappings:
    • Support custom mappings for functions and types via .map files
    • Added custom map directory parameter
  • Fill missing columns at notification .pam file.
  • Generate metrics data files (.pam) to specified reports folder

Release 0.1.2 - March 4th, 2022

Scala 0.1.351

  • Updated logos and text in UI and Documentation
  • Symbols
    • Support Generic Identifiers on Type Parameters for Generic Symbol
    • Exclusion of not required dependencies
  • ScalaParser:
    • Backticks idents
    • ArgAssign expressions
  • ScalaParser:
    • ExprLambda with ColonType next to ident
    • Try expression when try is not referring a keyword
    • Empty lambda expr with args
    • Underscore ("_") in TypeArgs
    • Files with all commented out source
    • New lines at SimpleExpr, SingleExpr, TailExpr nodes
  • ConversionTool:
    • Fix Crash of conversion due to javap parsing errors (related with jar dependencies)

Release 0.1.1 - February 14th, 2022

Scala 0.1.333

  • Command line interface.
  • Scala code assessment feature.
  • Consume multiple files or single files with multiple objects.
  • Conversion of basic Scala programs as defined by functions and syntax to be mutually agreed during the first 3 development sprints.
  • Comments in Scala code are re-inserted inline.
  • Insert comments in-line with any errors/warning/reviews.
  • Basic reporting including
    • Number of spark elements processed
    • Summary of elements transformed, files and locations of
    • Summary of errors/warnings/reviews encountered.
    • Summary of unsupported Spark APIs
  • Demonstrated inclusion of the following defined scenarios:
    • API mappings
    • Recreate project as SnowPark projects
      • Setup Proper project structure
      • Update to SnowPark supported Scala version
    • Helper Creation to reduce impedance mismatch
    • Define some pattern rewrite
    • Document guidelines for non-automatable concepts (e.g.: file usage patterns, data source configuration, or spark libraries without a direct equivalent, like Kafka stream reading)
  • Greater than 90% successful conversion rate for initial two customer code bases (basis code for the above scenarios) to be provided to Mobilize by Snowflake on the Effective Date.
    • Measured based upon number of compilable objects in Snowflake
    • Objects with unsupported/untranslatable functions not counted
    • Conversion rate for code will be based upon a complete code base containing all dependent objects.
    • Snowflake will provide access to all available private preview features for Mobilize development benefit