Links

Release Notes

Updates on each released version of Snowflake SnowConvert for Spark

Release 1.0.306 - February 14, 2023

Scala 0.2.13
SparkSnowConvert Core 1.1.27
New Features
- Jupyter notebooks (.ipynb) processing
- EWI generation when a dependency couldn't be added to the project config file
Improvements
- Lambda scopes opening and closing
Bug Fixes
- Bug 680497: The remaning to full qualified for functions is not working fine
- Bug 681704: Unable to generate final report

Release 1.0.273 - February 2, 2023

Scala 0.2.4
SparkSnowConvert Core 1.1.8.0
Hotfix
- API endpoints update

Release 1.0.263 - January 31, 2023

Scala 0.2.4
SparkSnowConvert Core 1.1.8.0
Added
  • .NET Core 6 Upgrade
  • ElementPackage column added to imports inventory
  • Sizing table added to assessment reports
  • Add conversion percentage in the reports synced with BDS
  • Add issues.csv file in the output
  • Generate SummaryReport.html and DetailedReport.html (mirror docx html) locally on Reports folder
  • Add ConversionStatus keywords to GenericScanner
  • Support full name conversion
Improvements
  • org.apache.spark.mllib mappings added to the core reference table
  • [UI] Fix wording when cancelling the execution
  • [UI] Change UI phase titles
  • Group issues by EWI code
  • Update TOOL_VERSION column value format on Execution info table
  • Simplified the Issue summary table so it is not too big
Bug Fixes
  • Resolved Issue with backslash
  • Resolved BreakLine Issue
  • Resolved Lambda blocks corner case
  • Remove AssessmentReport.html generation (local html report)

Release 1.0.191 - December 27, 2022

Scala 0.1.493
SparkSnowConvert Core 1.0.117.0
Added
  • Uploading packages inventory to cloud telemetry
Improvements
  • Detailed report
    • Minor visual improvements
    • Sorting issue table by:
      • Instances
      • Code
      • Description

Release 1.0.166 - December 21, 2022

Scala 0.1.492
SparkSnowConvert Core 1.0.105.0
Added
  • Added a margin of error description in the detailed report
Improvements
  • Improved sorting of issues table in the detailed report
  • Improved display of percentages in the detailed report
Bug Fixes
  • <#> character is showing issues
  • Compose is not recognized as a keyword
  • Parser is not working on 'join' argument
  • Scala code processor throwing critical error

Release 1.0.132 - December 13, 2022

Scala 0.1.487
SparkSnowConvert Core 1.0.88
Improvements
  • Customer information added to the detailed assessment report
  • Transformation logging messages
Bug fixes
  • An issue with expressions like (a, b) =>val c
  • compose not being recognized as a keyword

Release 1.0.107 - December 7, 2022

Scala 0.1.484
SparkSnowConvert Core 1.0.77
Added
  • Snowpark mappings update to 1.6.2 version
  • Functions without parentheses collection improvements on assessment
  • Maven project (pom.xml) file processing
  • ClassName column renamed to 'alias' on SparkUsagesInventory.pam and ImportUsagesInventory.pam
  • Added margin of error to the readiness score
Fixed
  • Snowpark Python and Scala posted version update
  • Issue with a new line after the name of functions

Release 1.0.59 - November 29, 2022

Scala 0.1.478
SparkSnowConvert Core 1.0.60
Added
  • Basic companion object support
  • org.apache.spark.sql.Column mappings update
  • org.apache.spark.sql.Expression mappings update
  • org.apache.spark.sql.functions mappings update
  • Reference extensions dependency from project config file (SBT)
  • Reference extensions dependency from project config file (Gradle)
Fixed
  • "Script" code is not supported

Release 1.0.17 - November 23, 2022

Scala 0.1.472
SparkSnowConvert Core 1.0.44
Added
  • Spark mappings update
  • Trim "FileId" column value on all .pam files
  • ConversionStatus and scala_spark_mappings_core.csv unification

Release 1.0.1 - November 17, 2022

Scala 0.1.472
SparkSnowConvert Core 1.0.37
Added
  • SparkSession, DataFrameReader, and DataFrameWriter mappings update
  • EWI Generation for unary and binary expressions
Fixed
  • Writer replacer supports csv, parquet, json, and options
  • Reader replacer is not supporting functions without parentheses
  • Writer replacer is not supporting functions without parentheses
  • Currently, the transformation of InsertInto is not a valid code.
  • Writer replacer is not including all functions.

Release 0.1.873 - November 11, 2022

Scala 0.1.468
SparkSnowConvert Core 1.0.23
Added:
  • Symbol resolution for function calls without parentheses
  • Scopes opening/closing exceptions handling(at Replacers)
  • EWI generation for not supported imports (complex cases)
  • EWI generation for not defined imports
  • SparkSession transformation improvements
  • DataFrame reader/writer transformation improvements
  • "Spark Usages by Support Category", "Scala Import Call Summary" sections added to Detailed report
  • RDD mappings update
Fixed:
  • Stack Overflow, output files were not generated
  • Expression without parentheses on Spark Session replacer transformation

Release 0.1.770 - October 21, 2022

Scala 0.1.458
SparkSnowConvert Core 0.1.530
Added:
  • Updated helper/extension .jar to latest version
  • Updated assessment .docx report template
  • Import usages inventory generation
  • Generating EWIs for not supported imports (simple case)
Fixed:
  • Indeterminism issue on SymblTable
  • Error when sorting spark usages inventory files
  • SclSingleExprPath must not contain null members
  • The collection was modified; the enumeration operation may not execute
  • Parsing does not finish when there are multiple closing multi-line in a row
  • Issue with expression
  • Error FileNotGenerated

Release 0.1.705 - October 04, 2022

Scala 0.1.442
SparkSnowConvert Core 0.1.499
Fixed:
  • The setting button is not refreshing when the license is changed.

Release 0.1.702 - September 28, 2022

Scala 0.1.442
SparkSnowConvert Core 0.1.498
Added:
  • Symbol table built ins loading improvements
  • Adding robustness to symbol table loaders
Fixed:
  • Error in the total of Scala files in the AssessmentReport
  • Symbol resolving for generic functions using an asterisk
  • Comments inside comments and id prefix and interpolation parsing error
  • The comma after identifier parsing error
  • Parsing error of the expression when the first statement is taking the pattern of the second statement
  • "and", "::","++" and "or" operators parsing errors

Release 0.1.687 - September 20, 2022

Scala 0.1.430
SparkSnowConvert Core 0.1.491.0
Added
  • Symbol loading/resolving - Add support for generic methods with asterisk params.
  • Symbol loading/resolving - Add type inference for type defs.
  • Symbol loading/resolving general improvements
Fixed
Issue related to the import usages not being stored if there are no Spark references.

Release 0.1.677 - September 15, 2022

Scala 0.1.427
SparkSnowConvert Core 0.1.486.0
Added
  • Cloud telemetry and sending email mechanism now available in Conversion Mode
  • Update contact information in the email template

Release 0.1.653 - September 06, 2022

Scala 0.1.426
SparkSnowConvert Core 0.1.476.0
Added
  • 'SnowConvert Version' and 'Snowpark version' columns to SparkUsagesInventory
  • Improvements to speed analysis

Release 0.1.624 - August 31st, 2022

Scala 0.1.422
SparkSnowConvert Core 0.1.454.0
Added
- Automated and Status columns added to SparkReferenceInventory.csv
- Summary and detailed html report uploading to Snowflake
- Mappings update
Fixed:
- Summary and detailed report wordings fixes
- Email template wording fixes.

Release 0.1.579 - August 23th, 2022

Scala 0.1.421 Spark
SnowConvert Core 0.1.414
Added - Email template update
Adding "Version information" section to Summary report
Adding "Resources" section to Detailed report
Final screen UI changes
Fixed:
Report missing spark functions on sparkUsagesInventory.pam
Detailed report logos update
Percentage values precision on summary and detailed assessment reports

Release 0.1.595 - August 17th, 2022

Scala 0.1.421
SparkSnowConvert Core 0.1.396
Added Spark read and write transformations improvement
Adding session id column to spark usages inventory

Release 0.1.479 - June 30th, 2022

Scala 0.1.411
SparkSnowConvert Core 0.1.279
Added
  • Spark read and write transformations
  • Spark trim, rtrim and ltrim function transformations
  • String interpolation parsing
  • Increasing sql extraction match patterns

Release 0.1.447 - June 14th, 2022

Scala 0.1.402
SparkSnowConvert Core 0.1.274
Added
  • File operations robustness
  • Output folders reorganization
  • SparkSession builder transformation
  • Adding "Scala files with embedded sql" count in assessment reports
Fixed
  • Cyclic dependencies issue on Symbol Table
  • Empty case clause parsing
  • Multiple statements on lambda block parsing
  • Case clause pattern parsing

Release 0.1.380 - June 1st, 2022

Scala 0.1.391
SparkSnowConvert Core 0.1.229
Added
  • Parsing robustness
  • .sbt configuration files processing
  • Issues breakdown section added in assessment html report
  • Look and feel improvements in assessment html report
  • Using RapidScanner inventories to calculate the spark usages assessment
  • macOS CLI & UI support
  • Improvements in import statements mappings

Release 0.1.7 - May 17th, 2022

Scala 0.1.380
Added
  • Scala Parser
    • Double exclamation mark support
  • Conversion tool
    • Sql extraction
    • object_struct function transformation
    • avg function transformation
    • Snowpark extensions .jar update
    • Lines of code report
    • Import mappings
    • Docx and html assessment reports
    • RapidScan integration
    • Linux OS support
Fixed
  • Binary expressions special cases parsing

Release 0.1.3 - March 18th, 2022

Scala 0.1.358
Added
  • Scala Parser
    • Support underscore followed by newline when parsing expressions
    • Improve parsing errors handling
  • Symbols
    • Improve support of Unresolved Symbols
    • Improve creation of Generic Symbols to reuse existing ones
    • Support Loading and Resolution of Lambda Expressions
  • Mappings:
    • Support custom mappings for functions and types via .map files
    • Added custom map directory parameter
Fixed
  • Fill missing columns at notification .pam file.
  • Generate metrics data files (.pam) to specified reports folder

Release 0.1.2 - March 4th, 2022

Scala 0.1.351

Added
  • Updated logos and text in UI and Documentation
  • Symbols
    • Support Generic Identifiers on Type Parameters for Generic Symbol
    • Exclusion of not required dependencies
  • ScalaParser:
    • Backticks idents
    • ArgAssign expressions
Fixed
  • ScalaParser:
    • ExprLambda with ColonType next to ident
    • Try expression when try is not referring a keyword
    • Empty lambda expr with args
    • Underscore ("_") in TypeArgs
    • Files with all commented out source
    • New lines at SimpleExpr, SingleExpr, TailExpr nodes
  • ConversionTool:
    • Fix Crash of conversion due to javap parsing errors (related with jar dependencies)

Release 0.1.1 - February 14th, 2022

Scala 0.1.333

Features
  • Command line interface.
  • Scala code assessment feature.
  • Consume multiple files or single files with multiple objects.
  • Conversion of basic Scala programs as defined by functions and syntax to be mutually agreed during the first 3 development sprints.
  • Comments in Scala code are re-inserted inline.
  • Insert comments in-line with any errors/warning/reviews.
  • Basic reporting including
    • Number of spark elements processed
    • Summary of elements transformed, files and locations of
    • Summary of errors/warnings/reviews encountered.
    • Summary of unsupported Spark APIs
  • Demonstrated inclusion of the following defined scenarios:
    • API mappings
    • Recreate project as SnowPark projects
      • Setup Proper project structure
      • Update to SnowPark supported Scala version
    • Helper Creation to reduce impedance mismatch
    • Define some pattern rewrite
    • Document guidelines for non-automatable concepts (e.g.: file usage patterns, data source configuration, or spark libraries without a direct equivalent, like Kafka stream reading)
  • Greater than 90% successful conversion rate for initial two customer code bases (basis code for the above scenarios) to be provided to Mobilize by Snowflake on the Effective Date.
    • Measured based upon number of compilable objects in Snowflake
    • Objects with unsupported/untranslatable functions not counted
    • Conversion rate for code will be based upon a complete code base containing all dependent objects.
    • Snowflake will provide access to all available private preview features for Mobilize development benefit