Data Migration

To migrate the data of your tables, you must have your S3 bucket in the same region as your Redshift cluster. Data migration with S3 buckets in regions other than the provided Redshift clusters will be added in the future.

Description

SnowConvert migrates the data of your Redshift tables by unloading it to PARQUET files in an S3 bucket that you must provide. After the files are created, the application will copy the data directly from those files to the tables deployed in Snowflake.

SnowConvert for Redshift Data Migration Process.

Prerequisites

Before executing the migration of your data you need the following prerequisites:

1. Have an S3 bucket in AWS in the same region as your Redshift cluster.

2. You need to create an IAM role associated with your Redshift cluster that can unload the data of your Redshift tables into your S3 bucket. The IAM role must have the following policy configuration (the following configuration can be used by any database user, to restrict access to this role you can read this guide):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<your_bucket_name>/*",
                "arn:aws:s3:::<your_bucket_name>"
            ]
        }
    ]
}

1. Have an IAM user that can read and delete objects of your S3 bucket, this is necessary to read the data from the files that were created in the S3 to the Snowflake tables. Here is an example of an IAM policy that can be used to load data from the S3 files into the Snowflake target tables:

If you don't provide the s3:DeleteObject and the s3:DeleteObectVersion permission to your IAM user the data migration process will not fail, but the data files that were created by the tool will not be deleted from the S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject", // DeleteObject and DeleteObjectVersion permissions are necessary to purge the data files once they are loaded.
                "s3:DeleteObjectVersion",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<your_bucket_name>/*",
                "arn:aws:s3:::<your_bucket_name>"
            ]
        }
    ]
}

1. Be connected to the Redshift cluster and the Snowflake account where the DDL code was deployed.

Execution

Ensure that the S3 bucket path you enter does not have any files, the process will fail if there are files in the given path.

1. Click on Set S3 Bucket Settings to add the following information:

   1. S3 Bucket URL (Ensure the URL you entered finishes with a “/“).

   2. IAM Role ARN to unload data from the tables into PARQUET files in the S3 bucket URL you provided.

   3. The access key of the IAM user that has permission to read and delete objects in the S3 bucket objects.

   4. The secret access key of the IAM user that has permission to read and delete objects in the S3 bucket objects.

   

   S3 Bucket Settings for Data Migration
2. Select the tables for which you want to have their data migrated to Snowflake.

3. Click on Migrate Data, this will start the data migration process by unloading the data into the S3 bucket and then copying the data from those files to the target tables in Snowflake.

4. The data migration column will be updated indicating if each table had their data migrated successfully or not.

Tables After Data Migration Process

Results

1. This page validates the number of rows moved from the source tables to the target tables.

2. Each row contains the following information about the migrated table: source schema and table name, target schema and table name, and the number of rows loaded.

3. If you want to execute another data migration process for more tables you can click on Go Back to Data Migration.

Data Migration Process Results