Amazon Redshift

Overview
Tonic supports moving data from one database to another within a single Redshift instance and can also move data between Redshift instances.  In both situations, Tonic uses S3 as an intermediate stage to host both real and masked data.
Diagram of the Tonic ETL process
Below is a high level diagram describing how Tonic orchestrates the processing and moving of data in Redshift.  This diagram should not be confused with Tonics architectural diagram.
Tonic orchestrates the moving and transforming of data between Redshift databases.  AWS services S3, SQS, and Lambda are used to accomplish this task.  Tonic manages the lifetimes of data and resources used in AWS and only requires the necessary permissions be assigned to the IAM role being used by Tonic.  The process at a high level is:
1.Create an AWS Lambda function for your version of Tonic.  This step is performed once per version of Tonic.  AWS Lambda function creation happens when you run your first job after a new installation or after a version upgrade.
2.Create AWS SQS queue and S3 Event Triggers.  This is done once per job and the resource names are scoped to your specific generation job.
3.COPY table data into S3.  The S3 bucket path is specific in the Tonic UI.
4.As files land in S3 this causes S3 Event notifications to place messages in SQS.  Messages in SQS trigger Lambda function invocations.  By default, each file placed in S3 has a maximum file size of 50MB and each lambda invocation will process a single file.  Lambda processes each file and writes them back to S3 in a different location, also specified by user in the UI.
5.Once all files for a table are processed Tonic copies data back into Redshift, into the destination database.
6.Once all tables are processed Tonic removes ephemeral AWS components such as SQS and S3 Event Notifications are removed.
Limitations
Redshift support includes all Standard Tonic features with the exception of Subsetting which is not currently supported.  If you'd like to use Subsetting in conjunction with Redshift please reach out to [email protected]
​

Working across AWS accounts

Initial Tonic setup

Last modified 19d ago

Copy link

Contents

Overview

Diagram of the Tonic ETL process

Limitations