Tonic process overview for Amazon Redshift
The following high-level diagram describes how Tonic orchestrates the processing and moving of data in Redshift.
Process flow for Redshift data
Tonic orchestrates the moving and transforming of data between Redshift databases. Tonic uses the S3, SQS, and Lambda services in AWS to accomplish this task.
Tonic manages the lifetimes of data and resources used in AWS. It only requires that the necessary permissions are assigned to the IAM role that Tonic uses.
At a high level, the process is:
- 1.Create an AWS Lambda function for your version of Tonic. This step is performed once per version of Tonic. The Lambda function is created when you run your first job after a new installation or after a version upgrade.
- 2.Create AWS SQS queue and S3 Event Triggers. This is done once per job. The resource names are scoped to your specific generation job.
- 3.COPY table data into S3. The S3 bucket path is specific in the Tonic UI.
- 4.As files land in S3, S3 Event notifications place messages in SQS. Messages in SQS trigger Lambda function invocations. By default, each file placed in S3 has a maximum file size of 50MB. Each Lambda invocation processes a single file. Lambda processes each file and then writes them back to S3 in a different location, which the user also specifies in the Tonic application.
- 5.Once all files for a table are processed, Tonic copies data back into Redshift, into the destination database.
- 6.Once all tables are processed, Tonic removes ephemeral AWS components such as SQS and S3 Event Notifications.