Tonic
Search…
Overview
Quick Start Guide
Release Notes
The Tonic Platform
Workspaces
Workspace Menu
Privacy Hub
Database View
Table View
Foreign Keys
Jobs
Schema Changes
Subsetting
Post-Job Scripts
Webhooks
Concepts
Generators
Consistency
Linking Generators
Smart Linking
Primary Key Generators
Table Modes
Workspace Sharing
Differential Privacy
On-premises Deployment Instructions
Sharing Logs and Telemetry with Tonic
API Documentation
Frequently Asked Questions
Performance
Database Specific Setup
Databricks
EMR
PostgreSQL
SQL Server
MySQL / MariaDB
Google BigQuery
Oracle
Snowflake on AWS
Amazon Redshift
Initial Tonic Setup
Connecting to Redshift
Redshift specific configuration options
Powered By GitBook
Amazon Redshift

Overview

Tonic supports moving data from one database to another within a single Redshift instance and can also move data between Redshift instances. In both situations, Tonic uses S3 as an intermediate stage to host both real and masked data. Below you can find a diagram describing what the Tonic ETL process looks like on Redshift.

Diagram

Below is a high level diagram describing how Tonic orchestrates the processing and moving of data in Redshift. This diagram should not be confused with Tonics architectural diagram.
Tonic orchestrates the moving and transforming of data between Redshift databases. AWS services S3, SQS, and Lambda are used to accomplish this task. Tonic manages the lifetimes of data and resources used in AWS and only requires the necessary permissions be assigned to the IAM role being used by Tonic. The process at a high level is:
    1.
    Create an AWS Lambda function for your version of Tonic. This step is performed once per version of Tonic. AWS Lambda function creation happens when you run your first job after a new installation or after a version upgrade.
    2.
    Create AWS SQS queue and S3 Event Triggers. This is done once per job and the resource names are scoped to your specific generation job.
    3.
    COPY table data into S3. The S3 bucket path is specific in the Tonic UI.
    4.
    As files land in S3 this causes S3 Event notifications to place messages in SQS. Messages in SQS trigger Lambda function invocations. By default, each file placed in S3 has a maximum file size of 50MB and each lambda invocation will process a single file. Lambda processes each file and writes them back to S3 in a different location, also specified by user in the UI.
    5.
    Once all files for a table are processed Tonic copies data back into Redshift, into the destination database.
    6.
    Once all tables are processed Tonic removes ephemeral AWS components such as SQS and S3 Event Notifications are removed.

Limitations

Redshift support includes all Standard Tonic features with the exception of Subsetting which is not currently supported. If you'd like to use Subsetting in conjunction with Redshift please reach out to [email protected]
​
Previous
Snowflake specific configuration options
Next
Initial Tonic Setup
Last modified 3mo ago
Copy link
Contents
Overview
Limitations