Types of data generation processes

Simple data generation

In the simplest type of data generation, Tonic Structural uses the configured table modes and generators to transform data in the source database and write the transformed data to the destination location. The destination location is usually a database server, but might also be:

• A storage location such as an S3 bucket

• A container repository

• A Tonic Ephemeral snapshot

For a file connector workspace, the data generation job uses the configured generators for each file group to transform the data in the source files. The transformed data is used to create output files that correspond to the source files.

Simple data generation process

Subsetting data generation

When subsetting is enabled, Structural first identifies the tables and rows to include in the subset. It uses the configured table modes and generators to transform the data. It then writes the transformed data to the destination location.

Data generation process with subsetting

Upsert data generation

Required license: Professional or Enterprise

When upsert is enabled, Structural runs a data generation job that writes the transformed data to an intermediate database. The data generation can include subsetting.

After the initial data generation, Structural runs an upsert job to add or update the appropriate records from the intermediate database to the destination database. The upsert job only adds and updates records. It does not remove any records from previous data generation jobs.

Data generation process with upsert

Before Structural can run an upsert job, the destination database must already exist and have the correct schema defined. To initialize the destination database:

1. Disable upsert.

2. Run a regular data generation.

3. Re-enable upsert.