Data generation process
Last updated
Last updated
The following diagram provides an overview of how Tonic Structural produces destination data during data generation.
To start, Structural analyzes the destination database to understand all of the database objects, including the schemas, tables, views, columns, and constraints.
For tables that use Preserve Destination or Incremental mode, Structural creates a copy of those tables in a temporary schema. The temporary schema holds those tables outside of the teardown process.
To remove all of the destination database objects, Structural issues logically ordered DROP and DELETE statements to the destination database. It continues to issue the statements until the destination database is empty.
After it tears down the destination database, Structural next analyzes the source database to understand all of the affected database objects.
In a logical order, Structural dumps the source database schema into pre-data and post-data schemas.
Pre-data schema - Schemas, tables, and columns with data types. The pre-data schema does not include keys or constraints.
Post-data schema - Other database elements, including keys, constraints, indexes, and views.
During data generation, Structural stores the pre-data and post-data schemas as temporary files.
Structural then applies the pre-data schema to the destination database.
Next, Structural de-identifies the source data, and uses the de-identified data to populate the destination database.
Structural processes the tables in the following order:
Tables that directly have generators applied to them.
Tables that have foreign key dependencies on those de-identified tables.
Tables for which no data is de-identified, also referred to as Passthrough tables.
For all of these tables, Structural:
Reads the data from the source database.
De-identifies the data based on the workspace configuration.
Writes the de-identified data to the destination database.
For tables that use Incremental mode, Structural copies the data back to the destination database. To understand the data to copy, Structural executes queries from the destination database to the source database.
After Structural copies all of the required data for a destination table, it applies the post-data schema elements.
Structural performs this process in an order that ensures that all of the dependencies for a database object are in place before Structural attempts to create the object.
Structural then updates any sequences that were created to ensure that they have the most accurate data.
For the Preserve Destination tables, Structural restores the data from the temporary schema. It then removes the temporary schema.
When the data generation job is complete, Structural runs any post-job tasks, including:
Webhooks
Post-job scripts
