Running a data generation job
Required workspace permission: Run data generation
The data generation job uses the configured tables modes and generators to transform the data in the source database or source files. The transformed data is used to create the destination database or to write transformed files to file storage.
In the simplest type of data generation, Tonic uses the configured table modes and generators to transform data in the source database and write the transformed data to the destination location. The destination location is usually a database, but might also be a storage location such as an S3 bucket.
For a file connector workspace, the data generation job uses the configured generators for each file group to transform the data in the source files. The transformed data is used to create output files that correspond to the source files.
Simple data generation process
When subsetting is enabled, Tonic first identifies the tables and rows to include in the subset. It uses the configured table modes and generators to transform the data. It then writes the transformed data to the destination location.
Data generation process with subsetting
Required license: Professional or Enterprise
Upsert is currently a beta feature
When upsert is enabled, Tonic runs a data generation job that writes the transformed data to an intermediate database. The data generation can include subsetting.
After the initial data generation, Tonic runs an upsert job to add or update the appropriate records from the intermediate database to the destination database. The upsert job only adds and updates records. It does not remove any records from previous data generation jobs.
Data generation process with upsert
Before Tonic can run an upsert job, the destination database must already exist and have the correct schema defined. To initialize the destination database:
- 1.Disable upsert.
- 2.Run a regular data generation.
- 3.Re-enable upsert.
You use the Generate Data button at the top right of the workspace management view to start the data generation.
As you configure the data generation options, Tonic runs checks to verify that you can use the current configuration to generate data.
If any of these checks do not pass, then when you click Generate Data, Tonic displays information about why you cannot run the data generation job.
If all of those checks pass, then when you click Generate Data, if there are no warnings, the Confirm Generation panel displays.
Data generation is always blocked by conflicting schema changes.
The workspace configuration includes whether to block data generation for all schema changes, including non-conflicting changes.
If this setting is turned off, then if there are non-conflicting schema changes, when you click Generate Data, a warning displays. Non-conflicting schema changes include new tables and columns. If the new columns contain sensitive data, then if you do not assign generators before you generate data, that sensitive data will be in the destination database.
If you are sure that the data in the new tables and columns is not sensitive, then to continue to the Confirm Generation panel, click Continue to Data Generation.
The Confirm Generation panel allows you to confirm the details for the data generation. If subsetting is configured, you can determine whether to generate the subset. Tonic can also provide tips on how to improve the data generation performance.
Confirm Generation panel
If you configured subsetting, then you can indicate whether to only generate the subset.
To create a subset based on the current subsetting configuration, toggle Use Subsetting to the on position.
The initial setting matches the current setting in the subsetting configuration. If Use subsetting is turned on on the Subsetting view, then it is on by default on the Generation Confirmation panel.
When you change the setting on the generation confirmation panel, it also updates the setting on the Subsetting view.
If upsert is enabled for the workspace, then you can also determine whether to use upsert for data generation.
If upsert is enabled for the workspace, then by default Use Upsert is in the on position.
Confirm Generation with upsert option
To not use upsert, toggle Use Upsert to the off position. When upsert is turned off, the data generation is a simple data generation that directly populates and replaces the destination database.
Tonic.ai has released an improved version of the data generation process. We are enrolling Tonic instances in the new process. Tonic.ai will contact you before we enroll your instance.
After your instance is enrolled, your PostgreSQL workspaces always use the new data generation process. For the new process, the job type is Data Pipeline Generation instead of Data Generation.
If your instance is not yet enrolled, then on the Confirm Generation panel, to use the new data generation process, toggle Data Pipeline V2 to the on position.
When upsert is enabled, the Confirm Generation panel provides access to the connection information for the intermediate database. To display the intermediate database connection details, click Intermediate Upsert Database.
If the intermediate database information is incorrect, to navigate to the workspace configuration view to make updates, click Edit Intermediate.
The Confirm Generation panel provides the destination information for the workspace. To display the destination database connection details, click Destination Database.
This is the connection information for the destination database or a storage location such as an S3 bucket.
If the destination information is incorrect, to navigate to the workspace configuration view to make updates, click Edit Destination.
For a file connector workspace, if the source files came from a local file system, then the destination files are written to the large file store in the Tonic application database. You can download the most recently generated files.
For data generation, assigning Truncate table mode to tables that you don't need data for can improve generation performance.
For subsetting, if an upstream table is very large, and the foreign key columns are not indexed, then it can make the subsetting process run more slowly.
The Want faster generations? message displays at the bottom of the Confirm Generation panel. It displays for all non-subsetting jobs. For subsetting jobs, the panel only displays if Tonic identified columns that you should consider indexing.
To display information about tips for faster generation, click Generation Tips.
On the Generation Tips panel for subsetting jobs, the Add Indexes panel displays the first few columns that you might consider indexing.
Generation Tips panel with indexing suggestions
To display a panel with a suggested SQL command to add the index, click the information icon next to the column.
Example SQL command for indexing
On the panel, to copy the command to the clipboard, click Copy SQL to Clipboard.
If there are additional columns that are not listed, then to display the full list of columns to index, click Show all columns.
Full list of indexing suggestions
On the full list, to download the list to a CSV file, click Download list of columns (.csv).
On the Generation Tips panel for non-subsetting jobs, the Truncate Tables panel displays the hint to truncate tables that contain data that you do not need in the destination database.
To navigate to Database View to change the current configuration, click Go to Database View.
On the Confirm Generation panel, after you confirm the generation details, to start the data generation, click Run Generation.
When upsert is enabled, to start the data generation and upsert jobs:
- 1.Click Run Generation + Upsert.
- 2.In the menu, click Run Generation + Upsert.
Upsert generation options
Tonic displays a notification that the job has started. To track the progress of the data generation job and view the results, click the View Job button on the notification, or go to the Job History view.
If upsert is enabled for a workspace, then on the Confirm Generation panel, the more common option is to run both data generation and upsert.
After you run at least one successful data generation to the intermediate database, then you can also choose to run only the upsert process.
For example, if the data generation succeeds but the upsert process fails, then after you address the issues that caused the upsert to fail, you can run the upsert process again.
You also must start the upsert job manually if you turn off Automatically Start Upsert After Successful Data Generation in the workspace settings.
From the Confirm Generation panel, to run upsert only:
- 1.Click the Run Generation + Upsert button.
- 2.In the menu, click Run Upsert Only.
When you run upsert only, the process uses the results of the most recent data generation.
The following issues prevent a data generation or subsetting job.
Issue | Description | How to resolve |
|---|---|---|
Tonic license expired | The current Tonic license is expired. | |
No workspace configured | The Tonic instance does not have any workspaces to generate data from. | |
Insufficient workspace permissions | You do not have permission to run data generation on this workspace. |
Issue | Description | How to resolve |
|---|---|---|
Unable to connect to the source database | The source database connection is either missing or incomplete. | Edit the workspace configuration to complete the source database connection.
Make sure to test the connection. |
Unable to connect to the destination database | The destination database connection is either missing or incomplete. | Edit the workspace configuration to complete the destination database connection.
Make sure to test the connection. |
Issue | Description | How to resolve |
|---|---|---|
Scale mode - Invalid generators | A table that uses Scale mode has columns with assigned generators that are not valid for Scale mode. | Change the selected generator for the columns. |
Scale mode - Passthrough generator or sub-generator for the Conditional generator | A table that uses Scale mode has columns that use the Conditional generator, and that are assigned Passthrough as a generator or sub-generator. | Change the selected generator, sub-generator, or default generator. |
Preserve Destination - cannot resolve foreign key references | A table that uses Preserve Destination mode is referenced from another table. | |
Truncate - cannot resolve foreign key references | A table that uses Truncate mode is referenced from another table. | |
Cross Table Sum - incomplete generation | A column is assigned the Cross Table Sum generator, but some required configuration field values are missing. | |
Incremental mode - circular foreign key dependency | There is a circular foreign key dependency between tables that use Incremental mode. | |
AI Synthesizer configured when it is disabled | A table is configured to use the AI Synthesizer, but the AI Synthesizer is not enabled. | Set the environment variable TONIC_NN_GENERATOR_ENABLED to true.
OR
Remove the AI Synthesizer from the generator configuration. |
Issue | Description | How to resolve |
|---|---|---|
Unresolved schema changes | There are detected schema changes that are not resolved. |
The following errors occur when you attempt to generate a subset. They do not apply if Use subsetting is turned off.
Issue | Description | How to resolve |
|---|---|---|
No target tables configured | The subsetting configuration does not include any target tables. | |
Invalid target table configuration | A target table has an invalid percentage value or WHERE clause. | |
In-subset table uses Scale mode | A table that is in the subset uses Scale table mode. | |
In-subset table uses Truncate mode. | A table that is in the subset uses Truncate table mode. | |
In-subset table uses Preserve Destination mode. | A table that is in the subset uses Preserve Destination table mode. | |
In-subset table uses Incremental mode. | A table that is in the subset uses Incremental table mode. |
When upsert is enabled, the following issues cause the upsert job to fail.
Issue | Description | How to resolve |
|---|---|---|
Destination database not populated | The destination database is empty. | Update the workspace to disable upsert, then run a regular data generation job to populate the destination database.
You can then re-enable upsert. |
Invalid table mode | A table is assigned a mode other than De-Identify or Truncate. | Change the table mode to De-Identify or Truncate. |
Unable to connect to the intermediate database | The intermediate database connection is either missing or incomplete. | Edit the workspace configuration to complete the intermediate database connection. |
Unresolved schema conflicts | There are schema conflicts between the source and destination databases. | Update the source or destination database schema to resolve the changes. |