# Running data generation manually ## Selecting the data generation option {% hint style="info" %} **Required workspace permission:** Run data generation {% endhint %} To start the data generation, at the top right of the workspace management view, click **Generate Data**.

Generate Data button to start a data generation

As you configure the data generation options, Structural runs checks to verify that you can use the current configuration to generate data. If any of these checks do not pass, then when you click **Generate Data**, Structural displays information about why you cannot run the data generation job. If all of those checks pass, then when you click **Generate Data**, if there are no warnings, the **Confirm Generation** panel displays. ## Warning for notification schema changes Data generation is always blocked by sensitive schema changes. Sensitive schema changes include new tables and columns that do not have an assigned generator. The workspace configuration includes whether to block data generation for all schema changes, including notification changes. Notification schema changes include removed tables and columns. If this setting is turned off, then if there are notification schema changes, when you click **Generate Data**, a warning displays. To continue to the **Confirm Generation** panel, click **Continue to Data Generation**. ## Confirming the generation details The **Confirm Generation** panel allows you to confirm the details for the data generation.

### Indicating whether to use upsert If upsert is available for the workspace, then you can also determine whether to use upsert for data generation. If upsert is enabled for the workspace, then by default **Use Upsert** is in the on position. To not use upsert, toggle **Use Upsert** to the off position. When upsert is turned off, the data generation is a simple data generation that directly populates and replaces the destination database. ### Indicating whether to generate a subset If you configured subsetting, then you can indicate whether to only generate the subset. To create a subset based on the current subsetting configuration, toggle **Use Subsetting** to the on position. The initial setting matches the current setting in the subsetting configuration. If **Use subsetting** is enabled in **Subsetting** view, then it is enabled by default on the **Generation Confirmation** panel. When you change the setting on the generation confirmation panel, it also updates the setting on **Subsetting** view. ### Determining the data generation process to use (Oracle only) Tonic.ai has released an improved version of the data generation process. It is used automatically for almost all data connectors. It is optional for Oracle. For the new process, the job type is Data Pipeline Generation instead of Data Generation. On the **Confirm Generation** panel, the **Data Pipeline V2** toggle indicates whether to use the new process: * When the toggle is in the off position, Structural uses the previous process. * When the toggle is in the on position, Structural uses the new process.

Confirm Generation panel with the Data Pipeline V2 option

## Configuring job logging and metrics ### Enabling diagnostic logging for a data generation job {% hint style="info" %} **Required global permission:** Enable diagnostic logging and uploading logs directly to Tonic.ai {% endhint %} By default, Structural redacts sensitive values from the logs. To help support troubleshooting, you can configure some Structural data connectors to use diagnostic logging, which generates unredacted versions of the log files. For details, go to [#diagnostic-log-environment-settings](https://docs.tonic.ai/app/admin/tonic-monitoring-logging/logs-redacted-diagnostic#diagnostic-log-environment-settings "mention"). If the data connector is not configured to use diagnostic logging, then you can choose whether to enable diagnostic logging for an individual data generation job. The option is also available for data connectors that do not have a diagnostic logging setting. On the **Confirm Generation** panel, to enable diagnostic logging for the job, toggle **Enable Diagnostic Logging** to the on position. Access to diagnostic logs is also controlled by the **Enable diagnostic logging** global permission. If you do not have this permission, then you cannot download diagnostic logs. ### **Sending the log package to Tonic.ai** {% hint style="info" %} **Required global permission:** Enable diagnostic logging and uploading logs directly to Tonic.ai {% endhint %} You can choose to send a log package to Tonic.ai: In most cases, you send the log package at the request of Tonic.ai support, who then uses the logs to troubleshoot an issue. On the **Confirm Generation** panel, to send the log package, toggle **Send Logs to Tonic.ai** to the on position. Structural creates the package, then uploads it to an S3 bucket. Packages are removed from the S3 bucket automatically after 30 days. ### Indicating whether to generate performance metrics {% hint style="info" %} **Required workspace permission:** Download job logs {% endhint %} To help to troubleshoot issues, for workspaces that use the newer data generation processing, you can configure the data generation job to also generate performance metrics. The performance metrics start when a specified table is processed, and continue for a specified length of time. To enable performance metrics for the data generation job:

Configuration options for generating performance traces

1. **Toggle Collect Performance Metrics** to the on position. 2. From the **Table Trigger** dropdown list, select the table that triggers the performance metrics. 3. From the **Trace Duration** dropdown list, select the length of time to run the performance metrics. ## Verifying connection information ### Viewing the destination location The **Confirm Generation** panel provides the destination information for the workspace. To display the destination database connection details, click **Destination Settings**. Depending on the workspace configuration and data connector type, the destination information is either: * Connection information for a database server. * A storage location such as an S3 bucket. * Information to create container artifacts. If the destination information is incorrect, to navigate to the workspace configuration view to make updates, click **Edit Destination Settings**. For a [file connector](https://docs.tonic.ai/app/setting-up-your-database/file-connector) workspace, if the source files came from a local file system, then the destination files are written to the large file store in the Structural application database. You can [download the most recently generated files](https://docs.tonic.ai/app/setting-up-your-database/file-connector/file-connector-download-generated-files). If the destination data is written to a container repository, then from the **Confirm Generation** panel, you can configure custom tag values to use for the artifacts that the data generation job generates. For information about how to configure the tag values, go to [#workspace-settings-containerization-tags](https://docs.tonic.ai/app/workspace/managing-workspaces/workspace-configuration-settings/workspace-config-write-to-container-artifacts#workspace-settings-containerization-tags "mention"). ### **Verifying the intermediate database connection information (for upsert)** When upsert is enabled, the **Confirm Generation** panel provides access to the connection information for the intermediate database. To display the intermediate database connection details, click **Intermediate Upsert Database**. If the intermediate database information is incorrect, to navigate to the workspace configuration view to make updates, click **Edit Intermediate**. ## Viewing generation performance tips For data generation, assigning Truncate table mode to tables that you don't need data for can improve generation performance. For subsetting, if an upstream table is very large, and the foreign key columns are not indexed, then it can make the subsetting process run more slowly. The **Want faster generations?** message displays at the bottom of the **Confirm Generation** panel. It displays for all non-subsetting jobs. For subsetting jobs, the panel only displays if Structural identified columns that you should consider indexing. To display information about tips for faster generation, click **Generation Tips**. ### Viewing suggested columns to index On the **Generation Tips** panel for subsetting jobs, the **Add Indexes** panel displays the first few columns that you might consider indexing.

Generation Tips panel with indexing suggestions

To display a panel with a suggested SQL command to add the index, click the information icon next to the column.

On the panel, to copy the command to the clipboard, click **Copy SQL to Clipboard**. If there are additional columns that are not listed, then to display the full list of columns to index, click **Show all columns**.

On the full list, to download the list to a CSV file, click **Download list of columns (.csv)**. ### Hint to truncate tables On the **Generation Tips** panel for non-subsetting jobs, the **Truncate Tables** panel displays the hint to truncate tables that contain data that you do not need in the destination database. To navigate to **Database View** to change the current configuration, click **Go to Database View**. ## Starting the generation job On the **Confirm Generation** panel, after you confirm the generation details, to start the data generation, click **Run Generation**. When upsert is enabled, to start the data generation and upsert jobs: 1. Click **Run Generation + Upsert**. 2. In the menu, click **Run Generation + Upsert**.

Structural displays a notification that the job has started. To track the progress of the data generation job and view the results, click the **View Job** button on the notification, or go to [**Jobs** view](https://docs.tonic.ai/app/workspace/jobs). ## Starting an upsert job based on the most recent data generation If upsert is enabled for a workspace, then on the **Confirm Generation** panel, the more common option is to run both data generation and upsert. After you run at least one successful data generation to the intermediate database, then you can also choose to run only the upsert process. For example, if the data generation succeeds but the upsert process fails, then after you address the issues that caused the upsert to fail, you can run the upsert process again. You also must start the upsert job manually if you turn off **Automatically Start Upsert After Successful Data Generation** in the workspace settings. From the **Confirm Generation** panel, to run upsert only: 1. Click the **Run Generation + Upsert** button. 2. In the menu, click **Run Upsert Only**. When you run upsert only, the process uses the results of the most recent data generation.