The Tonic Structural synthetic data platform combines sensitive data detection and data transformation to allow users to create safe, secure, and compliant datasets.

Common Structural use cases include creating staging and development environments and trying out a new cloud provider without complex data agreements. Structural allows you to reduce bug counts, shorten testing life cycles, and share data with partners, all while helping to ensure security and compliance with the latest regulations, from GDPR to CCPA.

You can use Structural APIs to integrate with CI/CD pipelines or to create automated processes that ensure that the generated data is available on demand.

Tonic Structural data generation combines sensitive data detection and data transformation to create safe, secure, and compliant datasets.

The Structural data generation workflow involves the following steps:

You can also view this video overview of the Structural data generation workflow.

1. To get started, you create a workspace. When you create a workspace, you identify the type of source data, such as PostgreSQL or MySQL, and establish the connections to the source database and the destination location. The source database contains the original data that you want to synthesize. The destination location is where Structural stores the synthesized data. It might be a database, a storage location, a container repository, or an Ephemeral database snapshhot.

2. Next, you analyze the results of the initial sensitivity scan. The sensitivity scan identifies columns that contain sensitive data. These columns need to be protected by a generator.

3. Based on the sensitivity scan results, you configure the data generation. The configuration includes:

   • Assigning table modes to tables. The table mode controls the number of rows and columns that are copied to the destination database.

   • Indicating column sensitivity. You can make adjustments to the initial sensitivity assignments. For example, you can mark additional columns as sensitive that the initial scan did not identify as sensitive.

   • Assigning and configuring column generators. To protect the data in a column, especially a sensitive column, you assign a generator to it. The generator replaces the source value with a different value in the destination database. For example, the generator might scramble the characters or assign a random value of the same type.

4. After you complete the configuration, you run the data generation job. The data generation job uses the configured table modes and generators to transform the data from the source database and write the transformed data to the destination location. You can track the job progress and view the job results.

A Tonic Structural implementation can involve the following roles - from those who set up the Structural environment to the consumers of the data that Structural processes.

Note that these roles are not related to role-based access (RBAC) within Structural, which is managed using .

Infrastructure engineers

For self-hosted instances of Structural.

Infrastructure engineers set up the Structural application and its relevant dependencies. They are typically DevOps, Site Reliability Engineering (SRE), or Kubernetes cluster administrators.

Infrastructure engineers perform the following Structural-related tasks:

• Ensure that the proper infrastructure is ready for Structural installation based on the .

• . Works with Tonic.ai support as needed.

• Perform routine maintenance of Structural and the Structural environment. and its dependencies as needed.

• Create Structural-processed data pipelines for development and testing workflows.

Database administrators

For both self-hosted instances of Structural and Structural Cloud.

Database administrators integrate Structural into your data architecture to support .

They ensure that source databases are available to Structural, and that Structural can write to destination databases.

perform the following Structural-related tasks:

• Set up the required Structural access to source databases.

• Set up destination databases for Structural to write transformed data to.

Structural users

Structural users are the actual users of the Structural application.

Depending on the use case, Structural users might be compliance analysts, DevOps, or data engineers.

Tonic users perform the following Structural-related tasks:

• Work with data consumers to produce usable data

Data consumers

Data consumers are the end users of transformed destination data.

They are typically QA testers, developers, or analysts.

Data consumers perform the following Structural-related tasks.

• Validate the usability of the destination data.

• Provide guidance on application-specific requirements for data.

Security and compliance

Security and compliance ensure and validate that the data that Structural produces meets expectations, and that Structural is compliant with other security-related processes.

Security and compliance specialists perform the following Structural-related tasks:

• Provide guidance on what data is sensitive.

• Sign off on proposed approaches to mask sensitive data.

• Approve data access and permissions.

Self-hosted Tonic Structural instance

You can deploy a self-hosted, on-premises instance of Tonic Structural.

For a self-hosted instance, Structural provides administrator tools that allow you to monitor Structural services and manage Structural users.

You can configure Structural environment settings to customize your instance.

On a self-hosted instance, based on your license plan, you have access to the full set of supported data connectors.

Structural Cloud

Structural Cloud is our secure hosted environment. On Structural Cloud, Tonic handles monitoring Structural services and updating Structural.

Structural Cloud does not include:

• Single sign-on (SSO) for user management

• Custom permission sets

• Environment setting configuration. Structural Cloud uses a single configuration.

• Structural data encryption

• Generator presets

• Custom sensitivity rules

• Access to the following data connectors:

  • Amazon EMR

  • Amazon Redshift

  • Spark SDK

Structural Cloud also supports a pay-as-you-go plan, where free trial users can move on to set up a monthly subscription. For more information, go to Setting up and managing a Structural Cloud pay-as-you-go subscription.

Each Structural Cloud user belongs to a Structural Cloud organization, which is determined either by the user's email domain or by a workspace invitation. Structural Cloud users do not have any access to workspaces or users from other organizations.

Each free trial user is in a separate organization, along with any users that they invite to have access to a free trial workspace.

For information about Structural Cloud organizations, go to Structural organizations.

The Account Admin permission set allows a Structural Cloud user to manage organization users and workspaces. For information about granting access to the Account Admin permission set, go to Granting Account Admin access for a Structural Cloud organization.

The Tonic Structural platform creates safe, realistic datasets to use in staging environments or for local development. It includes a web application and API that can be used by engineers, data analysts, or security experts.

Structural connects to source databases that contain sensitive data such as personally identifiable information (PII) or protected health information (PHI). To protect that data, Structural transforms the sensitive values and writes the transformed data to a destination location.

New to Structural? Review the Tonic Structural workflow overview. Go to Getting started with the Structural free trial for information on how to create a Structural account and start a Structural free trial.

Want to know what's in the latest Structural releases? Go to the Tonic Structural release notes.

The Structural application heading includes a feature updates icon, which displays a summary of the newest features, including a link to the Structural release notes.

Connect to your data

Configure and generate transformed data

Manage a self-hosted Tonic Structural instance

Need help with Structural? Contact support@tonic.ai.

Tonic Structural provides different license plans to accommodate organizations of different sizes who have more or less complex data architectures.

Basic license

The Basic license is designed for very small organizations who have a very simple data architecture. It provides access to Structural's core de-identification and data generation features.

Users

The Basic license allows access for a single user, with an option to purchase an additional two users.

There is no access to .

Data connectors

With a Basic license, you can create workspaces for one data connector type. The data connector type must be one of the following:

Concurrent jobs

With a Basic license, your Structural instance can have only one Structural worker. This means that only one sensitivity scan or data generation job can run at the same time.

Structural features

With a Basic license, you can create and configure workspaces, and run data generation for those workspaces.

The Basic license does NOT provide access to the following features:

• Custom generators

Structural API

With a Basic license, you only have access to the basic version of the Structural API.

You cannot use the basic Structural API to perform the following API tasks, which require the advanced API:

Professional license

The Professional license is designed for larger organizations that have more complex data architectures. The organization might have a larger team that supports multiple databases.

The Professional license provides access to a larger set of Structural features than the Basic license.

Users

The Professional license allows up to 10 users. You can purchase access for unlimited users as an add-on.

Data connectors

With a Professional license, you can create workspaces for up to two types of data connectors. You can purchase one additional data connector type as an add-on.

Concurrent jobs

With a Professional license, your Structural instance can have more than one Structural worker.

This means that you can run multiple jobs from different workspaces at the same time. You can never run multiple jobs from the same workspace at the same time.

Structural features

With a Professional license, you can do the following:

• Create and configure workspaces, and run data generation for those workspaces

The Professional license does NOT provide access to the following features:

Structural API

With a Professional license, you only have access to the basic version of the Structural API.

You cannot use the basic Structural API to perform the following API tasks, which require the advanced API:

Enterprise license

The Enterprise license is ideal for very large organizations that have multiple teams that support very large and complex data structures, and that might have more requirements related to scale and compliance.

It provides full access to all Structural features.

Users

An Enterprise instance does not limit the number of users.

Data connectors

You can use any number of any of the available data connectors.

Structural features

The following features are exclusive to the Enterprise license:

Structural API

The Enterprise license provides exclusive access to the advanced API.

The advanced Structural API provides access to all of the available API tasks, including the following tasks that are not available in the basic API:

Feature comparison across Structural license plans

The following table compares the available features for the Structural license plans.

When you go to Tonic Structural for the first time, you create an account. How you create an account depends on the type of user you are.

A new Structural user can be one of the following:

• A completely new user who is starting a Structural 14-day free trial. Free trial users use Structural Cloud to explore and experiment with Structural before they decide whether to purchase it.

• A new user on a self-hosted Structural instance. Self-hosted instances are installed on-premises. The customer administers the Structural users.

• A new user in an existing Structural Cloud organization. New users are added to existing organizations based on their email domain.

From the User Settings view, you can manage settings for your individual Tonic Structural account.

To display the User Settings view:

1. Click your user image at the top right.

2. In the menu, click User Settings.

The User Settings view includes options to:

• Configure your user image.

• View and copy your organization identifier.

• Configure email notifications for column comments.

• Generate and manage API tokens.

• Change your Structural password (if your Structural instance does not use SSO).

• Delete your Structural account.

Choosing your user image

You can select an image to associate with your account. The image is displayed next to your name and email address throughout Structural.

If your instance uses Google or Azure single sign-on (SSO) to manage Structural users, then by default your Structural account image is the image from the SSO.

Otherwise, the default image displays your initials.

To change your user image, click Upload, then select the image file.

Viewing and copying your organization identifier

Below your user image file name is the identifier of the organization that your account belongs to.

To copy the identifier, click the copy icon.

Configuring notifications for column comments

Structural allows users to provide comments on columns. You can do this from Privacy Hub and Database View.

From the Comment Notification Settings section of User Settings, you can configure when to receive email notifications for comments.

The available options are:

• I am an owner, editor, auditor, or am being replied to This is the default option. You receive email notifications when comments are made on columns in a workspace that you are an owner, editor, or auditor for. You also receive an email notification when someone replies to a comment that you made.

• I am @ mentioned You only receive an email notification if someone specifically mentions you in a comment.

• Never You never receive email notifications for column comments.

Generating and managing API tokens

Before you can use the Structural API, you must create an API token. From the User API Tokens section of the User Settings view, you can create and manage API tokens.

Creating an API token

To create an API token:

1. Click Create Token.

2. On the Create New Token dialog, enter a name for the new token.

3. Click Confirm. The token is added to the list.

Copying an API token

To copy an API token, click the copy icon for the token.

Revoking an API token

To revoke a token, click the Revoke option for the token.

Changing your Structural password

If your Structural account is not managed using SSO, then from User Settings, you can change your Structural password.

If your Structural instance uses SSO to manage users, then your user credentials are managed in the SSO system. You cannot change your user password in Structural.

Under Password Change, to change your Structural password:

1. In the Old Password field, type your current Structural password.

2. In the New Password field, type your new Structural password.

3. In the Repeat New Password field, type your new Structural password again.

4. Click Confirm.

Deleting your Structural account

From User Settings, you can delete your Structural account. If your instance uses SSO to manage users, then deleting your account only affects your access to Structural.

You cannot delete your Structural account if you are the owner of a workspace for which other users are granted access. Before you can delete your Structural account, you must either:

• Revoke access from other users.

• Transfer ownership to a different user.

To delete your Structural account, click Delete Account.

When you delete your account, you are logged out of Structural.

Use these tutorial videos to learn more about how to use Tonic Structural.

Structural workflow overview

Provides an overview of the workflow to use Structural to de-identify and synthesize your data. For more information, go to .

Creating a Structural workspace

Managing workspace access

Structural generators overview

Generator presets

File connector overview

Generating data with consistency

Subsetting your data

Upsert data generation

Writing destination data to a container repository

What is the minimum required screen width for the Tonic Structural application?

The minimum screen width is 1120 pixels.

How do you connect to a local database when running Structural in a Docker container locally?

If the locally running database that you want to connect to runs in a Docker container:

1. Run: docker inspect

2. In the networks section of the results, find the Gateway IP address. Use this IP address as the server address in Structural.

If the locally running database does NOT run in a container, but runs on the machine, then:

• On Windows or Mac, use host.docker.internal

• On Linux, use 172.17.0.1, which is the IP address of the docker0 interface.

I allowlist access to my database. What are your static IP addresses?

If you use Structural Cloud, and your database only allows connections from allowlisted IP addresses, then you need to allowlist Structural static IP addresses.

This is not required for self-hosted instances of Structural.

United States-based instance

For the United States-based instance (), the static IP addresses are:

• 54.92.217.68

• 52.22.13.250

The following IP addresses are used if needed for scaling or failover:

• 44.215.74.226

• 3.232.203.148

• 3.224.2.189

• 44.230.136.147

• 44.230.79.194

Europe-based instance

• 18.159.127.160

• 3.69.249.144

The following IP addresses are used if needed for scaling or failover:

• 18.159.179.95

• 3.120.214.225

• 3.75.12.1

• 16.16.71.42

• 16.170.51.237

I allowlist network calls. What do I need to allowlist?

URLs for telemetry sharing

The URL https://telemetry.tonic.ai/ is used for our Amplitude telemetry. https://telemetry.tonic.ai/logs is used specifically for log sharing.

Allowlist https://telemetry.tonic.ai/ or the following IP addresses:

• 75.2.74.76

• 99.83.246.105

Telemetry sharing is required. These metrics are valuable for us as we debug, make product roadmaps, and determine feature viability.

URLs for Structural version information

To support the one-click update option, Structural needs to be able to retrieve information about the latest Structural version.

How do I check my current version of Structural?

Click your user image at the top right. The menu includes the Tonic version.

How should we provision our source database?

We recommend that you use a static copy of your production database that was restored from a backup.

If that's not possible, consider the following when you connect Structural to your source data:

• Structural cannot guarantee referential integrity of the output data if the source database is written to while data is generated. For this reason we recommend that you connect to a static copy of production data.

• Read replicas and fast followers can be problematic for Structural because of how long it takes some queries run. Read replicas tend to have short query timeout limits, which causes the queries to timeout. Read replicas also reflect recent writes, which means that we cannot guarantee the referential integrity of the output.

What data does Tonic.ai collect from Structural?

Every workspace includes the following settings to identify the workspace and to select the type of data connector.

Fields to identify the workspace

All workspaces have the following fields that identify the workspace:

1. In the Workspace name field, enter the name of the workspace.

2. In the Workspace description field, provide a brief description of the workspace. The description can contain up to 200 characters.

3. In the Tags field, provide a comma-separated list of tags to assign to the workspace. For more information on managing tags, go to Assigning tags to a workspace.

Connection type

Under Connection Type, select the type of data connector to use for the workspace data. You cannot change the connection type on a child workspace.

The Basic and Professional licenses limit the number and type of data connectors you can use.

• A Basic instance can only use one data connector type, which can be either PostgreSQL or MySQL. After you create your first workspace, any subsequent workspaces must use the same data connector type.

• A Professional instance can use up two different data connector types, which can be any type other than Oracle or Db2 for LUW. After you create workspaces that use two different data connector types, any subsequent workspaces must use one of those data connector types.

If you don't see the database that you want to connect to, or you want to have different database types for your source and destination database, contact support@tonic.ai.

When you select a connector type, Structural updates the view to display the connection fields used for that connector type. The specific fields vary based on the connector type.

If you are a user who wants to set up an account in an existing Tonic Structural Cloud or self-hosted organization, go to Creating a new account in an existing organization.

About the Structural free trial

The Structural 14-day free trial allows you to explore and experiment in Structural Cloud before you decide whether to purchase Structural.

When you sign up for a free trial, Structural automatically creates a sample workspace for you to use. You can also create a workspace that uses your own database or files.

The free trial provides tools to introduce you to Structural and to guide you through configuring and completing a data generation.

Structural tracks and displays the amount of time remaining in your free trial. You can request a demonstration and contact support.

When the free trial period ends, you can continue to use Structural to configure workspaces. You can no longer generate data or train models. Contact Tonic.ai to discuss purchasing a Structural license, or select the option to start a Structural Cloud pay-as-you-go subscription.

Signing up for the free trial

To start a new free trial of Structural:

1. Go to app.tonic.ai.

2. Click Create Account.

On the Create your account dialog, to create an account, either:

• To use a corporate Google email address to create the account, click Create account using Google.

• To create a new Structural account, enter your email address, create and confirm a Structural password, then click Create Account. You cannot use a public email address for a free trial account.

Structural sends an activation link to your email address.

After you activate your account and log in, Structural next prompts you to select the use case that best matches why you are exploring Structural. If none of the provided use cases fits, use the Other option to tell us about your use case.

After you select a use case, click Next. The Create Your Workspace panel displays.

Determining whether to use your own data

When you sign up for a free trial, Structural automatically creates a sample PostgreSQL workspace that you can use to explore how to configure and run data generation.

You can also choose to create a workspace that uses your own data, either from local files or from a database.

On the Create your workspace panel:

• To use the sample workspace, click Use a sample workspace, then click Next. Structural displays Privacy Hub, which summarizes the protection status for the source data. It also displays the Getting Started Guide panel and the quick start checklist.

• To create a workspace that uses local files as the source data, click Upload Files, then click Next. Go to #uploading-files.

• To create a new workspace that uses your own data, click Bring your own data, then click Next. Go to #connecting-to-a-database.

Uploading files

The Upload files option creates a local files file connector workspace. The source data consists of groups of files selected from a local file system. The files in a file group must have the same type and structure. Each file group becomes a "table" in the source data.

For other workspaces that you create during the free trial, you can also create a file connector workspace that uses files from cloud storage ( Amazon S3 or Google Cloud Storage).

After you select Upload files and click Next, you are prompted to provide a name for the workspace.

In the field provided, enter the name to use for the workspace, then click Next.

Structural displays the File Groups view, where you can set up the file groups for the workspace.

It also displays the Getting Started Guide panel with links to resources to help you get started.

After you create at least one file group, you can start to use the other Structural features and functions.

Connecting to a database

Provide a name for your workspace

If you choose to create a workspace with your own data, then the first step is to provide a name for the workspace.

In the field provided, enter the name to use for your first workspace, then click Next.

The Invite others to Tonic panel displays.

Invite other users to Structural and your workspace

Under Invite others to Tonic, you can optionally invite other users with the same corporate email domain to start their own Structural free trial. The users that you invite are able to view and edit your workspace.

For example, you might want to invite other users if you don't have access to the connection information for the source data. You can invite a user who does have access. They can then update the workspace configuration to add the connection details.

To continue without inviting other users, click Skip this step.

To invite users:

1. For each user to invite, enter the email address, then press Enter. The email addresses must have the same corporate email domain as your email address.

2. After you create the list of users to invite, click Next.

The Add source data connection view displays.

Supported databases for free trial workspaces

The final step in the workspace creation is to provide the source data to use for your workspace.

Structural provides data connectors that allow you to connect to an existing database. Each data connector allows you to connect to a specific type of database. Structural supports several types of application databases, data warehouses, and Spark data solutions.

For the first workspace that you create using the free trial wizard, you can choose:

• Google BigQuery

• MongoDB

• MySQL

• PostgreSQL

• Snowflake on AWS

• Snowflake on Azure

• SQL Server

• Yugabyte

For subsequent workspaces that you create from Workspaces view, you can also choose Databricks and Salesforce.

Selecting the database type

To connect to an existing database, on the Add source data connection panel, click the data connector to use, then click Add connection details.

The panel also includes a Local files option, which creates a local files file connector workspace, the same as the Upload files option.

Use the connection details fields to provide the connection information for your source data. The specific fields depend on the type of data connector that you select.

After you provide the connection details, to test the connection, click Test Connection.

To save your workspace, click Save.

Structural displays Privacy Hub, which summarizes the protection status for the source data.

It also displays the Getting Started Guide panel with links to resources to help you get started.

Free trial resources

The Structural free trial includes a couple of resources to introduce you to Structural and to guide you through the tasks for your first data generation.

Getting Started Guide panel

The Getting Started Guide panel provides access to Structural information and support resources.

The Getting Started Guide panel displays automatically when you first start the free trial. To display the Getting Started Guide panel manually, in the Structural heading, click Getting Started.

The Getting Started Guide panel provides links to Structural instructional videos and this Structural documentation. It also contains links to request a Structural demo, contact Tonic.ai support, and purchase a Structural Cloud pay-as-you-go subscription.

Quick start checklist

For each free trial workspace, Structural provides access to a workspace checklist.

The checklist displays at the bottom left of the workspace management view. It displays automatically when you display the workspace management view. To hide the checklist, click the minimize icon. To display the checklist again, click the checklist icon.

The checklist provides a basic list of tasks to perform in order to complete a Structural data generation.

Each checklist task is linked to the Structural location where you can complete that task. Structural automatically detects and marks when a task is completed.

The checklist tasks are slightly different based on the type of workspace.

Checklist for database-based workspaces

For workspaces that are connected to a database, including the sample PostgreSQL workspace and workspaces that you connect to your own data, the checklist contains:

1. Connect a source database - Set the connection to the source database. In most cases, you set the source connection when you create the workspace. When you click this step, Structural navigates to the Source Settings section of the workspace details view.

2. Connect to destination database - Set the location where Structural writes the transformed data. When you click this step, Structural navigates to the Destination Settings section of the workspace details view.

3. Apply generators to modify dataset - Configure how Structural transforms at least one column in the source data. When you click this step:

   • If there are available generator recommendations, then Structural navigates to Privacy Hub and displays the generator recommendations panel.

   • If there are no available generator recommendations, then Structural navigates to Database View.

4. Generate data - Run the data generation to produce the destination data. When you click this item, Structural navigates to the Confirm Generation panel.

Checklist for local file workspaces

For workspaces that use data from local files, the checklist contains:

1. Create a file group - Create a file group with files that you upload from a local file system. Each file group becomes a table in the workspace. When you click this step, Structural navigates to the File Groups view for the workspace.

2. Apply generators to modify dataset - Configure how Structural transforms at least one column in the source files. When you click this step:

   • If there are available generator recommendations, then Structural navigates to Privacy Hub and displays the generator recommendations panel.

   • If there are no available generator recommendations, then Structural navigates to Database View.

3. Generate data - Run the data generation to produce transformed versions of the source files. When you click this step, Structural navigates to the Confirm Generation panel.

4. Download your dataset - Download the transformed files from the Structural application database.

Checklist for cloud storage file workspaces

For workspaces that use data from files in cloud storage (Amazon S3 or Google Cloud Storage), the checklist contains:

1. Configure output location - Configure the cloud storage location where Structural writes the transformed files. When you click this step, Structural navigates to the Output location section of the workspace details view.

2. Create a file group - Create a file group that contains files selected from cloud storage. When you click this step, Structural navigates to the File Groups view for the workspace.

3. Apply generators to modify dataset - Configure how Structural transforms at least one column in the source data. When you click this step:

   • If there are available generator recommendations, then Structural navigates to Privacy Hub and displays the generator recommendations panel.

   • If there are no available generator recommendations, then Structural navigates to Database View.

4. Generate data - Run the data generation to produce transformed versions of the source files. When you click this step, Structural navigates to the Confirm Generation panel.

Next step hints

In addition to the workspace checklists, Structural uses next step hints to help guide you through the workspace configuration and data generation.

When a next step hint is available, it displays as an animated marker next to the suggested next action.

When you hover over the highlighted action, Structural displays a help text popup that explains the recommended action.

When you click the highlighted action, the hint is removed, and the next hint is displayed.

Creating a file group

For a file connector workspace, to identify the source data, you create file groups. A file group is a set of files of the same type and with the same structure. Each file group becomes a table in the workspace. For CSV files, each column becomes a table column. For XML and JSON file groups, the table contains a single XML or JSON column.

On the File Groups view, click Create File Group.

Uploading local files

For a file connector workspace that uses local files, you can either drag and drop files from your local file system to the file group, or you can search for and select files to add. For more information, go to #adding-files-from-a-local-file-system.

Selecting files from cloud storage

For a file connector workspace that uses cloud storage, you select the files to include in the file group. For more information, go to #adding-files-from-amazon-s3-or-gcs.

Configuring file delimiters and settings

For files that contain CSV content, you also configure the delimiters and other file settings. For more information, go to #configuring-delimiters-and-file-settings-for-.csv-files.

Assigning a generator

To get value out of the data generation process, you assign generators to the data columns.

A generator indicates how to transform the data in a column. For example, for a column that contains a name value, you might assign the Name generator, which indicates how to generate a replacement name in the generation output.

Applying all recommendations

For sensitive columns that Structural detects, Structural can also provide a recommended generator configuration.

When there are recommendations available, Privacy Hub displays a link to review all of the recommendations.

The Recommended Generators by Sensitivity Type panel displays a list of sensitive columns that Structural detected, along with the suggested generators to apply.

After reviewing, to apply all of the suggested generators, click Apply All. For more information about using this panel, go to Reviewing and applying recommended generators.

Selecting a generator

You can also choose to apply an individual generator manually. You can do this from Privacy Hub, Database View, or Table View.

To display Database View, on the workspace management view, click Database View.

On Database View, in the column list, the Applied Generator column lists the currently assigned generator for each column. For a new workspace, the columns are all assigned the Passthrough generator. The Passthrough generator simply passes the source value through to the destination data without masking it.

Click a column that is marked as Passthrough, and that is not marked as sensitive. For example, in the sample workspace, the customers.Last_Transaction column. The column configuration panel displays. To select a generator, click the generator dropdown. The list contains generators that can be assigned to the column based on the column data type. For customers.Last_Transaction, the Timestamp Shift generator is a good option.

Assigning a recommended generator

For Passthrough columns that Structural identified as containing sensitive data, the column includes an icon to indicate that there is a recommended generator.

In Database View, click one of those columns. For example, in the sample workspace, the customers.email column is marked as containing an email address.

For customers.Email, click the Email label. Instead of the column configuration panel, you see a panel that indicates the recommended generator. For customers.Email, the recommended generator is Email. To assign the Email generator, click Apply. The column configuration panel displays with the generator assigned.

Configuring the destination location

To run a data generation, Structural must have a destination for the transformed data.

For a local files workspace, Structural saves the transformed files to the application database.

For workspaces that use data from a database, and for workspaces that use cloud storage files, you configure where Structural writes the output data.

Available output options

The destination location for data generation output can be one of the following:

• If the data connector supports Tonic Ephemeral, then the default option is to write the output data to Ephemeral.

• For database-based data connectors, you can write the transformed data to a destination database.

• For some Structural data connectors, Structural can write the transformed data to a data volume in a container repository.

• For file connector workspaces that transform files from cloud storage (Amazon S3 or Google Cloud Storage), you configure the cloud storage location where Structural writes the transformed files.

Displaying the current destination configuration

To display the destination configuration for the workspace:

1. Click the Settings tab.

2. Scroll to the Destination Settings section or, for a file connector workspace that uses cloud storage files, scroll to the Output location section.

Confirming or changing the destination configuration

Ephemeral snapshot

For data connectors that Ephemeral supports, the default option is to write the output to Ephemeral.

For the Ephemeral option, the default configuration is:

• Structural writes the output to Ephemeral Cloud. If you do not have an Ephemeral Cloud account, then we create an Ephemeral free trial account for you. If your organization has a self-hosted Ephemeral instance, then you can choose to write the output to that instance. Note that all workspaces in the same organization or for the same self-hosted Structural instance must use the same Ephemeral instance.

• Structural uses the output data to create an Ephemeral user snapshot. You can use the user snapshot to create Ephemeral databases.

• When Structural creates the user snapshot in Ephemeral, it creates a temporary Ephemeral database to use as the basis for the user snapshot. There is an option to keep that temporary database. For a free trial workspace, this option is enabled by default. The database expires after 48 hours.

Destination database

To write the data to a destination database, click Database Server. Structural displays the configuration fields for the destination database.

For information on how to configure the destination information for a specific data connector, go to the workspace configuration information for that data connector. The data connector summary contains a list of the available data connectors, and provides a link to the documentation for each data connector.

Container repository

To write the data to a data volume in a container repository, click Container Repository. Structural displays the configuration fields to select a base image and provide the details about the repository.

For more information, go to Writing output to a container repository.

Cloud storage files output location

For a file connector workspace that uses files from cloud storage (Amazon S3 or Google Cloud Storage), you configure the cloud storage output location where Structural writes the transformed files. The configuration includes the required credentials to use.

For more information, go to Configuring the file connector storage type and output options.

Running data generation

After you complete the workspace and generator configuration, you can run your first data generation.

The data generation process uses the assigned generators to transform the source data. It writes the transformed data to the configured destination location.

For a local files workspace, it writes the files to the Structural application database.

Starting the generation

The Generate Data option is at the top right of the Tonic heading.

When you click Generate Data, Structural displays the Confirm Generation panel.

The Confirm Generation panel provides access to the current destination configuration, along with other advanced generation options such as subsetting and upsert. It also indicates if there are any issues that prevent you from starting the data generation. For example, if the workspace does not have a configured destination, then Structural cannot run the data generation.

To start the data generation, click Run Generation. For more information about running data generation, go to Running data generation jobs.

For a new Tonic Ephemeral account, the first time that you run data generation, you also receive an activation email message for the account.

Viewing the job details and connecting to an Ephemeral database

To view the job status and details:

1. Click Jobs.

2. In the list, click the data generation job.

For a data generation that writes the output to an Ephemeral database, the Data Available in Tonic Ephemeral panel provides access to the database connection information.

To display the connection details, click Connecting to your database.

The connection details include:

• The database location and credentials. Each field contains a copy icon to allow you to copy the value.

• SSH tunnel information, including instructions on how to create an SSH tunnel from your local machine to the Ephemeral database.

Next steps for free trial users

The first time that you complete all of the steps in a checklist, Structural displays a panel with options to chat with our sales team, schedule a demo, or purchase a subscription.

You can also continue to get to know Structural and experiment with other Structural features such as subsetting or using composite generators to mask more complex values such as JSON or XML.

If your free trial has expired, to get an extension, you can reach out to us using either the in-app chat or an email message.

Creating a workspace

When you create a new workspace, you can either:

• Create a completely new workspace.

• Create a copy of an existing workspace. The copy initially uses the configuration from the original workspace. After the copy is created, it is completely independent from the original workspace.

• Create a child of an existing workspace. Child workspaces inherit configuration from the parent workspace. They continue to be updated automatically when the parent workspace is updated. For more information, go to About workspace inheritance.

You can also view this video overview of how to create a workspace.

Creating a completely new workspace

To create a completely new workspace, on Workspaces view, click Create Workspace > New Workspace.

Creating a copy of a workspace

To create a workspace based on an existing workspace, either:

• On the workspace management view of the workspace to copy, from the workspace actions menu, select Duplicate Workspace.

• On Workspaces view, click the actions menu for the workspace, then select Duplicate Workspace.

When you create a copy of a workspace, the copy initially inherits the following workspace configuration:

• Source and destination database connections

• Sensitivity designations, including manual designations that override the sensitivity scan results

• Table mode assignments

• Generator configuration

• Subsetting configuration

• Post-job scripts

Creating a child workspace

You can create a workspace that is a child of an existing workspace. You cannot create a child workspace of another child workspace.

The parent workspace must have a source database configured. You cannot create a child workspace from a workspace that uses the Databricks, Spark (Amazon EMR or self-managed Spark cluster), or MongoDB data connector.

To create a child workspace, either:

• On Workspaces view:

  • Click Create Workspace > Child Workspace.

  • Click the actions menu for the parent workspace, then select Create Child Workspace.

• On the workspace management view, from the workspace actions menu, select Create Child Workspace.

On the New Workspace view, under Child Workspace, Parent Workspace identifies the parent workspace.

If you used the Create Workspace > Child Workspace option to create the child workspace, then Parent Workspace is not populated. From the Parent Workspace dropdown list, select the parent workspace for the new child workspace.

If you selected the child workspace option for a specific workspace, then Parent Workspace is set to that workspace.

If you originally chose to create a completely new workspace, then on the New Workspace view:

1. To change to a child workspace, select Create Child Workspace from the Create a child workspace panel at the right. Structural adds the Child Workspace panel to the New Workspace view.

2. From the Parent Workspace dropdown list, select the parent workspace for the new child workspace.

Editing a workspace

To edit the configuration for an existing workspace, either:

• On the workspace management view:

  • On the workspace navigation bar, click Settings.

  • From the workspace actions menu, select Settings.

• On Workspaces view, click the actions menu for the workspace, then select Settings.

Deleting a workspace

You can delete workspaces that you no longer need.

You cannot delete a parent workspace. You must first delete all of its child workspaces.

To delete a workspace:

• On the workspace management view, from the workspace actions menu, select Delete Workspace.

• On the Workspaces view, click the actions menu for the workspace, then select Delete.

The workspace settings for a new workspace (New Workspace view) or edited workspace (workspace Settings tab) provide information about the workspace and its data.

Workspaces view lists the workspaces that you have access to. To display Workspaces view, in the Tonic Structural heading, click Workspaces.

How the workspace list is displayed

The workspace list contains:

• Workspaces that you own

• Workspaces that you are granted access to

If you have the global permission Copy any workspace or Manage user access to Tonic and to any workspace, then you see the complete list of workspaces.

The Permissions column lists the workspace permission sets that you are granted in each workspace. The permission sets include both permission sets that were granted to you directly as a user, and permission sets that were granted to an SSO group that you are a member of.

Child workspaces always display under their parent workspace. You can only see child workspaces that you have access to. If you have access to a child workspace, but not to its parent workspace, then the parent workspace is grayed out. You cannot select it.

Filtering the workspace list

You can filter the workspaces based on the following information:

• Name - In the filter field, begin to type text that is in the name of the workspaces to display in the list.

• Owner - From the Filter by Owner dropdown list, select the owner of the workspaces to display in the list.

• Database type - From the Filter by Database Type dropdown list, select the type of database for the workspaces to display in the list.

• Generation status - In the Generation Status column heading, click the filter icon. Check the checkbox next to the generation status values for the workspaces to display in the list.

• Tags - In the Tags column heading, click the filter icon. By default, the workspaces are not filtered by tag, and all of the checkboxes are unchecked. To only include workspaces that have specific tags, check the checkbox next to each tag to include. To uncheck all of the selected tags, click Reset Tags. When you filter by tag, Structural checks whether each workspace contains any of the selected tags.

• Permissions - In the Permissions column heading, click the filter icon. You can check and uncheck checkboxes to include or exclude specific permission sets. For example, you can filter the list to only display workspaces for which the Editor permission set is granted either to you or to an SSO group that you belong to. For users that have the global permission Copy any workspace, the Permissions filter panel also contains an Any permissions checkbox. By default, Any permissions is unchecked, and the list includes workspaces for which you are not assigned any workspace permission sets. To display all of the workspaces for which you have any assigned workspace permission sets, check Any permissions. If you filter the list based on a specific permission set, to clear the filter and show all workspaces for which you have any permission set, check Any permissions. To display all workspaces, including workspaces that you do not have any permissions for, uncheck Any permissions.

You can combine different filters. For example, you can filter the list to only include workspaces that use PostgreSQL and for which the generation status is Canceled or Failed.

Child workspaces always display under their parent workspace, even if the parent workspace does not match the filter.

Sorting the workspace list

You can sort the workspace list by name, status, or owner.

By default, the list is sorted alphabetically by name.

To sort by a column, click the column heading. To reverse the order of the sort, click the column heading again.

Child workspaces always display under their parent workspace. The child workspaces are sorted within the parent.

Workspace details on Workspaces view

Workspaces view provides the following information about each workspace:

• Name - Contains the name and database type for the workspace. To view the workspace description, hover over the name.

• Generation status - The status for the most recent generation job. To display the job details for the job, click the job status. To display more details about the date, time, and duration for the job, hover over the generation timestamp. If a job failed recently, you are given additional information about how long this job has been failing (the date of the first failure occurrence among a continuous series of failures).

• Schema changes - Indicates whether Structural detected changes to the source database schema. If there are changes, the column shows the number of changes. Hover over the column value to display additional details, and to navigate to the Schema Changes page. See Viewing and resolving schema changes.

• Tags - The tags that are assigned to the workspace.

• Permissions - The permission sets that are assigned to you for the workspace.

• Owner - The name and email address of the workspace owner.

Getting access to workspace tools and actions

Displaying the workspace management view

On Workspaces view, when you click the workspace name, the workspace management view for the workspace is displayed. The Privacy Hub tab is selected.

The Name column also provides access to a menu of workspace configuration options. When you select an option, the workspace management view is displayed, open to the view for the selected option.

Options column

The last column in the workspaces list provides additional workspace options:

• Subsetting icon - Displays the subsetting configuration for the workspace. See Viewing the current subsetting configuration.

• Post-job actions icon - Displays the post-job actions for the workspace. For more information, go to Post-job scripts and Webhooks.

• Actions menu - Provides access to additional options.

Actions menu for bulk actions

The Actions menu at the top left of the workspaces list allows you to to perform bulk actions on multiple workspaces. It is enabled when you check one or more of the checkboxes in the first column of each row. The Actions menu provides options for the selected workspaces.

A Tonic Structural workspace provides a context within which to configure and generate transformed data.

A workspace represents a path between the source data and the transformed output data. For example, postgres-prod-copy to postgres-staging.

A workspace includes:

• Where to find the source data to transform during data generation

• Where to write the transformed data

• The rules for the transformation

Manage workspaces

Workspace details and tools

Blocking data generation for all schema changes

Most workspaces that connect to a database have a Block data generation if schema changes detected toggle. The setting is usually in the Source Settings section.

By default, the option is turned off. When the option is off, Structural only blocks data generation when there are conflicting schema changes. Structural does not block data generation when there are non-conflicting schema changes.

If this option is turned on, then if Structural detects any changes at all to the schema, then data generation is blocked until you resolve the schema changes. For more information, go to .

Workspace statistics seed for cross-run consistency

For generators where consistency is enabled, a statistics seed enables consistency across data generation runs. The Structural-wide statistics seed value ensures consistency across both data generation runs and workspaces.

You use the Override Statistics Seed setting to override the Structural-wide statistics seed value. For workspaces that connect to a database, the setting is under Destination Settings. For a file connector workspace, the setting is under Output Location.

You can either disable consistency across data generations, or provide a seed value for the workspace. The workspace seed value ensures consistency across data generation runs for that workspace, and across other workspaces that have the same seed value.

For details about using seed values to ensure consistency across data generation runs and databases, go to .

After you select the connector type, you configure:

• Where to find the source data

• Where to write the data generation output

Source database connection

For data connectors that connect to a database, the Source Settings section provides connection information for the source database.

You cannot change the source data configuration for a .

For information about the source connection fields for a specific data connector, go to the workspace configuration topic for that .

Upsert configuration

For data connectors that support upsert, the workspace configuration includes an Upsert section to allow you to enable and configure upsert. Upsert adds and updates rows in the destination database, but keeps all other existing rows intact.

If you enable upsert, then you cannot write output to an Ephemeral database or to a container repository. You must write the output to a destination database.

For more information, go to .

Destination data location

For data connectors that connect to a database, the Destination Settings section provides information about where and how Structural writes the output data from data generation.

Depending on the data connector type, you might be able to write to either:

• Destination database - Writes the output data to a destination database on a database server.

• Ephemeral snapshot - Writes the output data to a Tonic Ephemeral user snapshot.

• Container repository - Writes the output data to a data volume in a container repository.

Destination database

When you write the output to a destination database, the destination database must be of the same type as the source database.

Structural does not create the destination database. It must exist before you generate data.

If available, the Copy Settings from Source allows you to copy the source connection details to the destination database, if both databases are in the same location. Structural does not copy the connection password.

Tonic Ephemeral snapshot

If Ephemeral supports your workspace database type, then you can write the destination data to a snapshot in Ephemeral. For data larger than 10 GB, this option is recommended instead of writing to a container repository.

From Ephemeral, you can use the snapshot to start new Ephemeral databases.

Container repository

Some data connectors allow you to write the transformed data to a data volume in a container repository instead of to a database server.

You can use the resulting data volume to create a database in Tonic Ephemeral. If you do plan to use the data to start an Ephemeral database, and the size of the data is larger than 10 GB, then the recommendation is to write the data to an Ephemeral user snapshot instead.

Testing database connections

When you provide connection details for a database server, Structural provides a Test Connection button to test the connection, and verify that Structural can use the connection details to connect to the database. Structural uses the connection details to try to reach the database, and indicates whether it succeeded or failed. We strongly recommend that you test the connections.

File connector source and destination data

For file connector workspaces, the File Location section indicates where the source files are obtained from - either a local file system or a cloud storage solution (Amazon S3 or Google Cloud Storage).

When the files come from cloud storage, the Output Location section indicates where to write the transformed files. You must also provide the cloud storage connection credentials.

Tonic Ephemeral is a separate Tonic.ai product that allows you to create temporary databases to use for testing and demos. For more information about Ephemeral, go to the .

If Ephemeral supports your workspace database type, then you can write the destination data to a snapshot in Ephemeral. You can then use the snapshot to start Ephemeral databases.

To write the transformed data to Ephemeral, under Destination Settings, click Ephemeral Database.

Selecting the Ephemeral instance type

Structural can write the data snapshot to either Ephemeral Cloud or to a self-hosted instance of Ephemeral. By default, Structural writes the data snapshot to Ephemeral Cloud.

All workspaces on the same self-hosted Structural instance or in the same Structural Cloud organization must write to the same instance of Ephemeral. When you change the Ephemeral output configuration in one workspace, it is automatically changed in other workspaces that write to Ephemeral.

For Ephemeral Cloud, Structural writes the snapshot to the account for the user who runs the data generation job. If that user has an Ephemeral account on Ephemeral Cloud, then Structural uses that account. If the user does not have an account, then Structural creates a two-week Ephemeral free trial account for the user.

Note that if you are on a self-hosted instance of Ephemeral, then you must always provide an Ephemeral API key.

To write a snapshot to Ephemeral Cloud:

1. Click Tonic Ephemeral cloud.

2. If you are on a self-hosted instance of Structural:

   1. In the API Key field, provide an Ephemeral API key from your Ephemeral account.

   2. To test the connection, click Test Connection.

To write the snapshot to a self-hosted instance of Ephemeral:

1. Click Tonic Ephemeral self-hosted.

2. In the API Key field, provide an Ephemeral API key from your Ephemeral account. Structural writes the snapshot to the Ephemeral account that is associated with the API key.

3. In the Tonic Ephemeral URL field, provide the URL to your self-hosted Ephemeral instance.

4. To test the connection, click Test Connection.

Selecting the image for Oracle

For Oracle, you select the base image to use when it creates the data snapshot.

If you are writing to Ephemeral Cloud, then you must use the Oracle 23c base image that comes with Ephemeral. This image has the following limitations:

• A maximum of 12GB of user data

• A maximum of 2CPU cores and 2GB of RAM

If you are writing to a self-hosted instance of Ephemeral, then you can also select a custom image that you created in Ephemeral.

Configuring advanced settings for the snapshot

If you do not configure any advanced settings, then:

• The snapshot uses the same name as the workspace, and has no description.

• The snapshot size allocation is determined by the source data size.

• Structural discards the temporary Ephemeral database that is created during the data generation.

To change any of these settings, click Advanced settings.

Providing a snapshot name and description

By default, the snapshot name uses the workspace name.

When you run data generation, if a snapshot with the same name already exists in Ephemeral, then Structural overwrites that snapshot with the new snapshot.

Under Advanced settings:

1. In the Snapshot name field, provide the name of the snapshot. The snapshot name can use the following placeholder values to help identify the snapshot:

   • {workspaceName} - Inserts the name of the workspace.

   • {workspaceId} - Inserts the identifier of the workspace.

   • {jobId} - Inserts the identifier of the data generation job that created the snapshot.

   • {timestamp} - Inserts the timestamp when the snapshot was created.

   Including the job ID or timestamp ensures that a data generation job does not overwrite a previous snapshot.

2. Optionally, in the Snapshot description field, provide a longer description of the snapshot.

Setting the pod resources for the snapshot

By default, the resources used for the snapshot are based on the size of the source data.

• For source data that is 25 GB or less, Nano is used.

• For source data larger than 25 GB, Micro is used.

To select a specific option:

1. Toggle Custom pod resources to the on position.

2. From the dropdown list, select the option to use for the combination of vCPUs and memory:

   • Nano - 0.125 vCPU with 0.5 GB RAM

   • Micro - 0.5 vCPU with 2 GB RAM

   • Small - 1 vCPU with 4 GB RAM

   • Medium - 2 vCPU with 8 GB RAM

   • Large - 4 vCPU with 16 GB RAM

Setting the size allocation for the snapshot

By default, the Ephemeral size allocation for the snapshot is based on the size of the source data.

To instead provide a custom data size allocation, under Advanced settings:

1. Toggle Custom data size allocation to the on position.

2. In the field, enter the size allocation in gigabytes.

Indicating whether to keep the temporary Ephemeral database

When Structural creates the Ephemeral snapshot, it creates a temporary Ephemeral database.

By default, Structural deletes that database when the data generation is complete.

To instead keep the database, under Advanced settings, toggle Keep database active in Tonic Ephemeral after data generation to the on position.

Providing a customization file for MySQL or PostgreSQL

For a MySQL or PostgreSQL workspace, you can provide a customization file that helps to ensure that the temporary Ephemeral database is configured correctly.

To provide the customization details:

1. Toggle Use custom configuration to the on position.

2. In the text area, paste the contents of the customization file.

You use the workspace management view to configure and run data generation for an individual workspace.

When you log in to Tonic Structural, it displays the workspace management view for the workspace that was selected when you logged out.

Components of the workspace management view

The workspace management view includes the following components.

Workspace information

The top left of the workspace management view provides information about the workspace, including:

• The workspace name

• When the workspace was last updated

• The user who last updated the workspace

Workspace options

The top right of the workspace management view provides general options for working with the workspace, including:

• Undo and redo options for configuration changes

• The workspace download menu to:

  • Download sensitivity scan and privacy reports
• The workspace actions menu

Workspace navigation bar

The workspace navigation bar provides access to workspace configuration options.

Displaying the workspace management view

To display the workspace management view for a workspace:

• On Workspaces view, in the Name column either:

  • Click the workspace name. The workspace management view opens to Privacy Hub.

  • Click the dropdown icon, then select a workspace management option.

• Click the search field at the top. A list of available type the name of the workspace. As you type, Tonic displays a list of matching workspaces. In the list, click the workspace name.

Collapsing and expanding the workspace heading

To reduce the amount of vertical space used by the heading of the workspace management view, you can collapse it.

To collapse the heading, click the collapse icon in the Structural heading.

When you collapse the workspace management heading:

• The workspace information is hidden. The workspace name is displayed in the search field.

• The workspace options are moved up into the Structural heading.

The workspace navigation bar remains visible.

When you collapse the heading, the collapse icon changes to an expand icon. To restore the full heading, click the expand icon.

By default, Tonic Structural data generation replaces the existing destination database with the transformed data from the current job.

Upsert adds and updates rows in the destination database, but keeps all of the other existing rows intact. For example, you might have a standard set of test records that you do not want to replace every time you generate data in Structural.

If you enable upsert, then you cannot write the destination data to a container repository or to a Tonic Ephemeral snapshot. You must write the data to a database server.

Upsert is currently only supported for the following data connectors:

• MySQL

• Oracle

• PostgreSQL

• SQL Server

For an overview of upsert, you can also view the .

About the upsert process

When upsert is enabled, the data generation job writes the generated data to an intermediate database. Structural then runs the upsert job to write the new and updated records to the destination database.

The destination database must already exist. Structural cannot run an upsert job to an empty destination database.

The upsert job adds and updates records based on the primary keys.

• If the primary key for a record already exists in the destination database, the upsert job updates the record.

• If the primary key for a record does not exist in the destination database, the upsert job inserts a new row.

To only update or insert records that Structural creates based on source records, and ignore other records that are already in the destination database, ensure that the primary keys for each set of records operate on different ranges. For example, allocate the integer range 1-1000 for existing destination database records that you add manually. Then ensure that the source database records, and by extension the records that Structural creates during data generation, use a different range.

Also note that when upsert is enabled, the Truncate table mode does not actually truncate the destination table. Instead, it works more like Preserve Destination table mode, which preserves existing records in the destination table.

Enabling upsert

To enable upsert, in the Upsert section of the workspace details, toggle Enable Upsert to the on position.

When you enable upsert for a workspace, you are prompted to configure the upsert processing and provide the connection details for the intermediate database.

Configuring upsert processing

When you enable upsert, Structural displays the following settings to configure the upsert process.

Connecting to migration scripts for schema changes

The intermediate database must have the same schema as the destination database. If the schemas do not match, then the upsert process fails.

To ensure that schema changes are automatically reflected in the intermediate database, you can connect the workspace to your own database migration script or tool. Structural then runs the migration script or tool whenever you run upsert data generation.

How upsert works with the migration process

When you start an upsert data generation job:

1. If migration is enabled, Structural calls the endpoint to start the migration.

2. Structural cannot start the upsert data generation until the migration completes successfully. It regularly calls the status check endpoint to check whether the migration is complete.

3. When the migration is complete, Structural starts the upsert data generation.

POST Start Schema Changes endpoint

Required. Structural calls this endpoint to start the migration process specified by the provided URL.

The request includes:

• Any custom parameter values that you add.

• The connection information for the intermediate database.

The request uses the following format:

The response contains the identifier of the migration task.

The response uses the following format:

GET Status of Schema Change endpoint

Required. Structural calls this endpoint to check the current status of the migration process.

The request includes the task identifier that was returned when the migration process started. The request URL must be able to pass the request identifier as either a path or query parameter.

The response provides the current status of the migration task. The possible status values are:

• Unknown

• Queued

• Running

• Canceled

• Completed

• Failed

The response uses the following format:

GET Schema Change Logs endpoint

Optional. Structural calls this endpoint to retrieve the log entries for the migration process. It adds the migration logs to the upsert logs.

The request includes the task identifier that was returned when the migration process started. The request URL must be able to pass the request identifier as either a path or query parameter

The response body of the request should be 'text/plain'. It contains the raw logs.

DELETE Cancel Schema Changes endpoint

Optional. Structural calls this endpoint to cancel the migration process.

The request includes the task identifier that was returned when the migration process started. The request URL must be able to pass the request identifier as either a path or query parameter.

Enabling and configuring the migration process

To enable the migration process, toggle Enable Migration Service to the on position.

When you enable the migration process, you must configure the POST Start Schema Changes and GET Status of Schema Change endpoints.

You can optionally configure the GET Schema Change Logs and DELETE Cancel Schema Changes endpoints.

To configure the endpoints:

1. To configure the POST Start Schema Changes endpoint:

   1. In the URL field, provide the URL of the migration script.

   2. Optionally, in the Parameters field, provide any additional parameter values that your migration scripts need.

2. To configure the GET Status of Schema Change endpoint, in the URL field, provide the URL for the status check.

   The URL must include an {id} placeholder. This is used to pass the identifier that is returned from the Start Schema Changes endpoint.

3. To configure the GET Schema Change Logs endpoint, in the URL field, provide the URL to use to retrieve the logs. The URL must include an {id} placeholder. This is used to pass the identifier that is returned from the Start Schema Changes endpoint.

4. To configure the DELETE Cancel Schema Changes endpoint, in the URL field, provide the URL to use for the cancellation. The URL must include an {id} placeholder. This is used to pass the identifier that is returned from the Start Schema Changes endpoint.

Connecting to the intermediate database

When you enable upsert, you must provide the connection information for the intermediate database.

For details, go to the workspace configuration information for the data connector.

You can configure a workspace to write destination data to a container repository instead of to a database server.

Indicating to write destination data to container artifacts

Under Destination Settings, to indicate to write the destination data to container artifacts, click Container Repository.

You can switch between writing to a database server and writing to a container repository at any time. Structural preserves the configuration details for both options. When you run data generation, it uses the currently selected option for the workspace.

Identifying the base image to use to create the container artifacts

From the Database Image dropdown list, select the image to use to create the container artifacts.

Select an image version that is compatible with the version of the database that is used in the workspace.

Providing a customization file for MySQL

For a MySQL workspace, you can provide a customization file that helps to ensure that the temporary destination database is configured correctly.

To provide the customization details:

1. Toggle Use customization to the on position.

2. In the text area, paste the contents of the customization file.

Setting the location for the container artifacts

To provide the location where Structural publishes the container artifacts:

1. In the Registry field, type the path to the container registry where Structural publishes the data volume.

2. In the Repository Path field, provide the path within the registry where Structural publishes the data volume.

Providing the credentials to write to the registry

You next provide the credentials that Structural uses to read from and write to the registry.

When you provide the registry, Structural detects whether the registry is from Amazon Elastic Container Registry (Amazon ECR), Google Artifact Registry (GAR), or a different container solution.

It displays the appropriate fields based on the registry type.

Fields for registries other than Amazon ECR or GAR

For a registry other than an Amazon ECR or a GAR registry, the credentials can be either a username and access token, or a secret.

In general, the credentials must be for a user that has read and write permissions for the registry.

To use a username and access token:

1. Click Access token.

2. In the Username field, provide the username.

3. In the Access Token field, provide the access token.

To use a secret:

1. Click Secret name.

2. In the Secret Name field, provide the name of the secret.

Azure Container Registry (ACR) permission requirements

For ACR, the provided credentials must be for a service principal that has sufficient permissions on the registry.

Providing a service file for GAR

For a GAR registry, you upload a service account file, which is a JSON file that contains credentials that provide access to Google Cloud Platform (GCP).

The associated service account must have the Artifact Registry Writer role.

For Service Account File, to search for and select the file, click Browse.

Amazon ECR registries

For an Amazon ECR registry, you can either:

• Provide the AWS access and secret key that is associated with the IAM user that will connect to the registry

• Provide an assumed role

• (Self-hosted only) Use the credentials configured in the Structural environment settings TONIC_AWS_ACCESS_KEY_ID and TONIC_AWS_SECRET_ACCESS_KEY.

• (Self-hosted only) If Structural is deployed in Amazon Elastic Kubernetes Service (Amazon EKS), then you can use the AWS credentials that live on the EC2 instance.

Using AWS access keys

To provide an AWS access key and secret key:

1. Click Access Keys.

2. In the Access Key field, enter an AWS access key that is associated with an IAM user or role.

3. In the Secret Key field, enter the secret key that is associated with the access key.

Using an assumed role

To provide an assumed role:

1. Click Assume Role.

2. In the Role ARN field, provide the Amazon Resource Name (ARN) for the role.

3. In the Session Name field, provide the role session name. If you do not provide a session name, then Structural automatically generates a default unique value. The generated value begins with TonicStructural.

4. In the Duration (in seconds) field, provide the maximum length in seconds of the session. The default is 3600, indicating that the session can be active for up to 1 hour. The provided value must be less than the maximum session duration that is allowed for the role.

For the assumed role, Structural generates the external ID that is used in the assume role request. Your role’s trust policy must be configured to condition on your unique external ID.

Here is an example trust policy:

Using the credentials from the environment settings (self-hosted only)

On a self-hosted instance, to use the credentials configured in the environment settings, click Environment Variables.

Using the AWS credentials from the EC2 instance (self-hosted only)

On a self-hosted instance, to use the AWS credentials from the EC2 instance, click Instance Profile.

Required permissions for the IAM user

The IAM user must have permission to list, push, and pull images from the registry. The following example policy includes the required permissions.

For additional security, a repository name filter allows you to limit access to only the repositories that are used in Structural. You need to make sure that the repositories that you create for Structural match the filter.

For example, you could prefix Structural repository names with tonic-. In the policy, you include a filter based on the tonic- prefix:

Providing tags for the container artifacts

In the Tags field, provide the tag values to apply to the container artifacts. You can also change the tag configuration for individual data generation jobs.

Use commas to separate the tags.

A tag cannot contain spaces. Structural provides the following built-in values for you to use in tags:

• {workspaceId} - The identifier of the workspace.

• {workspaceName} - The name of the workspace.

• {timestamp} - The timestamp when the data generation job that created the artifact completed.

• {jobId} - The identifier of the data generation job that created the artifact.

For example, the following creates a tag that contains the workspace name, job identifier, and timestamp:

{workspaceName}_{jobId}_{timestamp}

To also tag the artifacts as latest, check the Tag as "latest" in your repository checkbox.

Specifying custom resources for the Kubernetes pods

You can also optionally configure custom resource values for the Kubernetes pods. You can specify the ephemeral storage, memory, and CPU millicores.

To provide custom resources:

1. Toggle Set custom pod resources to the on position.

2. Under Storage Size:

   1. In the field, provide the number of megabytes or gigabytes of storage.

   2. From the dropdown list, select the unit to use.

   The storage can be between 32MB and 25GB.

3. Under Memory Size:

   1. In the field, provide the number of megabytes or gigabytes of RAM.

   2. From the dropdown list, select the unit to use.

   The memory can be between 512MB and 4 GB.

4. Under Processor Size:

   1. In the field, provide the number of millicores.

   2. From the dropdown list, select the unit.

   The processor size can be between 250m and 1000m.

Setting a custom database name

In the Custom Database Name field, provide the name to use for the destination database.

If you do not provide a custom database name, then the destination database uses the same name as the source database.

Setting a custom database user password

In the Custom Password field, provide the password for the destination database user.

If you do not provide a password, then Structural generates a password.

Configuring the required tolerations for datapacker node taints

If your Kubernetes nodes are configured with taints, then on a self-hosted instance, you can configure the tolerations that enable the datapacker pods to be scheduled on the nodes. The datapacker pod hosts the temporary database that Structural uses during the data generation.

• CONTAINERIZATION_POD_NODE_TOLERATION_KEY - The toleration key value to apply to the datapacker pods. This setting is required. If you do not configure this setting, then Structural ignores the other settings.

• CONTAINERIZATION_POD_NODE_TOLERATION_VALUES - A comma-separated list of toleration values to apply to the datapacker pods.

• CONTAINERIZATION_POD_NODE_TOLERATION_EFFECT - The toleration effect to apply to the datapacker pods.

• CONTAINERIZATION_POD_NODE_TOLERATION_OPERATOR - The toleration operator to apply to the datapacker pods.

Every workspace has an owner. The owner is always a user.

The user who creates the workspace is automatically the owner of the workspace.

By default, the workspace owner is assigned the built-in Manager workspace permission set. On Enterprise instances, you can choose a different workspace permission set to assign to all workspace owners.

You cannot remove that permission set from the workspace owner.

You can transfer a workspace to a different owner. The new owner is assigned the owner permission set. If the previous owner does not otherwise have access to the owner permission set, then that permission set is removed.

To transfer workspace ownership:

1. To transfer ownership of a single workspace, from the workspace actions menu, select Transfer Ownership.

2. To transfer ownership of multiple workspaces:

   1. Check the checkbox for each workspace to grant access to.

   2. From the Actions menu, select Transfer Ownership.

3. On the transfer ownership panel, from the User dropdown list, select the new owner.

4. If you are the current owner of the workspace, then to grant yourself non-owner access after you transfer the ownership:

   1. Toggle Receive access to workspace to the on position.

   2. Select the workspace permission set to assign to yourself.

5. Click Transfer Ownership.

You can associate custom tags with each workspace. Tags can help to organize and provide a quick glance into the workspace configuration.

Tags can be seen by every user that has access to the workspace.

Tags are stored in the workspace JSON, and are included in the workspace export. You can also use the API to get access to tags.

You can add and edit tags in the Tags field on the New Workspace and Settings pages.

• To add tags, enter a comma-separated list of the tags to add.

• To remove a tag, click its delete icon.

You can also manage tags directly from Workspaces view.

To add tags to a workspace that does not currently have tags:

1. Hover over the Tags column for the workspace.

2. Click Add Tags.

3. In the tag input field, type a comma-separated list of tags to apply.

4. Press Enter.

To edit the assigned tags:

1. Click the Tags column for the workspace.

2. In the tag input field, to remove tag, click its delete icon.

3. To add tags, type a comma-separated list of the tags to add.

4. To save the tag changes, press Enter.

If you have multiple workspaces, then it is likely that many of the workspace components and configurations are the same or similar. It can be difficult to maintain that consistency across separate, independent workspaces.

When you copy a workspace, the new workspace is completely independent of the original workspace. There is no visibility into or inheritance of changes from the original workspace.

Workspace inheritance allows you to create workspaces that are children of a selected workspace. Unlike a copy of a workspace, a child workspace remains tied to its parent workspace.

By default, a child workspace configuration is synchronized with the configuration of the parent. In other words, any changes to the parent workspace are copied to the child workspaces. Child workspaces can also override some of the parent configuration. You can track the child workspaces and how they are customized from the parent workspace.

For example, you might want separate workspaces for different development teams. Each team can make adjustments to suit their specific projects - such as different subsets - but inherit everything else.

What does a child workspace inherit?

By default, a child workspace inherits all of the configuration from the parent workspace, except for the following:

• Workspace name - A child workspace has its own name.

• Workspace description - A child workspace has its own description.

• Tags - A child workspace has its own tags.

• Destination database - A child workspace writes output data to its own destination database. You can copy the destination database from the parent workspace.

• Intermediate database - For upsert, a child workspace does not inherit the intermediate database.

• Webhooks - A child workspace has its own webhooks.

How parent workspace changes affect child workspaces

When you change the configuration of a parent workspace, the configuration is also updated in the child workspaces.

The exception is when a child workspace overrides the configuration. If the configuration is overridden, then the child workspace does not inherit the change.

Tonic Structural indicates on both the parent and child workspaces when the configuration is overridden.

What can a child workspace override?

A child workspace can override the following configuration items.

• Table modes - A child workspace can override the table mode for individual tables. The other tables continue to inherit the table mode that is configured in the parent workspace.

• Column generators - A child workspace can override the generator for individual columns. The other columns continue to inherit the generator that is configured in the parent workspace. For linked columns, a change to any of the linked columns overrides the inheritance for all of the columns.

• Subsetting - A child workspace can override the subsetting configuration from the parent workspace. Any change in the child workspace means that the child workspace no longer inherits any changes to the subsetting configuration from the parent workspace. For example, if you change the percentage setting on a single target table from 5 to 6, that eliminates the subsetting inheritance. The child workspace keeps the subsetting configuration that it already has, but it is not updated when the parent workspace is updated.

• Post-job scripts - A child workspace can override the post-job scripts. Any change to the post-job scripts in the child workspace means that the child workspace no longer inherits any changes to the post-job scripts configuration.

• Statistics seed - A child workspace can override the statistics seed configuration.

From each view, you can eliminate the overrides and restore the inheritance.

What must a child workspace inherit?

A child workspace cannot override the following configuration items:

• Data connector type and source database - A child workspace always uses the same source data as the parent workspace.

• Foreign keys - A child workspace always uses the same foreign key configuration as the parent workspace.

• Sensitivity designation for a column - A child workspace cannot change whether a column is marked as sensitive.

How schema changes are resolved in parent and child workspaces

For removed tables and columns, when a child workspace overrides the parent workspace configuration for the table or column, you must resolve the change in the child workspace.

If there is a conflicting change for the removed table or column in the parent workspace configuration, then regardless of whether the configuration is inherited, you must resolve that change in the parent workspace before the change is resolved for the child workspace.

For changes to column nullability or data type, you resolve the change separately in the child and parent workspaces.

You also dismiss notifications (new tables and columns) separately in the parent and child workspaces.

When you create a workspace, you become the owner of the workspace, and by default are assigned the built-in Manager workspace permission set for the workspace. The Manager permission set provides full access to the workspace configuration, data, and results.

With a Professional or Enterprise license, you can also assign workspace permission sets to other users and to SSO groups. You can also transfer a workspace to a different owner.

If you are granted access to any workspace permission set for a workspace, then you can see all of the workspace management views for that workspace. However, you can only perform tasks that you have permission for in that workspace.

Workspace access is managed from the Workspaces view. You cannot assign workspace permission sets from Structural Settings view.

You can also view an overview video tutorial about workspace access.

Tonic Structural uses workspace permission sets for role-based access (RBAC) of each workspace.

A workspace permission set is a set of workspace permissions. Each permission provides access to a specific workspace feature or function.

Structural provides built-in workspace permission sets. Enterprise instances can also configure custom permission sets.

You can assign workspace permission sets to users and to SSO groups, if you use SSO to manage Structural users. Before you assign a workspace permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group. For information on how to configure Structural to filter the allowed SSO groups, go to Synchronizing SSO groups with Tonic Structural.

You cannot remove the owner workspace permission set from the workspace owner. By default, the owner permission set is the built-in Manager permission set.

To change the current access to the workspace:

1. To manage access to a single workspace, either:

   • On the workspace management view, in the heading, click the share icon.

   • On Workspaces view, click the actions menu for the workspace, then select Share.

2. To manage access for multiple workspaces:

   1. Check the checkbox for each workspace to grant access to.

   2. From the Actions menu, select Share Workspaces.

3. The workspace access panel contains the current list of users and groups that have access to the workspace. To add a user or group to the list of users and groups, begin to type the user email address or group name. From the list of matching users or groups, select the user or group to add. Free trial users can invite other users to start their own free trial. Provide the email addresses of the users to invite. The email addresses must have the same corporate email domain as your email address. When the invited users sign up for the free trial, they are added to the Structural organization for the free trial user that invited them and have access to the workspace.

4. For a user or group, to change the assigned workspace permission sets:

   1. Click Access. The dropdown list is populated with the list of custom and built-in workspace permission sets. If you selected multiple workspaces, then on the initial display of the workspace sharing panel, for each permission set that a user or group currently has access to, the list shows the number of workspaces for which the user or group has that permission set. For example, you select three workspaces. A user currently has Editor access for one workspace and Viewer access for the other two. The Editor permission set has 1 next to it, and the Viewer permission set has 2 next to it.

   2. Under Custom Permission Sets, check the checkbox next to each workspace permission set to assign to the user or group. Uncheck the checkbox next to each workspace permission set to remove from the user or group.

   3. Under Built-In Permission Sets, check the workspace permission set to assign to the user or group. You can only select one built-in permission set to assign. By default, for an added user or group, the Editor permission set is selected. To select a built-in workspace permission set that is lower in access than the currently selected permission set, you must first uncheck the selected permission set. For example, if Editor is currently checked, then to change the selection to Viewer, you must first uncheck Editor.

5. To remove all access for a user or group, and remove the user or group from the list, click Access, then click Revoke.

6. To save the new access, click Save.

You can export a workspace configuration to a JSON file, and import configuration from a workspace configuration JSON file.

For example, you might want to preserve a version of the workspace configuration before you test other changes. You can then use the exported file to restore the original configuration.

Or you might want to use a script to make changes to an exported configuration file. You can then import the updated file to update the workspace configuration.

Information in the exported file

The workspace JSON configuration file includes the following information:

• Sensitivity designations that you assigned to columns

• Assigned table modes

• Assigned column generators

• Subsetting configuration

• Post-job script configuration

Exporting the workspace configuration

To export the workspace configuration, either:

• On the workspace management view, from the download menu, select Export Workspace.

• On Workspaces view, click the actions menu for the workspace, then select Export.

When you export a child workspace, the exported workspace does not retain any of the inheritance information. The exported information is the same for all exported workspaces.

Importing a workspace configuration file

To import a workspace configuration file:

1. Select the import option. Either:

   • On the workspace management view, from the download menu, select Import Workspace.

   • On Workspaces view, click the actions menu for the workspace, then select Import.

2. On the Import Workspace dialog, to select the file to import, click Browse.

3. After you select the file, click Import.

When you import a workspace configuration into a child workspace, Tonic Structural only updates the configuration that can be overridden. If a configuration must be inherited from the parent workspace, then it is not affected by the imported configuration. For more information, go to About workspace inheritance.

Database View provides a complete view of your source database structure and configuration.

To display Database View, either:

• On the workspace management view, in the workspace navigation bar, click Database View.

• On Workspaces view, from the dropdown menu in the Name column, select Database View.

Database View consists of:

• On the left, the list of tables in the source database.

• On the right, the list of columns in those tables.

For an individual column in Database View, you can configure the assigned generator and determine the column sensitivity.

Displaying the generator configuration panel

From the column list, to display the generator configuration panel, in the Applied Generator column, click the generator name tag.

Indicating whether a column is sensitive

The Structural sensitivity scan provides an initial indication of whether a column is sensitive and, if it is sensitive:

• The type of sensitive data that is in the column.

• The confidence level of the sensitivity detection.

For more information, go to Identifying sensitive data.

In a child workspace, you cannot configure whether a column is sensitive. A child workspace always inherits the sensitivity designations from its parent workspace.

Status column

From the Status column, to confirm or change the column sensitivity, click the Status value.

The status panel indicates whether the column is sensitive. It identifies the sensitivity type, and indicates how the sensitivity was determined - by a sensitivity scan or by a user.

Built-in sensitivity type

For a column that matches a built-in sensitivity type, the first time that you display the panel, the Sensitive data? setting displays Yes and No options for you to confirm or change the sensitivity.

• To indicate that the column is sensitive, click Yes.

• To indicate that the column is not sensitive, click No.

When you click Yes or No, the Yes and No options change to a simple toggle. When you click Yes, the sensitivity confidence level changes to full.

After that:

• To indicate that the column is sensitive, toggle Sensitive data? to the on position.

• To indicate that the column is not sensitive, toggle Sensitive data? to the off position.

Sensitivity rule match

When a column matches a sensitivity rule, the sensitivity panel indicates that the column matched a sensitivity rule.

You use the Sensitive data? toggle to indicate whether the column is actually sensitive.

No built-in sensitivity type or sensitivity rule match

When a column does not match a built-in sensitivity type or a custom sensitivity rule, the sensitivity panel indicates that column is not sensitive.

The Sensitive data? setting displays Yes and No options for you to confirm or change the sensitivity.

• To indicate that the column is sensitive, click Yes.

• To confirm that the column is not sensitive, click No.

When you click Yes or No, the Yes and No options change to a simple toggle.

If you click Yes:

• The panel updates to indicate that a user confirmed that the column is sensitive.

• The sensitivity confidence level is set to full confidence.

After that:

• To indicate that the column is sensitive, toggle Sensitive data? to the on position.

• To indicate that the column is not sensitive, toggle Sensitive data? to the off position.

Generator configuration panel

You can also configure the sensitivity from the generator configuration panel. On the generator configuration panel, the sensitivity setting is at the top right.

• To indicate that a column is sensitive, toggle the sensitivity setting to the on position.

• To indicate that the column is not sensitive, toggle the sensitivity setting to the off position.

When you change the sensitivity from the generator configuration panel, the Sensitive data? setting on the sensitivity panel also changes from the Yes and No options to the toggle.

Assigning or ignoring the recommended generator

When a sensitivity scan identifies a column, Structural recommends a generator for the column. For example, when the sensitivity scan identifies a column as a first name, Structural recommends the Name generator configured to generate a first name value.

In the Assigned Generator column on Database View, columns that do not have an assigned generator, and that have a recommended generator, display the available recommendation icon.

To review and either apply or ignore the recommended generator, click the generator dropdown.

The generator recommendation panel contains the following information:

• The sensitivity confidence level

• The recommended generator

• Sample source and destination values based on the recommended generator

From the panel, you choose whether to assign or ignore the recommended generator for that type.

• To assign the recommended generator, click Apply. Structural displays the generator configuration panel with the recommended generator selected. You can then adjust the configuration or select a different generator.

• To ignore the recommendation, click Ignore. Structural displays the generator configuration panel to allow you to select the generator to assign to the column.

Changing the column generator configuration

To change the generator that is assigned to a selected column:

1. Click the generator name tag for the column.

2. To assign a different generator to the column, from the Generator Type dropdown list, select the generator.

3. Configure the generator options.

To reset an assigned generator to Passthrough, which indicates to not transform the data:

1. Click the generator name tag.

2. On the generator configuration panel, click the delete icon next to the generator dropdown.

For details about the configuration options for each generator, go to the Generator reference.

For more information about selecting and configuring generators and generator presets, go to Assigning and configuring generators.

Tonic Structural runs the following types of jobs on a workspace:

• Sensitivity scans, which analyze the source database to identify sensitive data.

• Collection scans, which analyze the source data for a MongoDB workspace to determine the available fields in each collection, the field types, and how prevalent the fields are.

• Data generation, data pipeline generation, and containerized generation jobs, which generate the destination data from the source data.

• Upsert data generation jobs, which generate the intermediate database from the source database.

• Upsert jobs, which use data from the intermediate database to add new rows to and update changed rows in the destination database. If the migration process is enabled, then it is a step in the upsert job.

• SDK table statistics jobs. These jobs only run when you use the SDK to generate data in a Spark workspace, and the assigned generators require the statistics.

You can view a list of jobs that ran on the workspace, and view details for individual jobs.

Viewing the list of jobs

The Jobs view displays the list of jobs that ran on the workspace. The list includes the 100 most recent jobs.

To display the Jobs view:

• On the workspace management view, in the workspace navigation bar, click Jobs.

• On Workspaces view, from the dropdown menu in the Name column, select Jobs.

Information in the job list

For each job, the job list includes the following information:

• Job ID - The identifier of the job. To copy the job ID, click the icon at the left of the row.

• Type - The type of job.

• Submitted - The date and time when the job was submitted.

• Completed - The date and time when the job finished running.

Job statuses

A job can have one of the following statuses:

• Queued - The job is queued to run, but has not yet started. A job is queued for one of the following reasons:

  • Another job is currently running on the same workspace. For example, you cannot run a sensitivity scan and a data generation, or multiple data generations, at the same time on the same workspace. This is true regardless of the number of workers on the instance. On Structural Cloud, there is also a limit on the number of concurrent running jobs for each organization. When that maximum is reached, a new job remains queued until a current running job completes.

  • There isn't an available worker on the instance to run the job. A Structural instance with one worker can only run one job at a time. If a job from one workspace is currently running, a job from another workspace cannot start until the first job is finished.

  To view information about why a job is queued, click the status value.

• Running - The job is in progress.

• Canceled - The job is canceled.

• Completed - The job completed successfully.

• Failed - The job failed to complete.

Each of these statuses has a corresponding "with warnings" status. For example, Running with warnings, Completed with warnings. A "with warnings" status indicates that the job had at least one warning at the time of the request.

Filtering the job list

You can filter the list by either the type or the status.

To filter the list by the job type:

1. Click the filter icon in the Type column heading. By default, all types are included, and none of the checkboxes are checked.

2. To only include specific types of jobs, check the checkbox next to each type to include. Checking all of the checkboxes has the same effect as unchecking all of the checkboxes.

To filter the list by the job status:

1. Click the filter icon in the Status column heading. The status panel displays all of the statuses that are currently in the list. For example, if there are no Queued jobs, then the Queued status is not in the list. By default, all of the statuses are included, and none of the checkboxes are checked.

2. To only include jobs that have specific statuses, check the checkbox next to each status to include. Checking all of the checkboxes has the same effect as unchecking all of the checkboxes.

Sorting the job list

You can sort the jobs by either the submission or completion timestamp.

To sort by submission date, click the Submitted column heading. To reverse the sort order, click the heading again.

To sort by completion date, click the Completed column heading. To reverse the sort order, click the heading again.

Viewing details for a selected job

For jobs other than Queued jobs, you can display details about the workspace and the job progress.

From the Jobs view, to display the details for a job, click the job row.

Workspace information

The left side of the job details view contains the workspace information.

For a sensitivity scan, the workspace information is limited to the owner, database type, and worker version.

For a data generation job, the workspace information also includes:

• Whether subsetting, post-job scripts, or webhooks are used.

• The number of schemas, tables, and columns in the source database.

• The number of schemas, tables and columns in the destination database.

Job Log

The Job Log tab shows the start date, start time, and duration of the job, followed by the list of job process steps.

Privacy Report

For data generation jobs, the Privacy Report tab displays the number of at-risk, protected, and not sensitive columns in the source database.

At-risk columns contain sensitive data, but still have Passthrough as the assigned generator.

Protected columns have an assigned generator other than Passthrough.

Not sensitive columns have Passthrough as the assigned generator, but do not contain sensitive data.

Ephemeral output details

A workspace can write output to a Tonic Ephemeral snapshot, with an option to preserve the temporary Ephemeral database that is used to create the snapshot.

For data generation jobs that write to Ephemeral, the Data available in Tonic Ephemeral panel displays. It contains a link to Ephemeral, and access to either the snapshot or the database.

Snapshot details

When the temporary database is not preserved, the Data available in Tonic Ephemeral panel provides access to the snapshot.

To navigate to Ephemeral and view the details for an Ephemeral snapshot, click View Snapshot in Tonic Ephemeral.

Database connection details

When the temporary database is preserved, the Data available in Tonic Ephemeral panel provides access to the database.

To display the connection details for the Ephemeral database, click View connection info.

For an Ephemeral database, the connection details include:

• The database location and credentials. Each field contains a copy icon to allow you to copy the value.

• SSH tunnel information, including instructions on how to create an SSH tunnel from your local machine to the Ephemeral database.

Copying the job ID

The job identifier is a unique identifier for the job. To copy the job ID, either:

Canceling a job

You can cancel Queued or Running jobs.

For jobs with those statuses, the rightmost column in the job list contains a cancel icon.

To cancel the job, click the icon.

Downloading job information

For workspaces that are configured to write destination data to container artifacts, the Jobs view also provides access to those artifacts. For more information, go to Viewing and downloading container artifacts.

Job logs

For all jobs, the job logs provide detailed information about the job processing. Tonic.ai support might request the job logs to help diagnose issues.

For a failed data generation to Ephemeral, the job logs include the Ephemeral logs and the destination database pod logs.

For upsert jobs where the migration process is enabled, and you configured the GET Schema Change Logs endpoint, the upsert job logs include the migration process logs.

Where to download the job logs

You can download the job logs from the Jobs view or the job details view. The download includes up to 1MB of log entries.

On the Jobs view, to download the logs for a job, click the download icon in the rightmost column.

On the job details view, to download the logs for a job, click Reports and Logs, then select Job Logs.

Downloading diagnostic logs

By default, Structural redacts sensitive values from the job logs. To help support troubleshooting, you can configure data connectors or an individual data generation job to create unredacted versions of the log files, referred to as diagnostic logs. For more information, go to Redacted and diagnostic (unredacted) logs.

To access diagnostic log files, you must have the Enable diagnostic logging global permission.

If you do not have the Enable diagnostic logging global permission, then you cannot download the logs for that job. The download option is disabled.

Privacy Report for data generation

From the job details view, you can download a Privacy Report file that provides an overview of the current protection status of the database columns based on the workspace configuration at the time that the job ran.

You can download either:

• The Privacy Report .csv file, which provides details about the table columns, the column content, and the current protection configuration.

• The Privacy Report PDF file, which provides charts that summarize the privacy ranking scores for the table columns. It also includes the table from the .csv file.

To display the download options, click Reports and Logs. In the menu:

• To download the Privacy Report .csv file, click Privacy Report CSV.

• To download the Privacy Report PDF file, click Privacy Report PDF.

For more information about the Privacy Report files and their content, go to Using the Privacy Report to verify data protection.

Additional logs for output to a container repository

For a workspace that writes the output to a container repository, the job includes the following additional logs:

• Database logs - Logs for the database container that is used as the destination.

• Datapacker logs - Logs for creating the OCI artifact and uploading it to an OCI registry.

To download these logs for a data generation job, on the job details view, click Reports and Logs, then select Database Logs or Datapacker Logs.

CloudWatch logs for data generation

For workspaces that are connected to Amazon Redshift or Snowflake on AWS databases, the data generation job requires multiple calls to a Lambda function. For these data generation jobs, the CloudWatch logs monitor the progress of and display errors for these Lambda function calls.

To download the CloudWatch logs for a data generation job, on the job details view, click Reports and Logs, then select CloudWatch Logs.

The CloudWatch Logs option only displays for Amazon Redshift and Snowflake on AWS data generation jobs.

Oracle SQL Loader log files

For an Oracle data generation, if both of the following are true:

• The data generation job ran SQL Loader (sqlldr).

• sqlldr either failed or succeeded with errors.

Then to download the sqlldr log files, click Reports and Logs, then select sqlldr Logs.

Transformed files for file connector data generation