# Tonic.ai product documentation

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th></th><th></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td>De-identify, subset, and synthesize structured and semi-structured data.</td><td><a href="/spaces/-LSQCLFQ4bslJ-HYc8c3/pages/VrpHzzxQ8eWGmfWYHhCO">Structural User Guide</a></td><td><a href="https://app.tonic.ai/apidocs/index.html">Structural API Reference</a></td><td><a href="https://www.tonic.ai/product-release-notes/structural">Structural Release Notes</a></td><td><a href="/files/Kwdeac431CljnKDxkMIf">/files/Kwdeac431CljnKDxkMIf</a></td><td><a href="/spaces/-LSQCLFQ4bslJ-HYc8c3/pages/VrpHzzxQ8eWGmfWYHhCO">/spaces/-LSQCLFQ4bslJ-HYc8c3/pages/VrpHzzxQ8eWGmfWYHhCO</a></td></tr><tr><td>De-identify, redact, and synthesize unstructured data, free-text, and files.</td><td><a href="/spaces/vOPn7KQptPWmS5iKg5P0/pages/u4r36grREO84uKPzTDBL">Textual User Guide</a></td><td><a href="https://tonic-textual-sdk.readthedocs-hosted.com/en/latest/index.html">Textual Python SDK Reference</a></td><td><a href="https://www.tonic.ai/product-release-notes/textual">Textual Release Notes</a></td><td><a href="/files/FWlFr76M4wysIn60j1BA">/files/FWlFr76M4wysIn60j1BA</a></td><td><a href="/spaces/vOPn7KQptPWmS5iKg5P0/pages/u4r36grREO84uKPzTDBL">/spaces/vOPn7KQptPWmS5iKg5P0/pages/u4r36grREO84uKPzTDBL</a></td></tr><tr><td>Synthesize relational data, free-text, and mock APIs from scratch.</td><td><a href="/spaces/moU4gTR9LxlzHeWmQCUZ/pages/TnLqAbvcEQHt0TH4Sabr">Fabricate User Guide</a></td><td><a href="https://www.tonic.ai/product-release-notes/fabricate">Fabricate Release Notes</a></td><td></td><td><a href="/files/tq6aZIqI8TyMTtWZdo8D">/files/tq6aZIqI8TyMTtWZdo8D</a></td><td><a href="/spaces/moU4gTR9LxlzHeWmQCUZ/pages/TnLqAbvcEQHt0TH4Sabr">/spaces/moU4gTR9LxlzHeWmQCUZ/pages/TnLqAbvcEQHt0TH4Sabr</a></td></tr><tr><td>Learn about Tonic.ai security policies, practices, and resources.</td><td></td><td></td><td></td><td><a href="/files/NF3K6QX6fhdDyMqeliNu">/files/NF3K6QX6fhdDyMqeliNu</a></td><td><a href="/spaces/wQloIBJ3L8Y67Vy4o7dR/pages/cFKu0leeLhR4eTQJ0Hrh">/spaces/wQloIBJ3L8Y67Vy4o7dR/pages/cFKu0leeLhR4eTQJ0Hrh</a></td></tr></tbody></table>


# Tonic Structural User Guide

The Tonic Structural platform creates safe, realistic datasets to use in staging environments or for local development. The Structural web application, which includes the [Structural Agent](/app/structural-agent/agent-about) AI chat, and the [Structural API](/app/api/api-documentation) can be used by engineers, data analysts, or security experts.

Structural connects to source databases that contain sensitive data such as personally identifiable information (PII) or protected health information (PHI). To protect that data, Structural transforms the sensitive values and then writes the transformed data to a destination location.

![Data flow from the source database through Tonic Structural to the destination database](/files/IjCfcm0B78oR5EEjPPyn)

New to Structural? Review the [Tonic Structural workflow overview](/app/readme-1/tonic-workflows). For information on how to create a Structural account and start a Structural free trial, go to [Getting started with the Structural free trial](/app/quick-start-guide).

Want to know what's in the latest Structural releases? Go to the [Tonic Structural release notes](https://www.tonic.ai/product-release-notes/structural).

The Structural application heading includes a feature updates icon, which displays a summary of the newest features, and includes a link to the Structural release notes.

<figure><img src="/files/V0vzmKSOHakpLO5iu4PI" alt=""><figcaption><p>Feature updates icon</p></figcaption></figure>

## Connect to your data

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Workspaces</strong></td><td>A workspace contains the data connections and data generation configuration.</td><td></td><td><a href="/pages/iiQIClk0hypKcbpHcHp8">/pages/iiQIClk0hypKcbpHcHp8</a></td></tr><tr><td><strong>Data connectors</strong></td><td>Each data connector allows Structural to read from and write to a specific type of data source.</td><td></td><td><a href="/pages/o2o6ZU0DqKorKruBbDSj">/pages/o2o6ZU0DqKorKruBbDSj</a></td></tr></tbody></table>

## Configure and generate transformed data

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Privacy Hub</strong></td><td>View and update the current protection status based on the sensitivity scan and workspace configuration.</td><td></td><td><a href="/pages/-LxOaqqeGVI_OAV_GWhI">/pages/-LxOaqqeGVI_OAV_GWhI</a></td></tr><tr><td><strong>Database View</strong></td><td>Configure transformation options for tables and columns.</td><td></td><td><a href="/pages/-LxOaudlJ4axVcu_987i">/pages/-LxOaudlJ4axVcu_987i</a></td></tr><tr><td><strong>Generators</strong></td><td>A generator is assigned to a column and performs a data transformation.</td><td></td><td><a href="/pages/-Lu0NCf_P6o1RFziv_Ze">/pages/-Lu0NCf_P6o1RFziv_Ze</a></td></tr><tr><td><strong>Subsetting</strong></td><td>Configure a subset of source data to include in the transformed destination data.</td><td></td><td><a href="/pages/-LxOb7QudZOB9o-oSwlc">/pages/-LxOb7QudZOB9o-oSwlc</a></td></tr><tr><td><strong>Generate data</strong></td><td>Run the data generation process to produce transformed destination data.</td><td></td><td><a href="/pages/ImPbc0BuUKv2SpxVwugs">/pages/ImPbc0BuUKv2SpxVwugs</a></td></tr><tr><td><strong>Schema changes</strong></td><td>Review and address changes to the source data schema.</td><td></td><td><a href="/pages/-LxOb5d4FMwGy5a0FQt6">/pages/-LxOb5d4FMwGy5a0FQt6</a></td></tr></tbody></table>

## Manage a self-hosted Tonic Structural instance

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>User access</strong></td><td>Manage who has access to your instance.</td><td></td><td><a href="/pages/TJ113lVDxhMIaAzfk7WF">/pages/TJ113lVDxhMIaAzfk7WF</a></td></tr><tr><td><strong>Monitoring and logging</strong></td><td>Monitor Structural services and share logs with Tonic.ai.</td><td></td><td><a href="/pages/Z5YqqbQhpNYUEflPl7U0">/pages/Z5YqqbQhpNYUEflPl7U0</a></td></tr><tr><td><strong>Updating Structural</strong></td><td>Upgrade to the latest version of Structural.</td><td></td><td><a href="/pages/yaS3Sd1oFcRCXrppDngz">/pages/yaS3Sd1oFcRCXrppDngz</a></td></tr></tbody></table>

To check the current operational status of Structural Cloud, go to [status.tonic.ai](https://status.tonic.ai).

Need help with Structural? Contact <support@tonic.ai>.


# About Tonic Structural

The Tonic Structural synthetic data platform combines sensitive data detection and data transformation to allow users to create safe, secure, and compliant datasets.&#x20;

Common Structural use cases include creating staging and development environments, and trying out a new cloud provider without complex data agreements.

Structural allows you to reduce bug counts, shorten testing life cycles, and share data with partners, all while helping to ensure security and compliance with the latest regulations, from GDPR to CCPA.

You can use the Structural API to integrate with CI/CD pipelines or to create automated processes that ensure that the generated data is available on demand.

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Structural</strong> <strong>data generation workflow</strong></td><td>Overview of the Structural steps to generate de-identified data.</td><td></td><td><a href="/pages/RrxUElY9s48N5gULtmSX">/pages/RrxUElY9s48N5gULtmSX</a></td></tr><tr><td><strong>Structural deployment types</strong></td><td>You can use Structural Cloud or set up a self-hosted Structural instance.</td><td></td><td><a href="/pages/wwNkTUkfbuh1DgR3H6Lz">/pages/wwNkTUkfbuh1DgR3H6Lz</a></td></tr><tr><td><strong>Structural implementation roles</strong></td><td>Functions that participate in a Structural implementation.</td><td></td><td><a href="/pages/vVKFkDleWfO1yRjU4LD0">/pages/vVKFkDleWfO1yRjU4LD0</a></td></tr><tr><td><strong>Structural license plans</strong></td><td>View the license options and their available features.</td><td></td><td><a href="/pages/7fofr2YWJbNnU7dzooE7">/pages/7fofr2YWJbNnU7dzooE7</a></td></tr></tbody></table>


# Structural data generation workflow

Tonic Structural data generation combines sensitive data detection and data transformation to create safe, secure, and compliant datasets.

The Structural data generation workflow involves the following steps:

![Overview diagram of the Tonic Structural data generation workflow](/files/sqbeAWrArthlYBmhznVI)

You can also view this [video overview of the Structural data generation workflow](https://youtu.be/j1eVJgWoRyU).

1. To get started, you [create a workspace](/app/workspace/managing-workspaces/workspaces-create-edit-delete).\
   \
   When you create a workspace, you identify the type of source data, such as PostgreSQL or MySQL, and establish the connections to the source database and the destination location.\
   \
   The source database contains the original data that you want to synthesize. The destination location is where Structural stores the synthesized data. It might be a database, a storage location, or a container repository.
2. Next, you [analyze the results of the initial sensitivity scan](/app/generation/privacy-hub#privacy-hub-view-protection-status).\
   \
   The sensitivity scan identifies columns that contain sensitive data. These columns need to be protected by a generator.
3. Based on the sensitivity scan results, you configure the data generation. The configuration includes:
   * [Assigning table modes to tables.](/app/generation/table-modes#table-mode-selection)\
     \
     The table mode controls the number of rows and columns that are copied to the destination database.
   * [Indicating column sensitivity.](/app/generation/privacy-hub#privacy-hub-protection-status-flag-sensitive) You can make adjustments to the initial sensitivity assignments.\
     \
     For example, you can mark additional columns as sensitive that the initial scan did not identify as sensitive.
   * [Assigning and configuring column generators.](/app/generation/generators-assign-config/generator-assignment-and-config) To protect the data in a column, especially a sensitive column, you assign a generator to it. The generator replaces the source value with a different value in the destination database.\
     \
     For example, the generator might scramble the characters or assign a random value of the same type.
4. After you complete the configuration, you [run the data generation job](/app/workflows/data-generation-run-job).\
   \
   The data generation job uses the configured table modes and generators to transform the data from the source database and write the transformed data to the destination location.\
   \
   You can track the job progress and view the job results.


# Structural deployment types

## Self-hosted Tonic Structural instance

You can [deploy a self-hosted, on-premises instance of Tonic Structural](/app/admin/on-premise-deployment).

For a self-hosted instance, Structural provides administrator tools that allow you to [monitor Structural services](/app/admin/tonic-monitoring-logging/tonic-admin-service-list) and [manage Structural users](/app/admin/tonic-user-access).

You can [configure Structural environment settings](/app/admin/environment-variables-setting) to customize your instance.

On a self-hosted instance, based on your [license plan](/app/readme-1/tonic-license-plans), you have access to the full set of supported data connectors.

## Structural Cloud

Structural Cloud is our secure hosted environment. On Structural Cloud, Tonic.ai handles monitoring Structural services and updating Structural.

For [single sign-on (SSO)](/app/admin/tonic-user-access/single-sign-on), Structural Cloud only supports Okta.

Structural Cloud does not include:

* [Custom permission sets](/app/admin/tonic-user-access/permissions/permission-sets-about)
* [Environment setting configuration](/app/admin/environment-variables-setting), except for specific settings that can be set for a Cloud organization from the **Organization Settings** tab on **Structural Settings**. Other than that, Structural Cloud uses a single configuration.
* [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config)
* Access to the following data connectors:
  * [Amazon Redshift](/app/setting-up-your-database/amazon-redshift)
  * [Db2 for LUW](/app/setting-up-your-database/db2-luw)
  * [Spark SDK](/app/setting-up-your-database/spark-sdk)

Each Structural Cloud user belongs to a Structural Cloud organization, which is determined either by the user's email domain or by a workspace invitation. Structural Cloud users do not have any access to workspaces or users from other organizations.

Each free trial user is in a separate organization, along with any users that they invite to have access to a free trial workspace.

For information about Structural Cloud organizations, go to [Structural organizations](/app/admin/tonic-user-access/structural-organizations).&#x20;

The Account Admin permission set allows a Structural Cloud user to manage organization users and workspaces. For information about granting access to the Account Admin permission set, go to [Granting Account Admin access for a Structural Cloud organization](/app/admin/tonic-user-access/permissions/global-permissions/granting-account-admin-access-for-a-structural-cloud-organization).


# Structural implementation roles

A Tonic Structural implementation can involve the following roles - from those who set up the Structural environment to the consumers of the data that Structural processes.

Note that these roles are not related to role-based access (RBAC) within Structural, which is managed using [permission sets](/app/admin/tonic-user-access/permissions).

## Infrastructure engineers

For self-hosted instances of Structural.

Infrastructure engineers set up the Structural application and its relevant dependencies. They are typically DevOps, Site Reliability Engineering (SRE), or Kubernetes cluster administrators.&#x20;

Infrastructure engineers perform the following Structural-related tasks:

* Ensure that the proper infrastructure is ready for Structural installation based on the [deployment checklist](/app/admin/on-premise-deployment/tonic-deployment-checklist).
* [Follow the installation instructions](/app/admin/on-premise-deployment). Works with Tonic.ai support as needed.
* Perform routine maintenance of Structural and the Structural environment. [Updates Structural](/app/admin/updating-tonic) and its dependencies as needed.
* Create Structural-processed data pipelines for development and testing workflows.

## Database administrators

For both self-hosted instances of Structural and Structural Cloud.

Database administrators integrate Structural into your data architecture to support [Structural data connectors](/app/setting-up-your-database/database-connectors).

They ensure that source databases are available to Structural, and that Structural can write to destination databases.

[Database administrators](/app/setting-up-your-database/overview-for-database-administrators) perform the following Structural-related tasks:

* Set up the required Structural access to source databases.
* Set up destination databases for Structural to write transformed data to.

## Structural users

Structural users are the actual users of the Structural application.

Depending on the use case, Structural users might be compliance analysts, DevOps, or data engineers.

Tonic users perform the following Structural-related tasks:

* Use the [Structural data generation workflow](/app/readme-1/tonic-workflows) to configure the logic used to transform the source data and to generate the transformed data.&#x20;
* Work with data consumers to produce usable data.

## Data consumers

Data consumers are the end users of transformed destination data.

They are typically QA testers, developers, or analysts.

Data consumers perform the following Structural-related tasks:

* Validate the usability of the destination data.
* Provide guidance on application-specific requirements for data.

## Security and compliance

Security and compliance specialists ensure and validate that the data that Structural produces meets expectations, and that Structural is compliant with other security-related processes.

Security and compliance specialists perform the following Structural-related tasks:

* Provide guidance on what data is sensitive.&#x20;
* Sign off on proposed approaches to mask sensitive data.
* Approve data access and permissions.


# Structural license plans

The Tonic Structural license plans are designed to accommodate organizations that are of different sizes and that have more or less complex data architectures.

## Professional license <a href="#tonic-license-plans-professional" id="tonic-license-plans-professional"></a>

The Professional license is designed for larger organizations that have more complex data architectures. The organization might have a larger team that supports multiple databases.

### Users <a href="#tonic-license-plans-professional-users" id="tonic-license-plans-professional-users"></a>

The Professional license allows up to 10 users. You can purchase access for unlimited users as an add-on.

You can use [single sign-on (SSO)](/app/admin/tonic-user-access/single-sign-on) to manage your Structural users.

### Data connectors <a href="#tonic-license-plans-professional-data-connectors" id="tonic-license-plans-professional-data-connectors"></a>

With a Professional license, you can create workspaces for up to two types of data connectors. You can purchase one additional data connector type as an add-on.

Those data connectors can be of any type except for [Oracle](/app/setting-up-your-database/oracle) and [Db2 for LUW](/app/setting-up-your-database/db2-luw).

### Structural features <a href="#tonic-license-plans-professional-tonic-features" id="tonic-license-plans-professional-tonic-features"></a>

A Professional license provides access to most Structural features.

It does NOT provide access to the following features, which require an Enterprise license:

* [Custom permission sets](/app/admin/tonic-user-access/permissions/permission-sets-about)
* [Global permission set assignment](/app/admin/tonic-user-access/permissions/global-permissions/global-permission-set-access)
* [Privacy Report](/app/generation/privacy-report)
* [Workspace inheritance](/app/workspace/managing-workspaces/workspaces-inheritance)
* [Granting Viewer and Auditor access to workspaces](/app/workspace/workspace-access-management/workspace-sharing)
* [Secrets managers for database connections](/app/workspace/workspace-configuration-settings/secrets-manager)
* [Migration scripts for upsert](/app/workspace/workspace-configuration-settings/workspace-config-upsert#workspace-config-upsert-migration-server)

### Structural API <a href="#tonic-license-plans-professional-tonic-api" id="tonic-license-plans-professional-tonic-api"></a>

With a Professional license, you only have access to the basic version of the Structural API.

You cannot use the basic Structural API to perform the following API tasks, which require the advanced API:

* [Assigning table modes to tables](/app/api/quick-start-guide/tonic-api-table-modes)
* [Assigning generators to columns](/app/api/quick-start-guide/tonic-api-generator-assignment)

## Enterprise license <a href="#tonic-license-plans-enterprise" id="tonic-license-plans-enterprise"></a>

The Enterprise license is ideal for very large organizations that have multiple teams that support very large and complex data structures, and that might have more requirements related to scale and compliance.

It provides full access to all Structural features.

### Users <a href="#license-enterprise-users" id="license-enterprise-users"></a>

An Enterprise instance does not limit the number of users.

### Data connectors <a href="#tonic-license-plans-enterprise-data-connectors" id="tonic-license-plans-enterprise-data-connectors"></a>

You can use any number of any of the available data connectors.

The Enterprise license provides exclusive access to the [Oracle](/app/setting-up-your-database/oracle) and [Db2 for LUW](/app/setting-up-your-database/db2-luw) data connectors.

### Structural features <a href="#tonic-data-plans-enterprise-tonic-features" id="tonic-data-plans-enterprise-tonic-features"></a>

The following features are exclusive to the Enterprise license:

* [Custom permission sets](/app/admin/tonic-user-access/permissions/permission-sets-about)
* [Global permission set assignment](/app/admin/tonic-user-access/permissions/global-permissions/global-permission-set-access)
* [Privacy Report](/app/generation/privacy-report)
* [Workspace inheritance](/app/workspace/managing-workspaces/workspaces-inheritance)
* [Granting Viewer and Auditor access to workspaces](/app/workspace/workspace-access-management/workspace-sharing)
* [Secrets managers for database connections](/app/workspace/workspace-configuration-settings/secrets-manager)
* [Upsert migration service](/app/workspace/workspace-configuration-settings/workspace-config-upsert#workspace-config-upsert-migration-server)

### Structural API <a href="#tonic-license-plans-enterprise-tonic-api" id="tonic-license-plans-enterprise-tonic-api"></a>

The Enterprise license provides exclusive access to the advanced API.

The advanced Structural API provides access to all of the available API tasks, including the following tasks that are not available in the basic API:

* [Assigning table modes to tables](/app/api/quick-start-guide/tonic-api-table-modes)
* [Assigning generators to columns](/app/api/quick-start-guide/tonic-api-generator-assignment)
* [Managing generator presets](/app/api/quick-start-guide/api-generator-presets)
* [Managing custom sensitivity rules](/app/api/quick-start-guide/api-custom-sensitivity-rules)

## Feature comparison across Structural license plans <a href="#tonic-license-plans-feature-comparison" id="tonic-license-plans-feature-comparison"></a>

The following table compares the available features for the Structural license plans.

<table><thead><tr><th width="261.53125" valign="top">Feature</th><th width="277.16015625" valign="top">Professional</th><th valign="top">Enterprise</th></tr></thead><tbody><tr><td valign="top">Number of users</td><td valign="top"><p>10</p><p></p><p>Unlimited users available as an add-on</p></td><td valign="top">Unlimited</td></tr><tr><td valign="top"><a href="/pages/o2o6ZU0DqKorKruBbDSj">Data connectors</a></td><td valign="top"><p>2 data connectors</p><p></p><p>1 additional data connector available as an add-on</p><p></p><p>Any data connector except for Oracle or Db2 for LUW</p></td><td valign="top">Unlimited number from any available data connector</td></tr><tr><td valign="top"><a href="/pages/-MEx52iIIi8pXRp5K226">Workspace permission sets (built-in)</a></td><td valign="top">Manager, Editor</td><td valign="top">Manager, Editor, Auditor, Viewer</td></tr><tr><td valign="top">Custom generators</td><td valign="top">Available for purchase</td><td valign="top">2 included<br><br>Additional ones available for purchase</td></tr><tr><td valign="top"><a href="/pages/uARhWg4pWJJKxVrEbuGg">Create workspaces</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxOaqqeGVI_OAV_GWhI">View Privacy Hub</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxOaqqeGVI_OAV_GWhI#privacy-hub-run-sensitivity-scan">Run sensitivity scans</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxTS7xmOzg-RA895EmN">Assign table modes</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-Lu0NCf_P6o1RFziv_Ze">Assign generators to columns</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/ImPbc0BuUKv2SpxVwugs">Run data generation</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxOb2vhpn1VFAjFTH5e">View job details</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxOb5d4FMwGy5a0FQt6">Schema change monitoring (conflicting changes)</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/cBCvi4cNyjxf5FOWRSm9">View the Version History</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxOb5d4FMwGy5a0FQt6">Schema change monitoring (non-conflicting changes)</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/2XxQmCfzSDYtyZniXZHe">Single sign-on (SSO)</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/jFytQflKzDyiIg0WoCU1">Commenting and notifications</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/Zvc6nyNKEhQAl2zd8Y0u">Structural data encryption </a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/6i4UFk4vmOvz7ODJbIVQ">Custom value processors</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/QLbWXt57pqVFLN2hzOAb">Custom sensitivity rules</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/0iQUY8Ky525TRvsONu8N">Generator presets</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-M_wDKde1bdDMb1ryIFg">Virtual foreign keys</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-LxOb7QudZOB9o-oSwlc">Subsetting</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/Rlyl0zrCJVDRRw4NtDKL">Upsert</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-MI0IMsUb5bgBJp0WLXK">Post-job scripts</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/-MbrEGGVyulPzMX0191T">Webhooks</a></td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top">Concurrent jobs (more than 1 worker)</td><td valign="top">✓</td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/8xo2gopGT9L9GHrodJUY">Workspace inheritance</a></td><td valign="top"></td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/OHGvv4bKCSylB8hBp8EO">Secrets managers for database connections</a></td><td valign="top"></td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/AbvuarvFCjKCt5IDV3fL">Privacy Report</a></td><td valign="top"></td><td valign="top">✓</td></tr><tr><td valign="top"><a href="/pages/weejhycctSRY3TRwKQ74">Custom permission sets</a></td><td valign="top"></td><td valign="top">✓</td></tr><tr><td valign="top">Structural API</td><td valign="top"><a href="/pages/-LjYPhuSn0mEIHRixvig#basic-api">Basic</a></td><td valign="top"><a href="/pages/-LjYPhuSn0mEIHRixvig#advanced-api">Advanced</a></td></tr></tbody></table>


# Logging into Structural for the first time

When you go to Tonic Structural for the first time, you create an account. How you create an account depends on the type of user you are.

A new Structural user can be one of the following:

* [A completely new user who is starting a Structural 14-day free trial.](/app/quick-start-guide) Free trial users use Structural Cloud to explore and experiment with Structural before they decide whether to purchase it.
* [A new user on a self-hosted Structural instance.](/app/admin/tonic-user-access/user-access-new-account#getting-started-new-user-self-hosted) Self-hosted instances are installed on-premises. The customer administers the Structural users.
* [A new user in an existing Structural Cloud organization.](/app/admin/tonic-user-access/user-access-new-account#getting-started-new-user-hosted) New users are added to existing organizations based on their email domain.


# Getting started with the Structural free trial

If you are a user who wants to set up an account in an existing Tonic Structural Cloud or self-hosted organization, go to [Creating a new account in an existing organization](/app/admin/tonic-user-access/user-access-new-account).

## About the Structural free trial <a href="#free-trial-about" id="free-trial-about"></a>

The Structural 14-day free trial allows you to explore and experiment in Structural Cloud before you decide whether to purchase Structural.

Structural tracks and displays the amount of time remaining in your free trial. You can request a demonstration and contact support.

When the free trial period ends, you can continue to use Structural to configure workspaces. You can no longer generate data. Contact Tonic.ai to discuss purchasing a Structural license.

## Signing up for the free trial <a href="#getting-started-new-user-free-trial" id="getting-started-new-user-free-trial"></a>

To start a new free trial of Structural:

1. Go to [app.tonic.ai](https://app.tonic.ai/).
2. Click **Create Account**.

On the **Create your account** dialog, to create an account, either:

* To use a corporate Google email address to create the account, click **Create account using Google**.
* To create a new Structural account:
  1. Enter your email address. You cannot use a public email address for a free trial account.
  2. Create and confirm a Structural password.
  3. Click **Create Account**.

Structural sends an activation link to your email address.

## Creating your first workspace

After you complete the activation, Structural displays the **New Workspace** view to allow you to create your first workspace. The **New Workspace** view includes an option to create a sample workspace that uses data that we provide.

<figure><img src="/files/a4QynXquydbTrWesFpKR" alt=""><figcaption><p>New Workspace view with the Create Sample Workspace option to create a workspace that uses sample data</p></figcaption></figure>

For more information about workspace settings, go to [Workspace configuration settings](/app/workspace/workspace-configuration-settings).

After you create the workspace, Structural runs a sensitivity scan to identify sensitive values in the workspace source data.

The [Structural Agent](/app/structural-agent/agent-about) summarizes the results. Use the Agent to ask questions about Structural and to help you to apply [generators](/app/generation/generators) to transform the sensitive values. You can also explore other features such as [subsetting](/app/generation/subsetting/subsetting-about), which produces a smaller set of output data.

When the generator configuration is complete, [run a data generation](/app/workflows/data-generation-run-job) to create output data.

## Next steps for free trial users <a href="#free-trial-after-checklist" id="free-trial-after-checklist"></a>

During the trial, Structural prompts you to chat with our sales team or to schedule a demo.

If your free trial has expired, to get an extension, you can reach out to us using either the in-app chat or an email message.


# Managing your user account

From the **User Settings** view, you can manage settings for your individual Tonic Structural account.

To display the **User Settings** view:

1. Click your user image at the top right.
2. In the menu, click **User Settings**.

The **User Settings** view includes options to:

* [Configure your user image.](#user-settings-user-image)
* [View and copy your organization identifier.](#user-view-copy-org-id)
* [Configure email notifications for column comments.](#user-settings-notifications)
* [Generate and manage API tokens.](#user-settings-api-tokens)
* [Change your Structural password](#user-settings-change-password) (if your Structural instance does not use SSO).
* [Delete your Structural account.](#user-settings-delete-account)

## Choosing your user image <a href="#user-settings-user-image" id="user-settings-user-image"></a>

You can select an image to associate with your account. The image is displayed next to your name and email address throughout Structural.

If your instance uses Google or Azure single sign-on (SSO) to manage Structural users, then by default your Structural account image is the image from the SSO.

Otherwise, the default image displays your initials.

To change your user image, click **Upload**, then select the image fil&#x65;**.**

## Viewing and copying your organization identifier <a href="#user-view-copy-org-id" id="user-view-copy-org-id"></a>

Below your user image file name is the identifier of the organization that your account belongs to.

To copy the identifier, click the copy icon.

## Configuring notifications for column comments <a href="#user-settings-notifications" id="user-settings-notifications"></a>

{% hint style="info" %}
**Required license:** Professional or Enterprise
{% endhint %}

Structural allows users to provide comments on columns. You can do this from [Privacy Hub](/app/generation/privacy-hub#privacy-hub-protection-status-column-comments) and [Database View](/app/generation/database-view/database-view-column-comment).

From the **Comment Notification Settings** section of **User Settings**, you can configure when to receive email notifications for comments.

The available options are:

* **I am an owner, editor, auditor, or am being replied to**\
  \
  This is the default option. You receive email notifications when comments are made on columns in a workspace that you are an owner, editor, or auditor for.\
  \
  You also receive an email notification when someone replies to a comment that you made.
* **I am @ mentioned**\
  \
  You only receive an email notification if someone specifically mentions you in a comment.
* **Never**\
  \
  You never receive email notifications for column comments.

## Generating and managing API tokens <a href="#user-settings-api-tokens" id="user-settings-api-tokens"></a>

Before you can use the Structural API, you must create an API token. From the **User API Tokens** section of the **User Settings** view, you can create and manage API tokens.

<figure><img src="/files/BD9f1EJ9Pe1dFSB6uTyk" alt=""><figcaption><p>User API Tokens list on the User Settings view</p></figcaption></figure>

### Creating an API token <a href="#api-token-create" id="api-token-create"></a>

To create an API token:

1. Click **Create Token**.
2. On the **Create New Token** dialog, enter a name for the new token.
3. Click **Confirm**.

In the list, the new token displays as clear text. To copy the new token, click the copy icon next to the token.

The new token text and copy icon only display during the current session. After that, Structural masks the token and removes the copy icon.&#x20;

### Revoking an API token <a href="#api-token-revoke" id="api-token-revoke"></a>

To revoke a token, click the **Revoke** option for the token.

## Changing your Structural password <a href="#user-settings-change-password" id="user-settings-change-password"></a>

If your Structural account is not managed using SSO, then from **User Settings**, you can change your Structural password.

If your Structural instance uses SSO to manage users, then your user credentials are managed in the SSO system. You cannot change your user password in Structural.

Under **Password Change**, to change your Structural password:

1. In the **Old Password** field, type your current Structural password.
2. In the **New Password** field, type your new Structural password.
3. In the **Repeat New Password** field, type your new Structural password again.
4. Click **Confirm**.

## Deleting your Structural account <a href="#user-settings-delete-account" id="user-settings-delete-account"></a>

From **User Settings**, you can delete your Structural account. If your instance uses SSO to manage users, then deleting your account only affects your access to Structural.

You cannot delete your Structural account if you are the owner of a workspace for which other users are granted access. Before you can delete your Structural account, you must either:

* [Revoke access from other users.](/app/workspace/workspace-access-management/workspace-sharing)
* [Transfer ownership to a different user.](/app/workspace/workspace-access-management/workspace-transfer-ownership)

To delete your Structural account, click **Delete Account**.

When you delete your account, you are logged out of Structural.


# Frequently Asked Questions

## What is the minimum required screen width for the Tonic Structural application? <a href="#faq-minimum-screen-width" id="faq-minimum-screen-width"></a>

The minimum screen width is 1120 pixels.

## How do you connect to a local database when running Structural in a Docker container locally?

If the locally running database that you want to connect to runs in a Docker container:

1. Run: `docker inspect`
2. In the `networks` section of the results, find the Gateway IP address.\
   \
   Use this IP address as the server address in Structural.

If the locally running database does NOT run in a container, but runs on the machine, then:

* On Windows or Mac, use `host.docker.internal`.
* On Linux, use `172.17.0.1`, which is the IP address of the `docker0` interface.

## I allowlist access to my database. What are your static IP addresses?

If you use Structural Cloud, and your database only allows connections from allowlisted IP addresses, then you need to allowlist Structural static IP addresses.

This is not required for self-hosted instances of Structural.

### United States-based instance

For the United States-based instance ([app.tonic.ai](https://app.tonic.ai)), the static IP address is:

* 54.92.217.68&#x20;

### Europe-based instance

For the Europe-based instance ([app-de.tonic.ai](https://app-de.tonic.ai/)), the static IP address is:

* 3.69.249.144

## I allowlist network calls. What do I need to allowlist?

### URLs for telemetry sharing <a href="#faq-network-allowlist-amplitude" id="faq-network-allowlist-amplitude"></a>

The URL **<https://telemetry.tonic.ai/>** is used for our Amplitude telemetry.

**<https://telemetry.tonic.ai/logs>** is used specifically for log sharing.

Allowlist **<https://telemetry.tonic.ai/>** or the following IP address:

* 44.193.110.147

Telemetry sharing is required. These metrics are valuable for us as we debug, make product roadmaps, and determine feature viability.

No customer data is included. For more information about the specific telemetry data that we collect, go to [Data that Tonic.ai collects](/app/admin/tonic-monitoring-logging/tonic-data-collection).

For more information on how to verify that telemetry is shared, go to [Verifying and enabling telemetry sharing](/app/admin/tonic-monitoring-logging/sharing-logs-with-tonic).

### URLs for Structural version information <a href="#faq-network-allowlist-version-info" id="faq-network-allowlist-version-info"></a>

To support the one-click update option, Structural needs to be able to retrieve information about the latest Structural version.

For more information, go to [Updating Structural](/app/admin/updating-tonic#tonic-updating-allowlist-for-version-info).

### SaaS proxy for the Structural hosted LLM

On a self-hosted instance, you can use Structural's [hosted LLM](/app/admin/structural-ai-use/self-hosted-llm-configuration#hosted-llm) to support [Structural AI features](/app/admin/structural-ai-use/structural-ai-features).

If you do use the hosted LLM, then you must allowlist the SaaS proxy that the LLM requests are routed through, either:

* **<http://us-east-1.saasproxy.tonic.ai/>** or 98.84.248.119 (US location)
* **<http://eu-central-1.saasproxy.tonic.ai/>** or 3.120.214.225 (EU location)

## How do I check my current version of Structural?

Click your user image at the top right. The menu includes the Tonic version.

## How should we provision our source database?

We recommend that you use a static copy of your production database that was restored from a backup.

If that's not possible, consider the following when you connect Structural to your source data:

* Structural cannot guarantee referential integrity of the output data if the source database is written to while data is generated.\
  \
  For this reason we recommend that you connect to a static copy of production data.
* Read replicas and fast followers can be problematic for Structural because of how long it takes some queries to run.\
  \
  Read replicas tend to have short query timeout limits, which causes the queries to time out.\
  \
  Read replicas also reflect recent writes, which means that we cannot guarantee the referential integrity of the output.

## How does Structural use AI?

For details about the available AI features and how they are supported, go to [AI in Structural](/app/admin/structural-ai-use).

## What data does Tonic.ai collect from Structural? <a href="#faq-tonic-data-collection" id="faq-tonic-data-collection"></a>

For details about the types of data that Tonic.ai does and does not collect, go to [Data that Tonic.ai collects](/app/admin/tonic-monitoring-logging/tonic-data-collection).


# Tutorial videos

Use these tutorial videos to learn more about how to use Tonic Structural.

For the full set of tutorial and demo videos, go to the [Structural playlist](https://www.youtube.com/playlist?list=PLahjS-TeeUbVb1FSTb4_G7PiJlutiSDxv).

## Tonic Structural 101

Provides an overview of the Structural workflow and how to use Structural to generate de-identified data. For more information, go to [Structural data generation workflow](/app/readme-1/tonic-workflows).

{% embed url="<https://www.youtube.com/watch?v=A6-WfSO4dk4>" %}
Tutorial video: Tonic Structural 101
{% endembed %}

## Creating a Structural workspace

Provides an overview of what a Structural workspace is and how to create a new Structural workspace. For more information, go to [Creating and managing workspaces](/app/workspace/managing-workspaces).

{% embed url="<https://www.youtube.com/watch?v=fbfy74nOwsY>" %}
Tutorial video: Creating a Structural workspace
{% endembed %}

## Sensitivity detection and generator recommendations

Provides an overview of how Structural detects sensitive values and how you can apply recommended generators to the detected values.

{% embed url="<https://www.youtube.com/watch?v=vF73dllMJQI>" %}
Tutorial video: Sensitivity detection and generator recommendations
{% endembed %}

## Managing workspace access

Provides an overview of workspace owners, permissions, and permission sets. Explains how to share and transfer ownership of a workspace. For more information, go to [Managing access to workspaces](/app/workspace/workspace-access-management).

{% embed url="<https://www.youtube.com/watch?v=rEn_vpcAtgg>" %}
Tutorial video: Managing workspace access
{% endembed %}

## Structural generators overview

Identifies the types of generators and transformations that you can use in Structural, and explains how to assign a generator to a column. For more information, go to [Generator information](/app/generation/generators).

{% embed url="<https://youtu.be/UNngC2a6q94>" %}
Tutorial video: Tonic Structural generators overview
{% endembed %}

## Generator presets

Provides an overview of generator presets. Includes how to create and update them, and how to track where each generator preset is used. For more information, go to [Managing generator presets](/app/generation/generators-assign-config/generator-presets).

{% embed url="<https://www.youtube.com/watch?v=-OuhhG34HjA>" %}
Tutorial video: Generator presets
{% endembed %}

## File connector overview

Provides an overview of the file connector and how to manage file groups in a file connector workspace. For more information, go to [File connector](/app/setting-up-your-database/file-connector).

{% embed url="<https://youtu.be/Es_VPV9kCxs>" %}
Tutorial video: File connector overview
{% endembed %}

## Generating data with consistency

Provides an overview of the consistency generator property and how it works. For more information, go to [Enabling consistency](/app/generation/generators/generator-characteristics/consistency).

{% embed url="<https://www.youtube.com/watch?v=ejEnjLMm58E>" %}
Tutorial video: Generating data with consistency
{% endembed %}

## Using Document View to configure JSON columns

Provides an overview of how to enable **Document View** for a JSON column and how to use it to configure generators for JSON fields.

{% embed url="<https://www.youtube.com/watch?v=XCMezJ-Kst4>" %}
Tutorial video: Using Document View to configure JSON columns
{% endembed %}

## Subsetting your data

Provides an overview of subsetting, how it is configured, and how Structural uses the configuration to generate a subset. For more information, go to [Subsetting data](/app/generation/subsetting).

{% embed url="<https://youtu.be/j0-958NyTFc>" %}
Tutorial video: Subsetting your data
{% endembed %}

## Upsert data generation

Provides an overview of upsert data generation. Includes how it works and how to enable and run it for a workspace. For more information, go to [Enabling and configuring upsert](/app/workspace/workspace-configuration-settings/workspace-config-upsert).

{% embed url="<https://www.youtube.com/watch?v=veNRwlk8ArU>" %}
Tutorial video: Enabling upsert data generation
{% endembed %}

## Writing destination data to a container repository

Provides an overview of how to write destination data to a container repository instead of a database server. For more information, go to [Writing output to a container repository](/app/workspace/workspace-configuration-settings/workspace-config-write-to-container-artifacts).

{% embed url="<https://www.youtube.com/watch?v=mwmiZBMGWmQ>" %}
Tutorial video: Outputting data to a container repository
{% endembed %}


# About the Structural Agent

## What is the Structural Agent?

The Structural Agent is an LLM-powered, chat-based tool to help you to understand workspace data and to configure workspaces for data generation.

The Structural Agent chat initially starts when you save your first workspace.

## What can the Agent do automatically?

The Structural Agent can automatically perform the following tasks.

### Check for a source data connection

For workspaces that have not yet generated data, the Structural Agent checks whether the workspace has a valid connection to source data. If not, it prompts you to go to workspace settings and configure the connection.

If there is a source data connection, then the Agent checks the status of the initial sensitivity scan, and prompts you to run a scan if the scan hasn't started.

For a file connector workspace, the Structural Agent checks that you have created at least one file group with files.

### Summarize the sensitivity scan results

When the sensitivity scan is complete, the Structural Agent provides a summary of the scan results.

Depending on the number of tables and columns, the Structural Agent can provide a complete list of the sensitive columns, or a more condensed summary.

<figure><img src="/files/1RPZJtWyqKD0wwYMpijW" alt=""><figcaption><p>Agent summary of the sensitivity scan</p></figcaption></figure>

### Prompt to apply recommended generators

Based on the scan results, the Structural Agent suggests that you apply the recommended generators for the detected sensitive columns.

<figure><img src="/files/lFFkERjWWgkwQFeGSJ33" alt=""><figcaption><p>Agent prompt to assign generators</p></figcaption></figure>

### Prompt to generate data

When all of the sensitive columns are protected, after it verifies that the workspace has a valid destination connection, the Agent prompts you to run the data generation.

The Agent cannot start the data generation.


# What can you ask the Structural Agent to do?

You can ask the Structural Agent to perform a variety of analysis and configuration tasks. The following are just a few examples.

<table><thead><tr><th width="340.9765625" valign="top">Task</th><th valign="top">Example prompts</th></tr></thead><tbody><tr><td valign="top">Explore workspaces</td><td valign="top"><code>Which workspace did I update most recently?</code><br><br><code>Which workspace did I most recently generate data for?</code><br><br><code>What's the most recently failed job in my workspaces?</code><br><br><code>Give me a breakdown of the number of workspaces by data connector.</code><br><br><code>How many workspaces don't yet have a completed data generation job?</code><br><br><code>Are all of the name columns in my workspaces protected?</code></td></tr><tr><td valign="top">Navigate through the workspace</td><td valign="top"><code>Display Database View.</code><br><br><code>Show me Table View for the transactions table.</code></td></tr><tr><td valign="top">Ask questions about the workspace data</td><td valign="top"><p><code>How many name columns were detected in the data?</code><br></p><p><code>Are there any sensitive columns in the customers table?</code><br><br><code>Are there any JSON columns?</code></p></td></tr><tr><td valign="top">Filter the workspace data columns</td><td valign="top"><p><code>Show me all of the unprotected datetime columns.</code><br></p><p><code>Show me all of the primary key and foreign key columns.</code></p></td></tr><tr><td valign="top">Set column sensitivity</td><td valign="top"><code>Make sure that all datetime columns are marked as sensitive.</code><br><br><code>Mark the currently displayed columns as not sensitive.</code></td></tr><tr><td valign="top">Ask for generator recommendations</td><td valign="top"><code>Suggest a generator for the occupation column.</code></td></tr><tr><td valign="top">Configure column generators</td><td valign="top"><code>Apply the Timestamp Shift generator to all of the datetime columns. Shift the dates by 1 day before or after the current value.</code><br><code>Apply the Custom Categorical generator to the product column. Generate 20 product names and enable consistency.</code><br><br><code>Apply the Conditional generator to the seniority column. If the value of the column is less than 4, use the Random Integer generator to generate a replacement value between 0 and 4. If the value of the column is greater than or equal to 5, use the Random Integer generator to generate a value between 5 and 10.</code><br><br><code>Apply the JSON Mask generator to the customer_summary column so that properties under name have the Name generator applied with the correct name type, the location properties have the Address generator applied with the correct location type, and the name property under work/properties has the Business Name generator applied.</code><br><br><code>Apply the Regex Mask generator to the Product ID field to use the Random Integer generator to mask the numeric parts of the value.</code></td></tr><tr><td valign="top">Display sample data</td><td valign="top"><code>Show me sample source and destination values for the last name column in the customers table.</code></td></tr><tr><td valign="top">Manage document-based data</td><td valign="top"><p><code>Enable Document View for the customer_summary column.</code></p><p></p><p><code>How many name fields are there across the database collections?</code></p><p></p><p><code>Mark the occupation field as not sensitive.</code><br><br><code>Apply the recommended generators to the fields under customer/name.</code><br><br><code>Apply the Timestamp Shift generator to all fields that contain datetime values.</code></p></td></tr><tr><td valign="top">Manage subsetting</td><td valign="top"><code>Is subsetting enabled?</code><br><br><code>How many tables are in the subset?</code><br><br><code>How large will the subset be based on the current configuration?</code><br><br><code>What is the subset target table configuration?</code><br><br><code>Should any tables be lookup tables?</code><br><br><code>Enable subsetting and configure the transactions table as the target table. Use 5% of the records.</code><br><br><code>Change the target table configuration to use records where the date is within the past 3 months.</code><br><br><code>For the attendees table, filter optional records to include records where organization is test.</code><br><br><code>Make vendor_type a lookup table.</code><br><br><code>Enable subsetting and include tables that are not in the subset.</code></td></tr><tr><td valign="top">Create and configure post-job scripts</td><td valign="top"><p><code>Create a post-job script to add demo users to the users table in the destination database.</code></p><p></p><p><code>Enable the add demo users post-job script.</code></p><p></p><p><code>Change the order of the post-job scripts to make the add demo users script run first.</code></p><p></p><p><code>Delete the add demo users script.</code></p></td></tr><tr><td valign="top">Identify and resolve schema changes</td><td valign="top"><code>Are there any schema changes that need to be resolved?</code><br><br><code>Resolve the sensitive schema changes. Apply the Categorical generator to the new column.</code><br><br><code>Dismiss the schema change notifications.</code></td></tr><tr><td valign="top">Get information about jobs and diagnose job failures</td><td valign="top"><code>How many data generation jobs have run for this workspace?</code><br><br><code>How many columns were de-identified during the most recent successful data generation?</code><br><br><code>When was the most recent sensitivity scan for this workspace? How many sensitive columns did it identify?</code><br><br><code>Why did the most recent data generation job fail? What's the solution to enable the job to complete successfully?</code></td></tr><tr><td valign="top">Compose a draft email message to Tonic.ai support.<br><br>The Agent creates the draft message and then displays an <strong>Email support</strong> button that opens the draft message in your email client.</td><td valign="top"><code>Can you create a draft email message to support to report the issue with access to workspaces?</code></td></tr><tr><td valign="top">Generate scripts and API calls<br><br>Note that the Agent can only use the information available from <a href="/pages/-LjYPhuSn0mEIHRixvig">this guide</a>. It does not have access to the generated API reference.</td><td valign="top"><code>Can you show me the API call for assigning the Email generator to the email address columns?</code></td></tr></tbody></table>

Note that Structural Agent requests are still subject to your global and workspace permissions. You cannot ask the Structural Agent to perform a task that you do not have permission to do.


# Managing the Structural Agent and Agent chat conversations

## How chat conversations are stored

Tonic Structural stores chat conversations in browser storage.

## Using the AI Agent tab

The **AI Agent** tab provides a full-page view of the Structural Agent chat.

<figure><img src="/files/5MfiFaZFV7hC0JLZ039r" alt=""><figcaption><p>AI Agent tab on the workspace management tab</p></figcaption></figure>

## Using the AI Agent chat panel

In addition to the **AI Agent** tab, you can use the **AI Agent** chat panel, which allows you to interact with the Structural Agent while you view Structural pages.

### Available display types

The chat panel supports the following display options.

#### Side-by-side

With the **Side-by-side** option, the agent chat panel displays next to the Structural page.

To accommodate the chat panel, the Structural page becomes narrower.

You can display the chat panel on the left or right side of the screen.

<figure><img src="/files/dtEhj8Lf0KFBhVP0Jwoh" alt=""><figcaption><p>Structural Agent using the Side-by-side display </p></figcaption></figure>

#### Overlay

With the **Overlay** option, the agent chat panel displays on top of the Structural page.

You can display the chat panel on the left or right side of the screen.

<figure><img src="/files/78vnP2s9J3h6x7DK5A8N" alt=""><figcaption><p>Structural Agent using the Overlay display</p></figcaption></figure>

### Selecting the chat panel display type

From the chat panel, to select the display type:

1. Click the chat panel options menu.
2. In the menu, click the display type.

<figure><img src="/files/BrEOTAQERdvzI6lFMaLo" alt=""><figcaption><p>Agent options menu with the display types</p></figcaption></figure>

To also select the Agent chat panel location, click the location option under the display type.

For example, if you select the **Move to right side** option under **Side-by-side**, then the chat panel both changes to the side-by-side view and displays at the right.

If you don't select a specific location, then the chat panel uses the most recently selected location.

### Selecting the chat panel display location

To change the display location:

1. Click the chat panel options menu.
2. For the selected display option:
   * To move the chat panel to the left, click **Move to left side**.
   * To move the chat panel to the right, click **Move to right side**.

### Hiding and displaying the chat panel

To hide the chat panel, either:

* Click the hide icon at the top right.
* In the options menu, click **Hide**.

To display the chat panel, click the chat panel icon. The icon displays at the bottom right of each Structural page.

<figure><img src="/files/AToc0hnoH1gdBVcdXx84" alt=""><figcaption><p>Icon to show the chat panel</p></figcaption></figure>

### Adjusting the chat panel width

To adjust the chat panel width, click and drag the panel border.

## Identifying the chat context

The Structural Agent chat remains in place across the entire Structural application.

You can read information about data from any workspace that you have access to. For example, you can get a summary list of workspaces organized by data connector, identify unprotected columns in any workspace, or find the workspace with the most recent failed job.

However, you can only perform configuration or generation actions based on the current context. For example, you can only change the configuration for a workspace if you are currently in the workspace management view for that workspace. The Structural Agent can navigate to the workspace that you want to work with.

As you navigate to different workspaces or to views outside of a workspace, the Structural Agent chat notes the change in context, and provides links to allow you to navigate to each context.

<figure><img src="/files/mh4z1Rd9ZTQafGeWCtdc" alt=""><figcaption></figcaption></figure>

## Starting and managing chat conversations

You can create multiple conversations within the Structural Agent chat. For example, you might start a new conversation when you change to a different workspace.

### Starting a new conversation

From the Structural Agent chat, to start a new conversation, click the conversation dropdown, then select **New Conversation**.

<figure><img src="/files/jng57eMIZUiOFFuEIfd7" alt=""><figcaption><p>Conversation menu for the Structural Agent</p></figcaption></figure>

### Renaming the current conversation

Structural automatically assigns a name to each new conversation, based on the current context.

To assign a new name to the current conversation:

1. Click the conversation dropdown, then click **Rename**.
2. In the name field, provide the new name.
3. Click **Save**.

### Switching to a different conversation

To change to a different conversation, from the conversation dropdown, select the conversation to change to.

### Deleting the current conversation

To delete the current conversation:

1. Click the conversation dropdown, then click **Delete**.
2. On the confirmation panel, click **Delete**.

Structural deletes the current conversation and switches you to the first available conversation.

When you delete a conversation, Structural also removes it from the browser storage.

## Stopping an action

While the Agent is working, to stop the response and cancel the action, in the chat prompt, click the stop response icon.

<figure><img src="/files/OgTXsH2GMgaMJy0JOZZy" alt=""><figcaption><p>Stop response icon in the Structural Agent chat prompt</p></figcaption></figure>

## Undoing and redoing an action

Before it performs an action to modify the workspace, the Agent takes snapshots of the current workspace configuration.

To undo an action, or to revert to a previous point from the current conversation, prompt the Agent.

For example:

* `Undo that action.`
* `Revert the generator assignments for the datetime columns.`
* `Go back to before we assigned generators to the name columns.`

If needed, the Agent can ask you to select a snapshot to revert to.

To redo an action, cick the redo icon.

<figure><img src="/files/GMy5F6xHyjLLTzvB3kWa" alt=""><figcaption><p>Redo icon for the Structural Agent</p></figcaption></figure>

## Copying the most recent response from the Agent

A response from the Agent might be useful elsewhere. For example, you can prompt the Agent to generate Structural API calls, or the Agent might return a useful summary of the data and its status.

To copy the most recent Agent response, click the copy icon.

<figure><img src="/files/Qbwe3WSE9o6YOYnzueEn" alt=""><figcaption><p>Copy response icon for the Structural Agent</p></figcaption></figure>

## Providing feedback to the Agent

You can provide basic positive or negative feedback to indicate the helpfulness of a response. The Agent then uses that feedback to adjust its future responses in the current chat.

To provide positive feedback, click the thumbs up icon.

To provide negative feedback, click the thumbs down icon.


# Creating and managing workspaces

A Tonic Structural workspace provides a context within which to configure and generate transformed data.

A workspace represents a path between the source data and the transformed output data. For example, `postgres-prod-copy` to `postgres-staging`.

A workspace includes:

* Where to find the source data to transform during data generation
* Where to write the transformed data
* The rules for the transformation

<table data-view="cards"><thead><tr><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Workspaces view</strong><br><br>View, filter, and sort the list of workspaces that you have access to.</td><td><a href="/pages/VuZTv79aK60z2nx3kiI9">/pages/VuZTv79aK60z2nx3kiI9</a></td></tr><tr><td><strong>Create, edit, and delete workspaces</strong></td><td><a href="/pages/uARhWg4pWJJKxVrEbuGg">/pages/uARhWg4pWJJKxVrEbuGg</a></td></tr><tr><td><strong>Workspace management view</strong><br><br>Provides access to workspace configuration and generation tools.</td><td><a href="/pages/tY58NV84QsMNX33FlVbd">/pages/tY58NV84QsMNX33FlVbd</a></td></tr><tr><td><strong>Workspace inheritance</strong><br><br>Create child workspaces that inherit source data and configuration from their parent workspace.</td><td><a href="/pages/8xo2gopGT9L9GHrodJUY">/pages/8xo2gopGT9L9GHrodJUY</a></td></tr><tr><td><strong>Assign workspace tags</strong><br><br>Use tags to help organize and identify workspaces.</td><td><a href="/pages/tEVCNG3NxC4tRoN1bR6l">/pages/tEVCNG3NxC4tRoN1bR6l</a></td></tr><tr><td><strong>Export and import workspace configuration</strong></td><td><a href="/pages/0ZXfXdQ1oFJ9wItiPuLY">/pages/0ZXfXdQ1oFJ9wItiPuLY</a></td></tr></tbody></table>


# Viewing your list of workspaces

**Workspaces** view lists the workspaces that you have access to. To display **Workspaces** view, in the Tonic Structural heading, click **Workspaces**.

<figure><img src="/files/PQ2UlICBT8vDU4MrqnxR" alt=""><figcaption><p>Workspaces view</p></figcaption></figure>

## How the workspace list is displayed <a href="#workspaces-view-list-display" id="workspaces-view-list-display"></a>

The workspace list contains:

* Workspaces that you own
* Workspaces that you are granted access to

If you have the global permission **Copy any workspace** or **Manage user access to Tonic and to any workspace**, then list includes all of the workspaces.

The **Permissions** column lists the workspace permission sets that you are granted in each workspace. The permission sets include both permission sets that were granted to you directly as a user, and permission sets that were granted to an SSO group that you are a member of.&#x20;

[Child workspaces](/app/workspace/managing-workspaces/workspaces-inheritance) always display under their parent workspace. The list only includes child workspaces that you have access to. If you have access to a child workspace, but not to its parent workspace, then the parent workspace is grayed out. You cannot select it.

## Filtering the workspace list <a href="#workspaces-view-filtering" id="workspaces-view-filtering"></a>

You can filter the workspaces based on the following information:

* **Name -** In the filter field, begin to type text that is in the name of the workspaces to display in the list.
* **Owner -** From the **Filter by Owner** dropdown list, select the owner of the workspaces to display in the list.
* **Database type -** From the **Filter by Database Type** dropdown list, select the type of database for the workspaces to display in the list.
* **Generation status -** In the **Generation Status** column heading, click the filter icon. Check the checkbox next to the generation status values for the workspaces to display in the list.
* **Tags -** In the **Tags** column heading, click the filter icon. By default, the workspaces are not filtered by tag, and all of the checkboxes are unchecked.\
  \
  To only include workspaces that have specific tags, check the checkbox next to each tag to include.\
  \
  To uncheck all of the selected tags, click **Reset Tags**.\
  \
  When you filter by tag, Structural checks whether each workspace contains any of the selected tags.
* **Permissions -** In the **Permissions** column heading, click the filter icon. You can check and uncheck checkboxes to include or exclude specific permission sets. For example, you can filter the list to only display workspaces for which the Editor permission set is granted either to you or to an SSO group that you belong to.\
  \
  For users that have the global permission **Copy any workspace**, the **Permissions** filter panel also contains an **Any permissions** checkbox. By default, **Any permissions** is unchecked, and the list includes workspaces for which you are not assigned any workspace permission sets. To display all of the workspaces for which you have any assigned workspace permission sets, check **Any permissions**.\
  \
  If you filter the list based on a specific permission set, to clear the filter and show all workspaces for which you have any permission set, check **Any permissions**. To display all workspaces, including workspaces that you do not have any permissions for, uncheck **Any permissions**.

You can combine different filters. For example, you can filter the list to only include workspaces that use PostgreSQL and for which the generation status is Canceled or Failed.

Child workspaces always display under their parent workspace, even if the parent workspace does not match the filter.

## Sorting the workspace list <a href="#workspaces-view-sorting" id="workspaces-view-sorting"></a>

You can sort the workspace list by name, status, or owner.

By default, the list is sorted alphabetically by name.

To sort by a column, click the column heading. To reverse the order of the sort, click the column heading again.

Child workspaces always display under their parent workspace. The child workspaces are sorted within the parent.

## Workspace details on Workspaces view <a href="#workspaces-view-details" id="workspaces-view-details"></a>

**Workspaces** view provides the following information about each workspace:

* **Name -** Contains the name and database type for the workspace. To view the workspace description, hover over the name.
* **Generation status -** The status for the most recent generation job.\
  \
  To display the job details for the job, click the job status.\
  \
  To display more details about the date, time, and duration for the job, hover over the generation timestamp. If a job failed recently, you are given additional information about how long this job has been failing (the date of the first failure occurrence among a continuous series of failures).
* **Schema changes -** Indicates whether Structural detected changes to the source database schema. If there are changes, the column shows the number of changes.\
  \
  Hover over the column value to display additional details, and to navigate to the **Schema Changes** view. Go to [Viewing and resolving schema changes](/app/generation/schema-changes).
* **Tags -** The tags that are assigned to the workspace.
* **Permissions -** The permission sets that are assigned to you for the workspace.
* **Owner -** The name and email address of the workspace owner.

## Getting access to workspace tools and actions

### Displaying the workspace management view

On **Workspaces** view, when you click the workspace name, the [workspace management view](/app/workspace/managing-workspaces/selecting-a-workspace-to-manage) for the workspace is displayed. The **Privacy Hub** tab is selected.

The **Name** column also provides access to a menu of workspace configuration options. When you select an option, the [workspace management view](/app/workspace/managing-workspaces/selecting-a-workspace-to-manage) is displayed, open to the view for the selected option.

<figure><img src="/files/iWn46n7ahDhCXDcgcjum" alt=""><figcaption><p>Workspace tools menu</p></figcaption></figure>

### Options column

The last column in the workspaces list provides additional workspace options:

![Options column and dropdown
for a workspace](/files/39M0yYtXgiLtBPn7OdH1)

* **Subsetting icon -** Displays the subsetting configuration for the workspace. Go to [Viewing the current subsetting configuration](/app/generation/subsetting/subsetting-view-config).
* **Post-job actions icon -** Displays the post-job actions for the workspace. For more information, go to [Post-job scripts](/app/workflows/scripts) and [Webhooks](/app/workflows/webhooks).
* **Actions menu -** Provides access to additional options.

### Actions menu for bulk actions

The **Actions** menu at the top left of the workspaces list allows you to to perform bulk actions on multiple workspaces. It is enabled when you check one or more of the checkboxes in the first column of each row. The **Actions** menu provides options for the selected workspaces.

![Actions menu for selected workspaces](/files/k1w5R5AGZSStvcDgOzHh)


# Creating, editing, or deleting a workspace

## Creating a workspace <a href="#workspaces-create-options" id="workspaces-create-options"></a>

When you create a new workspace, you can either:

* [Create a completely new workspace.](#workspace-create-new)
* [Create a copy of an existing workspace.](#workspace-create-copy)\
  \
  The copy initially uses the configuration from the original workspace.\
  \
  After the copy is created, it is completely independent from the original workspace.
* [Create a child of an existing workspace.](#workspace-create-child)\
  \
  Child workspaces inherit configuration from the parent workspace. They continue to be updated automatically when the parent workspace is updated. For more information, go to [About workspace inheritance](/app/workspace/managing-workspaces/workspaces-inheritance).

You can also view this [video overview of how to create a workspace](https://youtu.be/iH-wBYJLQlk).

When you create a workspace, a new [Structural Agent](/app/structural-agent/agent-about) chat starts for the workspace.

### Creating a completely new workspace <a href="#workspace-create-new" id="workspace-create-new"></a>

{% hint style="info" %}
**Required global permission:** Create workspaces
{% endhint %}

To create a completely new workspace, on **Workspaces** view, click **Create Workspace > New Workspace**.

For information about the configuration settings for a workspace, go to [Workspace configuration settings](/app/workspace/workspace-configuration-settings).

On the new workspace view, the carousel at the bottom left includes an option to create a workspace that uses sample data that we provide. On the carousel, click the right arrow until the sample dataset option is selected.

<figure><img src="/files/eXFVZdnItTLVOFchN7GC" alt=""><figcaption><p>New Workspace option to create an example workspace that uses sample data</p></figcaption></figure>

### Creating a copy of a workspace <a href="#workspace-create-copy" id="workspace-create-copy"></a>

{% hint style="info" %}
**Required workspace permission:** Copy workspace (in the workspace to copy)

Or

**Required global permission:** Copy any workspace
{% endhint %}

To create a workspace based on an existing workspace, either:

* On the workspace management view of the workspace to copy, from the workspace actions menu, select **Duplicate Workspace**.

<figure><img src="/files/QnEOAa9aL9ds6tlMatib" alt=""><figcaption><p>Workspace actions menu</p></figcaption></figure>

* On **Workspaces** view, click the actions menu for the workspace, then select **Duplicate Workspace**.

![Workspace options column and dropdown list](/files/39M0yYtXgiLtBPn7OdH1)

When you create a copy of a workspace, the copy initially inherits the following workspace configuration:

* Source and destination database connections
* Sensitivity designations, including manual designations that override the sensitivity scan results
* Table mode assignments
* Generator configuration
* Subsetting configuration
* Post-job scripts

### Creating a child workspace <a href="#workspace-create-child" id="workspace-create-child"></a>

{% hint style="info" %}
**Required license:** Enterprise

**Required workspace permission:** Create child workspaces (in the parent workspace)
{% endhint %}

You can create a workspace that is a child of an existing workspace. You cannot create a child workspace of another child workspace.

The parent workspace must have a source database configured. You cannot create a child workspace from a workspace that uses the Databricks, self-managed Spark cluster, or MongoDB data connector.

#### From within a specific workspace

To create a child workspace from within a specific workspace, either:

* On **Workspaces** view, click the actions menu for the parent workspace, then select **Create Child Workspace**.
* On the workspace management view, from the workspace actions menu, select **Create Child Workspace**.

On the new child workspace view, **Parent Workspace** identifies the parent workspace.

<figure><img src="/files/ij3Fjgp4xaFFebaKQLza" alt=""><figcaption><p>New Child Workspace view for a child workspace created directly from a </p></figcaption></figure>

#### From the Workspaces list or New Workspace view

To create a child workspace from outside of a specific workspace:

* On **Workspaces** view, click **Create Workspace > Child Workspace**.
* On the new workspace view, in the carousel at the bottom left, click to navigate to the **Create a child workspace** view, then click **Create child workspace**.&#x20;

<figure><img src="/files/WJ0nzH2t256cf1SaFqg7" alt=""><figcaption><p>Create Child Workspace option on the New Workspace view</p></figcaption></figure>

When you use one of these options to create a child workspace, then **Parent Workspace** is not populated.

<figure><img src="/files/SWBRPDY03osZyqNtWEjw" alt=""><figcaption><p>New Child Workspace view for a new child workspace without a parent selected</p></figcaption></figure>

From the **Parent Workspace** dropdown list, select the parent workspace for the new child workspace.

## Editing a workspace <a href="#workspaces-edit" id="workspaces-edit"></a>

{% hint style="info" %}
**Required workspace permission:** Configure workspace settings
{% endhint %}

To edit the configuration for an existing workspace, either:

* On the workspace management view:
  * On the workspace navigation bar, click **Workspace Settings**.
  * From the workspace actions menu, select **Workspace Settings**.
* On **Workspaces** view, click the actions menu for the workspace, then select **Workspace Settings**.

## Deleting a workspace <a href="#workspaces-delete" id="workspaces-delete"></a>

{% hint style="info" %}
**Required workspace permission:** Delete workspace
{% endhint %}

You can delete workspaces that you no longer need.

You cannot delete a parent workspace. You must first delete all of its child workspaces.

To delete a workspace:

* On the workspace management view, from the workspace actions menu, select **Delete Workspace**.
* On the **Workspaces** view, click the actions menu for the workspace, then select **Delete**.
* On the **Workspace Settings** view, click **Delete Workspace**.


# About the workspace management view

You use the workspace management view to configure and run data generation for an individual workspace.

When you log in to Tonic Structural, it displays the workspace management view for the workspace that was selected when you logged out.

## Components of the workspace management view

<figure><img src="/files/XCm2gMjBAHvctRIWaImS" alt=""><figcaption><p>Workspace management view for a workspace</p></figcaption></figure>

The workspace management view includes the following components.

### Workspace information

The top left of the workspace management view provides information about the workspace, including:

<figure><img src="/files/XDmSNvUDyUwrmdaRy8Tr" alt=""><figcaption><p>Workspace information</p></figcaption></figure>

* The workspace name
* When the workspace was last updated
* The user who last updated the workspace
* Whether the workspace is a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance)

### Workspace options

The top right of the workspace management view provides general options for working with the workspace, including:

<figure><img src="/files/cFgKMPJKupEIu2jjXTNc" alt=""><figcaption><p>Workspace options</p></figcaption></figure>

* The workspace share icon, to [grant workspace access to other users and groups](/app/workspace/workspace-access-management/workspace-sharing)
* The workspace download menu to:
  * Download sensitivity scan and privacy reports
  * [Export and import workspace configuration](/app/workspace/managing-workspaces/workspace-export-import-config)
* The history icon, to [display the **Version History**](/app/generation/protection-audit-trail)
* The workspace actions menu
* The **Generate Data** button, to [start a data generation job](/app/workflows/data-generation-run-job)

### Workspace navigation bar

The workspace navigation bar provides access to workspace configuration options.

<figure><img src="/files/efLJH2WXCtinoPbnLYYa" alt=""><figcaption><p>Workspace navigation bar</p></figcaption></figure>

## Displaying the workspace management view

To display the workspace management view for a workspace:

* On **Workspaces** view, in the **Name** column either:
  * Click the workspace name. The workspace management view opens to **Privacy Hub**.
  * Click the dropdown icon, then select a workspace management option.

<figure><img src="/files/GuVhdMII4Uv8d4BgEInA" alt=""><figcaption><p>Workspace tools menu</p></figcaption></figure>

* Click the search field at the top. A list of available type the name of the workspace. As you type, Tonic displays a list of matching workspaces. In the list, click the workspace name.

<figure><img src="/files/GJA56z1xhnF9u8KQYNwf" alt=""><figcaption><p>Workspace search</p></figcaption></figure>

## Collapsing and expanding the workspace heading

To reduce the amount of vertical space used by the heading of the workspace management view, you can collapse it.

To collapse the heading, click the collapse icon in the Structural heading.

<figure><img src="/files/8wKT9RnByfJ3PAEh27ak" alt=""><figcaption><p>Workspace heading with collapse option highlighted</p></figcaption></figure>

When you collapse the workspace management heading:

* The workspace information is hidden. The workspace name is displayed in the search field.
* The workspace options are moved up into the Structural heading.

The workspace navigation bar remains visible.

When you collapse the heading, the collapse icon changes to an expand icon. To restore the full heading, click the expand icon.

<figure><img src="/files/8nJFpIHe7RkepB2MuUXn" alt=""><figcaption><p>Collapsed workspace heading with expand option highlighted</p></figcaption></figure>


# About workspace inheritance

{% hint style="info" %}
**Required license:** Enterprise
{% endhint %}

If you have multiple workspaces, then it is likely that many of the workspace components and configurations are the same or similar. It can be difficult to maintain that consistency across separate, independent workspaces.

When you copy a workspace, the new workspace is completely independent of the original workspace. There is no visibility into or inheritance of changes from the original workspace.

Workspace inheritance allows you to create workspaces that are children of a selected workspace. Unlike a copy of a workspace, a child workspace remains tied to its parent workspace.

By default, a child workspace's configuration is synchronized with the configuration of the parent. In other words, any changes to the parent workspace are copied to its child workspaces. Child workspaces can also override some of the parent configuration. From the parent workspace, you can track the child workspaces and how they are customized .

For example, you might want separate workspaces for different development teams. Each team can make adjustments to suit their specific projects - such as different subsets - but inherit everything else.

## What does a child workspace inherit? <a href="#workspace-inheritance-inherited-items" id="workspace-inheritance-inherited-items"></a>

By default, a child workspace inherits all of the configuration from the parent workspace, except for the following:

* **Workspace name -** A child workspace has its own name.
* **Workspace description -** A child workspace has its own description.
* **Tags -** A child workspace has its own tags.
* **Destination database -** A child workspace writes output data to its own destination database. You can copy the destination database from the parent workspace.
* **Intermediate database -** For upsert, a child workspace does not inherit the intermediate database.
* **Webhooks -** A child workspace has its own webhooks.

## How parent workspace changes affect child workspaces <a href="#workspace-inheritance-parent-effects" id="workspace-inheritance-parent-effects"></a>

When you change the configuration of a parent workspace, the configuration is also updated in the child workspaces.

The exception is when a child workspace overrides the configuration. If the configuration is overridden, then the child workspace does not inherit the change.

Tonic Structural indicates on both the parent and child workspaces when the configuration is overridden.

## What can a child workspace override? <a href="#workspace-inheritance-child-overrides" id="workspace-inheritance-child-overrides"></a>

A child workspace can override the following configuration items.

* **Schema management settings** **-** A child workspace can override the settings to determine how to respond to schema changes and whether to cache the source database schema.
* **Table modes -** A child workspace can override the table mode for individual tables. The other tables continue to inherit the table mode that is configured in the parent workspace.
* **Column generators -** A child workspace can override the generator for individual columns. The other columns continue to inherit the generator that is configured in the parent workspace.\
  \
  For linked columns, a change to any of the linked columns overrides the inheritance for all of the columns.
* **Subsetting -** A child workspace can override the subsetting configuration from the parent workspace.\
  \
  Any change in the child workspace means that the child workspace no longer inherits any changes to the subsetting configuration from the parent workspace. For example, if you change the percentage setting on a single target table from 5 to 6, that eliminates the subsetting inheritance.\
  \
  The child workspace keeps the subsetting configuration that it already has, but it is not updated when the parent workspace is updated.
* **Post-job scripts -** A child workspace can override the post-job scripts.\
  \
  Any change to the post-job scripts in the child workspace means that the child workspace no longer inherits any changes to the post-job scripts configuration.
* **Statistics seed -** A child workspace can override the [statistics seed configuration](/app/generation/generators/generator-characteristics/consistency#enabling-consistency-across-runs-or-multiple-databases).

From each view, you can eliminate the overrides and restore the inheritance.

## What must a child workspace inherit? <a href="#workspace-inheritance-cannot-override" id="workspace-inheritance-cannot-override"></a>

A child workspace cannot override the following configuration items:

* **Data connector type and source database -** A child workspace always uses the same source data as the parent workspace.
* **Advanced workspace overrides -** A child workspace cannot add, remove, or change any [advanced workspace overrides](/app/workspace/workspace-configuration-settings/advanced-overrides). It always uses the same overrides as the parent workspace.
* **Foreign keys -** A child workspace always uses the same foreign key configuration as the parent workspace.
* **Sensitivity designation for a column -** A child workspace cannot change whether a column is marked as sensitive.

## How schema changes are resolved in parent and child workspaces <a href="#workspace-inheritance-schema-changes" id="workspace-inheritance-schema-changes"></a>

For removed tables and columns, when a child workspace overrides the parent workspace configuration for the table or column, you must resolve the change in the child workspace.

If there is a conflicting change for the removed table or column in the parent workspace configuration, then regardless of whether the configuration is inherited, you must resolve that change in the parent workspace before the change is resolved for the child workspace.

For changes to column nullability or data type, you resolve the change separately in the child and parent workspaces.

You also dismiss notifications (new tables and columns) separately in the parent and child workspaces.&#x20;


# Assigning tags to a workspace

{% hint style="info" %}
**Required workspace permission:** Configure workspace settings
{% endhint %}

You can associate custom tags with each workspace. Tags can help to organize and provide a quick glance into the workspace configuration.

Tags are accessible to every user that has access to the workspace.

Tags are stored in the workspace JSON, and are included in the workspace export. You can also use the API to get access to tags.

## Managing tags from workspace settings <a href="#tags-manage-workspace-settings" id="tags-manage-workspace-settings"></a>

You can add and edit tags in the **Tags** field on the **New Workspace** and **Workspace** **Settings** views.

* To add tags, enter a comma-separated list of the tags to add.
* To remove a tag, click its delete icon.

## Managing tags from Workspaces view <a href="#tags-manage-workspaces-view" id="tags-manage-workspaces-view"></a>

You can also manage tags directly from **Workspaces** view.

### Assigning tags <a href="#tags-assign-new" id="tags-assign-new"></a>

To add tags to a workspace that does not currently have tags:

1. Hover over the **Tags** column for the workspace.
2. Click **Add Tags**.
3. In the tag input field, type a comma-separated list of tags to apply.
4. Press **Enter**.

### Editing the assigned tags <a href="#tags-edit" id="tags-edit"></a>

To edit the assigned tags:

1. Click the **Tags** column for the workspace.
2. In the tag input field, to remove tag, click its delete icon.
3. To add tags, type a comma-separated list of the tags to add.
4. To save the tag changes, press **Enter**.


# Exporting and importing the workspace configuration

{% hint style="info" %}
**Required workspace permission:** Export and import workspace
{% endhint %}

You can export a workspace configuration to a JSON file, and import configuration from a workspace configuration JSON file.

For example, you might want to preserve a version of the workspace configuration before you test other changes. You can then use the exported file to restore the original configuration.

Or you might want to use a script to make changes to an exported configuration file. You can then import the updated file to update the workspace configuration.

## Information in the exported file <a href="#workspace-config-file-info" id="workspace-config-file-info"></a>

The workspace JSON configuration file includes the following information:

* Sensitivity designations that you assigned to columns
* Assigned table modes
* Assigned column generators
* Subsetting configuration
* Post-job script configuration

## Exporting the workspace configuration <a href="#workspace-export-config" id="workspace-export-config"></a>

To export the workspace configuration, either:

* On the workspace management view, from the download menu, select **Export Workspace**.

<figure><img src="/files/5wQ496YyNKVrpEalTP3E" alt=""><figcaption><p>Download menu for a workspace</p></figcaption></figure>

* On **Workspaces** view, click the actions menu for the workspace, then select **Export**.

![Workspace options column and dropdown list](/files/39M0yYtXgiLtBPn7OdH1)

When you export a child workspace, the exported workspace does not retain any of the inheritance information. The exported information is the same for all exported workspaces.

## Importing a workspace configuration file <a href="#workspace-import-config" id="workspace-import-config"></a>

To import a workspace configuration file:

1. Select the import option. Either:
   * On the workspace management view, from the download menu, select **Import Workspace**.
   * On **Workspaces** view, click the actions menu for the workspace, then select **Import**.
2. On the **Import Workspace** dialog, to select the file to import, click **Browse**.
3. After you select the file, click **Import**.

When you import a workspace configuration into a child workspace, Tonic Structural only updates the configuration that can be overridden. If a configuration must be inherited from the parent workspace, then it is not affected by the imported configuration. For more information, go to [About workspace inheritance](/app/workspace/managing-workspaces/workspaces-inheritance).


# Workspace configuration settings

The workspace settings for a new workspace (**New Workspace** view) or edited workspace (**Workspace** **Settings** tab) provide information about the workspace and its data.

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Identification and worker zone</strong></td><td>Settings to identify the workspace and to route the workspace jobs to a specific set of Structural workers.</td><td></td><td><a href="/pages/MBBpZJ4384LdUTv3Cnot">/pages/MBBpZJ4384LdUTv3Cnot</a></td></tr><tr><td><strong>Data connection settings</strong></td><td>Select the connector type. Connect to source and destination data.</td><td></td><td><a href="/pages/vHxsFKBu5epnRv3T5HgO">/pages/vHxsFKBu5epnRv3T5HgO</a></td></tr><tr><td><strong>Use secrets managers</strong><br><br>Configure and select secrets managers secrets to use for authentication.</td><td></td><td></td><td><a href="/pages/OHGvv4bKCSylB8hBp8EO">/pages/OHGvv4bKCSylB8hBp8EO</a></td></tr><tr><td><strong>Schema management settings</strong></td><td>Block data generation on schema changes.</td><td></td><td><a href="/pages/KCqlYisVjYzohWVPLIlw">/pages/KCqlYisVjYzohWVPLIlw</a></td></tr><tr><td><strong>Enable and configure upsert</strong></td><td>Add new destination records and update changed destination records. Ignore other unchanged destination records.</td><td></td><td><a href="/pages/Rlyl0zrCJVDRRw4NtDKL">/pages/Rlyl0zrCJVDRRw4NtDKL</a></td></tr><tr><td><strong>Write output to a container repository</strong></td><td>Use the data generation output to populate a container data volume.</td><td></td><td><a href="/pages/VtpkGsrBMiGpGZwcRG7y">/pages/VtpkGsrBMiGpGZwcRG7y</a></td></tr><tr><td><strong>Advanced workspace overrides</strong></td><td>Workspace-specific settings for cross-run consistency and data generation performance.</td><td></td><td><a href="/pages/9fuBxxyFNpJHTWtMpKFD">/pages/9fuBxxyFNpJHTWtMpKFD</a></td></tr></tbody></table>


# Workspace identification and worker zone

The **Workspace Details** section for every workspace includes the following settings to identify the workspace and to route the workspace jobs to a specific set of Structural workers.

## Fields to identify the workspace <a href="#workspaces-config-common-fields" id="workspaces-config-common-fields"></a>

All workspaces have the following fields that identify the workspace:

1. In the **Name** field, enter the name of the workspace.
2. In the **Tags** field, provide a comma-separated list of tags to assign to the workspace. For more information on managing tags, go to [Assigning tags to a workspace](/app/workspace/managing-workspaces/workspace-tags).
3. In the **Description** field, provide a brief description of the workspace. The description can contain up to 200 characters.

## Workspace job routing zone

### About worker zones

Workspace jobs are processed by Tonic Structural workers.

Each can optionally be assigned to a zone. Each zone can contain multiple workers.

Structural Cloud provides a set of zones, as well as a set of workers that are not assigned to zones.

On a self-hosted instance, you can [configure the list of zones and assign your Structural workers to those zones](/app/admin/on-premise-deployment/configuring-worker-zones).

### Assigning a workspace to a zone

On the workspace settings view, from the **Zone** dropdown list, select the name of the zone to route the workspace jobs to.

If no worker zones are configured, then the dropdown is not displayed.

By default, a workspace is not assigned to a zone. The workspace jobs are processed by a worker that is also not assigned to a zone.

Note that the following cases cause the workspace jobs to not be processed:

* A workspace is not assigned to a zone, but all of the workers are assigned to zones.
* A workspace is assigned to a zone, but the zone has no assigned workers.


# Data connection settings

For each workspace, you select the connector type.

After you select the connector type, you configure:

* Where to find the source data
* Where to write the data generation output
* How to make the connections

Not all connectors support all of the connection options. The workspace configuration information for each connector type lists the supported settings.

<table data-view="cards"><thead><tr><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Connection type</strong><br><br>Select the data connector to use for the workspace data.<br></td><td><a href="/pages/BCmegwBkttnAKc3XN2Lc">/pages/BCmegwBkttnAKc3XN2Lc</a></td></tr><tr><td><strong>Data locations</strong><br><br>Configure the data source and destination information.</td><td><a href="/pages/nGePNQFOULEkO4GpSxfN">/pages/nGePNQFOULEkO4GpSxfN</a></td></tr><tr><td><strong>Authentication encryption</strong><br><br>Use SSL/TLS to encrypt authentication.</td><td><a href="/pages/2sZpDmNv2ovxK2LiC6nG">/pages/2sZpDmNv2ovxK2LiC6nG</a></td></tr><tr><td><strong>Trust the server certificate</strong><br><br>Indicate whether to trust the server certificate for authentication.</td><td><a href="/pages/zPhi1vsOV8bfy8bs4YQY">/pages/zPhi1vsOV8bfy8bs4YQY</a></td></tr><tr><td><strong>Provide custom certificates</strong><br><br>Provide your own client and root certificates.</td><td><a href="/pages/lJyzuYLCUrSUVDFUj6Zz">/pages/lJyzuYLCUrSUVDFUj6Zz</a></td></tr><tr><td><strong>Use connection tunneling</strong><br><br>Use SSH or Tailscale tunneling for database connection.</td><td><a href="/pages/9dES3kKFU5xiJOIpxSKE">/pages/9dES3kKFU5xiJOIpxSKE</a></td></tr><tr><td><strong>Connection testing</strong><br><br>Test your data connections.</td><td><a href="/pages/HzsLCqngU3HxZkxdXJQn">/pages/HzsLCqngU3HxZkxdXJQn</a></td></tr></tbody></table>


# Connection type

When you create a workspace, under **Workspace Details**, from the **Connection Type** dropdown list, select the type of data connector to use for the workspace data.

You cannot change the connection type on a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance).

<figure><img src="/files/IxXpcoJjVwlJr5S6vLHN" alt=""><figcaption><p>Connection Type dropdown list for a new workspace</p></figcaption></figure>

The Professional license limits the number and type of data connectors you can use. A Professional instance can use up two different data connector types, which can be any type other than Oracle or Db2 for LUW. After you create workspaces that use two different data connector types, any subsequent workspaces must use one of those data connector types.

If the database that you want to connect to isn't in the list, or you want to have different database types for your source and destination database, contact <support@tonic.ai>.

When you select a connector type, Structural updates the view to display the connection fields used for that connector type. The specific fields vary based on the [connector type](/app/setting-up-your-database/data-connector-summary).

After you save a new workspace, you cannot change the workspace connection type.

<figure><img src="/files/GVoY7OSLLdj5LTIJ8FZk" alt=""><figcaption><p>Locked connection type value on a saved workspace</p></figcaption></figure>


# Data locations

## Source database connection <a href="#workspace-config-source-database" id="workspace-config-source-database"></a>

For data connectors that connect to a database, the **Source Settings** section provides connection information for the source database.

You cannot change the source data configuration for a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance).&#x20;

For information about the source connection fields for a specific data connector, go to the workspace configuration topic for that [connector type](/app/setting-up-your-database/data-connector-summary).

## Upsert configuration <a href="#workspace-config-connection-upsert" id="workspace-config-connection-upsert"></a>

For data connectors that support upsert, the workspace configuration includes an **Upsert** section to allow you to enable and configure upsert. Upsert adds and updates rows in the destination database, but keeps all other existing rows intact.&#x20;

If you enable upsert, then you cannot write output to a container repository. You must write the output to a destination database.

For more information, go to [Enabling and configuring upsert](/app/workspace/workspace-configuration-settings/workspace-config-upsert).

## Destination data location <a href="#workspace-config-destination-data" id="workspace-config-destination-data"></a>

For data connectors that connect to a database, the **Destination Settings** section provides information about where and how Structural writes the output data from data generation.

Depending on the data connector type, you might be able to write to either:

* Destination database - Writes the output data to a destination database on a database server.
* Container repository - Writes the output data to a data volume in a container repository.

### Destination database <a href="#workspace-config-connection-source-destination" id="workspace-config-connection-source-destination"></a>

When you write the output to a destination database, the destination database must be of the same type as the source database.

Structural does not create the destination database. It must exist before you generate data.

In **Destination Settings**, you provide the connection information for the destination database. For information about the destination database connection fields for a specific data connector, go to the workspace configuration topic for that [connector type](/app/setting-up-your-database/data-connector-summary).

If available, the **Copy Settings from Source** allows you to copy the source connection details to the destination database, if both databases are in the same location. Structural does not copy the connection password.

### Container repository <a href="#workspace-connection-other-config-containers" id="workspace-connection-other-config-containers"></a>

Some data connectors allow you to write the transformed data to a data volume in a container repository instead of to a database server.

For more information, go to [Writing output to a container repository](/app/workspace/workspace-configuration-settings/workspace-config-write-to-container-artifacts).

## File connector source and destination data <a href="#workspace-config-connection-file-connector" id="workspace-config-connection-file-connector"></a>

A [file connector](/app/setting-up-your-database/file-connector) workspace uses files as its source data and produces transformed versions of those files as its output.

For file connector workspaces, the **File Location** section indicates where the source files are obtained from - either a local file system or a cloud storage solution (Amazon S3 or Google Cloud Storage).

When the files come from cloud storage, the **Output Location** section indicates where to write the transformed files. You must also provide the cloud storage connection credentials.

For more information, go to [Configuring the file connector storage type and output options](/app/setting-up-your-database/file-connector/file-connector-workspace-config).


# Ensuring encryption of database authentication

For data connectors that support it, the **Enable SSL/TLS** setting indicates whether to encrypt the database authentication.

By default, it is in the on position. We strongly recommend that you do not turn off this setting.


# Trusting the server certificate

For data connectors that support it, to indicate that Tonic Structural should trust the server certificate, toggle **Trust Server Certificate** to the on position.<br>


# Providing your own client certificates

For data connectors that support it, to specify your own client certificates for authentication:

1. Click the expand icon for **Client certificate settings**.
2. The settings can include one or more of the following:
   1. For **Client Cert**, choose the client certificate file.
   2. For **Client Key**, choose the key file for the client certificate.
   3. For **Root Cert**, choose the root certificate file.

For all of these settings, if secrets managers are available, you can instead [select a secret name from a secrets manager](/app/workspace/workspace-configuration-settings/secrets-manager/selecting-a-secrets-manager-secret).


# Using a connection tunneling option

For additional security, for data connectors that support it, you can specify a connection tunneling option for your data connections.

Structural currently supports:

* Routing the data connections through an SSH bastion.
* Routing the data connections through a [Tailscale](https://tailscale.com/blog/how-tailscale-works) tailnet. Tailscale is currently supported on MySQL, Oracle, PostgreSQL, and SQL Server. Snowflake also supports Tailscale as an option instead of a proxy connection for the [source](/app/setting-up-your-database/snowflake/connecting-to-snowflake/snowflake-source-connection) and [destination](/app/setting-up-your-database/snowflake/connecting-to-snowflake/snowflake-destination-connection) connections.

<table data-view="cards"><thead><tr><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Connect through an SSH bastion</strong><br><br>Set up and use an SSH bastion for connection tunneling.</td><td><a href="/pages/Ip0rEzfvBRyJUaOHpAbD">/pages/Ip0rEzfvBRyJUaOHpAbD</a></td></tr><tr><td><strong>Connect through Tailscale</strong><br><br>Use Tailscale for connection tunneling.</td><td><a href="/pages/ecJPbpT1zRd6GOJTDcIn">/pages/ecJPbpT1zRd6GOJTDcIn</a></td></tr></tbody></table>


# Connecting through an SSH bastion

An SSH bastion is a hardened, publicly accessible server that provides a single entry point into a private network.

When you use an SSH bastion, you do not need to allow direct access from Structural to to your database servers. Instead, you only allow access to the bastion. Structural then SSHes through the bastion to reach the servers.

## Setting up your SSH bastion

Before you configure a workspace to connect through an SSH bastion, make sure to configure the bastion as follows.

### **CPU architecture**

To offload encryption tasks, the CPU must support the AES-NI instruction set.

For example, on AWS, you would use Generation 4 instances, such as C4 or M4, or newer.

### **Instance type**

Avoid burstable instances, such as AWS t3 and t3a. During high-throughput transfers, these can be throttled, which can lead to dropped connections.

Use instances that have dedicated CPU resources.

### **Software and cipher compatibility**

High-speed ciphers require a modern version of OpenSSH (OpenSSH 6.5+ ).

We recommend that you privilege ChaCha20-Poly1305 and AES-GCM.

### **Prioritize high-speed encryption**

To prioritize high-speed encryption, in `/etc/ssh/sshd_config`, add or update the `Ciphers` line:

{% code overflow="wrap" %}

```
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
```

{% endcode %}

### **Allowlist Structural**

To allow for high-concurrency and multiplexing, and prevent Structural from being throttled, at the end of `/etc/ssh/sshd_config`, add this `Match` block:

{% code overflow="wrap" %}

```
# Global setting for unauthenticated connections
MaxStartups 100:30:200

# Specific limits for Structural Source IP
Match Address <Structural_Source_IP>
    MaxAuthTries 100
    MaxSessions 1000
```

{% endcode %}

## **Connecting to an SSH bastion from a Structural workspace**

In the workspace configuration, to connect to an SSH bastion :

1. If the data connector supports multiple tunneling options:

   1. Toggle **Enable Connection Tunnel** to the on position.
   2. From the **Tunnel type** dropdown list, select **SSH Tunnel**.

   If the data connector only supports SSH tunneling, then toggle **Enable SSH Tunnel** to the on position.
2. In the **SSH Host** field, provide the host for the SSH bastion.
3. In the **SSH Port** field, provide the port for the SSH bastion.
4. In the **SSH User** field, provide the name of the user to use to connect to the SSH bastion.
5. If you do not use a private key, then in the **SSH Passphrase** field, provide the passphrase to use for authentication.\
   \
   If secrets managers are available, you can instead [select a secret name from a secrets manager](/app/workspace/workspace-configuration-settings/secrets-manager/selecting-a-secrets-manager-secret).
6. If you do use a private key, then in the **SSH Private Key** field, provide the private key.\
   \
   If secrets managers are available, you can instead [select a secret name from a secrets manager](/app/workspace/workspace-configuration-settings/secrets-manager/selecting-a-secrets-manager-secret).\
   \
   If the private key uses a passphrase, then in the **SSH Passphrase** field, provide the passphrase for the private key.


# Connecting through Tailscale

[Tailscale](https://tailscale.com/blog/how-tailscale-works) is a managed virtual private network (VPN). It creates a private mesh network called a tailnet.

Each device on the Tailscale network receives a stable private IP address, and can communicate directly with other devices on the tailnet, without the need to open ports to the public internet or to configure VPN gateways.

Tonic Structural can join your corporate tailnet as a node. It can then reach database servers that are on the same tailnet, even if those servers are in a private network and have no public IP address.

## Overview of the steps to enable the connection <a href="#tailscale-enable-connection" id="tailscale-enable-connection"></a>

To allow Tonic Structural to reach a database through Tailscale, you:

1. [**Allow connections from the tailnet to your database server.**](#allowing-connections-to-the-database-server-from-the-tailnet)
2. [**Add your database server to the tailnet**](#adding-your-database-server-to-the-tailnet) - Install the Tailscale agent on the server, or configure a subnet router.
3. [**Declare an access control list (ACL) tag**](#declaring-an-acl-tag) - Register the tag in the tailnet policy that Structural uses to identify itself.
4. [**Create an OAuth credential**](#creating-an-oauth-credential) - Generate the client secret that Structural uses to authenticate to your tailnet.
5. [**Configure the Structural workspace**](#configuring-a-workspace-to-use-tailscale-tunneling) - Enable the Tailscale tunnel and provide the connection details.

## Allowing connections to the database server from the tailnet

Connections from Structural arrive at the server's Tailscale IP address, not the server's actual IP address.

You must configure your database server to allow connections from your tailnet's IP address range.

By default, the range is `100.64.0.0/10`, although it can be customized.

For a PostgreSQL database server, you add an entry for the subnet in `pg_hba.conf`.

## Adding your database server to the tailnet

To make your database server reachable through Tailscale, you can either:

* Install Tailscale directly on the database server.
* Use a subnet router.

### Installing Tailscale directly on the database server

If you have shell access to the database server, then this is the simplest approach.

1. Install Tailscale on the server.\
   \
   For platform-specific instructions, go to [tailscale.com/download](https://tailscale.com/download).
2. If a browser is available:
   1. Run `sudo tailscale up`.\
      \
      This returns a link that looks like:<br>

      ```
      https://login.tailscale.com/a/0123456789abcdef
      ```
   2. Open the URL in a browser.\
      \
      Tailscale prompts for authentication and then adds the server to your tailnet.
3. For headless servers where no browser is available, use the `--auth-key` flag instead:
   1. To generate an auth key, go to `https://login.tailscale.com/admin/settings/keys` .
   2. Click **Generate auth key**, then indicate whether the key is:

      * &#x20;A one-off key - single use, for a single server.
      * A reusable key - for multiple servers.

      The generated key starts with `tskey-`.
   3. After you generate the key, run:<br>

      ```bash
      sudo tailscale up --auth-key=<key>
      ```
4. Retrieve the server's Tailscale IP address:<br>

   ```bash
   tailscale ip -4
   ```
5. Verify that the device appears at `https://login.tailscale.com/admin/machines`.

The server now has a stable Tailscale IP address, which is by default in the `100.x.x.x` range.

You use the Tailscale IP address as the value of **Server** in the workspace data connection settings.

### Using a subnet router

If you cannot install Tailscale on the database server itself, for example because it is a managed database service, you can instead:

1. Install Tailscale on any other machine that is on the same private network as the database server.
2. Configure the machine as a [subnet router](https://tailscale.com/kb/1019/subnets).

A subnet router advertises a CIDR range to your tailnet. Other nodes such as Structural can then use regular IP addresses to reach the devices on that subnet. Tailscale does not need to be installed on those devices.

To configure a subnet router:

1. Install Tailscale on a machine that can reach the database server.
2. Start Tailscale with the `--advertise-routes` flag, which specifies the subnet that your database is on:<br>

   ```bash
   sudo tailscale up --advertise-routes=10.0.1.0/24
   ```
3. Approve the advertised route:
   1. Go to `https://login.tailscale.com/admin/machines`**.**
   2. Click the machine.
   3. Click **Edit route settings**.
   4. Enable the route.

When you use a subnet router, then in Structural, you set the value of **Server** to the database server's regular private IP address - for example, `10.0.1.50` - instead of a Tailscale IP address.

## Declaring an ACL tag

Tailscale uses ACL tags to control the devices that a given credential can authenticate to.

Before you create an OAuth credential, you must declare the tag that you intend to use in your tailnet's ACL policy.

1. Go to `https://login.tailscale.com/admin/acls`**.**
2. Edit the policy JSON.
3. If the policy does not already contain a `tagOwners` section, then add the section.
4. Add the tag to `tagOwners`. For example, to create a tag called `tag:tonic`:

```json
"tagOwners": {
    "tag:tonic": ["your-tailscale-email@example.com"],
},
```

3. Save the policy.

## Creating an OAuth credential

Structural authenticates to your tailnet using an OAuth client credential. You create this from the Tailscale admin console under **Trust credentials**.

1. Go to `https://login.tailscale.com/admin/settings/trust-credentials`.
2. Click **+ Credential**.
3. Select **OAuth** as the credential type.

<figure><img src="/files/AiL4IdfIilmPWd0OBL9R" alt=""><figcaption><p>New Tailscale credential with OAuth selected as the credential type</p></figcaption></figure>

4. Add a description, then click **Next**.
5. To set the credential scope:
   1. From the **Scopes** dropdown, select **Custom scopes.**
   2. Under **Keys**, for **Auth Keys**, check the **Write** checkbox.
   3. In the **Tags** field, provide the ACL tag that you created.
6. Click **Generate credential**.\
   \
   Tailscale displays the **Credential created** panel with the client ID and the client secret.

<figure><img src="/files/6Lq12hPYVeRG3pV9bTLv" alt=""><figcaption><p>Credential created panel with the client ID and client secret for the Tailscale OAuth credential</p></figcaption></figure>

7. Copy the **Client secret** immediately. It is only shown once.\
   \
   The client secret starts with `tskey-client-`.\
   \
   In the Structural workspace configuration, you paste this value into the **Tailscale** **Auth Key** field. Structural does not use the client ID.

## Configuring a workspace to use Tailscale tunneling

After you complete the setup in Tailscale, you can configure a Structural workspace to connect through Tailscale.

### Setting the database connection server value

When you connect through Tailscale, in the database connection fields, to set the value of **Server**:

* If you installed Tailscale on the database server, set **Server** to the Tailscale IP address, which is the stable address that is assigned to the device on your tailnet. By default, the address in the  `100.x.x.x` range.
* If you use a subnet router, then in Structural, set **Server** to the database server's regular private IP address - for example, `10.0.1.50`.

### Enabling and configuring the Tailscale connection

To enable and configure the Tailscale connection:

1. If the connector supports multiple tunneling options:

   1. Toggle **Enable Connection Tunnel** to the on position.
   2. From the **Tunnel type** dropdown list, select **Tailscale Tunnel**.

   If the data connector only supports Tailscale tunneling, then it is selected automatically.
2. In the **Tags** field, provide a comma-separated list of ACL tags to use to join the tailnet. For example, `tag:tonic-worker`.\
   \
   Each tag name must exactly match the tag name that you configured in Tailscale, and must include the `tag:` prefix.\
   \
   If the tags don't match, then when Structural tries to connect, Tailscale returns an authentication error.
3. Optionally, in the **Control Server URL** field, provide the URL. This is intended for customers who use Headscale or a self-hosted Tailscale control plane.
4. In the **Tailscale Auth Key** field, you can either:
   * Provide the Tailscale OAuth client credential (value starting with `tskey-client-`) that you generated and copied from Tailscale. Structural does not support other types of Tailscale keys, such as user auth keys.
   * If secrets managers are available, [select a secret name from a secrets manager](/app/workspace/workspace-configuration-settings/secrets-manager/selecting-a-secrets-manager-secret).

## How the connection works

When Structural uses Tailscale tunneling to establish a connection:

1. Structural uses the provided credential (OAuth credential or secrets manager secret) to authenticate as a new ephemeral node on your tailnet.
2. The node is tagged with the specified ACL tags. The tags control what the node can reach based on your tailnet policy.
3. Structural uses its Tailscale IP address to connect to the database server. Traffic flows through the encrypted tailnet mesh, not the public internet.
4. When the connection is torn down, the ephemeral node is removed from your tailnet.

## Troubleshooting the connection

Here are some errors that might occur during the connection:

<table><thead><tr><th width="254.19921875" valign="top">Error</th><th valign="top">Likely cause</th></tr></thead><tbody><tr><td valign="top">Tailscale authentication failed</td><td valign="top"><p>Either:<br></p><ul><li>The tag in the <strong>Tags</strong> field does not match the tag on the OAuth credential's <code>auth_keys</code> scope.</li><li>The tag is not declared in <code>tagOwners</code> in your ACL policy.</li></ul></td></tr><tr><td valign="top">Connection times out</td><td valign="top"><p>Either:</p><p></p><ul><li>The database server is not on the tailnet.</li><li>The ACL policy does not permit the tagged node to reach it.</li></ul></td></tr><tr><td valign="top">Invalid auth key type</td><td valign="top">You used a user auth key or a pre-auth key instead of an OAuth client secret.</td></tr></tbody></table>


# Connection testing

When you provide connection details for a database server, Structural provides a **Test Connection** button to test the connection. We strongly recommend that you test your database connections.

Structural also runs connection tests when you open an existing workspace and when you start data generation.&#x20;

Structural is currently migrating its data connectors from the legacy connection test to a newer suite of connection tests.

## Legacy connection test

**Used by:** Connectors other than file connector, MySQL, Oracle, PostgreSQL, Snowflake, SQL Server

The legacy connection test verifies whether Structural is able to connect to the database server.

For some data connectors, the legacy connection test also checks whether the database user has sufficient permissions to perform the required Structural tasks.

When you run the legacy connection test, Structural indicates whether the test was successful.

## New connection test suite

**Used by:** File connector, MySQL, Oracle, PostgreSQL, Snowflake, SQL Server

For the new connection test suite, the exact tests vary based on the data connector and the connection role, such as source or destination.

At a high level, the new tests can include:

* **Configuration -** Is the connection configuration valid?
* **Connectivity -** Can Structural connect to the server?
* **Schema queries -** Can Structural retrieve the data schema?
* **Data queries -** Can Structural retrieve the data?
* **Data generation -** Can Structural start and run a data generation job? Can Structural write generated data?

When you run the new connection test suite, Structural:

* Lists the tests that it ran.
* Indicates whether each test was successful.
* For failed tests, the results:
  * Indicate whether the failure is fatal, a warning, or informational.\
    \
    Fatal issues must be resolved. Other types of failures might not block functionality, but might have unexpected side effects.
  * When possible, provide a hint as to how to resolve the failure. You can also use the **Ask AI** option to ask an LLM to troubleshoot the issue and suggest next steps.

## Configuring timeouts for the connection tests

To configure the timeouts for the connection tests, set the following [environment settings](/app/admin/environment-variables-setting). You can set these settings from the **Environment Settings** tab on **Structural Settings**.

* `TONIC_TEST_CONNECTION_TIMEOUT_IN_SECONDS` - Used by both the legacy and the new connection tests. The number of seconds before a general connectivity test times out. By default, the connection test times out after 15 seconds.
* `TONIC_DATASOURCE_TEST_TIMEOUT_SECONDS` - Used by the new connection tests only. The number of seconds before an individual test times out. By default, the test times out after 300 seconds.
* `TONIC_DATASOURCE_TEST_SUITE_TIMEOUT_SECONDS` - Used by the new connection tests only. The number of seconds before a connection test suite times out. By default, the suite of tests times out after 600 seconds.

## Testing the connection speed

For the following data connectors, if Structural is able to connect to the database, then it also tests and reports on the connection speed.

* MySQL
* Oracle
* PostgreSQL
* SQL Server


# Using secrets managers for authentication

{% hint style="info" %}
**Required license:** Enterprise
{% endhint %}

Your organization might use a secrets manager to secure credentials, including database connection credentials.

You can configure a set of available secrets managers. In the workspace configuration, for fields that support it, you can then select a secret name from a secrets manager.

<table data-view="cards"><thead><tr><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Configure available secrets managers</strong><br><br>Set up and test the secrets managers that can be used.<br><br></td><td><a href="/pages/MdlymJIKnMcSiuENCE6b">/pages/MdlymJIKnMcSiuENCE6b</a></td></tr><tr><td><strong>Select a secret from a secrets manager</strong><br><br>In the workspace configuration, identify a secret to use for the connection.</td><td><a href="/pages/wtPF0ynXXCQVoIemxlAM">/pages/wtPF0ynXXCQVoIemxlAM</a></td></tr></tbody></table>


# Configuring the available secrets managers

{% hint style="info" %}
**Required global permission:** Manage secrets managers
{% endhint %}

## Supported secrets manager tools and formats <a href="#secrets-manager-platforms-formats" id="secrets-manager-platforms-formats"></a>

Structural currently supports:

* AWS Secrets Manager
* HashiCorp Vault
* Cyberark Central Credential Provider
* Azure Key Vault

## Viewing the secrets manager list <a href="#secrets-manager-list" id="secrets-manager-list"></a>

To display the list of secrets managers, on **Structural Settings** view, click **Secrets Managers**.

<figure><img src="/files/kqP98qQNRACXsYcewVA5" alt=""><figcaption><p>Secrets Manager stab on Structural Settings</p></figcaption></figure>

## Working with secrets managers <a href="#secrets-manager-config" id="secrets-manager-config"></a>

### Creating a secrets manager <a href="#secrets-manager-create" id="secrets-manager-create"></a>

To create a secrets manager:

1. On the **Secrets Managers** tab, click **Add Secrets Manager**.
2. On the **Create Secrets Manager** panel, in the **Name** field, provide a name to use to identify the secrets manager.\
   \
   Secrets manager names must be unique.\
   \
   The name is used in the secrets manager dropdown list on the workspace settings view.
3. From the **Type** dropdown list, select the secrets manager product.
4. Configure the credentials to use to connect to the secrets manager.
5. Click **Save**.

### Editing an existing secrets manager <a href="#secrets-manager-edit" id="secrets-manager-edit"></a>

For an existing secrets manager, you can change the name and the credentials configuration.

You cannot change the type.

To edit an existing secrets manager:

1. In the secrets manager list, click the edit icon for the secrets manager.
2. On the **Edit Secrets Manager** panel, update the configuration.

<figure><img src="/files/iE8UxH6TWrF5fgoD7HxT" alt=""><figcaption><p>Edit Secrets Manager panel</p></figcaption></figure>

3. Click **Save**.

### Deleting a secrets manager <a href="#secrets-manager-delete" id="secrets-manager-delete"></a>

When you delete a secrets manager, it is removed from the workspace database connections that use it. Structural is no longer able to connect to those databases.

To delete a secrets manager:

1. In the secrets manager list, click the delete icon for the secrets manager.
2. On the confirmation panel, click **Delete**.

## Providing credentials for AWS Secrets Manager <a href="#credentials-aws-secrets-manager" id="credentials-aws-secrets-manager"></a>

### Required AWS Secrets Manager permissions <a href="#secrets-manager-aws-permissions" id="secrets-manager-aws-permissions"></a>

The AWS Secrets Manager credentials that you provide must have the following permissions:

* On each secret to use, `secretsmanager:GetSecretValue`&#x20;
* On the encryption key for secrets that are encrypted with a customer managed key (CMK), `kms:Decrypt`

Here is an example policy that grants the required Secrets Manager permissions:

{% code overflow="wrap" %}

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:111111111111:secret:mySecretNamespace/*"
    }
  ]
}
```

{% endcode %}

### Selecting the source of the credentials <a href="#secrets-manager-aws-credentials-source" id="secrets-manager-aws-credentials-source"></a>

For AWS Secrets Manager, from the **Authentication** dropdown list, select the source of the credentials:

<figure><img src="/files/o5chLqBGQNZlwHSmLMqA" alt=""><figcaption><p>Authentication options for AWS Secrets Manager</p></figcaption></figure>

* **Environment -** Only available on self-hosted instances.\
  \
  Indicates to use either:
  * The credentials for the AWS Identity and Access Management (IAM) role on the host machine.
  * The credentials set in the following [environment settings](/app/admin/environment-variables-setting):
    * `TONIC_AWS_ACCESS_KEY_ID` - An AWS access key that is associated with an IAM user or role
    * `TONIC_AWS_SECRET_ACCESS_KEY` - The secret key that is associated with the access key
    * `TONIC_AWS_REGION` - The AWS Region to send the authentication request to
* **Assume Role -** Indicates to use the specified assumed role.
* **User Credentials -** Indicates to use the provided user credentials.

### Providing an assumed role <a href="#credentials-aws-assumed-role" id="credentials-aws-assumed-role"></a>

To provide an assumed role, select **Assume Role**, then:

<figure><img src="/files/ACFhJ9jhunVeqrQc3Ckw" alt=""><figcaption><p>Configuration fields for the Assume Role option for AWS Secrets Manager credentials</p></figcaption></figure>

1. In the **Role ARN** field, provide the Amazon Resource Name (ARN) for the role.
2. In the **Session Name** field, provide the role session name.\
   \
   If you do not provide a session name, then Structural automatically generates a default unique value. The generated value begins with `TonicStructural`.
3. In the **Duration (in seconds)** field, provide the maximum length in seconds of the session. \
   \
   The default is 3600, indicating that the session can be active for up to 1 hour.\
   \
   The provided value must be less than the maximum session duration that is allowed for the role.
4. From the **AWS Region** dropdown list, select the AWS Region to send the authentication request to.

Structural generates the external ID that is used in the assume role request. Your role’s trust policy must be configured to condition on your unique external ID.

Here is an example trust policy:

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "AWS": "<originating-account-id>"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "<external-id>"
      }
    }
  }
}
```

### Providing AWS user credentials <a href="#credentials-aws-credentials" id="credentials-aws-credentials"></a>

To provide the credentials, select **User Credentials**, then:

<figure><img src="/files/AJtJPQ9GWBx3MCgGLREh" alt=""><figcaption><p>Configuration fields for the User Credentials option for AWS Secrets Manager credentials</p></figcaption></figure>

1. In the **AWS Access Key** field, enter the AWS access key that is associated with an IAM user or role.
2. In the **AWS Secret Key** field, enter the secret key that is associated with the access key.
3. Optional. In the **AWS Session Token** field, provide the session token to use.
4. From the **AWS Region** dropdown list, select the AWS Region to send the authentication request to.

## Configuring a HashiCorp Vault secrets manager

For a HashiCorp Vault secrets manager:

<figure><img src="/files/Y5fthRpAlR4HEHs0IT8b" alt=""><figcaption><p>Configuration panel for a HashiCorp Vault secrets manager</p></figcaption></figure>

1. In the **Vault Server** field, provide the server name or IP address where the secrets manager is located.
2. From the **Secrets Engine** dropdown list, select the version of the secrets engine used by the secrets manager.
3. From the **Authentication Method** dropdown list, select how to authenticate to the secrets manager. The options are:
   * **AppRole**
   * **Token**
   * **LDAP**
4. For app role authentication:

   1. In the **Role ID** field, provide the identifier of the application role.
   2. In the **Secret ID** field, provide the secret identifier of the application role.

   On a self-hosted instance, if you do not provide a role identifier and secret identifier, then Structural uses the values of the [environment settings](/app/admin/environment-variables-setting) `TONIC_SECRET_MANAGERS_HASHICORP_VAULT_APPROLE_ROLE_ID` and `TONIC_SECRET_MANAGERS_HASHICORP_VAULT_APPROLE_SECRET_ID`.
5. For token authentication, in the **Token** field, provide the authentication token to use.\
   \
   On a self-hosted instance, if you do not provide a token, Structural uses the value of the [environment setting](/app/admin/environment-variables-setting) `TONIC_SECRET_MANAGERS_HASHICORP_VAULT_TOKEN` to authenticate.
6. For LDAP authentication:

   1. In the **Username** field, provide the LDAP username.
   2. In the **Password** field, provide the password for the LDAP user.

   On a self-hosted instance, if you do not provide a username and password, then Structural uses the values of the [environment settings](/app/admin/environment-variables-setting) `TONIC_SECRET_MANAGERS_HASHICORP_VAULT_LDAP_USERNAME` and `TONIC_SECRET_MANAGERS_HASHICORP_VAULT_LDAP_PASSWORD`.
7. If the authentication method is enabled in a specific namespace, then in the **Namespace** field, provide the namespace.
8. If the selected authentication method does not use the default mount path, then in the **Mount path** field, provide the mount path.

## Configuring a CyberArk Central Credential Provider secrets manager

For a CyberArk secrets manager:

<figure><img src="/files/nKTg9joSqojDAnLWvWtF" alt=""><figcaption><p>Configuration panel for a CyberArk Credential Provider secrets manager</p></figcaption></figure>

1. In the **CyberArk Server** field, provide the server name or IP address where the secrets manager is located.
2. In the **CyberArk Port** field, provide the port to use to connect to the secrets manager.
3. In the **CyberArk Application ID** field, provide the identifier of the CyberArk application that contains the secrets manager.
4. In the **CyberArk Safe** field, provide the name of the CyberArk safe that contains the secrets.
5. Optional. In the **CyberArk Folder** field, provide the name of the folder within the safe that contains the secrets.\
   \
   If you do not specify a folder here or in the workspace configuration, the folder defaults to `Root`.\
   \
   To specify a folder path, use `Root` followed by the rest of the path, with each path component separated by backslashes. For example: `Root\OS\Linux`.

By default, Structural uses the application identifier to authenticate to CyberArk.

You can instead use a CyberArk authentication certificate.

When you use a certificate, then by default, during authentication, Structural validates the certificate that the server sends. You might need to install a root certificate on the Structural web server and workers. Instead, to bypass the validation, set the [environment setting](/app/admin/environment-variables-setting) `TONIC_SECRET_MANAGERS_CYBER_ARK_CCP_VERIFY_SERVER_CERTIFICATE`  to `false`.

To configure the CyberArk authentication certificate:

1. Click **Certificate**.

<figure><img src="/files/a63A3aX1m71BIHDBgVzT" alt=""><figcaption><p>Configuration for a CyberArk authentication certificate</p></figcaption></figure>

2. Under **CyberArk Authentication Certificate**, to search for and select the certificate file, click **Browse**.\
   \
   The certificate must be either .pfx or .p12.
3. In the **CyberArk Certificate Password** field, provide the password that is associated with the certificate.

## Configuring an Azure Key Vault secrets manager

For an Azure Key Vault secrets manager:

<figure><img src="/files/ClUb1RV5Vm0u0IZTrojb" alt=""><figcaption><p>Configuration panel for an Azure Key Vault secrets manager</p></figcaption></figure>

1. In the **Vault Name** field, provide the name of the key vault.
2. In the **Tenant ID** field, provide your Azure tenant identifier to use to authenticate to the vault.
3. In the **Client ID** field, provide your Azure client identifier to use to authenticate to the vault.
4. In the **Client Secret** field, provide your Azure client secret to use to authentiate to the vault.

## Testing a secrets manager configuration

From the configuration panel, you can test whether Structural can use the configured credentials to connect to the secrets manager and retrieve a specific secret.

### Running the configuration test

To test a secrets manager configuration:

1. Click **Test Secrets Manager**.
2. On the **Test Secrets Manager** panel, in the **Secret Name** field, provide the name of the secret.
3. If needed for the secrets manager type, provide any other information that is needed for the connection.
4. Click **Run Test**.

### Additional test information for HashiCorp Vault

For a HashiCorp Vault secrets manager, in addition to the secret name, you can provide the following if applicable:

* **Namespace -** If the secret is in a specific namespace, the name of the namespace.
* **Mount Path -** If the secrets engine does not use the default mount path, the mount path to use.

### Additional test information for CyberArk Central Credential Provider

For a CyberArk Central Credential Provider secrets manager, in addition to the secret name, you can optionally provide:

* **CyberArk Safe -** The name of the CyberArk safe that contains the secrets manager.
* **CyberArk Folder** - The name of the folder within the safe that contains the secrets manager.

If you do not provide these values, they fall back to the values that are configured for the secrets manager.


# Selecting a secrets manager secret

For fields that support secrets managers, such as database password fields, if at least one secrets manager is available, then at the top right of the field is a **Use Secret** link.

<figure><img src="/files/3ICIyePPefvJcfVSGgzc" alt=""><figcaption><p>Use Secret option for a field that supports using a secret from a secrets manager</p></figcaption></figure>

## Indicating to use a secret

To use a secret to populate the value:

1. Click **Use Secret**.

<figure><img src="/files/BKXHfZ8DeztNcwnUZv4z" alt=""><figcaption><p>Use Secret panel to select the secret to use</p></figcaption></figure>

2. On the **Use Secret** panel, from the **Secrets Manager** dropdown list, select the name of the secrets manager that contains the secret.
3. Based on the secrets manager type, Structural prompts you for the information needed to identify and retrieve the secret.
4. After you provide the required information, click **Confirm**.
5. Structural changes the link to **Using Secret**, and disables the field.

<figure><img src="/files/P8DGqqyKw5E19A9fiRJX" alt=""><figcaption><p>Field marked as using a secret from a secrets manager</p></figcaption></figure>

## Updating the secret selection

To change the secret selection or other information about the selected secret:

1. Click the **Using Secret** link.
2. On the **Use Secret** panel, update the information.
3. Click **Confirm**.

## Selecting a secret from AWS Secrets Manager

When you select a secrets manager from AWS Secrets Manager:

1. In the **Secret ARN or Name** field, provide the name or ARN of the secret.
2. If the secret is part of a structured key-value pair, then in the **Property Name** field, provide the property name that contains the secret value.

<figure><img src="/files/PorMqW0rLuWHPsjxYBVF" alt=""><figcaption><p>Fields to identify a secret from an AWS secrets manager</p></figcaption></figure>

## Selecting a secret from HashiCorp Vault

When you select a secrets manager from HashiCorp Vault:

<figure><img src="/files/g8XqgZBrizuveeHy1ru2" alt=""><figcaption><p>Secret selection panel for a HashiCorp Vault secret</p></figcaption></figure>

### Using chained credentials

For HashiCorp vault, for additional security, you can choose to use chained credentials. When you enable chained credentials, you provide a set of credentials that is used in turn to retrieve the credentials that are used to retrieve the specified secret.

To enable chained credentials, toggle **Use chained credentials** to the on position. The **Chained Credentials Configuration** is displayed.

<figure><img src="/files/WHtXw5Wr6TIKs8Mi6Oyh" alt=""><figcaption><p>Chained credentials configuration for a HashiCorp Vault secret</p></figcaption></figure>

#### Selecting the authentication method

From the **Method** dropdown list, select the type of authentication to use:

* **AppRole**
* **LDAP**
* **Token**

#### Configuring the shared authentication settings

For all authentication types:

1. If the selected authentication method is enabled in a specific namespace, then in the first **Namespace** field, provide the namespace.
2. If the selected authentication method does not use the default mount path, then in the first **Mount path** field, provide the mount path.
3. In the **Secret Name** field, provide the name of the secret.
4. If the vault is enabled in a specific namespace, then in the second **Namespace** field, provide the namespace.
5. If the secrets engine does not use the default mount path, then in the second **Mount path** field, provide the mount path.

#### Configuring app role authentication

For app role authentication:

1. In the **Role ID** field, provide the name of the secret property that contains the identifier of the application role.
2. In the **Secret ID** field, provide the name of the secret property that contains the secret identifier of the application role.

#### Configuring token authentication

For token authentication, in the **Token** field, provide the name of the secret property that contains the authentication token to use.

#### Configuring LDAP authentication

For LDAP authentication:

1. In the **LDAP Username** field, provide the name of the secret property that contains the LDAP username.
2. In the **LDAP Password** field, provide the name of the secret property that contains the password for the LDAP user.

### Providing the database secret

If you use chained credentials, then the database secret fields are under **Database Secret Configuration**.

To provide information about the secret to retrieve:

1. In the **Secret Name** field, provide the name of the secret.
2. If the secret is in a specific namespace, then in the **Namespace** field, provide the namespace.
3. If the authentication does not use the default mount path, then in the **Mount Path** field, provide the mount path.
4. If the secret is part of a structured key-value pair, then in the **Property Name** field, provide the property name that contains the secret value.

## Selecting a secret from CyberArk Central Credential Provider

When you select a secrets manager from CyberArk Central Credential Provider:

<figure><img src="/files/g8jBB9sB9bTlwxwonnNq" alt=""><figcaption><p>Secret selection panel for a CyberArk Central Credential Provider secret</p></figcaption></figure>

1. In the **Secret Name** field, provide the name of the secret.
2. Optionally:

   1. In the **CyberArk Safe** field, provide the name of the CyberArk safe that contains the secrets manager.
   2. In the **CyberArk Folder** field, provide the name of the folder within the safe that contains the secrets manager.\
      \
      If you do not specify a folder here, Structural uses the folder configured in the secrets manager. If a folder is not configured in the secrets manager, then the folder defaults to `Root`.\
      \
      To specify a folder path, use Root followed by the rest of the path, with each path component separated by backslashes. For example: `Root\OS\Linux`.

   If you do not provide these values, they fall back to the values that are configured for the secrets manager.

## Selecting a secret from Azure Key Vault

When you select a secrets manager from Azure Key Vault:

<figure><img src="/files/bpj2h5bXUNhW3hqQLhFb" alt=""><figcaption><p>Secret selection panel for an Azure Key Vault secret</p></figcaption></figure>

1. In the **Secret Name** field, provide the name of the secret.
2. If the secret is part of a structured key-value pair, then in the **Property Name** field, provide the property name that contains the secret value.

## Removing the secret selection

To remove the secret selection entirely, and enable a value to be entered manually.

1. Click the **Using Secret** link.
2. On the **Use Secret** panel, click **Remove**.


# Schema management settings

On **Workspace Settings** view for a workspace, the **Schema Changes** section contains the schema management settings..

<figure><img src="/files/mw4JiK6AqxwZJYxGqZ6V" alt=""><figcaption><p>Schema management settings for a workspace</p></figcaption></figure>

## **Responding to schema changes** <a href="#workspace-config-schema-change-handling" id="workspace-config-schema-change-handling"></a>

Schema changes include:

* Schema changes that could expose data, which if not addressed can result in data leakage. These changes include new tables and columns, and changes to data types.
* Notifications, which Structural can handle automatically during each data generation. These include removed tables and columns.

For more information, go to [Viewing and resolving schema changes](/app/generation/schema-changes).

### Selecting the handling option

On the **Workspace Settings** view, under **Block Data Generation on Schema Changes**, select how Structural responds when there are unaddressed changes to the database schema.

The options are:

* **Do Not Block -** With this option, schema changes never block data generation.\
  \
  When you select this option, then Structural automatically handles notifications during data generation.<br>

  The **Automatically apply generators** toggle determines how Structural responds to changes that could expose data.

  * When this is enabled, then when Structural detects schema changes, it automatically applies generators to the affected columns.
  * When it is disabled, Structural does not change the column configuration.
* **Block On Changes That Could Expose Data -** Indicates to only block data generation if there are schema changes that might expose data, such as new columns.\
  \
  Structural automatically handles notifications during data generation.\
  \
  For this option, Structural does not block data generation for schema changes on truncated tables.
* **Block On All Changes -** For this option, if there are any unaddressed schema changes at all, either sensitive changes or notifications, then data generation fails.

### How Structural selects generators to apply automatically

When Structural automatically applies generators to columns affected by a schema change:

* For new columns or document fields that are detected as sensitive, Structural applies the recommended generator for the sensitivity type.
* For new columns or document fields that are not sensitive, Structural applies an appropriate generator based on the data type.
* For changes to column data types, nullability, or uniqueness, Structural applies a new recommended generator or an appropriate generator for the data type.

Here is a summary of how Structural determines the generator to apply to columns that are not sensitive and do not match an existing sensitivity type or custom sensitivity rule:

<table><thead><tr><th width="244.41796875" valign="top">Data type</th><th valign="top">Generator selection</th></tr></thead><tbody><tr><td valign="top">Integer</td><td valign="top">If unique or a primary key, <a href="/pages/2KDUZbkl4mGozBdm7vog">Integer Key</a>.<br><br>Otherwise <a href="/pages/J0no3tPglJjbX7pcB3Rj">Random Integer</a>.</td></tr><tr><td valign="top">Boolean</td><td valign="top"><a href="/pages/00UvmmnzkwyY2aVyrRkZ">Random Boolean</a></td></tr><tr><td valign="top">Datetime</td><td valign="top">Either <a href="/pages/fk3ME8SWpUNErj5wHXU9">Timestamp Shift</a>, <a href="/pages/zFcGQfMbzYUhu2FxIBsP">Date Truncation</a>, or <a href="/pages/y3Ko1JJbu1073EdEk87c">Random Timestamp</a></td></tr><tr><td valign="top">Text</td><td valign="top"><p>If unique or a primary key, either <a href="/pages/CfuJagRPVBZmPM0oxBhU">Aphanumeric String Key</a>, <a href="/pages/qeq2VIVXiMjO54X2J9Ez">ASCII Key</a>, or <a href="/pages/dV9aZtcRZdnNEXx7CJEr">Numeric String Key</a>.<br></p><p>Otherwise either <a href="/pages/YkyIbAnIucBzXXgehrmT">Character Scramble</a> or <a href="/pages/JRbQS9Aj4nQn2To8ncff">Categorical</a>.</p></td></tr><tr><td valign="top">Continuous</td><td valign="top"><p>If unique or a primary key, <a href="/pages/2KDUZbkl4mGozBdm7vog">Integer Key</a>.<br></p><p>Otherwise <a href="/pages/XlbTgf3f5IPlg1ePNLuX">Random Double</a>.</p></td></tr><tr><td valign="top">Network</td><td valign="top"><a href="/pages/KduWaTvOoFwua7IXNaWx">IP Address</a></td></tr><tr><td valign="top">MAC address</td><td valign="top"><a href="/pages/DCIwJMeIf96Pkx82inIz">MAC Address</a></td></tr><tr><td valign="top">UUID</td><td valign="top"><p>If unique or a primary key, <a href="/pages/ltEMEZ8lClXIOLV3EWrk">UUID Key</a>.<br></p><p>Otherwise <a href="/pages/6mcOp7hMjQG64xzgQmHX">Random UUID</a>.</p></td></tr><tr><td valign="top">JSON</td><td valign="top"><p>If Struct format, <a href="/pages/O93C5heA0jclbt3E0m1a">Struct Mask</a>.<br></p><p>Otherwise <a href="/pages/DuCeR6mVkXvBqA8p34h5">JSON Mask</a>.</p></td></tr><tr><td valign="top">XML</td><td valign="top">Either <a href="/pages/ApVlvOeIW4OwiEmivdhr">XML Mask</a> or <a href="/pages/EJBjN3p71wRjD4x5ja7c">HTML Mask</a></td></tr><tr><td valign="top">Other</td><td valign="top">If HStore format, <a href="/pages/VTTM6x7sMxY6mOYl1nLS">HStore Mask</a></td></tr><tr><td valign="top">User-defined</td><td valign="top"><a href="/pages/JRbQS9Aj4nQn2To8ncff">Categorical</a></td></tr></tbody></table>

## Indicating whether to cache the source schema <a href="#schema-cache-config" id="schema-cache-config"></a>

{% hint style="info" %}
Schema caching is not available for document-based databases (MongoDB).
{% endhint %}

### About the schema cache <a href="#schema-cache-about" id="schema-cache-about"></a>

By default, every time you load a workspace, Structural queries the source database to retrieve the schema.

You can instead configure the workspace to cache the schema. Structural then updates the cache at a regular interval, and whenever a change to the workspace triggers a schema cache update.

You can also trigger a cache update manually.

By default, the schema cache is only used by calls from within Structural. To enable an external API request to use the cached schema, add the query parameter `useSchemaCache=true` to the request.

In the application, each update to the schema cache is represented by a schema retrieval job. Schema retrieval jobs are short-lived, and run on the Structural web server. You can view the schema retrieval jobs from the [workspace **Jobs** view](/app/workspace/jobs).

Note that the schema cache does not include the schema for JSON columns that use **Document View**. Those schemas are detected by a different scan.

### Enabling and configuring schema caching <a href="#schema-cache-enable-configure" id="schema-cache-enable-configure"></a>

To enable and configure the caching:

1. On the **Workspace Settings** view, toggle **Cache source schema for faster loading** to the on position.
2. Under **Schema Freshness**, configure the maximum length of time between schema retrievals.

   1. In the field, provide the value.
   2. From the dropdown list, select the unit of time. You can configure the length of time in minutes, hours, or days.

   If the cached schema is older than that length of time, then the next time the application loads, it queries the source database for the current schema.\
   \
   The default value is 6 hours.\
   \
   Note that for some data connectors, schema retrievals run automatically in the background. This setting does not affect the frequency of those schema retrievals.\
   \
   For example, a schema retrieval runs automatically in the background every 2 hours. If you set the schema freshness to 6 hours, the background retrieval still runs every 2 hours. However, if you set the schema freshness to 1 hour, then schema retrieval occurs no more than 1 hour after the previous schema retrieval.
3. You can optionally enable diagnostic logging for the schema retrieval. Diagnostic logging adds additional diagnostic errors to help with troubleshooting.\
   \
   Note that this additional information might contain sensitive information such as schema identifiers.\
   \
   To enable diagnostic logging:
   1. Click **Show advanced options**.
   2. Toggle **Enable diagnostic logging** to the on position.


# Enabling and configuring upsert

{% hint style="info" %}
Not compatible with writing output to a container repository.
{% endhint %}

By default, Tonic Structural data generation replaces the existing destination database with the transformed data from the current job.

Upsert adds and updates rows in the destination database, but keeps all of the other existing rows intact. For example, you might have a standard set of test records that you do not want to replace every time you generate data in Structural.

If you enable upsert, then you cannot write the destination data to a container repository. You must write the data to a database server.

Upsert is currently only supported for the following data connectors:

* MySQL
* Oracle
* PostgreSQL
* SQL Server

For an overview of upsert, you can also view the [video tutorial](https://www.youtube.com/watch?v=veNRwlk8ArU).

## About the upsert process <a href="#workspace-config-upsert-about" id="workspace-config-upsert-about"></a>

When upsert is enabled, the data generation job writes the generated data to an intermediate database. Structural then runs the upsert job to write the new and updated records to the destination database.&#x20;

<figure><img src="/files/Fqqt2r6gaq3tyM5pT4Ya" alt=""><figcaption><p>Data generation process with upsert</p></figcaption></figure>

The destination database must already exist. Structural cannot run an upsert job to an empty destination database.

The upsert job adds and updates records based on the primary keys.

* If the primary key for a record already exists in the destination database, the upsert job updates the record.
* If the primary key for a record does not exist in the destination database, the upsert job inserts a new row.

To only update or insert records that Structural creates based on source records, and ignore other records that are already in the destination database, ensure that the primary keys for each set of records operate on different ranges. For example, allocate the integer range 1-1000 for existing destination database records that you add manually. Then ensure that the source database records, and by extension the records that Structural creates during data generation, use a different range.

Also note that when upsert is enabled, the Truncate [table mode](/app/generation/table-modes) does not actually truncate the destination table. Instead, it works more like Preserve Destination table mode, which preserves existing records in the destination table.

## Enabling upsert <a href="#workspace-config-upsert-enable" id="workspace-config-upsert-enable"></a>

To enable upsert, in the **Upsert** section of the workspace details, toggle **Enable Upsert** to the on position.

When you enable upsert for a workspace, you are prompted to configure the upsert processing and provide the connection details for the intermediate database.

## Configuring upsert processing <a href="#workspace-config-upsert-process-config" id="workspace-config-upsert-process-config"></a>

When you enable upsert, Structural displays the following settings to configure the upsert process.

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Disable Triggers</strong></td><td valign="top">Indicates whether to disable any user-defined triggers before the upsert job runs. This prevents duplicate rows from being added to the destination database.<br><br>By default, this is enabled.</td></tr><tr><td valign="top"><strong>Automatically Start Upsert After Successful Data Generation</strong></td><td valign="top">Indicates whether to immediately run the upsert job after the initial data data generation to the intermediate database.<br><br>By default, this is enabled.<br><br>If you turn this off, then after the initial data generation, you must start the upsert job manually. For more information, go to <a data-mention href="/pages/XuTwfC9rv4Cc3OxeXCyJ#data-gen-run-upsert-only">/pages/XuTwfC9rv4Cc3OxeXCyJ#data-gen-run-upsert-only</a>.</td></tr><tr><td valign="top"><strong>Persist Conflicting Data Tables</strong></td><td valign="top">When an upsert job cannot process rows with unique constraint conflicts, as well as rows that have foreign keys to those rows, this setting indicates whether to preserve the temporary tables that contain those rows.<br><br>By default, this is disabled.<br><br>Structural only keeps the applicable temporary tables from the most recent upsert job.</td></tr><tr><td valign="top"><strong>Warn on Mismatched Constraints</strong></td><td valign="top">Indicates whether to treat mismatched foreign key and unique constraints between the source and destination databases as warnings instead of errors, so that the upsert job does not fail.<br><br>By default, this is disabled.</td></tr></tbody></table>

## Connecting to migration scripts for schema changes <a href="#workspace-config-upsert-migration-server" id="workspace-config-upsert-migration-server"></a>

{% hint style="info" %}
**Required license:** Enterprise
{% endhint %}

The intermediate database must have the same schema as the destination database. If the schemas do not match, then the upsert process fails.

To ensure that schema changes are automatically reflected in the intermediate database, you can connect the workspace to your own database migration script or tool. Structural then runs the migration script or tool whenever you run upsert data generation.

### **How upsert works with the migration process** <a href="#upsert-with-data-migration-process-overview" id="upsert-with-data-migration-process-overview"></a>

When you start an upsert data generation job:

<figure><img src="/files/X57ygIPl0PSNu3NKIEeO" alt=""><figcaption><p>Upsert data generation process with migration</p></figcaption></figure>

1. If migration is enabled, Structural calls the endpoint to start the migration.
2. Structural cannot start the upsert data generation until the migration completes successfully. It regularly calls the status check endpoint to check whether the migration is complete.
3. When the migration is complete, Structural starts the upsert data generation.

### **POST Start Schema Changes endpoint**

Required. Structural calls this endpoint to start the migration process specified by the provided URL.

The request includes:

* Any custom parameter values that you add.
* The connection information for the intermediate database.

The request uses the following format:

```json
{ 
  "parameters": {/* user supplied parameters */ },
  "databaseConnectionDetails": {
        "server": "rds.amazon.com",
        "port": "54321",
        "username": "user",
        "password": "password",
        "databaseName": "tonic_upsert",
        "schemaName": "<Oracle schema to use>",
        "sslEnabled": true,
        "trustServerCertificate": false
  }
}
```

The response contains the identifier of the migration task.

The response uses the following format:

```json
{ "id": "<unique-string-identifier>" }
```

### **GET Status of Schema Change endpoint**

Required. Structural calls this endpoint to check the current status of the migration process.

The request includes the task identifier that was returned when the migration process started. The request URL must be able to pass the request identifier as either a path or a query parameter.

The response provides the current status of the migration task. The possible status values are:

* `Unknown`
* `Queued`
* `Running`
* `Canceled`
* `Completed`
* `Failed`

The response uses the following format:

```json
{
  "id": "a0c5c4c3-a593-4daa-a935-53c45ec255ea",
  "status": "Completed",
  "errors": []
}
```

### **GET Schema Change Logs endpoint**

Optional. Structural calls this endpoint to retrieve the log entries for the migration process. It adds the migration logs to the upsert logs.

The request includes the task identifier that was returned when the migration process started. The request URL must be able to pass the request identifier as either a path or a query parameter

The response body of the request should be `'text/plain'`. It contains the raw logs.

### **DELETE Cancel Schema Changes endpoint**

Optional. Structural calls this endpoint to cancel the migration process.

The request includes the task identifier that was returned when the migration process started. The request URL must be able to pass the request identifier as either a path or query parameter.

### **Enabling and configuring the migration process**

To enable the migration process, toggle **Enable Migration Service** to the on position.

When you enable the migration process, you must configure the `POST Start Schema Changes` and `GET Status of Schema Change` endpoints.

You can optionally configure the `GET Schema Change Logs` and `DELETE Cancel Schema Changes` endpoints.

To configure the endpoints:

1. To configure the `POST Start Schema Changes` endpoint:
   1. In the **URL** field, provide the URL of the migration script.
   2. Optionally, in the **Parameters** field, provide any additional parameter values that your migration scripts need.
2. To configure the `GET Status of Schema Change` endpoint, in the **URL** field, provide the URL for the status check.

   \
   The URL must include an `{id}` placeholder. This is used to pass the identifier that is returned from the `Start Schema Changes` endpoint.
3. To configure the `GET Schema Change Logs` endpoint, in the **URL** field, provide the URL to use to retrieve the logs.\
   \
   The URL must include an `{id}` placeholder. This is used to pass the identifier that is returned from the `Start Schema Changes` endpoint.
4. To configure the `DELETE Cancel Schema Changes` endpoint, in the **URL** field, provide the URL to use for the cancellation.\
   \
   The URL must include an `{id}` placeholder. This is used to pass the identifier that is returned from the `Start Schema Changes` endpoint.

## Connecting to the intermediate database

When you enable upsert, you must provide the connection information for the intermediate database.

For details, go to the workspace configuration information for the [data connector](/app/setting-up-your-database/data-connector-summary).

## How Structural responds to inconsistencies in the source and destination schemas <a href="#upsert-schema-change-handling" id="upsert-schema-change-handling"></a>

During upsert data generation, when Structural finds inconsistencies between the source and destination database schemas:

* Where possible, Structural attempts to address the issue so that the data generation can succeed.
* Structural does not change the schema of the destination database.
* For constraint-related schema issues, Structural only attempts to address the issues if **Warn on Mismatched Constraints** is enabled for the workspace. If the setting is turned off, then the job fails.

Here are some common schema issues that can occur, and how Structural responds to them.

### Source column is not in the destination schema <a href="#upsert-schema-column-not-in-dest" id="upsert-schema-column-not-in-dest"></a>

In this case, a column that is present in the source schema is not present in the destination schema.

<figure><img src="/files/G8LjU3YOojekCmyOySFP" alt=""><figcaption><p>Source column missing from destination schema</p></figcaption></figure>

For example, a new column is added to a production source table, but is not in the schema of the de-identified destination database that is used for testing.

When this occurs, Structural ignores the column. It does not add the column to the destination schema.

Structural adds a warning to the job logs.

### Destination column is not in the source schema <a href="#upsert-schema-column-not-in-source" id="upsert-schema-column-not-in-source"></a>

In this case, a column that is present in the destination schema is not present in the source schema.

<figure><img src="/files/4MuU0iP6sSK8aGuu6HGY" alt=""><figcaption><p>Destination column is missing from source schema</p></figcaption></figure>

For example, a developer adds a column to the de-identified destination database so that they can test a new feature. The new feature is not yet released, so the source production data doesn't include the column.

When this occurs:

<figure><img src="/files/8Oosh4l8XIv9ElYMU5J0" alt=""><figcaption><p>Flow to address a column missing from the source schema</p></figcaption></figure>

1. If the destination column is nullable, then Structural sets the value to NULL.
2. If the destination column is not nullable, but the column has a default value, then Structural sets the destination value to the default.
3. If the non-nullable destination column does not have a default value, then Structural attempts to set a value based on the column data type.\
   \
   For example, Structural might set an integer column to 0, or a varchar column to an empty string.
4. If Structural is unable to set a value, then the data generation fails and Structural returns an error.

### Source and destination columns have different data types <a href="#upsert-schema-data-type-mismatch" id="upsert-schema-data-type-mismatch"></a>

In this case, the same column has different data types in the source and destination schemas.

<figure><img src="/files/oJ9qNFgB6cjChbY6kNYR" alt=""><figcaption><p>Data type mismatch between the source and destination columns</p></figcaption></figure>

For example, a column might be a string in the source schema and a timestamp in the destination schema.

When this occurs, for each record:

<figure><img src="/files/gefTlXR88zORqgZ9rpU7" alt=""><figcaption><p>Flow to address a data type mismatch</p></figcaption></figure>

1. If possible, Structural converts the values.\
   \
   For example, the source column is a string and contains datetime values. The generator also produces datetime values. In that case, Structural should be able to populate a datetime destination column.
2. If it cannot convert the value, and the column is nullable, then Structural sets the destination column value to NULL.
3. If it cannot convert the value, and the column is not nullable, then the record is excluded from the upsert.

For each of these actions, Structural also adds warnings to the job logs.

If Structural cannot perform any of those actions to work around the issue, then the data generation fails and Structural returns an error.

### Source constraint is not in the destination schema <a href="#upsert-schema-constraint-not-in-dest" id="upsert-schema-constraint-not-in-dest"></a>

In this case, a constraint on a source column is not present in the destination schema.

<figure><img src="/files/vo4ttx3nD49Z7MyYrpHT" alt=""><figcaption><p>Constraint in source schema is not in the destination schema</p></figcaption></figure>

For example, a column is required in the source schema but optional in the destination schema.

If **Warn On Mismatched Constraints** is enabled for the workspace, then Structural does not have to make any changes to the data. It populates the destination column correctly.

Structural also adds a warning to the job logs.

If **Warn On Mismatched Constraints** is turned off, then the job fails.

### Destination constraint is not in the source schema <a href="#upsert-schema-constraint-not-in-source" id="upsert-schema-constraint-not-in-source"></a>

In this case, a constraint on a destination column is not present in the source schema.

<figure><img src="/files/4Nwaftcq0wPyRHCLLmPR" alt=""><figcaption><p>Constraint in destination schema is not in the destination schema</p></figcaption></figure>

For example, a column has no constraints in the source schema, but has a uniqueness constraint in the destination schema.

When this occurs, if **Warn on Mismatched Constraints** is enabled for the workspace, Structural removes any records that fail the constraint. For example, for a uniqueness constraint, Structural removes duplicate records.

Structural also adds warnings to the job logs.

If **Warn on Mismatched Constraints** is turned off, then the job fails.

### Source table is not in the destination schema <a href="#upsert-schema-table-not-in-dest" id="upsert-schema-table-not-in-dest"></a>

In this case, a table in the source schema is not present in the destination schema.

<figure><img src="/files/fMfX3wD4ENfnIsyx0c40" alt=""><figcaption><p>Source table column is not in the destination schema</p></figcaption></figure>

For example, a new table is added to a production source table, but is not yet in the schema of the de-identified destination database that is used for testing.

When this occurs, Structural ignores the table. It does not add the table to the destination schema.

Structural also adds warnings to the job logs.

### Destination table is not in the source schema <a href="#upsert-schema-table-not-in-source" id="upsert-schema-table-not-in-source"></a>

In this case, a table in the destination schema is not present in the source schema.

<figure><img src="/files/rxr83EFGAteMWHeL2gUC" alt=""><figcaption><p>Destination table is not in the source schema</p></figcaption></figure>

For example, a developer adds a table to the de-identified destination database so that they can test a new feature. Because the new feature is not yet released, the source production data doesn't include the table.

When this occurs, Structural ignores the table. It does not attempt to populate the destination table.

Structural also adds warnings to the job logs.

### Source or destination table is renamed <a href="#upsert-schema-table-rename" id="upsert-schema-table-rename"></a>

Structural cannot detect that a table is renamed.

From Structural's perspective, the original table is removed, and the table with the new name is added.

For example, a source and destination schema both contain a table called `Users`.

In the source database, the `Users` table is renamed to `People`.

Structural would detect the following schema issues:

* The source schema contains a `People` table that is not in the destination schema. For information about how Structural addresses this, go to [#upsert-schema-table-not-in-dest](#upsert-schema-table-not-in-dest "mention").
* The destination schema contains a `Users` table that is not in the source schema. For information about how Structural addresses this, go to [#upsert-schema-table-not-in-source](#upsert-schema-table-not-in-source "mention").


# Writing output to a container repository

{% hint style="info" %}
Requires Kubernetes.

For self-hosted Docker deployments, you can install and configure a separate Kubernetes cluster to use. For more information, go to [Setting up a Kubernetes cluster to use to write output data to a container repository](/app/admin/on-premise-deployment/enable-output-to-container-artifacts/container-output-separate-kubernetes-cluster).

For information about required Kubernetes permissions, go to [Required access to write destination data to a container repository](/app/admin/on-premise-deployment/enable-output-to-container-artifacts/kubernetes-required-containerization-access).

Not compatible with upsert.

Not compatible with Preserve Destination or Incremental table modes.
{% endhint %}

{% hint style="info" %}
Only supported for PostgreSQL, MySQL, and SQL Server.
{% endhint %}

You can configure a workspace to write destination data to a container repository instead of to a database server.

<figure><img src="/files/jg9AfNqPCHJhvR8C1o4V" alt=""><figcaption><p>Diagram showing how data is written to and accessed from a container artifact</p></figcaption></figure>

When Structural writes data generation output to a repository, it writes the destination data to a container volume. From the list of container artifacts, you can copy the volume digest, and download a Docker Compose file that provides connection settings for the database on the volume. Structural generates the Compose file when you make the request to download it. For more information about getting access to the container artifacts, go to [Viewing and downloading container artifacts](/app/workflows/container-artifacts-view-download).

For an overview of writing destination data to container artifacts, you can also view the [video tutorial](https://www.youtube.com/watch?v=mwmiZBMGWmQ).

## Indicating to write destination data to container artifacts <a href="#workspace-settings-containerization-enabling" id="workspace-settings-containerization-enabling"></a>

Under **Destination Settings**, to indicate to write the destination data to container artifacts, click **Container Repository**.

For a Structural instance that is deployed on Docker, unless you [set up a separate Kubernetes cluster](/app/admin/on-premise-deployment/enable-output-to-container-artifacts/container-output-separate-kubernetes-cluster), the **Container Repository** option is hidden.

You can switch between writing to a database server and writing to a container repository at any time. Structural preserves the configuration details for both options. When you run data generation, it uses the currently selected option for the workspace.

## Identifying the base image to use to create the container artifacts <a href="#workspace-settings-containerization-base-image" id="workspace-settings-containerization-base-image"></a>

From the **Database Image** dropdown list, select the image to use to create the container artifacts.

Select an image version that is compatible with the version of the database that is used in the workspace.

## Providing a customization file for MySQL <a href="#workspace-container-mysql-customization-file" id="workspace-container-mysql-customization-file"></a>

For a MySQL workspace, you can provide a customization file that helps to ensure that the temporary destination database is configured correctly.

To provide the customization details:

1. Toggle **Use customization** to the on position.
2. In the text area, paste the contents of the customization file.

## Setting the location for the container artifacts <a href="#workspace-settings-containerization-artifact-location" id="workspace-settings-containerization-artifact-location"></a>

To provide the location where Structural publishes the container artifacts:

1. In the **Registry** field, type the path to the container registry where Structural publishes the data volume.

   \
   Do not include the HTTP protocol, such as `http://` or `https://`.
2. In the **Repository Path** field, provide the path within the registry where Structural publishes the data volume.

   \
   For a Google Artifact Registry (GAR) repository, the path format is `PROJECT-ID/REPOSITORY/IMAGE`.

   For more information about repository and image names, go to the [Google Cloud documentation](https://cloud.google.com/artifact-registry/docs/docker/names#containers).

## Providing the credentials to write to the registry <a href="#workspace-settings-container-credentials" id="workspace-settings-container-credentials"></a>

You next provide the credentials that Structural uses to read from and write to the registry.

When you provide the registry, Structural detects whether the registry is from Amazon Elastic Container Registry (Amazon ECR), Google Artifact Registry (GAR), or a different container solution.

It displays the appropriate fields based on the registry type.

### Fields for registries other than Amazon ECR or GAR <a href="#container-credentials-non-ecr" id="container-credentials-non-ecr"></a>

For a registry other than an Amazon ECR or a GAR registry, the credentials can be either a username and access token, or a secret.

{% hint style="info" %}
The option to use a secret is not available on Structural Cloud.
{% endhint %}

In general, the credentials must be for a user that has read and write permissions for the registry.

The secret is the name of a Kubernetes secret that lives on the pod that the Structural worker runs on. The secret type must be `kubernetes.io/dockerconfigjson`. The Kubernetes documentation provides information on [how to create a registry credentials secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).

To use a username and access token:

1. Click **Access token**.
2. In the **Username** field, provide the username.
3. In the **Access Token** field, provide the access token.

To use a secret:

1. Click **Secret name**.
2. In the **Secret Name** field, provide the name of the secret.

### Azure Container Registry (ACR) permission requirements <a href="#container-credentials-acr-permissions" id="container-credentials-acr-permissions"></a>

For ACR, the provided credentials must be for a service principal that has sufficient permissions on the registry.

For Structural, the service principal must at least have the permissions that are associated with the[ AcrPush role](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-roles?tabs=azure-cli).

### Providing a service file for GAR <a href="#workspace-container-credentials-gar" id="workspace-container-credentials-gar"></a>

{% hint style="info" %}
Structural only supports Google Artifact Registry (GAR). It does not support Google Container Registry (GCR).
{% endhint %}

For a GAR registry, you upload a service account file, which is a JSON file that contains credentials that provide access to Google Cloud Platform (GCP).&#x20;

The associated service account must have the Artifact Registry Writer role.

For **Service Account File**, to search for and select the file, click **Browse**.

### Amazon ECR registries <a href="#container-credentials-ecr" id="container-credentials-ecr"></a>

For an Amazon ECR registry, you can either:

* Provide the AWS access and secret key that is associated with the IAM user that will connect to the registry
* Provide an assumed role
* (Self-hosted only) Use the credentials configured in the Structural environment settings `TONIC_AWS_ACCESS_KEY_ID` and `TONIC_AWS_SECRET_ACCESS_KEY`.
* (Self-hosted only) If Structural is deployed in Amazon Elastic Kubernetes Service (Amazon EKS), then you can use the AWS credentials that live on the EC2 instance.

#### Using AWS access keys <a href="#container-credentials-ecr-access-keys" id="container-credentials-ecr-access-keys"></a>

To provide an AWS access key and secret key:

1. Click **Access Keys**.
2. In the **AWS Access Key** field, enter an AWS access key that is associated with an IAM user or role.
3. In the **AWS** **Secret Key** field, enter the secret key that is associated with the access key.
4. Optionally, in the **AWS Session Token** field, enter the session token to use for the connection.

#### Using an assumed role <a href="#container-credentials-ecr-assumed-role" id="container-credentials-ecr-assumed-role"></a>

To provide an assumed role:

1. Click **Assume Role**.
2. In the **Role ARN** field, provide the Amazon Resource Name (ARN) for the role.
3. In the **Session Name** field, provide the role session name.\
   \
   If you do not provide a session name, then Structural automatically generates a default unique value. The generated value begins with `TonicStructural`.
4. In the **Duration (in seconds)** field, provide the maximum length in seconds of the session. \
   \
   The default is `3600`, indicating that the session can be active for up to 1 hour.\
   \
   The provided value must be less than the maximum session duration that is allowed for the role.

For the assumed role, Structural generates the external ID that is used in the assume role request. Your role’s trust policy must be configured to condition on your unique external ID.

Here is an example trust policy:

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "AWS": "<originating-account-id>"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "<external-id>"
      }
    }
  }
}
```

#### Using the credentials from the environment settings (self-hosted only) <a href="#container-credentials-ecr-env-settings" id="container-credentials-ecr-env-settings"></a>

On a self-hosted instance, to use the credentials configured in the environment settings, click **Environment Variables**.&#x20;

#### Using the AWS credentials from the EC2 instance (self-hosted only) <a href="#container-credentials-ecr-ec2" id="container-credentials-ecr-ec2"></a>

On a self-hosted instance, to use the AWS credentials from the EC2 instance, click **Instance Profile**.

#### Required permissions for the IAM user <a href="#container-credentials-ecr-iam-perms" id="container-credentials-ecr-iam-perms"></a>

The IAM user must have permission to list, push, and pull images from the registry. The following example policy includes the required permissions.

```
{
  {
    "Sid": "ManageTonicRepositoryContents",
    "Effect": "Allow",
    "Action": [
      "ecr:DescribeRepositories",
      "ecr:ListImages",
      "ecr:DescribeImages",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability",
      "ecr:InitiateLayerUpload",
      "ecr:UploadLayerPart",
      "ecr:CompleteLayerUpload",
      "ecr:PutImage"
    ],
    "Resource": [
       "arn:aws:ecr:<region>:<account_id>:repository/<optional name filter>"
    ]
  },
  {
    "Sid": "GetAuthorizationToken",
    "Effect": "Allow",
    "Action": [
      "ecr:GetAuthorizationToken"
    ],
    "Resource": "*"
  }
}
```

For additional security, a repository name filter allows you to limit access to only the repositories that are used in Structural. You need to make sure that the repositories that you create for Structural match the filter.

For example, you could prefix Structural repository names with `tonic-`. In the policy, you include a filter based on the `tonic-` prefix:

```
"Resource": [
  "arn:aws:ecr:<region>:<account_id>:repository/tonic-*"
]
```

## Providing tags for the container artifacts <a href="#workspace-settings-containerization-tags" id="workspace-settings-containerization-tags"></a>

In the **Tags** field, provide the tag values to apply to the container artifacts. You can also change the tag configuration for individual data generation jobs.

Use commas to separate the tags.

A tag cannot contain spaces. Structural provides the following built-in values for you to use in tags:

* `{workspaceId}` - The identifier of the workspace.
* `{workspaceName}` - The name of the workspace.
* `{timestamp}` - The timestamp when the data generation job that created the artifact completed.
* `{jobId}` - The identifier of the data generation job that created the artifact.

For example, the following creates a tag that contains the workspace name, job identifier, and timestamp:

`{workspaceName}_{jobId}_{timestamp}`

To also tag the artifacts as latest, check the **Tag as "latest" in your repository** checkbox.

## Specifying custom resources for the Kubernetes pods <a href="#workspace-settings-containerization-custom-resources" id="workspace-settings-containerization-custom-resources"></a>

You can also optionally configure custom resource values for the Kubernetes pods. You can specify the ephemeral storage, memory, and CPU millicores.

To provide custom resources:

1. Toggle **Set custom pod resources** to the on position.
2. Under **Storage Size**:

   1. In the field, provide the number of megabytes or gigabytes of storage.
   2. From the dropdown list, select the unit to use.

   The storage can be between 32MB and 25GB.
3. Under **Memory Size**:

   1. In the field, provide the number of megabytes or gigabytes of RAM.
   2. From the dropdown list, select the unit to use.

   The memory can be between 512MB and 4 GB.
4. Under **Processor Size**:

   1. In the field, provide the number of millicores.
   2. From the dropdown list, select the unit.

   The processor size can be between 250m and 1000m.

## Setting a custom database name <a href="#output-to-repos-database-name" id="output-to-repos-database-name"></a>

{% hint style="info" %}
Only available for PostgreSQL and SQL Server. Not available for MySQL.
{% endhint %}

In the **Custom Database Name** field, provide the name to use for the destination database.

If you do not provide a custom database name, then the destination database uses the same name as the source database.

## Setting a custom database user password <a href="#output-to-repos-custom-password" id="output-to-repos-custom-password"></a>

In the **Custom Password** field, provide the password for the destination database user.

If you do not provide a password, then Structural generates a password.

The destination database username is always the default user for the database:

* For PostgreSQL, `postgres`
* For MySQL, `root`
* For SQL Server, `sa`

## Configuring the required tolerations for datapacker node taints <a href="#output-to-repos-tolerations" id="output-to-repos-tolerations"></a>

If your Kubernetes nodes are configured with taints, then on a self-hosted instance, you can configure the tolerations that enable the datapacker pods to be scheduled on the nodes. The datapacker pod hosts the temporary database that Structural uses during the data generation.

For an overview of taints and tolerations, go to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).

To configure the tolerations, you configure the following [environment settings](/app/admin/environment-variables-setting). You can add these settings to the **Environment Settings** list on **Structural Settings**.

* `CONTAINERIZATION_POD_NODE_TOLERATION_KEY` - The toleration key value to apply to the datapacker pods. This setting is required. If you do not configure this setting, then Structural ignores the other settings.
* `CONTAINERIZATION_POD_NODE_TOLERATION_VALUES` - A comma-separated list of toleration values to apply to the datapacker pods.
* `CONTAINERIZATION_POD_NODE_TOLERATION_EFFECT` - The toleration effect to apply to the datapacker pods.
* `CONTAINERIZATION_POD_NODE_TOLERATION_OPERATOR` - The toleration operator to apply to the datapacker pods.


# Advanced workspace overrides

For self-hosted instances, Structural provides [environment settings](/app/admin/environment-variables-setting) to configure features that include:

* Consistency across runs and databases
* Data generation performance

The **Advanced Workspace Overrides** section of the workspace details view allows you to override those environment settings for an individual workspace.

For example, the environment setting `TONIC_TABLE_PARALLELISM` determines the number of tables that Structural processes simultaneously. You can then override that value within individual workspaces.

The workspace overrides are available on both self-hosted instances and on Structural Cloud.

## **Configuring the overrides** <a href="#workspace-overrides-configure" id="workspace-overrides-configure"></a>

To display the available override settings, expand **Advanced Workspace Overrides**.

<figure><img src="/files/U2MqhAYyBnpGcFt7pFPY" alt=""><figcaption><p>Advanced Workspace Overrides section for a workspace</p></figcaption></figure>

### Enabling and setting an override <a href="#override-set-initial" id="override-set-initial"></a>

You can use the search field to find a specific setting.

<figure><img src="/files/K8vY1xMTZWRtCVUTN2TL" alt=""><figcaption><p>Applying a search filter to the advanced workspace overrides</p></figcaption></figure>

For information on how to configure the statistics seed, go to [Enabling consistency](/app/generation/generators/generator-characteristics/consistency#consistency-cross-run-workspace-override).

For other settings, to enable the override and set the override value:

1. Toggle the setting to the on position.

<figure><img src="/files/iSRRxfoq5anskTJ9c5Q2" alt=""><figcaption><p>Enabling and setting an override value</p></figcaption></figure>

2. Set the value.

### Removing an override <a href="#override-remove" id="override-remove"></a>

To remove the override, toggle the setting to the off position.

## Available overrides <a href="#overrides-available" id="overrides-available"></a>

### **Workspace statistics seed for cross-run consistency** <a href="#workspace-statistics-seed" id="workspace-statistics-seed"></a>

For generators where [consistency](/app/generation/generators/generator-characteristics/consistency) is enabled, a statistics seed enables consistency across data generation runs. The Structural-wide statistics seed value ensures consistency across both data generation runs and workspaces.

You use the **Override Statistics Seed** setting to override the Structural-wide statistics seed value.

You can either disable consistency across data generations, or provide a seed value for the workspace. The workspace seed value ensures consistency across data generation runs for that workspace, and across other workspaces that have the same seed value.

For details about using seed values to ensure consistency across data generation runs and databases, go to [Enabling consistency](/app/generation/generators/generator-characteristics/consistency#enabling-consistency-across-runs-or-multiple-databases).

### Collection and sensitivity scan timeouts

For document-based databases, you can configure the maximum amount of time in seconds to scan a schema.

For sensitivity scans, you can configure the number of minutes after which a scheduled scan times out.

From **Advanced Workspace Overrides**, you can override both of these timeouts.

### Data generation performance settings <a href="#data-gen-performance" id="data-gen-performance"></a>

Structural provides environment settings to manage [data generation performance](/app/workflows/performance). For example, these settings include configuration for parallel processing and client-side compression.

From **Advanced Workspace Overrides**, you can override some of these data generation performance settings for an individual workspace.

### Data encryption and decryption keys <a href="#encryption-decryption-keys" id="encryption-decryption-keys"></a>

To use Structural data encryption, you must [provide encryption and decryption keys](/app/generation/generators-assign-config/generators-data-encryption-config#data-encryption-keys).

You use the **Override Data Decryption Key** and **Override Data Encryption Key** settings to override the Structural-wide keys that are provided in the environment settings.

### Destination database schema creation

Some data connectors allow you to configure whether you provide the schema for the destination database. For more information, go to related information for [Databricks](/app/setting-up-your-database/databricks/before-you-create-a-databricks-workspace/databricks-config-create-dest-schema), [MySQL](/app/setting-up-your-database/mysql/mysql-before-create-workspace#mysql-schema-creation-config), [Oracle](/app/setting-up-your-database/oracle/oracle-before-workspace-creation#oracle-config-skip-db-creation), [Snowflake](/app/setting-up-your-database/snowflake), and [SQL Server](/app/setting-up-your-database/sql-server/sql-server-before-create-workspace#sql-server-skip-db-creation).

From **Advanced Workspace Overrides**, you can override the instance-wide configuration.

### Overwrite handling for Databricks <a href="#databricks-emr-overwrite-handling" id="databricks-emr-overwrite-handling"></a>

Databricks allows you to configure how Structural handles overwrites of existing data.&#x20;

You use the **Workspace Default Error on Override** and **Workspace Default Save Mode** settings to override the instance-wide configuration.

### Enable or disable event-based sensitivity scans

Structural provides an environment setting to determine whether to [automatically run a sensitivity scan](/app/generation/identify-sensitive-data/running-the-structural-sensitivity-scan#sensitivity-scan-automatic) when you create a workspace, copy a workspace, or change the connection details.

From **Advanced Workspace Overrides**, you can override the instance-wide configuration.

### Enable or disable LLM-based sensitivity detection on Structural Cloud

By default, LLM-based sensitivity detection is enabled on Structural Cloud.

You use the **Tonic LLM Enable Enhanced Recommendations** setting to determine whether to enable LLM-based sensitivity detection in the workspace.

For information about LLM-based sensitivity detection, go to [Running the Structural sensitivity scan](/app/generation/identify-sensitive-data/running-the-structural-sensitivity-scan#llm-based-sensitivity-detection-medium-confidence).


# Managing access to workspaces

When you create a workspace, you become the owner of the workspace, and by default are assigned the built-in Manager workspace permission set for the workspace. The Manager permission set provides full access to the workspace configuration, data, and results.

You can also assign workspace permission sets to other users and to SSO groups. You can also transfer a workspace to a different owner.

If you are granted access to any workspace permission set for a workspace, then you have access to all of the workspace management views for that workspace. However, you can only perform tasks that you have permission for in that workspace.

Workspace access is managed from the **Workspaces** view. You cannot assign workspace permission sets from **Structural Settings** view.

You can also view an [overview video tutorial about workspace access](https://www.youtube.com/watch?v=rEn_vpcAtgg).

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Share workspace access</strong></td><td>Grant access to other users. The assigned workspace permission sets determine the level of access.</td><td></td><td><a href="/pages/-MEx52iIIi8pXRp5K226">/pages/-MEx52iIIi8pXRp5K226</a></td></tr><tr><td><strong>Transfer ownership of a workspace</strong></td><td>Make another user the workspace owner. You can also assign yourself workspace permission sets.</td><td></td><td><a href="/pages/U6bO1ihwRXysoGnr6oWY">/pages/U6bO1ihwRXysoGnr6oWY</a></td></tr></tbody></table>


# Sharing workspace access

{% hint style="info" %}
**Required permission**

* **Global permission:** View organization users. This permission is only required for the Tonic Structural application. It is not needed when you use the Structural API.
* **Either:**
  * **Workspace permission:** Share workspace access
  * **Global permission:** Manage user access to Tonic and to any workspace
    {% endhint %}

## About workspace access

Tonic Structural uses workspace permission sets for role-based access (RBAC) of each workspace.

A workspace permission set is a set of workspace permissions. Each permission provides access to a specific workspace feature or function.

Structural provides [built-in workspace permission sets](/app/admin/tonic-user-access/permissions/workspace-permissions/workspace-permission-built-in#permission-sets-builtin-workspace). Enterprise instances can also [configure custom workspace permission sets](/app/admin/tonic-user-access/permissions/workspace-permissions/workspace-permission-sets-custom).

To share workspace access, you assign workspace permission sets to users and, if you use SSO to manage Structural users, to SSO groups.

Before you assign a workspace permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group. For information on how to configure Structural to filter the allowed SSO groups, go to [Synchronizing SSO groups with Structural](/app/admin/tonic-user-access/single-sign-on/sso-limit-groups).

## Limitations on workspace sharing

### Cannot remove the owner permission set from the owner

You cannot remove the [owner workspace permission set](/app/admin/tonic-user-access/permissions/workspace-permissions/workspace-owner-permission-set) from the workspace owner. By default, the owner permission set is the built-in Manager permission set.

### Cannot grant or remove a permission you do not have

Within a workspace, the **Share workspace access** permission grants you the ability to share access to the workspace.

However, you cannot either grant or revoke access to a workspace permission that you do not have.

For example, for a given workspace, the workspace permission set that is assigned to you includes **Share workspace access**, but does not include **Run data generation**. Because of this, when you share workspace access:

* You cannot grant access to a workspace permission set that includes **Run data generation**.
* You cannot remove access to a workspace permission set that includes **Run data generation**.

Note that this requirement does not apply to users who have the global permission **Manage user access to Tonic and to any workspace**, which is by default granted to **Admin** users. Those users can grant or revoke any workspace permission set.

## Changing the workspace access

To change the current access to the workspace:

<figure><img src="/files/U9T3434ftopM347FgM8J" alt=""><figcaption><p>Workspace access panel</p></figcaption></figure>

1. To manage access to a single workspace, either:
   * On the workspace management view, in the heading, click the share icon.
   * On **Workspaces** view, click the actions menu for the workspace, then select **Share**.&#x20;
2. To manage access for multiple workspaces:
   1. Check the checkbox for each workspace to grant access to.
   2. From the **Actions** menu, select **Share Workspaces**.
3. The workspace access panel contains the current list of users and groups that have access to the workspace.\
   \
   To add a user or group to the list of users and groups, begin to type the user email address or group name. From the list of matching users or groups, select the user or group to add.\
   \
   Free trial users can invite other users to start their own free trial. Provide the email addresses of the users to invite. The email addresses must have the same corporate email domain as your email address. When the invited users sign up for the free trial, they are added to the Structural organization for the free trial user that invited them and have access to the workspace.
4. For a user or group, to change the assigned workspace permission sets:
   1. Click **Access**.\
      \
      The dropdown list is populated with the list of custom and built-in workspace permission sets.\
      \
      If you selected multiple workspaces, then on the initial display of the workspace sharing panel, for each permission set that a user or group currently has access to, the list shows the number of workspaces for which the user or group has that permission set.\
      \
      For example, you select three workspaces. A user currently has Editor access for one workspace and Viewer access for the other two. The Editor permission set has 1 next to it, and the Viewer permission set has 2 next to it.
   2. Under **Custom Permission Sets**, check the checkbox next to each workspace permission set to assign to the user or group.\
      \
      Uncheck the checkbox next to each workspace permission set to remove from the user or group.
   3. Under **Built-In Permission Sets**, check the workspace permission set to assign to the user or group. You can only assign one built-in permission set.\
      \
      By default, for an added user or group, the Editor permission set is selected.\
      \
      To select a built-in workspace permission set that is lower in access than the currently selected permission set, you must first uncheck the selected permission set.\
      \
      For example, if Editor is currently checked, then to change the selection to Viewer, you must first uncheck Editor.
5. To remove all access for a user or group, and remove the user or group from the list, click **Access**, then click **Revoke**.
6. To save the new access, click **Save**.


# Transferring ownership of a workspace

{% hint style="info" %}
**Required permission**

* **Global permission:** View organization users. This permission is only required for the Tonic Structural application. It is not needed when you use the Structural API.
* Either:
  * **Workspace permission:** Transfer workspace ownership
  * **Global permission:** Manage access to Tonic Structural and to any workspace

**To grant yourself access after the transfer:**

* **Workspace permission:** Share workspace access
  {% endhint %}

## About workspace ownership

Every workspace has an owner. The owner is always a user.

The user who creates the workspace is automatically the owner of the workspace.

By default, the workspace owner is assigned the built-in Manager workspace permission set. On Enterprise instances, you can [choose a different workspace permission set to assign to all workspace owners](/app/admin/tonic-user-access/permissions/workspace-permissions/workspace-owner-permission-set).

You cannot remove that permission set from the workspace owner.

## About ownership transfer

You can transfer a workspace to a different owner.

When you transfer ownership of a workspace, the new owner is assigned the owner permission set.

If the previous owner was not separately granted access to the owner permission set, then that permission set is removed.

## Completing the ownership transfer

To transfer workspace ownership:

1. To transfer ownership of a single workspace, from the workspace actions menu, select **Transfer Ownership**.
2. To transfer ownership of multiple workspaces:
   1. Check the checkbox for each workspace to grant access to.
   2. From the **Actions** menu, select **Transfer Ownership**.
3. On the transfer ownership panel, from the **User** dropdown list, select the new owner.
4. If you are the current owner of the workspace, then to grant yourself non-owner access after you transfer the ownership:
   1. Toggle **Receive access to workspace** to the on position.
   2. Select the workspace permission set to assign to yourself.
5. Click **Transfer Ownership**.


# Managing the workspace schema cache

You can configure a workspace to [cache the source database schema](/app/workspace/workspace-configuration-settings/schema-management-settings#schema-cache-config), to reduce the number of times that Tonic Structural needs to query the source database.

## Viewing the schema cache status <a href="#schema-cache-status" id="schema-cache-status"></a>

When schema caching is enabled for a workspace, then in the [expanded version of the workspace management view header](/app/workspace/managing-workspaces/selecting-a-workspace-to-manage#collapsing-and-expanding-the-workspace-heading), below the workspace name, Structural shows the current status of the schema cache.

<figure><img src="/files/elENUjZwk6VdbkBdvieM" alt=""><figcaption><p>Schema cache status for a workspace</p></figcaption></figure>

The status indicates when:

* Structural is retrieving the schema information.
* Structural is checking for schema updates.
* Structural is refreshing the schema cache.
* Structural fails to connect to the source database.
* The schema cache refresh fails.
* The schema cache is updated. The status includes the timestamp of the most recent refresh.

## Refreshing the schema cache <a href="#schema-cache-refresh" id="schema-cache-refresh"></a>

When the schema cache is updated, or when Structural has detected updates to the schema, then the schema cache status includes an option to refresh the schema cache.

<figure><img src="/files/kvmnEFbwsemqj5qzakHv" alt=""><figcaption><p>Refresh Schema option for a workspace schema cache</p></figcaption></figure>

To refresh the schema:

1. Click the schema cache status.
2. On the panel, click **Refresh Schema**.

Structural starts a new schema retrieval job. To track the progress of the job, go to the [workspace **Jobs** view](/app/workspace/jobs).


# Viewing workspace jobs and job details

You can view a list of jobs for the workspace, and view details for individual jobs.

You can also use the Structural Agent to get information about jobs and diagnose failed jobs. For example:

`How many data generation jobs have run for this workspace?`

`How many columns were de-identified during the most recent successful data generation?`

`When was the most recent sensitivity scan for this workspace? How many sensitive columns did it identify?`

`Why did the most recent data generation job fail? What's the solution to enable the job to complete successfully?`

## Job types

Tonic Structural runs the following types of jobs on a workspace:

* **Sensitivity scans -** Analyze the source database to identify sensitive data.
* **Collection scans -** Analyze the source data for a MongoDB workspace to determine the available fields in each collection, the field types, and how prevalent the fields are.
* **Schema retrieval jobs -** Refresh the cached version of the source database schema.
* **Data generation, data pipeline generation, and containerized generation jobs -** Generate the destination data from the source data.
* **Upsert data generation jobs -** Generate the intermediate database from the source database.
* **Upsert jobs -** Use data from the intermediate database to add new rows to and update changed rows in the destination database. If the migration process is enabled, then it is a step in the upsert job.
* **SDK table statistics jobs -** Only run when you use the SDK to generate data in a Spark workspace, and the assigned generators require the statistics.

## Job statuses <a href="#job-statuses" id="job-statuses"></a>

A job can have one of the following statuses:

* **Queued -** The job is queued to run, but has not yet started. A job is queued for one of the following reasons:

  * Another job is currently running on the same workspace.\
    \
    For example, you cannot run a sensitivity scan and a data generation, or multiple data generations, at the same time on the same workspace. This is true regardless of the number of workers on the instance.\
    \
    On Structural Cloud, there is also a limit on the number of concurrent running jobs for each organization. When that maximum is reached, a new job remains queued until a current running job completes.
  * There isn't an available worker on the instance to run the job. A Structural instance with one worker can only run one job at a time. If a job from one workspace is currently running, a job from another workspace cannot start until the first job is finished.

  To view information about why a job is queued, click the status value.
* **Running -** The job is in progress.
* **Canceled -** The job is canceled.
* **Completed -** The job completed successfully.
* **Failed -** The job failed to complete.

Each of these statuses has a corresponding "with warnings" status. For example, **Running with warnings**, **Completed with warnings**. A "with warnings" status indicates that the job had at least one warning at the time of the request.

## Viewing the list of jobs <a href="#jobs-list-view" id="jobs-list-view"></a>

**Jobs** view displays the list of jobs for the workspace.&#x20;

![Jobs view](/files/Mcbn1sYtUoIb2ZiBL0PY)

### Displaying Jobs view

To display **Jobs** view:

* On the workspace management view, in the workspace navigation bar, click **Jobs**.
* On **Workspaces** view, from the dropdown menu in the **Name** column, select **Jobs**.

### Filtering the jobs by type

On **Jobs** view, you use the tabs to filter the jobs based on the job type.

The possible tabs are:

* **All Jobs -** Always displayed. Contains all of the workspace jobs across all job types. When you first display **Jobs** view, **All Jobs** is selected.
* **Data Generation -** Always displayed. Includes the following types of jobs:
  * Data generation
  * Data pipeline generation
  * Containerized data generation
  * Upsert data generation
  * Upsert
* **Sensitivity Scan -** Always displayed. Lists the sensitivity scans for the workspace.
* **Collection Scan -** Displays for MongoDB and Amazon DynamoDB workspaces. Lists the collection scans on the source data.
* **Schema -** Lists the schema retrieval jobs for the workspace.
* **Statistics -** Displays for workspaces that use Spark-based data. Lists the SDK table statistics jobs.

### Information in a jobs list <a href="#jobs-list-info" id="jobs-list-info"></a>

The list is always sorted by the submission date, with the most recent jobs at the top of the list.

For each job, the job list includes the following information:

* **Job ID -** The identifier of the job. To copy the job identifier, click the icon at the left of the row.
* **Type -** The type of job.
* **Status -** The current status of the job, and how long ago the job reached that status.\
  \
  When you hover over the status, a tooltip displays the actual timestamp for the status change, and the length of time that the job ran.\
  \
  For queued jobs, to display a panel with information about why the job is queued, click the status value.\
  \
  ![](/files/cLk3a7ULtw5pYh4TOxOf)
* **Submitted -** The date and time when the job was submitted.
* **Completed -** The date and time when the job finished running.

### Filtering jobs by job status

To filter the list by the job status:

![Job status filter options for Jobs view](/files/rowXCzYasyHD91FDhBJL)

1. Click the filter icon in the **Status** column heading.\
   \
   The status panel displays all of the statuses that are currently in the list. For example, if there are no Queued jobs, then the Queued status is not in the list.\
   \
   By default, all of the statuses are included, and none of the checkboxes are checked.
2. To only include jobs that have specific statuses, check the checkbox next to each status to include.\
   \
   Checking all of the checkboxes has the same effect as unchecking all of the checkboxes.

### Filtering jobs by identifier

To filter the list by the job identifier, in the filter field, provide the full identifier.

## Viewing details for a selected job <a href="#jobs-view-details" id="jobs-view-details"></a>

For jobs other than Queued jobs, you can display details about the workspace and the job progress.

From the **Jobs** view, to display the details for a job, click the job row.

![Job details page for a data generation job](/files/GiwVBmSxY9hlFgS53dWN)

### Workspace information <a href="#job-details-workspace-info" id="job-details-workspace-info"></a>

The left side of the job details view contains the workspace information.

For a sensitivity scan, the workspace information is limited to the owner, database type, and worker version.

For a data generation job, the workspace information also includes:

* Whether subsetting, post-job scripts, or webhooks are used.
* The number of schemas, tables, and columns in the source database.
* The number of schemas, tables, and columns in the destination database.

### Job Log <a href="#job-details-job-log" id="job-details-job-log"></a>

The **Job Log** tab shows the start date, start time, and duration of the job, followed by the list of job process steps.

### Privacy Report <a href="#job-details-privacy-report" id="job-details-privacy-report"></a>

For data generation jobs, the **Privacy Report** tab displays the number of at-risk, protected, and not sensitive columns in the source database.

![Privacy Report tab on the job details page](/files/gSfMIjf9KEcIISrLvxVZ)

At-risk columns contain sensitive data, but still have Passthrough as the assigned generator.

Protected columns have an assigned generator other than Passthrough.

Not sensitive columns have Passthrough as the assigned generator, but do not contain sensitive data.

## Copying the job identifier

The job identifier is a unique identifier for the job. To copy the job identifier, either:

* From **Jobs** view, click the copy (![](/files/SDkEfCNjxzgSJiLkqwbW)) icon in the leftmost column.
* From the job details view, click **Copy Job ID**.

## Using AI to troubleshoot failed jobs

{% hint style="info" %}
**Required workspace permission:** Download job logs
{% endhint %}

For the the following types of jobs:

* Sensitivity scans
* Data generation jobs
* Upsert jobs
* Collection scans

You can ask for AI help to troubleshoot the failure.

On the job details, to start the AI troubleshooting, click **Ask AI**.

Structural sends the job log information to the LLM, which returns an analysis of the failure and suggested next steps.

The diagnosis includes an **Open in Agent chat** option to allow you to use the Structural Agent to continue the conversation.

## Canceling a job <a href="#jobs-cancel" id="jobs-cancel"></a>

You can cancel Queued or Running jobs.

For jobs with those statuses, the rightmost column in the job list contains a cancel icon.

![Job list with a Running job that can be canceled](/files/9W7AiP6Oaf6bkJKdddWH)

To cancel the job, click the icon.

## Downloading job information

For workspaces that are configured to write destination data to a container repository, the **Jobs** view also provides access to the generated artifacts. For more information, go to [Viewing and downloading container artifacts](/app/workflows/container-artifacts-view-download).

### Job logs <a href="#jobs-download-job-logs" id="jobs-download-job-logs"></a>

{% hint style="info" %}
**Required workspace permission:** Download job logs

To download diagnostic logs, you must have the **Enable diagnostic logging** global permission.
{% endhint %}

For all jobs, the job logs provide detailed information about the job processing. Tonic.ai support might request the job logs to help diagnose issues.

For upsert jobs where the migration process is enabled, and you configured the `GET Schema Change Logs` endpoint, the upsert job logs include the migration process logs.

#### Where to download the job logs <a href="#job-logs-download-location" id="job-logs-download-location"></a>

You can download the job logs from the **Jobs** view or the job details view. The download includes up to 1MB of log entries.

On the **Jobs** view, to download the logs for a job, click the download icon in the rightmost column.

On the job details view, to download the logs for a job, click **Reports and Logs**, then select **Job Logs**.

#### Downloading diagnostic logs <a href="#job-logs-download-diagnostic" id="job-logs-download-diagnostic"></a>

By default, Structural redacts sensitive values from the job logs. To help support troubleshooting, you can configure data connectors or an individual data generation job to create unredacted versions of the log files, referred to as diagnostic logs. For more information, go to [Redacted and diagnostic (unredacted) logs](/app/admin/tonic-monitoring-logging/logs-redacted-diagnostic).

To access diagnostic log files, you must have the **Enable diagnostic logging** global permission.

If you do not have the **Enable diagnostic logging** global permission, then you cannot download the logs for that job. The download option is disabled.

### Privacy Report for data generation <a href="#jobs-download-privacy-report" id="jobs-download-privacy-report"></a>

{% hint style="info" %}
**Required workspace permission:** View and download Privacy Report
{% endhint %}

From the job details view, you can download a Privacy Report file that provides an overview of the current protection status of the database columns based on the workspace configuration at the time that the job ran.

You can download either:

* The Privacy Report .csv file, which provides details about the table columns, the column content, and the current protection configuration.
* The Privacy Report PDF file, which provides charts that summarize the privacy ranking scores for the table columns. It also includes the table from the .csv file.

To display the download options, click **Reports and Logs**. In the menu:

<figure><img src="/files/izQOYnmVFYH0ROliwUuL" alt=""><figcaption><p>Reports and Logs menu for a data generation job</p></figcaption></figure>

* To download the Privacy Report .csv file, click **Privacy Report CSV**.
* To download the Privacy Report PDF file, click **Privacy Report PDF**.

For more information about the **Privacy Report** files and their content, go to [Using the Privacy Report to verify data protection](/app/generation/privacy-report).

### Additional logs for output to a container repository <a href="#logs-output-to-repos" id="logs-output-to-repos"></a>

For a workspace that [writes the output to a container repository](/app/workspace/workspace-configuration-settings/workspace-config-write-to-container-artifacts), the job includes the following additional logs:

* **Database logs -** Logs for the database container that is used as the destination.
* **Datapacker logs -** Logs for creating the OCI artifact and uploading it to an OCI registry.

To download these logs for a data generation job, on the job details view, click **Reports and Logs**, then select **Database Logs** or **Datapacker Logs**.

### CloudWatch logs for data generation <a href="#jobs-download-cloudwatch-logs" id="jobs-download-cloudwatch-logs"></a>

For workspaces that are connected to Amazon Redshift or Snowflake on AWS databases, the data generation job requires multiple calls to a Lambda function. For these data generation jobs, the CloudWatch logs monitor the progress of and display errors for these Lambda function calls.

To download the CloudWatch logs for a data generation job, on the job details view, click **Reports and Logs**, then select **CloudWatch Logs**.

The **CloudWatch Logs** option only displays for Amazon Redshift and Snowflake on AWS data generation jobs.

### Oracle SQL Loader log files <a href="#jobs-download-oracle-sql-loader-logs" id="jobs-download-oracle-sql-loader-logs"></a>

{% hint style="info" %}
**Required workspace permission:**  Download SqlLdr Files
{% endhint %}

For an Oracle data generation, if both of the following are true:

* The data generation job ran SQL Loader (sqlldr).
* sqlldr either failed or succeeded with errors.

Then to download the sqlldr log files, click **Reports and Logs**, then select **sqlldr Logs**.

### Sending a log package to Tonic.ai <a href="#job-upload-logs" id="job-upload-logs"></a>

{% hint style="info" %}
**Required global permission:** Enable diagnostic logging and uploading logs directly to Tonic.ai
{% endhint %}

The job details include an option to send a log package to Tonic.ai.

You would likely send the log package at the request of the Structural support team, to help to troubleshoot a data generation issue.

To send the package, from the **Reports and Logs** dropdown list, select **Send logs to Tonic.ai**.

Structural creates the package, then uploads it to an S3 bucket. Packages are removed from the S3 bucket automatically after 30 days.

### Transformed files for file connector data generation <a href="#job-details-download-file-connector-files" id="job-details-download-file-connector-files"></a>

For a data generation from a file connector workspace that uses local files, you can download the transformed files for that job.

The download is a .zip file that contains the files for a selected file group.

On the job details view, when files are available to download, the **Data available for file groups** panel displays.

To download the files for a file group:

1. Click **Download Results**.
2. From the list, select the file group. Use the filter field to filter the list by the file group name.

<figure><img src="/files/WFHGPyCXV6bMzxs2P5jk" alt=""><figcaption><p>Job details option to download transformed file connector files</p></figcaption></figure>

### Performance metrics for data generation

{% hint style="info" %}
**Required workspace permission:** Download job logs
{% endhint %}

For workspaces that use the newer data generation processing, users can configure a data generation job to also [generate performance metrics](/app/workflows/data-generation-run-job/data-generation-manual#data-gen-manual-performance-metrics). This is usually done for troubleshooting purposes.

On the job details view, to download the performance metrics for the job, click **Reports and Logs**, then click **Performance Metrics**.

## Viewing a Gantt chart of a data generation job flow <a href="#job-gantt-visualization" id="job-gantt-visualization"></a>

{% hint style="info" %}
This feature is currently in beta.
{% endhint %}

From the job details view, you can display a Gantt chart that shows the flow of a data generation job over time. The chart can help you to understand the different steps of a data generation job and how long it takes Structural to complete each step.

Note that this option is only available for data generation jobs that use the newer data generation process. For more information, go to [Running data generation manually](/app/workflows/data-generation-run-job/data-generation-manual#confirm-gen-details-data-pipeline-v2). Data generation jobs that use the older process do not produce the Gantt chart.

To display the chart, click **Reports and Logs**, then select **View Gantt**.

The **Job Visualization** page displays the Gantt chart of the job progress.

<figure><img src="/files/4YG7t1oJKWdAkFECXW8B" alt=""><figcaption><p>Job Visualization page with a Gantt chart of the job flow</p></figcaption></figure>


# Privacy Hub

## About Privacy Hub <a href="#privacy-hub-about" id="privacy-hub-about"></a>

**Privacy Hub** tracks the current protection status of source data columns based on:

* [Column sensitivity](/app/generation/identify-sensitive-data), either from the most recent sensitivity scan or from manual assignments
* Assigned [table modes](/app/generation/table-modes)
* Assigned [generators](/app/generation/generators)

![Privacy Hub](/files/tPlc8VKIs6O5eJLzCng8)

To display **Privacy Hub**, either:

* On the workspace management view, in the workspace navigation bar, click **Privacy Hub**.
* On **Workspaces** view, click the workspace name.

From **Privacy Hub**, you can:

* Review and apply the recommended generators for all detected sensitive columns
* View the current protection status of columns
* Manually mark columns as sensitive or not sensitive
* Configure protection for sensitive columns
* Download a preview Privacy Report
* Run a new sensitivity scan

## Viewing the count of detected sensitive columns that are not protected <a href="#privacy-hub-view-sensitive-column-recommendations-banner" id="privacy-hub-view-sensitive-column-recommendations-banner"></a>

The sensitivity scan detects specific types of sensitive data.

If your workspace contains any columns that the sensitivity scan identified, and for which you have not either:

* Assigned a generator
* Marked as not sensitive

Then Tonic Structural displays a **Sensitivity Recommendations** banner that contains a count of those columns.

<figure><img src="/files/mpCXCBBoylgLPAFlAD3t" alt=""><figcaption><p>Sensitivity Recommendations banner on Privacy Hub</p></figcaption></figure>

The count only includes sensitive columns that the sensitivity scan detects. If you manually mark a column as sensitive, it is not included in the list.

On the banner, the **Review Recommendations** option allows you to review the detected columns and the recommended generators for each detected sensitive data type.

You can then apply the recommended generators or ignore the recommendations. When you ignore a recommendation, you either:

* Indicate to remove the generator recommendation for the column.
* Indicate that the column data is not sensitive.

For more information, go to [Reviewing and applying recommended generators](/app/generation/generators-assign-config/generators-review-apply-recommended).

## Viewing the protection status for each column <a href="#privacy-hub-view-protection-status" id="privacy-hub-view-protection-status"></a>

The protection status panels at the top of **Privacy Hub** provide an overview of the current protection status of the columns in the source data.

![Protection status panels](/files/ZECwK6ThNzqRIjJOLlg2)

Each panel displays:

* The number of columns that are in that category.
* The estimated percentage of columns that are in that category.

Note that for a [JSON column that uses **Document View**](/app/generation/working-with-document-based-data/json-document-view), the protection status displays a separate box for each combination of JSON path and data type.

From each panel, you can [display details for and configure protection for each column](#viewing-and-configuring-columns).

The column counts do not include columns that do not have data in the destination database. For example, if a table is assigned Truncate table mode, then **Privacy Hub** ignores the columns in that table.

The information on these panels updates automatically as you change whether columns are sensitive and assign generators to columns.

### At-Risk Columns <a href="#privacy-hub-unprotected-sensitive-columns" id="privacy-hub-unprotected-sensitive-columns"></a>

The **At-Risk Columns** panel reflects columns that:

* Are populated in the destination database.
* Are marked as sensitive.
* Have the generator set to Passthrough, which indicates that Structural does not perform any transformation on the data.

For each column, the **At-Risk Columns** panel also indicates the sensitivity confidence, from full confidence (completely red) to low confidence (a small percentage of red).

The goal is to have 0 at-risk columns.

When you click **Open in Database View**, you navigate to [Database View](/app/generation/database-view). The column list is filtered to show columns that are at risk.

### Protected Columns <a href="#privacy-hub-protected-columns" id="privacy-hub-protected-columns"></a>

The **Protected Columns** panel reflects columns that:

* Are populated in the destination database.
* Are assigned a generator other than Passthrough.

It includes both sensitive and non-sensitive columns.

Note that a column is considered protected based solely on the assigned generator. Some more complex generators, such as JSON Mask or Conditional, allow you to apply different generators to specific portions of a value or based on a specific condition. However, the protection status does not reflect these sub-generators. An applied sub-generator could be Passthrough.

When you click **Open in Database View**, you navigate to [Database View](/app/generation/database-view). The column list is filtered to show all included columns that are protected.

### Not Sensitive Columns <a href="#privacy-hub-nonsensitive-columns" id="privacy-hub-nonsensitive-columns"></a>

The **Not Sensitive Columns** panel reflects columns that:

* Are populated in the destination database.
* Are marked as not sensitive.
* Have the generator set to Passthrough.

When you click **Open in Database View**, you navigate to [Database View](/app/generation/database-view). The column list is filtered to show included columns that are not sensitive and are not protected.

## Viewing the protection status for each table <a href="#privacy-hub-database-tables" id="privacy-hub-database-tables"></a>

The **Database Tables** list shows the protection status for each table in the source database. You can view the number of columns that have each protection status, and update the column configuration.

The list does not include tables where the table mode is Truncate or Preserve Destination. Truncated tables are not populated in the destination database. For Preserve Destination tables, the existing data in the destination database does not change.

### Information in the list <a href="#privacy-hub-database-table-columns" id="privacy-hub-database-table-columns"></a>

For each table, **Database Tables** provides the following information:

* **Name -** The table name. For a [file connector](/app/setting-up-your-database/file-connector) workspace, each table corresponds to a file group.\
  \
  Each [JSON column that uses **Document View**](/app/generation/working-with-document-based-data/json-document-view) is also in a separate row. For JSON columns, the Name column displays both the table name and the column name.\
  \
  When you click a table name, you can navigate to either:
  * **Database View**, filtered to to display the columns for that table.
  * **Table View** for that table.
* **Not Sensitive -** The number of not sensitive columns in the table. Not sensitive columns are not marked as sensitive and have Passthrough as the generator.\
  \
  When you click the value, you navigate to [Database View](/app/generation/database-view), filtered to display the not sensitive columns for the table.
* **Protected -** The number of protected columns in the table. Protected columns have an assigned generator. A protected column can be either sensitive or not sensitive.\
  \
  When you click the value, you navigate to [Database View](/app/generation/database-view), filtered to display the protected columns for the table.
* **At-Risk -** The number of at-risk columns in the table. These columns are marked as sensitive, but have Passthrough as the generator. The goal is to have 0 unprotected sensitive columns.\
  \
  When you click the value, you navigate to [Database View](/app/generation/database-view), filtered to display the at-risk columns for the table.
* **Privacy Status -** Indicates the current protection status of the columns in the table. It provides the same view and configuration options as the protection status panels at the top of **Privacy Hub**.

### Filtering the list <a href="#privacy-hub-database-tables-filter" id="privacy-hub-database-tables-filter"></a>

You can filter the **Database Tables** list either by the table name or by the schema.

#### Filtering by table name

To filter the list by table name, in the filter field, begin to type text that is in the table name. As you type, Structural updates the list to only display matching tables.

#### Filtering by schema

To filter the list to only include tables that belong to a specific schema:

1. Click **Filter by Schema**.
2. From the schema dropdown list, select the schema.

When you select a schema, Structural adds it to the filter field.

### Sorting the list <a href="#privacy-hub-database-tables-sort" id="privacy-hub-database-tables-sort"></a>

You can sort the **Database Tables** list by any column except for the **Privacy Status** column.

To sort by a column, click the column heading. To reverse the sort order, click the heading again.

### Managing columns from the table list <a href="#privacy-hub-database-tables-manage-columns" id="privacy-hub-database-tables-manage-columns"></a>

The **Privacy Status** column in the **Database Tables** list indicates the protection status of the columns in the table.

This column provides the same [options to view and configure columns](#viewing-and-configuring-columns) as the protection status panels at the top of **Privacy Hub**, but is limited to the columns in a specific table.

## Viewing and configuring columns

### Navigating through columns and viewing column details <a href="#privacy-hub-protection-status-column-details" id="privacy-hub-protection-status-column-details"></a>

Each protection status panel displays a series of boxes to represent the columns that apply to that status. For example, if the source data contains four columns that are at-risk, then the **At-Risk Columns** panel displays four boxes, one for each column.

The **Privacy Status** column in the **Database Tables** list displays the same set of boxes for the columns in an individual table.

If the number of columns is too large to fit, then the last box shows the number of additional columns that apply. For example, if there are 15 columns that don't fit, then the last box is labeled +15.

When you hover over a box, the column name displays in a tooltip.

When you click a box, the details panel for that column displays.

![Settings view of column details panel](/files/q2ISx2ljWFLGSkKz7EuT)

When you click the box for remaining columns, the details panel for the first column in the remaining columns displays.

You can use the next and previous icons at the bottom right of the details panel to display the details for the next or previous column.

The column details panel opens to the settings view. The settings view contains the following information:

* The table and column name.
* Whether the column is flagged as sensitive.
* The type of sensitive data that the column contains.
* The data type for the column data.
* The generator that is assigned to the column.
* For a child workspace, whether the column configuration is inherited from the parent workspace. For columns that have overrides, you can reset to the parent configuration.

### Indicating whether a column is sensitive <a href="#privacy-hub-protection-status-flag-sensitive" id="privacy-hub-protection-status-flag-sensitive"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column sensitivity
{% endhint %}

From the settings view of the column details, you can configure the column sensitivity.

You cannot change the sensitivity of columns in a child workspace. A child workspace always inherits the sensitivity from its parent workspace. For more information, go to [About workspace inheritance](/app/workspace/managing-workspaces/workspaces-inheritance).

As you change the column sensitivity, Structural updates the protection status panels.

To change whether the column is sensitive, toggle the **Sensitive** option. The column is moved if needed to reflect its new status. However, you remain on the current panel.

For example, from the **At-Risk Columns** panel, you change a column to be not sensitive. The column is moved to the **Not Sensitive Columns** panel. When you click the next or previous icons, you view the details for the next or previous column on the **At-Risk Columns** panel.

### Selecting and configuring a generator for the column <a href="#privacy-hub-protection-status-generator-assignment" id="privacy-hub-protection-status-generator-assignment"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

From the column details, you can assign and configure the column generator.

When you change the column generator, Structural updates the protection status panels.

If the column generator was previously Passthrough, then the column is moved to the **Protected Columns** panel. However, you remain on the current panel. For example, you assign a generator to a column that is on the **At-Risk Columns** panel. The column is moved to the **Protected Columns** panel, but when you click the next or previous icons, you view the details for the next or previous column on the **At-Risk Columns** panel.

#### Selecting the generator <a href="#privacy-hub-protection-status-select-generator" id="privacy-hub-protection-status-select-generator"></a>

For sensitive columns that are not protected, Structural displays the recommended generator as a button.

For self-hosted instances that have an Enterprise license, the recommended generator is the built-in generator preset.

To assign the recommended generator to the column, click the button.

Otherwise, select the generator from the **Generator Type** dropdown list.

For more information about selecting a generator, go to [Assigning and configuring generators](/app/generation/generators-assign-config/generator-assignment-and-config).

#### Configuring the generator <a href="#privacy-hub-protection-status-configure-generator" id="privacy-hub-protection-status-configure-generator"></a>

If the selected generator requires additional configuration, then below the **Generator Type** dropdown list is an **Edit Generator Options** link.

![Column details panel with generator selected](/files/Ojn2Grhv2C3SQXMYjJlJ)

To display the configuration fields for the generator, click **Edit** **Generator Options**.

![Configuration options for a selected generator](/files/fzmRfGJ640DnQ8gtfHHQ)

For information about configuring a selected generator or generator preset, go to [Assigning and configuring generators](/app/generation/generators-assign-config/generator-assignment-and-config).

After you configure the generator, to return to the settings view, click **Back**.

### Displaying sample data for a column <a href="#privacy-hub-protection-status-display-sample-data" id="privacy-hub-protection-status-display-sample-data"></a>

{% hint style="info" %}
**Required workspace permission:**

* **Source data:** Preview source data
* **Destination data:** Preview destination data
  {% endhint %}

From the column details, you can display sample data for the column. The sample data allows you to compare the source and destination versions of the column values.

To display the sample data, click the view sample (magnifying glass) icon.

On the sample data view of the column details:

* The **Original Data** tab shows the values in the source data.
* The **Protected Output** tab shows the values that the generator produced.

![Sample data view on the column details panel](/files/Mn6Mv8pMWWMAYFJmwYjD)

### Enabling Document View for JSON columns <a href="#column-json-document-view" id="column-json-document-view"></a>

{% hint style="info" %}
Supported only for the file connector and PostgreSQL.
{% endhint %}

For a JSON column, instead of assigning a generator, you can enable **Document View**.

From **Document View**, you can view the JSON schema structure and assign generators to individual JSON fields. For more information, go to [Using Document View for JSON columns](/app/generation/working-with-document-based-data/json-document-view).

To enable **Document View**, on the column details panel, toggle **Use Document View** to the on position. When **Document View** is enabled, the generator dropdown is replaced with the **Open in Document View** option.

### Commenting on a column <a href="#privacy-hub-protection-status-column-comments" id="privacy-hub-protection-status-column-comments"></a>

{% hint style="info" %}
**Required license:** Professional or Enterprise
{% endhint %}

From the column details, you can view and add comments on the column. You might use a comment to explain why you selected a particular generator or marked a column as sensitive or not sensitive.

From the column details, to display the comments for the column, click the comment icon.

The comments view displays any existing comments on the column. The most recent comment is at the bottom of the list. Each comment includes the name of the user who made the comment.

To add the first comment to a column, type the comment into the comment text area, then click **Comment**.

![Comment view of the column details panel](/files/DFk4w0Szn1Uan4Q1Ookt)

To add an additional comment, type the comment into the comment text area, then click **Reply**.

## Downloading a preview Privacy Report <a href="#privacy-hub-preview-privacy-report" id="privacy-hub-preview-privacy-report"></a>

{% hint style="info" %}
**Required license:** Enterprise
{% endhint %}

The Privacy Report files that you download from **Privacy Hub** or the workspace download menu provide an overview of the current protection status based on the current configuration.

This is different from the Privacy Report files that you download from the data generation job details, which show the protection status for the data produced by that data generation.

You can download either:

* The Privacy Report .csv file, which provides details about the table columns, the column content, and the current protection configuration.
* The Privacy Report PDF file, which provides charts that summarize the privacy ranking scores for the table columns. It also includes the table from the .csv file.

For more information about the Privacy Report files and their content, go to [Using the Privacy Report to verify data protection](/app/generation/privacy-report).

### From workspace management view

To download the report from the workspace management view, click the download icon. In the download menu:

<figure><img src="/files/gt3sZUo6JJNhzRaXQlnt" alt=""><figcaption><p>Download menu for a workspace</p></figcaption></figure>

* To download the Privacy Report PDF file, click **Download Privacy Report PDF**.
* To download the Privacy Report .csv file, click **Download Privacy Report CSV**.

### From Privacy Hub

To download the report from **Privacy Hub**, click **Reports and Logs**, then:

<figure><img src="/files/ibv40Cm8ZBoVImzqMWGu" alt=""><figcaption><p>Reports and Logs menu on Privacy Hub</p></figcaption></figure>

* To download the Privacy Report .csv file, click **Privacy Report CSV**.
* To download the Privacy Report PDF file, click **Privacy Report PDF**.

## Running a new sensitivity scan on the data <a href="#privacy-hub-run-sensitivity-scan" id="privacy-hub-run-sensitivity-scan"></a>

{% hint style="info" %}
**Required workspace permission:** Run sensitivity scan
{% endhint %}

**Privacy Hub** provides an option to manually start a new [sensitivity scan](/app/generation/identify-sensitive-data/running-the-structural-sensitivity-scan). For example, you might want to run a new sensitivity scan when:

* You add columns to the source database. The new scan identifies whether the new columns contain sensitive data.
* The data in a column changes significantly, and a column that Structural originally marked as not sensitive might now contain sensitive data.

You cannot run a sensitivity scan on a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance). Child workspaces always inherit the sensitivity results from their parent workspace.

To run a new sensitivity scan, click **Run Sensitivity Scan**.

![Buttons at the top of Privacy Hub](/files/dArZtehv7Xfvem1j838K)

When Structural runs a new sensitivity scan:

* Structural analyzes and determines the sensitivity of any new columns.
* It does not change the sensitivity of existing columns that you marked as sensitive or not sensitive.
* For existing columns that you did not change the sensitivity of:
  * Structural does not change the sensitivity of columns that the original scan marked as sensitive.
  * It can change the sensitivity of columns that the original scan marked as not sensitive.

The protection status panels are updated to reflect the results of the new scan.


# Database View

**Database View** provides a complete view of your source database structure and configuration.

To display **Database View**, either:

* On the workspace management view, in the workspace navigation bar, click **Database View**.
* On **Workspaces** view, from the dropdown menu in the **Name** column, select **Database View**.

**Database View** consists of:

* On the left, the list of tables in the source database.
* On the right, the list of columns in those tables.

![Database View](/files/jS5T16HD50FT3l2lsRwR)

## View table and column information <a href="#database-view-table-column-info" id="database-view-table-column-info"></a>

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>View and configure tables</strong></td><td>Filter the table list, and assign table modes to tables.</td><td></td><td><a href="/pages/a2QlRg9XN9n74MXCYolJ">/pages/a2QlRg9XN9n74MXCYolJ</a></td></tr><tr><td><strong>View the column list</strong></td><td>Apply filters to and sort the list of columns.</td><td></td><td><a href="/pages/anyvT0muaqefZj2maMCt">/pages/anyvT0muaqefZj2maMCt</a></td></tr><tr><td><strong>View sample data</strong></td><td>View example source and destination data for a column.</td><td></td><td><a href="/pages/AkeumuzHh62CCE7AqE4V">/pages/AkeumuzHh62CCE7AqE4V</a></td></tr></tbody></table>

## Configure and comment on columns <a href="#database-view-column-config-comment" id="database-view-column-config-comment"></a>

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Configure an individual column</strong></td><td>Assign a generator and determine the column sensitivity.</td><td></td><td><a href="/pages/fYEzPpYXuBs0MDe1G7Cp">/pages/fYEzPpYXuBs0MDe1G7Cp</a></td></tr><tr><td><strong>Configure multiple columns</strong></td><td>Use the bulk edit option to configure multiple columns.</td><td></td><td><a href="/pages/hm4Pr7TWzPRW9G8wjTla">/pages/hm4Pr7TWzPRW9G8wjTla</a></td></tr><tr><td><strong>Identify similar columns</strong></td><td>Identify and filter to columns that are similar to a column, based on the column name.</td><td></td><td><a href="/pages/XyOVG1XOt2rUX5pDdDKt">/pages/XyOVG1XOt2rUX5pDdDKt</a></td></tr><tr><td><strong>Comment on columns</strong></td><td>Add and respond to column comments.</td><td></td><td><a href="/pages/jFytQflKzDyiIg0WoCU1">/pages/jFytQflKzDyiIg0WoCU1</a></td></tr></tbody></table>


# Viewing and configuring tables

The table list at the left of **Database View** contains the list of tables in the source database. You can filter the table list and assign tables modes to the tables.

You can use the [Structural Agent](/app/structural-agent/agent-about) to filter **Database View** to a specific table. For example:

* `Display the products table on Database View.`
* `Show me the stores and vendors tables on Database View.`

## Information in the table list <a href="#database-view-info-table" id="database-view-info-table"></a>

The table list is grouped by schema. You can expand and collapse the list of tables in each schema. This does not affect the displayed columns.

For a [file connector](/app/setting-up-your-database/file-connector) workspace, each table corresponds to a file group.

![Table list on Database View](/files/SIJqEYOJZTmEiNLrT08o)

For each table, the table list includes the following information:

* The name of the table.
* The number of columns that have an assigned generator (a generator other than Passthrough).\
  \
  The number does not display if none of the table columns has an assigned generator.
* The [assigned table mode](#database-view-tables-assign-mode). The table list only shows the first letter of the table mode:
  * D = De-identify
  * T = Truncate
  * P = Preserve Destination
  * I = Incremental

For a child workspace, if the selected table mode overrides the parent workspace configuration, then the override icon displays.

<figure><img src="/files/MSuEpnZSZnz7tFCgAAIs" alt=""><figcaption><p>Table that overrides the configuration from the parent workspace</p></figcaption></figure>

To display [Table View](/app/generation/table-view) for a table, click the arrow icon to the right of the table entry.

## Filtering the table list <a href="#database-view-tables-filter" id="database-view-tables-filter"></a>

You can filter the table list [by name](#database-view-tables-filter-name) and [by the assigned table mode](#database-view-tables-filter-mode). You can also filter the tables based on [whether any of the columns have assigned generators](#database-view-tables-filter-has-generators).

As you filter the table list, the column list also is filtered to only include the columns for the filtered tables.

### Filtering by table name <a href="#database-view-tables-filter-name" id="database-view-tables-filter-name"></a>

To filter the table list by name, in the filter field, begin to type text that is in the table name.

As you type, Tonic Structural filters the list to only display tables with names that contain the filter text.

![Filtering the table list by table name](/files/0b1RdWyMwAMdewf3L0uV)

### Filtering by the assigned table mode <a href="#database-view-tables-filter-mode" id="database-view-tables-filter-mode"></a>

To filter the table list based on the assigned table mode:

![Filtering the table list by table mode](/files/ZnuF1q2EOrgKNlkthzGZ)

1. Click **Filters**.
2. On the filter panel, check the checkbox next to each table mode to include.\
   \
   By default, the list includes all of the table modes.\
   \
   As you check and uncheck the table mode checkboxes, Structural adds and removes the associated tables from the list.

### Filtering to exclude tables that have assigned generators <a href="#database-view-tables-filter-has-generators" id="database-view-tables-filter-has-generators"></a>

You can filter the table list to only display tables that have no assigned generators:

1. Click **Filters**.
2. On the filter panel, to only show tables that do not have assigned generators, check the **No Generators Applied** checkbox.

## Assigning table modes to tables <a href="#database-view-tables-assign-mode" id="database-view-tables-assign-mode"></a>

{% hint style="info" %}
**Required workspace permission:** Assign table modes
{% endhint %}

The table mode determines the number of rows and columns in the destination database. For details about the available table modes and how they work, go to [Table modes](/app/generation/table-modes).

### Updating a single table <a href="#database-view-table-mode-single-table" id="database-view-table-mode-single-table"></a>

To change the assigned table mode for a single table:

![Assigning a table mode to a single table](/files/kD5OAJ4DAaB5HWfVYZB9)

1. Click the table mode dropdown next to the table name.
2. From the table mode dropdown list, select the table mode.
3. For a child workspace, the table mode selection panel indicates whether the selected table mode is inherited from the parent workspace.\
   \
   If the child workspace currently overrides the parent workspace configuration, then to reset the table mode to the table mode that is assigned in the parent workspace, click **Reset**.

### Updating multiple tables <a href="#database-view-table-mode-multiple-tables" id="database-view-table-mode-multiple-tables"></a>

To change the assigned table mode for multiple tables:

![Assigning a table mode to multiple tables](/files/XpGnmh7lF31xS6A7cvmr)

1. Check the checkbox for each table to change the table mode for.\
   \
   To select a continuous range of tables, click the first table in the range, then Shift-click the last table in the range.\
   \
   To select all of the tables in a schema, click the schema name.
2. Click **Bulk Edit**.
3. On the panel, click the radio button for the table mode to assign to the selected tables.


# Viewing the column list

The column list on **Database View** contains information about the sensitivity and generator configuration for each column.

![Column list on Database View](/files/F0g4Vd1YuOgU8GV9jSgC)

## Column - Column name and type <a href="#database-view-column-column" id="database-view-column-column"></a>

The **Column** column provides general information about the columns and their content, including:

* Table and column name. When you click the column name, [**Table View**](/app/generation/table-view) for the column table displays.
* The name of the schema that contains the table.
* The data type for the column.
* An indicator when the column is a primary key

The **Column** column also contains the option to [display sample data for the column](/app/generation/database-view/database-view-sample-data).

## Status - Protection and sensitivity status <a href="#database-view-status-column" id="database-view-status-column"></a>

The **Status** column provides information about whether the column contains sensitive data and whether it has an assigned generator.

The protection status can be one of the following values:

<figure><img src="/files/SCUffU8gME0kUnXzifNx" alt=""><figcaption><p>Status Values for a column</p></figcaption></figure>

* **Protected -** The column has an assigned generator.
* **Not Sensitive -** The column is marked as not sensitive.
* **At Risk -** The column is sensitive and does not have an assigned generator.

At the right of the **Status** column is a confidence indicator. For **At Risk** columns, the confidence indicator shows how confident Structural is that the column is sensitive and contains values of the displayed sensitivity type. Protected columns also reflect the original confidence level.

<figure><img src="/files/XtH60c9beLO65DQC8WVg" alt=""><figcaption><p>Confidence level indicators for database columns</p></figcaption></figure>

For more information about how Structural identifies values and assigns the confidence level, go to [Running the Structural sensitivity scan](/app/generation/identify-sensitive-data/running-the-structural-sensitivity-scan#sensitive-data-how-identified).

From the **Status** column, you can [change whether a column is sensitive](/app/generation/database-view/database-view-configure-column#database-view-column-config-single-sensitivity).

## Applied Generator - Column configuration <a href="#database-view-applied-generator" id="database-view-applied-generator"></a>

The **Applied Generator** column is where you [select and configure the generator to assign](/app/generation/database-view/database-view-configure-column).

The generator dropdown indicates the currently assigned generator. It also indicates when an unprotected column has a recommended generator.

<figure><img src="/files/60eDLJDNPkjRho1PCOUy" alt=""><figcaption><p>Unprotected column that has a recommended generator</p></figcaption></figure>

For foreign key columns, the generator dropdown is disabled and the column is marked as a foreign key. Foreign key columns always inherit the generator that is assigned to the primary key.

<figure><img src="/files/9TOk3oQKBgPGNpm9kDas" alt=""><figcaption><p>Disable generator dropdown for a foreign key column</p></figcaption></figure>

In a child workspace, when the generator configuration overrides the parent workspace, the generator dropdown displays the override icon.

<figure><img src="/files/LQ7LSzljqiNyETJhUV0C" alt=""><figcaption><p>Column with a generator configuration that overrides the parent workspace</p></figcaption></figure>

The **Applied Generator** column also contains the option to [display and create column comments](/app/generation/database-view/database-view-column-comment).

## Filtering the column list <a href="#database-view-columns-filter" id="database-view-columns-filter"></a>

You can use the [Structural Agent](broken://pages/LEAQtFWJ2Y868wHKb8G7) to display a filtered list of columns. For example:

* `Display Database View filtered to show all of the unprotected columns.`
* `Display Database View filtered to show non-sensitive columns in the stores and vendors tables.`

To filter the column list manually from **Database View**, you can:

* Use the table list to filter the displayed columns based on the table that the columns belong to.
* Use the filter field to filter the columns by table or column name.
* Use the **Filters** panel to filter the columns based on column attributes and generator configuration.

You can use column filters to quickly find columns that you want to verify or to update the configuration for.

### Filter by table <a href="#database-view-columns-filter-table" id="database-view-columns-filter-table"></a>

To filter the column list to only include columns for specific tables, either:

* [Apply a filter to the table list](/app/generation/database-view/database-view-tables#database-view-tables-filter).
* Check the checkbox for each table to display columns for.

### Filter by table or column name <a href="#database-view-columns-filter-name" id="database-view-columns-filter-name"></a>

To filter the column list by table or column name, in the filter field, begin to type text that is in the table or column name.

As you type, Structural filters the column list.

![Filtering columns by name](/files/6RippuaGaTRq3gdcSn5O)

### Using the Filters panel <a href="#database-view-columns-filter-panel" id="database-view-columns-filter-panel"></a>

The **Filters** panel provides access to column filters other than the table and column name.

To display the **Filters** panel, click **Filters**. The list only includes the filters that apply to the displayed data.

<figure><img src="/files/V2TAydrThzyB7QyIZ7N4" alt=""><figcaption><p>Filters panel for columns</p></figcaption></figure>

#### **Searching for a filter** <a href="#database-view-columns-filter-panel-search" id="database-view-columns-filter-panel-search"></a>

To search for a filter or a filter value, in the search field, start to type the value. The search looks for text in the individual settings.

For each filter, the **Filters** panel indicates the number of matching columns, based on the selected tables and the current filters.

<figure><img src="/files/uQl7fwUWVXpWME7jowwf" alt=""><figcaption><p>Using the column filter search</p></figcaption></figure>

#### **Adding a filter** <a href="#database-view-columns-filter-panel-add" id="database-view-columns-filter-panel-add"></a>

To add a filter, depending on the filter type, either check the checkbox or select a filter option. As you add filters, Structural applies them to the column list. Above the list, Structural displays tags for the selected filters.

<figure><img src="/files/hYPWArKK0pHnZflJuCaH" alt=""><figcaption><p>Filters panel with filters selected</p></figcaption></figure>

#### **Clearing the selected filters** <a href="#database-view-columns-filter-panel-clear" id="database-view-columns-filter-panel-clear"></a>

To clear all of the currently selected filters, click **Clear All**.

## Filters panel filters <a href="#database-view-column-filter-panel-filters" id="database-view-column-filter-panel-filters"></a>

### Columns with generator recommendations

To only display detected sensitive columns for which there is a recommended generator, on the **Filters** panel, check **Has Generator Recommendation**.

### At-risk columns <a href="#database-view-columns-filter-at-risk" id="database-view-columns-filter-at-risk"></a>

An at-risk column:

* Is marked as sensitive.
* Is included in the destination data.
* Is assigned the Passthrough generator.

To only display at-risk columns, on the **Filters** panel, check **At-Risk Column**.

When you check **At-Risk Column**, Structural adds the following filters under **Privacy Settings**:

* Sets the sensitivity filter to **Sensitive**
* Sets the protection status filter to **Not protected**
* Sets the column inclusion filter to **Included**

### Sensitivity <a href="#database-view-columns-filter-sensitivity" id="database-view-columns-filter-sensitivity"></a>

You can filter the columns based on the column sensitivity.

On the **Filters** panel, under **Privacy Settings**, the sensitivity filter is by default set to **All**, which indicates to display both sensitive and non-sensitive columns.

* To only display sensitive columns, click **Sensitive**.
* To only display non-sensitive columns, click **Not sensitive**.

Note that when you check **At-risk Column**, Structural automatically selects **Sensitive**.

### Protection status <a href="#database-view-columns-filter-has-generator" id="database-view-columns-filter-has-generator"></a>

You can filter the columns based on whether they have any generator other than Passthrough assigned. To filter the columns based on specific assigned generators, use the **Applied Generator** filter.

On the **Filters** panel, under **Privacy Settings**, the column protection filter is by default set to **All**, which indicates to display both protected and not protected columns.

* To only display columns that have an assigned generator, click **Protected**.
* To only display columns that do not have an assigned generator, click **Not protected**.

Note that when you check **At-Risk Column**, Structural automatically selects **Not protected**.

### Inclusion in the destination database <a href="#database-view-columns-filter-destination-data" id="database-view-columns-filter-destination-data"></a>

You can filter the columns based on whether they are populated in the destination database. For example, if a table is truncated, then the columns in that table are not populated.

On the **Filters** panel, under **Privacy Settings**, the column inclusion filter is by default set to **All**, which indicates to display both included and not included columns.

* To only display columns that are populated in the destination database, click **Included**.
* To only display columns that are not populated in the destination database, click **Not included**.

Note that when you check **At-Risk Column**, Structural automatically selects **Included**.

### Assigned generator <a href="#database-view-columns-filter-assigned-generator" id="database-view-columns-filter-assigned-generator"></a>

To only display columns that are assigned specific generators, on the **Filters** panel, under **Applied Generator,** check the checkbox for each generator to include.

The list of generators only includes generators that are assigned to the currently displayed columns and that are compatible with other applied filters.

To search for a specific generator, in the **Filters** search field, begin to type the generator name.

### Column data type <a href="#database-view-columns-filter-data-type" id="database-view-columns-filter-data-type"></a>

You can filter the columns by the column data type. For example, you can only display `varchar` columns, or only columns that contain either numeric or integer values.

To only display columns that have specific data types, on the **Filters** panel, under **Database Data Types**, check the checkbox for each data type to include.

The list of data types only includes data types that are present in the currently displayed columns and that are compatible with other applied filters.

To search for a specific data type, in the **Filters** search field, begin to type the data type.

### Unresolved schema changes <a href="#filtering-for-unresolved-schema-changes" id="filtering-for-unresolved-schema-changes"></a>

When the source database schema changes, you might need to update the configuration to reflect those changes. If you do not resolve the schema changes, then the data generation might fail. The data generation fails if there are unresolved conflicting changes, or if you configure Structural to always fail data generation when there are any unresolved changes.

For more information about schema changes, go to [Viewing and resolving schema changes](/app/generation/schema-changes).

To only display columns that have unresolved schema changes, on the **Filters** panel, check **Unresolved Schema Changes**.

### Sensitivity type <a href="#database-view-column-filter-sensitivity-type" id="database-view-column-filter-sensitivity-type"></a>

For detected sensitive columns, the sensitivity type indicates the type of data that was detected. Examples of sensitivity types include First Name, Address, and Email.

To only display columns that contain specific sensitivity types, on the **Filters** panel, under **Sensitivity Type**, check the checkbox for each sensitivity type to include.

The list of sensitivity types only includes sensitivity types that are present in the currently displayed columns.

To search for a specific sensitivity type, in the **Filters** search field, type the sensitivity type.

### **Sensitivity confidence** <a href="#database-view-filter-sensitivity-confidence" id="database-view-filter-sensitivity-confidence"></a>

When the Structural sensitivity scan [identifies a value as belonging to a sensitivity type](/app/generation/identify-sensitive-data/running-the-structural-sensitivity-scan#sensitive-data-how-identified), it also determines how confident it is in that determination. The **Status** column displays the confidence level.

You can filter the columns based on the confidence level.

To only display columns that have a specific confidence level, on the **Filters** panel, under **Sensitivity confidence**, check the checkbox next to each confidence level to include.

### Column nullability <a href="#database-view-columns-filter-nullability" id="database-view-columns-filter-nullability"></a>

You can filter the column list based on whether the column is nullable.

On the **Filters** panel, under **Data Attributes**, the nullability filter is by default set to **All**, which indicates to display both nullable and non-nullable columns.

* To only display columns that are nullable, click **Nullable**.
* To only display columns that are not nullable, click **Non-nullable**.

### Column uniqueness <a href="#database-view-columns-filter-uniqueness" id="database-view-columns-filter-uniqueness"></a>

You can filter the column list based on whether the column must be unique.

On the **Filters** panel, under **Data Attributes**, the uniqueness filter is by default set to **All**, which indicates to display both unique and not unique columns.

* To only display columns that must be unique, click **Unique**.
* To only display columns that do not require uniqueness, click **Not unique**.

### Primary or foreign keys <a href="#database-view-columns-filter-is-key" id="database-view-columns-filter-is-key"></a>

You can filter the column list to indicate whether to include:

* Columns that are not primary or foreign keys.
* Columns that are foreign keys.
* Columns that are primary keys.

On the **Filters** panel, under **Column Type**:

* To display columns that are neither a primary key nor a foreign key, check **Non-keyed**.
* To display columns that are primary keys, check **Primary key**.
* To display columns that are foreign keys, check **Foreign key**.

### Generator overrides in a child workspace

In a child workspace, to only display columns that override the generator configuration that is in the parent workspace, on the **Filters** panel, check **Overrides Inheritance**.

### Uses Structural data encryption

You can enable Structural data encryption, a configuration that allows Structural to:

* Decrypt source data before applying the generator.
* Encrypt generated data before writing it to the destination database.

For more information, go to [Configuring and using Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config).

When Structural data encryption is enabled, the generator configuration panel includes an option to use Structural data encryption.

To only display columns that are configured to use Structural data encryption, on the **Filters** panel, check **Uses Data Encryption**.

### Columns with automatically assigned generators

You can filter the columns to only display those for which Structural automatically assigned the generator. Based on the [workspace configuration](/app/workspace/workspace-configuration-settings/schema-management-settings#workspace-config-schema-change-handling), this can happen during data generation in response to changes to the source data schema.

To only display columns that have automatically applied generators, on the **Filters** panel, toggle **Automatically Applied Generators** to the on position.

## Sorting the column list <a href="#database-view-columns-sort" id="database-view-columns-sort"></a>

By default, the column list is sorted first by table name, then by column name. The columns for each table display together. Within each table, the columns are in alphabetical order.

You can also sort the column list by column name first, then by table. Columns that have the same name display together. Those columns are sorted by the name of the table.

The button at the right of the **Column** column heading indicates the current sort order.

<figure><img src="/files/byPSZ1M21jYUVFeBqMDi" alt=""><figcaption><p>Sort button in the Column heading</p></figcaption></figure>

* **T.C** indicates that the table is sorted by table, then by column
* **C.T** indicates that the table is sorted by column, then by table

To switch the sort order, click the button.


# Displaying sample data for a column

{% hint style="info" %}
**Required workspace permission:**

* **Source data:** Preview source data
* **Destination data:** Preview destination data
  {% endhint %}

For each column on **Database View**, you can display a sample list of the column values.

For columns that have an assigned generator, the sample shows both the current values and the possible values after the generator is applied.

To display the sample values, in the **Column** column, click the magnifying glass icon.

If the generator is Passthrough, then the sample data panel contains only **Original Data**.

![Sample data for a column that does not have an assigned generator](/files/zPafD77gSIishk1urtHF)

If a different generator is assigned, then the sample data panel contains both **Original Data** and **Protected Output**.

![Sample data for a column that has an assigned generator](/files/liRZyhR8eRqqd0EeuxyC)


# Configuring an individual column

For an individual column in **Database View**, you can configure the assigned generator and determine the column sensitivity.

## Displaying the generator configuration panel <a href="#database-view-display-generator-config" id="database-view-display-generator-config"></a>

From the column list, to display the generator configuration panel, in the **Applied Generator** column, click the generator name tag.

<figure><img src="/files/nbWFQaywXo7YcnBltCbo" alt=""><figcaption><p>Generator configuration panel</p></figcaption></figure>

## Indicating whether a column is sensitive <a href="#database-view-column-config-single-sensitivity" id="database-view-column-config-single-sensitivity"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column sensitivity
{% endhint %}

The Structural sensitivity scan provides an initial indication of whether a column is sensitive and, if it is sensitive:

* The type of sensitive data that is in the column.
* The confidence level of the sensitivity detection.

For more information, go to [Identifying sensitive data](/app/generation/identify-sensitive-data).

In a child workspace, you cannot configure whether a column is sensitive. A child workspace always inherits the sensitivity designations from its parent workspace.

### **Status column** <a href="#database-view-sensitivity-status-column" id="database-view-sensitivity-status-column"></a>

From the **Status** column, to confirm or change the column sensitivity, click the **Status** value.

The status panel indicates whether the column is sensitive. It identifies the sensitivity type, and indicates how the sensitivity was determined - by a sensitivity scan or by a user.

#### Built-in sensitivity type <a href="#status-column-structural-detected" id="status-column-structural-detected"></a>

For a column that matches a built-in sensitivity type, the first time that you display the panel, the **Sensitive data?** setting displays **Yes** and **No** options for you to confirm or change the sensitivity.

<figure><img src="/files/6jiZlWFDZR18hhUvZBr6" alt=""><figcaption><p>Column status panel with confirmation options</p></figcaption></figure>

* To indicate that the column is sensitive, click **Yes**.
* To indicate that the column is not sensitive, click **No**.

When you click **Yes** or **No**, the **Yes** and **No** options change to a simple toggle. When you click **Yes**, the sensitivity confidence level changes to full.

<figure><img src="/files/6qnaUXKLa6HEWXXMe3XO" alt=""><figcaption><p>Status panel after you select No or Yes to indicate the sensitivity</p></figcaption></figure>

After that:

* To indicate that the column is sensitive, toggle **Sensitive data?** to the on position.
* To indicate that the column is not sensitive, toggle **Sensitive data?** to the off position.

#### Sensitivity rule match <a href="#status-column-sensitivity-rule" id="status-column-sensitivity-rule"></a>

When a column matches a sensitivity rule, the sensitivity panel indicates that the column matched a sensitivity rule.

<figure><img src="/files/Dt6irYuIYdOoieob9iDy" alt=""><figcaption><p>Sensitivity panel for a column that matched a sensitivity rule</p></figcaption></figure>

You use the **Sensitive data?** toggle to indicate whether the column is actually sensitive.

#### No built-in sensitivity type or sensitivity rule match <a href="#status-column-not-sensitive" id="status-column-not-sensitive"></a>

When a column does not match a built-in sensitivity type or a custom sensitivity rule, the sensitivity panel indicates that column is not sensitive.

<figure><img src="/files/frRrwwYOZ5ICZYvz4pur" alt=""><figcaption><p>Sensitivity panel for a not sensitive column</p></figcaption></figure>

The **Sensitive data?** setting displays **Yes** and **No** options for you to confirm or change the sensitivity.

* To indicate that the column is sensitive, click **Yes**.
* To confirm that the column is not sensitive, click **No**.

When you click **Yes** or **No**, the **Yes** and **No** options change to a simple toggle.

If you click **Yes**:

* The panel updates to indicate that a user confirmed that the column is sensitive.
* The sensitivity confidence level is set to full confidence.

<figure><img src="/files/nL20KgSUlkLEKe3fapmf" alt=""><figcaption><p>Sensitivity panel after you change the sensitivity on a not sensitive column</p></figcaption></figure>

After that:

* To indicate that the column is sensitive, toggle **Sensitive data?** to the on position.
* To indicate that the column is not sensitive, toggle **Sensitive data?** to the off position.

### **Column configuration panel** <a href="#database-view-sensitivity-generator-config" id="database-view-sensitivity-generator-config"></a>

To configure the sensitivity, you can also use the **Sensitive Data** toggle on the column configuration panel.

<figure><img src="/files/T9nbembWcmJnyXt1rRh4" alt=""><figcaption><p>Sensitivity toggle on the column configuration panel</p></figcaption></figure>

* To indicate that a column is sensitive, toggle the sensitivity setting to the on position.
* To indicate that the column is not sensitive, toggle the sensitivity setting to the off position.

When you change the sensitivity from the generator configuration panel, the **Sensitive data?** setting on the sensitivity panel also changes from the **Yes** and **No** options to the toggle.

## Assigning or ignoring the recommended generator <a href="#database-view-single-column-recommended-generator" id="database-view-single-column-recommended-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

When a sensitivity scan identifies a column, Structural recommends a generator for the column. For example, when the sensitivity scan identifies a column as a first name, Structural recommends the Name generator configured to generate a first name value.

In the **Assigned Generator** column on **Database View**, columns that do not have an assigned generator, and that have a recommended generator, display the available recommendation icon.

<figure><img src="/files/60eDLJDNPkjRho1PCOUy" alt=""><figcaption><p>Column with the available recommendation icon</p></figcaption></figure>

When you click the generator dropdown, the column configuration panel includes the following information:

* The sensitivity confidence level.
* The recommended generator.
* Sample source and destination values based on the recommended generator.

<figure><img src="/files/xHNsWwS1Bs7QD5obCgmx" alt=""><figcaption><p>Recommended generator panel for a column</p></figcaption></figure>

From the panel, you choose whether to assign or ignore the recommended generator for that type.

* To assign the recommended generator, click **Apply**.
* To ignore the recommendation, click **Ignore**. Structural clears the recommendation.

## Changing the column generator configuration <a href="#database-view-column-config-single-generator" id="database-view-column-config-single-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

To change the generator that is assigned to a selected column:

1. Click the generator name tag for the column.
2. To assign a different generator to the column, from the **Generator Type** dropdown list, select the generator.
3. Configure the generator options.

To reset an assigned generator to Passthrough, which indicates to not transform the data, click **Reset**, then click **Reset to Passthrough**.

For details about the configuration options for each generator, go to the [Generator reference](/app/generation/generators/generator-reference).

For more information about selecting and configuring generators and generator presets, go to [Assigning and configuring generators](/app/generation/generators-assign-config/generator-assignment-and-config).

## Enabling Document View for JSON columns <a href="#column-enable-document-view" id="column-enable-document-view"></a>

{% hint style="info" %}
Supported only for the file connector and PostgreSQL.
{% endhint %}

For a JSON column, instead of assigning a generator, you can enable **Document View**.

From **Document View**, you can view the JSON schema structure and assign generators to individual JSON fields. For more information, go to [Using Document View for JSON columns](/app/generation/working-with-document-based-data/json-document-view).

To enable **Document View**, on the column configuration panel, toggle **Use Document View** to the on position. Note that if you have [custom value processors](/app/generation/generators-assign-config/custom-value-processors), or enabled [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config), then the **Use Document View** toggle is in the advanced options.

&#x20;When **Document View** is enabled, the generator dropdown is replaced with the **Open in Document View** option.


# Configuring multiple columns

The bulk edit option on **Database View** allows you to configure multiple columns at the same time. From the bulk editing panel, you can:

* Mark the selected columns as sensitive or not sensitive.
* Assign a generator to the selected columns.
* Apply the recommended generator to the selected columns.
* Reset the generator configuration to the baseline. This option requires that all of the selected columns are assigned the same preset.

Depending on the column selection, you can also create a new sensitivity rule.

## Displaying the bulk edit option <a href="#database-view-column-display-bulk-option" id="database-view-column-display-bulk-option"></a>

To select the columns and display the bulk edit option:

1. Check the checkbox next to each column to update.
2. Click **Bulk Edit**.

<figure><img src="/files/JI7inE8bulkC92LSSndJ" alt=""><figcaption><p>Bulk Edit panel to update multiple columns</p></figcaption></figure>

## Marking the columns as sensitive or not sensitive <a href="#database-view-column-config-multi-sensitivity" id="database-view-column-config-multi-sensitivity"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column sensitivity
{% endhint %}

On the **Bulk Edit** panel, under **Sensitivity**:

* To mark the selected columns as sensitive, click **Sensitive**.
* To mark the selected columns as not sensitive, click **Not Sensitive**.

## Changing the assigned generator <a href="#database-view-column-config-multi-generator" id="database-view-column-config-multi-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

On the **Bulk Edit** panel, under **Bulk Edit Applied Generator**, select and configure the generator to assign to the selected columns.

## Assigning the recommended generator to the columns <a href="#database-view-bulk-column-apply-recommendations" id="database-view-bulk-column-apply-recommendations"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

If any of the selected columns have a recommended generator, then on the **Bulk Edit** panel, the **Generator recommendations found** panel displays. The panel indicates the number of selected columns that have a recommendation.

To assign the recommended generators to those columns, click **Apply**.

## Restoring the baseline configuration for the columns

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

For a generator preset, the baseline configuration is the configuration that is saved for that preset. The baseline configuration determines the default configuration to use when you assign the preset to a column. After you select the preset, you can override the baseline configuration.

If all of the selected columns are assigned the same preset, then to restore the baseline configuration for all of the columns, click **Reset**, then select **Reset to Generator Preset**.

## Creating a sensitivity rule <a href="#database-view-bulk-sensitivity-rule" id="database-view-bulk-sensitivity-rule"></a>

{% hint style="info" %}
**Required license:** Enterprise

**Required global permission:** Create and manage sensitivity rules
{% endhint %}

You might bulk edit columns that could benefit from a custom sensitivity rule.

For example, in your data, the Widget column is in multiple tables and contains sensitive data that Structural cannot identify. You select all of the Widget columns so that you can mark them as sensitive and apply the Character Scramble generator to them.

However, a custom sensitivity rule would ensure that in the future, Widget columns are always marked as sensitive and have the Character Scramble generator recommended.

On the **Bulk Edit** panel, when all of the selected columns:

* Have the same data type.
* Do not have a generator assigned.
* Do not have a recommended generator.

Then Structural displays the **Create a Sensitivity Rule** panel, which contains the option to create a new sensitivity rule.

<figure><img src="/files/CxtHIqRofKZx21YWr6aR" alt=""><figcaption><p>Bulk Edit panel with the option to create a sensitivity rule</p></figcaption></figure>

To create a sensitivity rule:

1. Click **Create Custom Rule**.
2. On the **Create Custom Rule** view, configure the new sensitivity rule.\
   \
   Structural automatically selects a data type based on the selected columns.\
   \
   The current workspace is used as the testing workspace to verify the columns that match the rule configuration.\
   \
   For details about the sensitivity rule configuration, go to [Creating and managing custom sensitivity rules](/app/generation/identify-sensitive-data/custom-sensitivity-rules#sensitivity-rule-config).
3. When you finish configuring the new rule:
   * To both save the rule and apply the generator preset to all workspace columns that match the rule, click **Save and Apply**. On the confirmation panel, click **Confirm Auto Apply**.
   * To save the rule, but not apply the configured generator preset to matching columns, click **Save**.

Structural closes the sensitivity rule configuration view and returns you to **Database View**. It maintains the previous column selection.

If you did not apply the generator preset, then the sensitivity rule is included in the next sensitivity scan.


# Identifying similar columns

During sensitivity scans and schema change scans, Tonic Structural identifies groups of similar columns.

To identify similar columns, Structural uses a text embedding model to calculate the semantic similarity between any two column names in the database. When a column name's semantic similarity  to the name of a given column is above a specified threshold, then the column is similar to the given column.

If a column has similar columns, then the **Applied Generator** column contains an icon that includes the count of similar columns.

<figure><img src="/files/HNiTVFPmsHNiMAs7G6TW" alt=""><figcaption><p>Similar columns icon with the count of similar columns</p></figcaption></figure>

By default, the similar columns icon is hidden. To display the similar columns icon, hover over the column row.

When you assign a generator to a column, the similar columns icon for that column remains visible during your current session.

When you click the similar columns icon, Structural displays a panel with an option to filter the list to display the current column and its similar columns. To apply the filter, click **Filter**.&#x20;

<figure><img src="/files/P8Cxq1L0gD5EVngyBoBa" alt=""><figcaption><p>Similar columns panel with filter option</p></figcaption></figure>

The similar columns filter is applied, and other table and column filters are removed.

<figure><img src="/files/Jf5OGYeQvlALw0PXS4tk" alt=""><figcaption><p>Database View column list with a similar columns filter applied</p></figcaption></figure>


# Commenting on columns

{% hint style="info" %}
**Required license:** Professional or Enterprise
{% endhint %}

From **Database View**, you can add comments to columns. For example, you might use a comment to explain why you selected a particular generator or marked a column as sensitive or not sensitive.

## Creating a new comment <a href="#comment-new" id="comment-new"></a>

If a column does not have any comments, then to add a comment:

![Comment panel for a column that has no comments](/files/eB9o05wTTuiCWDQSp01n)

1. In the **Applied Generator** column, click the comment icon.
2. In the comment field, type the comment text.
3. Click **Comment**.

## Responding to existing comments <a href="#comment-respond" id="comment-respond"></a>

When a column has existing comments, the comment icon is green. To add comments:

![Replying to existing comments on a column](/files/3nFm2yGJn3bB3GMPhGRz)

1. Click the comment icon.\
   \
   The comments panel shows the previous comments. Each comment includes the comment user.
2. In the comment field, type the comment text.
3. Click **Reply**.


# Table View

**Table View** displays source or preview data for a single table. For a [file connector](/app/setting-up-your-database/file-connector) workspace, each table corresponds to a file group.

{% hint style="info" %}
**Required workspace permission:**

* **Source data:** Preview source data
* **Destination data:** Preview destination data

If you do not have either of these permissions, then you cannot display **Table View**.
{% endhint %}

To display **Table View**:

* On the workspace management view, click **Table View**.
* On **Workspaces** view, from the dropdown menu in the **Name** column, select **Table View**.
* From **Database View**, either click the arrow icon for the table, or click a row in the table.

From **Table View**, you can view and update the table and column configuration.

![Table View with highlighted sections](/files/SPmMRHxb0vbuymLRbaVm)

## Selecting and configuring tables <a href="#table-view-select-table" id="table-view-select-table"></a>

### Selecting the table to view <a href="#table-view-select-table" id="table-view-select-table"></a>

When you display **Table View** from **Database View**, it displays the data for the selected table.

When you display **Table View** from the workspace management view or **Workspaces** view, it displays the most recently displayed table.

If **Table View** was never displayed before, then it displays the first table in the workspace.

To change the selected table, from the **Table** dropdown list, select the table to view.

### Selecting the table mode <a href="#table-view-table-mode" id="table-view-table-mode"></a>

{% hint style="info" %}
**Required workspace permission:** Assign table modes
{% endhint %}

To change the table mode that is assigned to the table:

1. Click the current table mode.
2. On the table mode panel, from the table mode dropdown list, select the new table mode.

![Table Mode selection on Table View](/files/17gUGGszUUZ7FH0apcgu)

When you change the table mode, Tonic Structural updates the preview data as needed. For example, if you change the table mode to Truncate, then the preview data is empty.

For a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance), the table mode selection panel indicates whether the selected table mode is inherited from the parent workspace.

<figure><img src="/files/TEFIfpCPF00Xja5gqT92" alt=""><figcaption><p>Table mode configuration that overrides the parent workspace</p></figcaption></figure>

If the child workspace currently overrides the parent workspace configuration, then to reset the table mode to the table that is assigned in the parent workspace, click **Reset**.

## Viewing the generator configuration summary <a href="#table-view-model" id="table-view-model"></a>

The **Model** section of **Table View** displays the configured generators for the table columns.

![Model section of Table View](/files/Lxw7oVeq4ne1H0ga9Hn6)

The header for each **Model** entry is the column name.

Linked columns share an entry. The heading is a comma-separated list of the linked columns.

<figure><img src="/files/aAc9EgGByawWJhSXTSxn" alt=""><figcaption><p>Model entry for linked columns</p></figcaption></figure>

Each entry contains the following information:

* The column and generator, in the format `Column Name >> Generator Name`. For example, `First_Name >> Name` indicates that the `First_Name` column has the Name generator applied.\
  \
  For linked columns, there is a `Column Name >> Generator Name` entry for each column.
* The selected configuration options for the generator.

For a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance), each **Model** entry indicates whether the configuration overrides the parent configuration. For configurations that override the parent, to remove the overrides and restore the inheritance, click **Reset**.

<figure><img src="/files/1JUeUmB53zB2K98y4HOZ" alt=""><figcaption><p>Model entry for a configuration that overrides the parent workspace</p></figcaption></figure>

The **Model** entry also indicates when [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled for the column.

To remove the generator from a column, click the delete icon.

## Changing the column data display <a href="#table-view-toggle-source-preview-data" id="table-view-toggle-source-preview-data"></a>

### Toggling between source and preview data <a href="#table-view-toggle-source-preview-data" id="table-view-toggle-source-preview-data"></a>

The **Preview** toggle at the top right of **Table View** allows you to choose whether to display original source data or the transformed data. You can switch back and forth to understand exactly how Structural transforms the data based on the table and column configuration.

By default, the **Preview** toggle is in the on position, and the displayed data reflects the selected table mode and the assigned generators. For tables that use Truncate mode, the preview data is empty. Truncated tables do not have data in the destination database.

To display the original source data, toggle **Preview** to the off position.

Note that for [JSON columns that use **Document View**](#column-json-document-view), you cannot preview the destination data from **Table View**. You must preview the data from **Document View**.

### Using a query to filter the source data <a href="#table-view-query-filter" id="table-view-query-filter"></a>

You can provide a query to filter the source data. The query is always against the source data, not the preview data, regardless of whether the **Preview** toggle is off or on.

For example, you configure a first name field to use the Name generator and enable consistency. You can then query the source data for a specific first name value to check that the preview data uses the same destination value for all of those records.

To apply a query to the source data:

1. Click the query filter icon, located between the table name and the table mode.
2. On the **Table Filter** dialog, provide the `WHERE` clause for the query.
3. To apply the query, click **Apply**.
4. To close the dialog, click **Close**.

![Table Filter dialog for Table View](/files/bjBv5oVlV2WZyiQxquH8)

To clear an applied query, on the **Table Filter** dialog, click **Clear**.

If no filter is applied, then the query filter icon has a white background.

![Query filter icon when no query is applied](/files/LFqWyffBTnVIWSEZhbv6)

If a valid filter is applied, then the query filter icon has a gray background.

![Query filter icon when a query is applied](/files/AmC0wMuiadvjTsR54wbv)

If the provided `WHERE` clause is not valid, then the query filter icon has a red background.

![Query filter icon when a bad query is applied](/files/zIAcMwhwLdJ5F0fKQ7ff)

## Navigating to a specific column <a href="#table-view-jump-to-column" id="table-view-jump-to-column"></a>

When the width of the table is more than 1.5 times the visible display area, then the **Jump to column** option displays.

To bring a specific column into view:

1. At the top right of **Table View**, click **Jump to column**.\
   \
   The list of table columns is displayed.
2. To filter the list, in the filter field, begin to type text that is in the column name.

<figure><img src="/files/hiXsag833oVbfHkzxva0" alt=""><figcaption><p>Jump to column list with column name filter text</p></figcaption></figure>

3. Click the column.\
   \
   Structural scrolls the columns to make the selected column visible.

## Information in the column headings <a href="#table-view-column-attributes-data" id="table-view-column-attributes-data"></a>

In addition to the column name, the column heading provides details about the column type and protection status. It also provides access to change the column configuration.

### Primary and foreign key indicators <a href="#column-primary-foreign-key" id="column-primary-foreign-key"></a>

The column heading indicates when a column is either a primary key or a foreign key.

<figure><img src="/files/3wmiz7e4MA9ZCgennY7H" alt=""><figcaption><p>Column headings for primary and foreign keys</p></figcaption></figure>

### Protection status

The column heading indicates the column protection status:

<figure><img src="/files/ZGLPIlGWYTCyDIvlUlsY" alt=""><figcaption><p>Protection status indicators in the column headings</p></figcaption></figure>

* At risk columns are sensitive and do not have an assigned generator.
* Protected columns have an assigned generator.
* Not sensitive columns are not sensitive and do not have an assigned generator.

### Sensitivity confidence

The sensitivity confidence indicator indicates the confidence in the detection.

<figure><img src="/files/AwZVCXFiTlM3EWTCMcXM" alt=""><figcaption><p>Sensitivity confidence level indicator</p></figcaption></figure>

For sensitive columns that Structural detected, the confidence level can be high, medium, or low.

For custom sensitivity rule matches or columns that you manually marked as sensitive, the confidence level is full confidence.

For more information about how Structural identifies values and assigns the confidence level, go to [Running the Structural sensitivity scan](/app/generation/identify-sensitive-data/running-the-structural-sensitivity-scan#sensitive-data-how-identified).

### Column data type

The column heading displays the type of data that the column contains.

<figure><img src="/files/u7wXHmGLax592NljBMXV" alt=""><figcaption><p>Data type information in the column headings</p></figcaption></figure>

### Child workspace overrides <a href="#table-view-view-overrides" id="table-view-view-overrides"></a>

{% hint style="info" %}
**Required license:** Enterprise
{% endhint %}

In a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance), when a column overrides the parent configuration, an **Overriding** label displays in the column heading.

<figure><img src="/files/qzTUYnGmLmDJWwylQ61h" alt=""><figcaption><p>Table View column with a configuration override</p></figcaption></figure>

To filter **Table View** to only display columns with overrides, toggle **Show Overrides Only** to the on position.

<figure><img src="/files/mNKJCo7XnH6RI74Kdcpa" alt=""><figcaption><p>Table View with Show Overrides Only enabled</p></figcaption></figure>

## Configuring a column <a href="#table-view-configure-columns" id="table-view-configure-columns"></a>

### Applying or ignoring a recommended generator <a href="#table-view-recommended-generator" id="table-view-recommended-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

When a sensitivity scan identifies a column, Structural recommends a generator for the column. For example, when the sensitivity scan identifies a column as a first name, Structural recommends the Name generator configured to generate a first name value.

For unprotected columns that have a recommended generator, the column heading displays the available recommendation icon.

<figure><img src="/files/mWhUfpDT8xDd6f9Oiuib" alt=""><figcaption><p>Table View column heading with the recommended generator icon</p></figcaption></figure>

When you click the dropdown, the column configuration panel includes the following information:

<figure><img src="/files/IWsRdSG0vl9OhiQC5tJD" alt=""><figcaption><p>Recommended generator panel for a column on Table View</p></figcaption></figure>

* The sensitivity confidence level
* The recommended generator
* Sample source and destination values based on the recommended generator

From the panel, you can choose whether to assign or ignore the recommended generator for that type.

* To assign the recommended generator, click **Apply**.
* To ignore the recommendation, click **Ignore**. Structural clears the recommendation.

### Changing the column generator configuration <a href="#table-view-column-generator" id="table-view-column-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

To assign a generator to a column that does not have an assigned generator, or to change the current configuration, click the dropdown in the column heading.

On the generator configuration panel, from the generator type dropdown list, select the generator to assign to the column.

<figure><img src="/files/S7efwUHTAuopL3trOSk2" alt=""><figcaption><p>Generator dropdown list for a Table View column</p></figcaption></figure>

Structural displays the available configuration options for the selected generator. For details about the configuration options for each generator, go to the [Generator reference](/app/generation/generators/generator-reference).

To remove the selected generator or generator preset, and reset the generator to Passthrough, click **Reset**, then click **Reset to Passthrough**.

For more information about selecting and configuring generators and generator presets, go to [Assigning and configuring generators](/app/generation/generators-assign-config/generator-assignment-and-config).

### Indicating whether a column is sensitive <a href="#table-view-configure-column-sensitivity" id="table-view-configure-column-sensitivity"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column sensitivity
{% endhint %}

On the column configuration panel, the **Sensitive Data** toggle indicates whether the column is marked as sensitive. The initial configuration is based on the sensitivity scan.

* To mark a column as sensitive, toggle the setting to the on position.
* To mark a column as not sensitive, toggle the setting to the off position.

In a [child workspace](/app/workspace/managing-workspaces/workspaces-inheritance), you cannot configure whether a column is sensitive. A child workspace always inherits the sensitivity designation from its parent workspace.

When you copy a workspace, Structural performs a new sensitivity scan on the copy. It does not copy the sensitivity designations from the original workspace.

### Enabling Document View for JSON columns <a href="#column-json-document-view" id="column-json-document-view"></a>

{% hint style="info" %}
Supported only for the file connector and PostgreSQL.
{% endhint %}

For a JSON column, instead of assigning a generator, you can enable **Document View**.

From **Document View**, you can view the JSON schema structure and assign generators to individual JSON fields. For more information, go to [Using Document View for JSON columns](/app/generation/working-with-document-based-data/json-document-view).

To enable **Document View**, on the column configuration panel, toggle **Use Document View** to the on position. Note that if you have [custom value processors](/app/generation/generators-assign-config/custom-value-processors), or enabled [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config), then the **Use Document View** toggle is in the advanced options.

&#x20;When **Document View** is enabled, the generator dropdown is replaced with the **Open in Document View** option.


# Working with document-based data

For document-based data connectors - currently [MongoDB](/app/setting-up-your-database/mongodb) - **Database View** and **Table View** are replaced by **Collection View**. "Collection" is the term that Structural uses to refer to MongoDB collections.

For JSON columns in file connector and PostgreSQL workspaces, you can use **Document View** to view and assign generators to JSON fields.

<table data-view="cards"><thead><tr><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Scan collections</strong><br><br>Collection scans identify the fields and data types in a collection.</td><td></td></tr><tr><td><strong>Use Collection View</strong><br><br>Configure generators and collection modes.</td><td><a href="/pages/GCwZu14RRvB5QWvA5GYx">/pages/GCwZu14RRvB5QWvA5GYx</a></td></tr><tr><td><strong>Use Document View for JSON columns</strong><br><br>Assign generators to fields in JSON columns.</td><td><a href="/pages/ECML3cFwndLoP2YxJwvS">/pages/ECML3cFwndLoP2YxJwvS</a></td></tr><tr><td><strong>Assign generators to path expressions</strong><br><br>In <strong>Collection View</strong> or <strong>Document View</strong>, assign a generator to fields that match a JSONPath expression.</td><td><a href="/pages/zgxOyU7jo4MjaIkIiyBV">/pages/zgxOyU7jo4MjaIkIiyBV</a></td></tr></tbody></table>


# Performing scans on collections

{% hint style="info" %}
**Required workspace permission:** Run collection scan
{% endhint %}

When you first connect to a [MongoDB](/app/setting-up-your-database/mongodb) database, Tonic Structural performs a scan to determine the available fields in each collection, the field types, and how prevalent the fields are. It performs this scan at the same time as the initial sensitivity scan.

For each collection, Structural creates a hybrid document, which is a superset of all of the fields contained in the collection documents.

## Configuring the collection scan <a href="#collection-scan-configure" id="collection-scan-configure"></a>

By default, for each collection:

* The scan includes all of the documents in the collection, and continues until the scan is finished.
* Every unique path (field+data type) in the collection is added to the hybrid document.

You can change the default scan behavior. To change the scan configuration, use the following [environment settings](/app/admin/environment-variables-setting). You can add these settings manually to the **Environment Settings** list on **Structural Settings**.

### Configuring how schemas are scanned

The following options control the number of documents that Structural scans in a collection.

These options allow you to limit the number of scanned documents when the additional documents do not add fields to the hybrid document.

For large homogenous collections, where all or most documents have the same structure, configuring these options can improve performance.

<table data-header-hidden><thead><tr><th width="414" valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><code>TONIC_DOCUMENT_SCAN_MAX_DOCS_COUNT</code></td><td valign="top">The maximum number of documents to scan for each schema in a collection.<br><br>For example, if this is 10, then Structural scans up to 10 documents, and ignores the remaining documents.<br><br>When this value is empty, Structural scans all of the documents. </td></tr><tr><td valign="top"><code>TONIC_DOCUMENT_SCAN_MAX_TIME_SECONDS</code></td><td valign="top">The maximum amount of time in seconds to scan a schema.<br><br>For example, if this is 360, then Structural scans a schema for up to 360 seconds.<br><br>When this value is empty, Structural continues the scan until it is complete.<br><br>You can override this setting in an individual workspace.</td></tr></tbody></table>

If you set both options, then the scan completes when it reaches either limit. For example, if the maximum document count is 10 and the maximum scan time is 360 seconds, then the scan completes either after 10 documents or after 360 seconds, whichever comes first.

### Configuring how fields are collapsed in the hybrid document

Typically, the number of unique fields in a collection is small relative to the number of documents. However, in some cases the number of fields is similar to or greater than the number of documents.  This most commonly occurs when documents have "data as keys", such as keys that are ObjectIds, UUIDs, or incrementing integers.

In these cases, adding every unique field to the hybrid document can result in a large hybrid document that has an undesirable structure.

Structural offers configuration options to "collapse" fields within the hybrid document. This shrinks the size of the hybrid document. It also allows you to assign a generator to the collapsed group instead of to each unique key.

By default, Structural does not collapse fields.

#### Collapsing fields when the key is an ObjectId <a href="#collection-scan-config-collapse-objectid" id="collection-scan-config-collapse-objectid"></a>

To enable this, set the [environment setting](/app/admin/environment-variables-setting) `TONIC_MONGO_OBJECT_ID_COLLAPSE_THRESHOLD` to the number of ObjectId keys that an object can contain before Structural collapses the object schema into a single key.

For example, if this is 10, then any object that has 10 or more ObjectId keys is collapsed into a single key.

A negative value indicates to not collapse the keys.

The default value is -1.

#### Collapsing fields when the key matches a custom pattern <a href="#collection-scan-config-collapse-regex" id="collection-scan-config-collapse-regex"></a>

To enable Structural to collapse fields, you provide a regular expression to identify the fields that can be collapsed into the same field. You then configure the number of matches that must exist before Structural collapses the fields.

To configure how the fields are collapsed, use the following [environment settings](/app/admin/environment-variables-setting):

<table data-header-hidden><thead><tr><th width="407" valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><code>TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX</code></td><td valign="top">The regular expression that identifies the fields that can be collapsed into a single field.<br><br>By default, this value is empty.</td></tr><tr><td valign="top"><code>TONIC_DOCUMENT_COLLAPSE_FIELDS_REGEX_THRESHOLD</code></td><td valign="top">The number of fields that match the regular expression before Structural collapses the fields into a single field.<br><br>For example, if this is 5, then after Structural finds 5 fields that match the regular expression, it collapses all of the matching fields into a single field.<br><br>A negative value indicates to not collapse the fields.<br><br>The default value is -1.</td></tr></tbody></table>

For example:

* To collapse keys that are integer values, use the regular expression `[0-9]+` or `\d+`
* To collapse keys that are UUIDs, use the regular expression `[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}`

## Viewing the most recent scans for each collection <a href="#collection-scan-list" id="collection-scan-list"></a>

On **Privacy Hub**, the **Latest Collection Scan** table shows the most recent scans on each scanned collection.

The **Build Schema** option runs a new scan on the collection.

<figure><img src="/files/LTLFNmgq4yWkQezjvBd2" alt=""><figcaption><p>Latest Collection Scan on Privacy Hub</p></figcaption></figure>

## Starting a collection scan <a href="#collection-scan-start" id="collection-scan-start"></a>

When the source database has a new collection, then on **Collection View**, you are prompted to run a scan either on that collection or on all collections.

<figure><img src="/files/6tIoZ3R5c0IMTu3M30NV" alt=""><figcaption><p>Collection scan prompt on Collection View</p></figcaption></figure>

If a collection scan fails, you can [ask for AI assistance to troubleshoot the failure](/app/workspace/jobs#using-ai-to-troubleshoot-failed-jobs).


# Using Collection View

For [MongoDB](/app/setting-up-your-database/mongodb), **Collection View** replaces **Database View** and **Table View**. From **Collection View**, you can view the fields in a selected collection. You can then assign a collection mode to the collection, and assign generators to fields.

<figure><img src="/files/irM5iKkIeeGDGgTgoJqF" alt=""><figcaption><p>Collection View for a MongoDB workspace</p></figcaption></figure>

## Selecting the collection to view

From the **Collection** dropdown list, select the collection to view.

You can also use the Structural Agent to navigate between views. For example, `Display the companies collection`.

## Assigning a collection mode to the collection <a href="#mongodb-collection-view-collection-mode" id="mongodb-collection-view-collection-mode"></a>

Collection mode is the term used for table mode. The collection mode determines at the collection level how Structural uses the collection data to generate the destination database.

### Available collection modes <a href="#collection-modes-available" id="collection-modes-available"></a>

By default, the collection mode is De-Identify. In this mode, Structural uses the assigned generators to transform the source database into the destination database.

For [MongoDB](/app/setting-up-your-database/mongodb), the only other options are Truncate and Preserve Destination.

* Truncate means that only the collection structure is included in the destination database. The collection has no data in the destination database.
* Preserve Destination means that Tonic does not change the data that is currently in the destination database.

### Assigning the collection mode <a href="#collection-mode-assign" id="collection-mode-assign"></a>

{% hint style="info" %}
**Required workspace permission:** Assign table modes
{% endhint %}

To assign the collection mode:

1. Click the **Collection Mode** dropdown list.
2. On the panel, click the current collection mode.
3. From the drop-down list, select the mode to use.

## Selecting the type of view <a href="#mongodb-collection-view-select-view-type" id="mongodb-collection-view-select-view-type"></a>

You can view a collection either as a hybrid document or as single documents. From the **View** dropdown list, select the view to use.

### Hybrid document view <a href="#collection-view-hybrid" id="collection-view-hybrid"></a>

The default view is **Hybrid Document**. For the hybrid document view, the key list reflects all of the permutations of every field from every document. For example, a field might sometimes be a datetime value and sometimes a string. Hybrid document view lists both types.

<figure><img src="/files/irM5iKkIeeGDGgTgoJqF" alt=""><figcaption><p>Hybrid Document view of Collection View</p></figcaption></figure>

### Single document view <a href="#collection-view-single-document" id="collection-view-single-document"></a>

**Single Document** view displays a single document at a time. You can then page through up to 100 documents. **Single Document** view displays the structure for each document.

<figure><img src="/files/RkMmfXWx4BIQprJw19Ql" alt=""><figcaption><p>Single Document view of Collection View</p></figcaption></figure>

## Information on the field list <a href="#mongodb-collection-view-field-list-information" id="mongodb-collection-view-field-list-information"></a>

For each field, **Collection View** always displays:

* The field name and type.
* For fields that you configured as primary or foreign keys, a key icon.
* The assigned generator.
* An example value. For the hybrid view, you can use the magnifying glass icon to display additional example values.

For the hybrid document view, there is also a **Field Freq** column. **Field Freq** shows the percentage of documents that contain that permutation of field and type.

For example, a field might be Null 33% of the time and contain a numeric value 67% of the time. Or a field value might be an Int32 value 3% of the time and an Int64 value 6% of the time. The percentages apply to the first 100 documents.

## Toggling between source and preview data <a href="#mongodb-collection-view-toggle-preview" id="mongodb-collection-view-toggle-preview"></a>

{% hint style="info" %}
**Required workspace permission:**

* **Source data:** Preview source data
* **Destination data:** Preview destination data
  {% endhint %}

The **Preview** toggle at the top right of **Collection View** allows you to choose whether to display original source data or the transformed data. You can switch back and forth to determine exactly how Tonic Structural transforms the data based on the collection and field configuration.

By default, the **Preview** toggle is in the on position, and the displayed data reflects the selected collection mode and the assigned generators. For collections that use Truncate mode, the preview data is empty. Truncated collections do not have data in the destination database.

To display the original source data, toggle **Preview** to the off position.

## Filtering collection fields <a href="#collection-view-filter-fields" id="collection-view-filter-fields"></a>

In the single document view, you can filter the fields by either the field name or the field value.

In the hybrid document view, you can filter the fields based on either the field name or field properties.

### Filtering single document view by field name or value <a href="#collection-view-filter-single" id="collection-view-filter-single"></a>

You can filter single document view to only display fields that have specific text in either the field name or the field value.

To filter by value, toggle **Search by Value** to the on position.

<figure><img src="/files/Z4Bm1kARp0pM0hfxhFCg" alt=""><figcaption><p>Filter field and Search by Value toggle for single document view</p></figcaption></figure>

After you select the filter type, in the search field, type text that is in the field name or value. As you type, Structural filters the list to only include fields that contain the filter text.

### Filtering hybrid view by field name <a href="#collection-view-filter-hybrid-field-name" id="collection-view-filter-hybrid-field-name"></a>

To filter hybrid view by field name, in the search field, begin to type text that is in the field name. As you type, Structural filters the list to only include fields with names that include the filter text.

<figure><img src="/files/SEUZckU3vcHgNCcoO5cr" alt=""><figcaption><p>Filter field and Filters button for hybrid view</p></figcaption></figure>

### Filtering hybrid view by field properties <a href="#collection-view-filter-hybrid-field-props" id="collection-view-filter-hybrid-field-props"></a>

From the hybrid document view, you can filter the fields based on field properties.

To display the **Filters** panel, click **Filters**.

<figure><img src="/files/tpJrvMzWyJlzlMxWNFs4" alt=""><figcaption><p>Filters panel for hybrid view on Collection View</p></figcaption></figure>

#### **Searching for a filter** <a href="#collection-view-filter-hybrid-search" id="collection-view-filter-hybrid-search"></a>

To search for a filter or a filter value, in the search field, start to type the value. The search looks for text in the individual settings.

#### **Adding a filter** <a href="#collection-view-filter-hybrid-add" id="collection-view-filter-hybrid-add"></a>

To add a filter, depending on the filter type, either check the checkbox or select a filter option. As you add filters, Structural applies them to the field list.

Above the list, Structural displays tags for the selected filters.

#### **Clearing the selected filters** <a href="#collection-view-filter-hybrid-clear" id="collection-view-filter-hybrid-clear"></a>

To clear all of the currently selected filters, click **Clear All**.

## Filters panel filters <a href="#collection-view-filters-panel-filters" id="collection-view-filters-panel-filters"></a>

The **Filters** panel in hybrid view includes the following fields.

### At-risk fields <a href="#collection-view-filter-panel-at-risk" id="collection-view-filter-panel-at-risk"></a>

An at-risk field:

* Is marked as sensitive
* Is assigned the Passthrough generator.

To only display at-risk fields, on the **Filters** panel, check **At-Risk Field**.

When you check **At-Risk Field**, Structural adds the following filters under **Privacy Settings**:

* Sets the sensitivity filter to **Sensitive**.
* Sets the protection status filter to **Not protected**.

### Sensitivity <a href="#collection-view-filter-panel-sensitivity" id="collection-view-filter-panel-sensitivity"></a>

You can filter the fields based on the field sensitivity.

On the **Filters** panel, under **Privacy Settings**, the sensitivity filter is by default set to **All**, which indicates to display both sensitive and non-sensitive fields.

* To only display sensitive fields, click **Sensitive**.
* To only display non-sensitive fields, click **Not sensitive**.

Note that when you check **At-risk Field**, Structural automatically selects **Sensitive**.

### Protection status <a href="#collection-view-filter-panel-protection" id="collection-view-filter-panel-protection"></a>

You can filter the fields based on whether they have any generator other than Passthrough assigned.

On the **Filters** panel, under **Privacy Settings**, the field protection filter is by default set to **All**, which indicates to display both protected and not protected fields.

* To only display fields that have an assigned generator, click **Protected**.
* To only display fields that do not have an assigned generator, click **Not protected**.

Note that when you check **At-Risk Field**, Structural automatically selects **Not protected**.

### Recommended generators <a href="#collection-view-filter-panel-recommended-generators" id="collection-view-filter-panel-recommended-generators"></a>

When Structural detects that a field is sensitive, it can also determine a recommended generator.

For example, when it detects a name value, it also recommends the Name generator.

You can filter the fields to display the fields that have recommended generators.

On the **Filters** panel, under **Recommended Generators**, check the checkbox next to the recommended generator for which to display the fields that have that recommendation.

### Field data type <a href="#collection-view-filter-panel-data-type" id="collection-view-filter-panel-data-type"></a>

You can filter the fields by the field data type. For example, you might only display columns that contain either numeric or integer values.

To only display fields that have specific data types, on the **Filters** panel, under **Database Data Types**, check the checkbox for each data type to include.

The list of data types only includes data types that are present in the currently displayed fields and that are compatible with other applied filters.

To search for a specific data type, in the **Filters** search field, begin to type the data type.

### Unresolved schema changes <a href="#collection-view-filter-panel-schema-changes" id="collection-view-filter-panel-schema-changes"></a>

When the source database schema changes, you might need to update the configuration to reflect those changes. If you do not resolve the schema changes, then the data generation might fail. The data generation fails if there are unresolved conflicting changes, or if you configure Structural to always fail data generation when there are any unresolved changes.

For more information about schema changes, go to [Viewing and resolving schema changes](/app/generation/schema-changes).

To only display fields that have unresolved schema changes, on the **Filters** panel, check **Unresolved Schema Changes**.

### Sensitivity type <a href="#collection-view-filter-panel-sensitivity-type" id="collection-view-filter-panel-sensitivity-type"></a>

For detected sensitive fields, the sensitivity type indicates the type of data that was detected. Examples of sensitivity types include First Name, Address, and Email.

To only display fields that contain specific sensitivity types, on the **Filters** panel, under **Sensitivity Type**, check the checkbox for each sensitivity type to include.

The list of sensitivity types only includes sensitivity types that are present in the currently displayed fields.

To search for a specific sensitivity type, in the **Filters** search field, type the sensitivity type.

### Sensitivity confidence

When the Structural sensitivity scan identifies a value as belonging to a sensitivity type, it also determines how confident it is in that determination.

You can filter the columns based on the confidence level.

To only display columns that have a specific confidence level, on the **Filters** panel, under **Sensitivity confidence**, check the checkbox next to each confidence level to include.

### Primary or foreign keys <a href="#collection-view-filter-panel-keys" id="collection-view-filter-panel-keys"></a>

You can filter the column list to indicate whether to include:

* Columns that are not primary or foreign keys.
* Columns that are foreign keys.
* Columns that are primary keys.

On the **Filters** panel, under **Field Type**:

* To display fields that are neither a primary key nor a foreign key, check **Non-keyed**.
* To display fields that are primary keys, check **Primary key**.
* To display fields that are foreign keys, check **Foreign key**.

## Commenting on fields <a href="#mongodb-collection-view-field-comments" id="mongodb-collection-view-field-comments"></a>

{% hint style="info" %}
**Required license:** Professional or Enterprise
{% endhint %}

You can add comments to fields. For example, you might use a comment to explain why you selected a particular generator or marked a field as sensitive or not sensitive.

### Adding a new comment <a href="#collection-view-comment-new" id="collection-view-comment-new"></a>

If a field does not have any comments, then to add a comment:

1. Click the comment icon.
2. In the comment field, type the comment text.
3. Click **Comment**.

### Replying to an existing comment <a href="#collection-view-comment-reply" id="collection-view-comment-reply"></a>

When a field has existing comments, the comment icon is green. To add comments:

1. Click the comment icon.\
   \
   The comments panel shows the previous comments. Each comment includes the comment user and timestamp.
2. In the comment field, type the comment text.
3. Click **Reply**.

## Indicating whether a field is sensitive <a href="#mongodb-collection-view-field-sensitivity" id="mongodb-collection-view-field-sensitivity"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column sensitivity
{% endhint %}

On the field configuration panel, the sensitivity toggle at the top right indicates whether the field is marked as sensitive.

To mark a field as sensitive, toggle the setting to the **Sensitive** position.

To mark a field as not sensitive, toggle the setting to the **Not Sensitive** position.

You can also use the Structural Agent to set sensitivity. For example, `Mark the occupation field as not sensitive`.

## Assigning a generator to a field and type <a href="#mongodb-collection-view-assign-generator" id="mongodb-collection-view-assign-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

You can assign a generator to each combination of field and type. For example, depending on the document, the data type for a field might be either string or integer. You can indicate to use the Character Scramble generator when the field type is a string and the Random Integer generator when the field type is integer.

In hybrid document view, the Null type reflects when the column value is Null. You do not assign a generator to it.

To assign a generator:

1. Click the generator value for the field.
2. On the configuration panel, from the **Generator Type** dropdown list, select the generator.
3. Configure the generator options. For details about the available configuration options for each generator, go to the [Generator reference](/app/generation/generators/generator-reference).

You can also use the Structural Agent to apply generators to fields. For example:

* `Apply the recommended generators to all name fields.`
* `Apply the Timestamp Shift generator to the renewal_date field.`

## Assigning generators to fields that match JSONPath expressions <a href="#collection-view-path-expressions" id="collection-view-path-expressions"></a>

In addition to assigning generators to individual fields, you can assign generators to generic paths. The paths use JSONPath syntax.

For more information, go to [Assigning generators to path expressions](/app/generation/working-with-document-based-data/document-path-expressions).

## Disabling examples for sparse collections <a href="#collection-config-disable-sparse-examples" id="collection-config-disable-sparse-examples"></a>

By default, Structural retrieves 100 documents. It then uses the data in these documents to populate example values in the hybrid document.

For sparsely populated collections, where less common fields are not present in those 100 documents, Structural retrieves extra documents until it has example values for all fields. For very sparsely populated collections, this might cause the collection view to load slowly, because it must retrieve many documents.

To disable examples for sparse collections, set the [environment setting](/app/admin/environment-variables-setting) `TONIC_MONGO_DISABLE_EXTRA_EXAMPLES` to `true`. You can add this setting manually to the **Environment Settings** list on **Structural Settings**.

When this setting is `true`, fields that do not have a retrieved value use a dummy default value that is based on the data type.


# Using Document View for JSON columns

{% hint style="info" %}
Only supported for the file connector and PostgreSQL.

Note that for PostgreSQL, **Document View** cannot be used when the entire JSON document is an array. The JSON can contain arrays, but the document itself cannot be an array.
{% endhint %}

For columns that contain JSON content, you can use the JSON Mask generator to assign generators to individual JSON fields. To identify the fields, you use JSONPath expressions.

Another option is to use **Document View**, which allows you to view the structure of the JSON content and then assign generators to individual JSON fields.

You can also view this [video overview of **Document View**](https://www.youtube.com/watch?v=XCMezJ-Kst4).

## Enabling Document View for a JSON column <a href="#column-enable-document-view" id="column-enable-document-view"></a>

For a JSON column, the **Document View** option is available from **Privacy Hub**, **Database View**, and **Table View**.

On the column configuration panel, to enable **Document View**, toggle **Use Document View** to the on position.

You can also use the Structural Agent to enable Document View. For example, `Enable document view for the customer_summary column`.

When you enable **Document View**:

* The generator dropdown changes to an **Open in Document View** button.
* If this is the first column that you enabled **Document View** for, then the **Document View** tab becomes visible.
* Any existing generator assignment is discarded.
* On **Privacy Hub**, in the protection status display, each JSON path is displayed as a separate column. In the **Database Tables** list, each JSON path is a separate entry.

Structural also runs a scan on the column to detect the JSON structure and identify sensitive fields.

## Displaying Document View <a href="#document-view-display" id="document-view-display"></a>

On workspace management view, you use **Document View** to view the JSON structure.

**Document View** is only available when it is enabled for at least one JSON column.

## Selecting the JSON column to configure <a href="#document-view-select-column" id="document-view-select-column"></a>

From the **Column** dropdown list, select the JSON column to configure. The dropdown contains the columns that have **Document View** enabled.

<figure><img src="/files/6FA0Rn2XbE26I1NJpeHO" alt=""><figcaption><p>Column dropdown list on Document View</p></figcaption></figure>

## Selecting the type of view <a href="#document-view-select-view" id="document-view-select-view"></a>

From the **View** dropdown list, select the view to use for the selected column.

<figure><img src="/files/8fw6C6tI9AS54F8MXuqF" alt=""><figcaption><p>View dropdown on Document View</p></figcaption></figure>

### Hybrid view

Hybrid view provides a consolidated view of the schema across all of the rows.

<figure><img src="/files/kGmmcLsv8jO6wlUKk5b2" alt=""><figcaption><p>Hybrid view of Document view</p></figcaption></figure>

For example, for an array, hybrid view contains a single entry with all of the possible fields.&#x20;

### Single view

Single view shows the structure for one row at a time. You can then page through up to 100 rows. For each row, Structural displays the row structure.

<figure><img src="/files/8E1bVloMJ9kMBAQfCkLW" alt=""><figcaption><p>Single view of Document view</p></figcaption></figure>

For example, for an array, single view shows the actual array entries for each record.

## Information in the field list <a href="#document-view-field-list-info" id="document-view-field-list-info"></a>

For each JSON field, **Document View** always displays:

* The field name and data type.
* The assigned generator.
* An example value. In hybrid view, you can use the magnifying glass icon to display additional example values.

Hybrid view also displays a **Field Freq** column. **Field Freq** shows the percentage of rows that contain that permutation of field and type. For example, a field might be Null 33% of the time and contain a numeric value 67% of the time. Or a field value might be an Int32 value 3% of the time and an Int64 value 6% of the time. The percentages apply to the first 100 rows.

## Toggling between source and preview data <a href="#document-view-source-preview-toggle" id="document-view-source-preview-toggle"></a>

{% hint style="info" %}

**Required workspace permission:**

* **Source data:** Preview source data
* **Destination data:** Preview destination data
  {% endhint %}

The **Preview** toggle at the top right of **Document View** allows you to choose whether to display original source data or the transformed data. You can switch back and forth to determine exactly how Tonic Structural transforms the data based on the field configuration.

By default, the **Preview** toggle is in the on position, and the displayed data reflects the assigned generators.

To display the original source data, toggle **Preview** to the off position.

## Filtering Document View fields <a href="#document-view-filter-fields" id="document-view-filter-fields"></a>

In single view, you can filter by either a field name or a field value.

In hybrid view, you can filter by either field name or field properties.

### **Filtering single document view by field name or value** <a href="#filter-single-name-value" id="filter-single-name-value"></a>

You can filter single view to only display fields that have specific text in either the field name or the field value.

To filter by value, toggle **Search by Value** to the on position.

After you select the filter type, in the search field, type text that is in the field name or value. As you type, Structural filters the list to only include fields that contain the filter text.

<figure><img src="/files/OepYEMg1KcuePQLOe64b" alt=""><figcaption><p>Searching single view by field value</p></figcaption></figure>

### **Filtering hybrid view by field name** <a href="#filter-hybrid-name" id="filter-hybrid-name"></a>

To filter hybrid view by field name, in the search field, begin typing text that is in the field name. As you type, Structural filters the list to only include fields with names that include the filter text.

<figure><img src="/files/Siq9HS9q8EyS4UNfYcAS" alt=""><figcaption><p>Searching hybrid view by field name</p></figcaption></figure>

### **Filtering hybrid view by field properties** <a href="#filter-hybrid-field-properties" id="filter-hybrid-field-properties"></a>

From the hybrid document view, you can filter the fields based on field properties.

To display the **Filters** panel, click **Filters**.

<figure><img src="/files/xx6Bcr9yoUk2we0egT5K" alt=""><figcaption><p>Filters panel on Document View</p></figcaption></figure>

#### **Searching for a filter** <a href="#filter-properties-search" id="filter-properties-search"></a>

To search for a filter or a filter value, in the search field, start to type the value. The search looks for text in the individual settings.

<figure><img src="/files/E94hbQdERtWf4mLo3Z5r" alt=""><figcaption><p>Filter search for Document View</p></figcaption></figure>

#### **Adding a filter** <a href="#filter-properties-add" id="filter-properties-add"></a>

To add a filter, depending on the filter type, either check the checkbox or select a filter option. As you add filters, Structural applies them to the field list.

Above the list, Structural displays tags for the selected filters.

<figure><img src="/files/B2Njnq196COW8jY0Jjho" alt=""><figcaption><p>Document View with applied filters</p></figcaption></figure>

#### **Clearing the selected filters** <a href="#filter-properties-clear" id="filter-properties-clear"></a>

To clear all of the currently selected filters, click **Clear All**.

## Filters panel filters

The **Filters** panel in hybrid view includes the following options.

### **At-risk JSON fields**

An at-risk JSON field:

* Is marked as sensitive
* Is assigned the Passthrough generator.

To only display at-risk JSON fields, on the **Filters** panel, check **At-Risk Field**.

When you check **At-Risk Field**, Structural adds the following filters under **Privacy Settings**:

* Sets the sensitivity filter to **Sensitive**.
* Sets the protection status filter to **Not protected**.

### **Sensitivity**

You can filter the JSON fields based on the field sensitivity.

On the **Filters** panel, under **Privacy Settings**, the sensitivity filter is by default set to **All**, which indicates to display both sensitive and non-sensitive JSON fields.

* To only display sensitive JSON fields, click **Sensitive**.
* To only display non-sensitive JSON fields, click **Not sensitive**.

Note that when you check **At-risk Field**, Structural automatically selects **Sensitive**.

### **Protection status**

You can filter the JSON fields based on whether they have any generator other than Passthrough assigned.

On the **Filters** panel, under **Privacy Settings**, the field protection filter is by default set to **All**, which indicates to display both protected and not protected JSON fields.

* To only display JSON fields that have an assigned generator, click **Protected**.
* To only display JSON fields that do not have an assigned generator, click **Not protected**.

Note that when you check **At-Risk Field**, Structural automatically selects **Not protected**.

### **Recommended generators**

When Structural detects that a JSON field is sensitive, it can also determine a recommended generator.

For example, when it detects a name value, it also recommends the Name generator.

You can filter the fields to display the fields that have recommended generators.

On the **Filters** panel, under **Recommended Generators**, check the checkbox next to the recommended generator for which to display the fields that have that recommendation.

### **Field data type**

You can filter the fields by the field data type. For example, you might only display columns that contain either numeric or integer values.

To only display fields that have specific data types, on the **Filters** panel, under **Database Data Types**, check the checkbox for each data type to include.

The list of data types only includes data types that are present in the currently displayed fields and that are compatible with other applied filters.

To search for a specific data type, in the **Filters** search field, begin to type the data type.

### **Unresolved schema changes**

When the structure of the JSON changes, you might need to update the configuration to reflect those changes. If you do not resolve the changes, then the data generation might fail.

To only display fields that have unresolved changes to the JSON structure, on the **Filters** panel, check **Unresolved Schema Changes**.

### **Sensitivity type**

For detected sensitive fields, the sensitivity type indicates the type of data that was detected. Examples of sensitivity types include First Name, Address, and Email.

To only display fields that contain specific sensitivity types, on the **Filters** panel, under **Sensitivity Type**, check the checkbox for each sensitivity type to include.

The list of sensitivity types only includes sensitivity types that are present in the currently displayed fields.

To search for a specific sensitivity type, in the **Filters** search field, type the sensitivity type.

### **Sensitivity confidence**

When the document scan identifies a value as belonging to a sensitivity type, it also determines how confident it is in that determination.

You can filter the columns based on the confidence level.

To only display columns that have a specific confidence level, on the **Filters** panel, under **Sensitivity confidence**, check the checkbox next to each confidence level to include.

## Indicating whether a JSON field is sensitive <a href="#json-field-sensitivity" id="json-field-sensitivity"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column sensitivity
{% endhint %}

On the field configuration panel, the sensitivity toggle at the top right indicates whether the field is marked as sensitive.

<figure><img src="/files/M7p5Dq5m3fWMHLEu95MU" alt=""><figcaption><p>Field configuration panel on Document View</p></figcaption></figure>

To mark a field as sensitive, toggle the setting to the **Sensitive** position.

To mark a field as not sensitive, toggle the setting to the **Not Sensitive** position.

You can also use the Structural Agent to set sensitivity. For example, `Mark the occupation field as not sensitive.`

## Assigning a generator to a JSON field <a href="#json-field-generator" id="json-field-generator"></a>

{% hint style="info" %}
**Required workspace permission:** Configure column generators
{% endhint %}

For each node, you assign a generator.

To assign a generator:

1. Click the generator value for the JSON field.
2. On the configuration panel, from the **Generator Type** dropdown list, select the generator.\
   \
   Other than the Conditional and Regex Mask generators, you cannot assign a composite generator to a JSON field.
3. Configure the generator options. For details about the available configuration options for each generator, go to the [generator reference](/app/generation/generators/generator-reference).

When you configure a generator in **Document View**:

* You can only link to other JSON fields.
* You can only enable self-consistency.

You can also use the Structural Agent to apply generators to fields. For example:

* `Apply the recommended generators to all name fields.`
* `Apply the Timestamp Shift generator to the renewal_date field.`

## Assigning generators to fields that match JSONPath expressions <a href="#document-view-path-expression" id="document-view-path-expression"></a>

In addition to assigning generators to individual fields, you can assign generators to generic paths. The paths use JSONPath syntax.

For more information, go to [Assigning generators to path expressions](/app/generation/working-with-document-based-data/document-path-expressions).


# Assigning generators to path expressions

On **Collection View** and **Document View**, Structural can automatically assign generators to fields that match a configured path expression. Each collection or JSON column has its own set of path generators.

Structural always applies the path generator to matching fields that do not have an assigned generator (are set to Passthrough).

Structural does not apply the path generator to matching fields that have a generator configuration applied directly.

However, in a child workspace, Structural does apply the path generator to matching fields that inherit their current generator configuration from the parent workspace.

## Displaying the list of path generators

To display the list of path generators for the current collection or JSON column, click **Path Generators**.

For each path generator, the list includes:

<figure><img src="/files/BJBwXTjVImxg2B062s8t" alt=""><figcaption><p>Path generators list</p></figcaption></figure>

* The priority order. Structural checks the fields against the paths in the order that the paths are displayed. The first matching path wins.
* The path expression to identify matching fields.
* The data type filter for matching fields. You can configure a path generator to only apply to fields of a specific type or types.
* The name of the generator preset that Structural applies to matching fields.

## Creating a path generator

To create a path generator, you can either create a completely new path generator, or start from a duplicate of an existing path generator.

Structural saves the new generator automatically when the configuration is complete. The **Saved** button at the bottom right indicates when the generator is saved.

<figure><img src="/files/kkGbIFKMfEAJi6yfl3io" alt=""><figcaption></figcaption></figure>

### **Creating a completely new path generator**

To create a path generator:

1. On the path generators panel, click **Add path generator**.
2. On the path generator details panel, in the **Path Expression** field, provide the path expression to use to identify matching paths.\
   \
   The path expression uses the [JSONPath](https://goessner.net/articles/JsonPath/index.html#e2) syntax. Note that for a path generator, you cannot use the expression to check for a field value. For more information about the supported operators and some examples, go to [#path-gen-jsonpath-examples](#path-gen-jsonpath-examples "mention").\
   \
   When you provide a path expression, the matching fields list displays the fields that match the expression.
3. You can optionally filter the matching fields based on the data type. For example, you might only want to apply a generator to text or integer fields.\
   \
   By default, the data type filter list is empty.\
   \
   The available data types are general types that map to specific data types in a given database.\
   \
   Under **Data Types**, to add a data type to the filter, select it from the dropdown list.\
   \
   To remove a data type, click its delete icon.\
   \
   When you configure data type filters, the matching fields list is updated to only include fields that have one of the specified data types.
4. From the generator dropdown list, select the generator to apply to matching fields.\
   \
   The available generators are affected by the data type filter. When the data type filter is empty, you can only select from generators that can be used for any type of column.\
   \
   When you specify a list of data types, you can only select from generators that can be used for all of those data types.
5. Configure the selected generator.\
   \
   For a generator assigned to a path expression:
   * Linking is not supported.
   * Consistency with other columns is not supported.

### **Copying an existing path generator**

You can create a path generator based on an existing one. For example, for the same path expression, you might want to assign a different generator based on the data type.

To create a new path generator based on an existing path generator:

1. On the path generators list, click the options menu for the path generator to copy.

<figure><img src="/files/W3WqQ1Y0nc1t1vzMlFwG" alt=""><figcaption><p>Options menu for a path generator</p></figcaption></figure>

2. Click **Duplicate path generator**.
3. On the path generator details panel, edit the configuration.

## Updating a path generator

To update a path generator:

1. On the path generators list, click the options menu for the path generator.
2. Click **Edit path generator**.
3. On the path generator details panel, edit the configuration.

Structural saves the changes automatically.

For fields that were assigned a generator based on the previous configuration, but that do not match the updated path generator configuration:

* If the field matches other path generators, then the next matching configuration is applied.
* If the field does not match any other path generators, then the field reverts to Passthrough.

## Deleting a path generator

When you delete a path generator, the generator assignment is removed from the matching fields. If a field matched more than one path generator, then the next match is used.

To delete a path configuration:

1. On the path generators list, click the options menu for the path configuration.
2. Click **Delete path generator**.

For fields that were assigned a generator based on the path generator:

* If the field matches other path generators, then the next matching path generator is applied.
* If the field does not match any other path generators, then the field reverts to Passthrough.

## Supported JSONPath operators and examples <a href="#path-gen-jsonpath-examples" id="path-gen-jsonpath-examples"></a>

For a path generator path expression, Structural supports the following operators:

* `$` - Root
* `.` - Child operator
* `..` - Recursive descent operator
* `*` - Wildcard operator
* `[*]` - Array operator. Note that a path generator must always target all of the items in an array. Any use of the array operator must include the wildcard operator.

For example, a document includes an array of objects. Each object contains name, address, and email address fields.

```json
{
  [
    {
      "name": "John Smith",
      "address": "1 Main Street",
      "email_address": "jsmith@example.com"
    },
    {
      "name": "Mary Jones",
      "address": "5 Elm Avenue",
      "email_address": "mjones@example.com"
    },
  ]
}
```

You can configure a path generator that assigns a generator to the address field in all of the array objects. You cannot only assign a generator to the address field in one of the array objects.

Here are some example path expressions, based on the following JSON:

```json
{
  "bookstore_name": "Read & Brew Books",
  "mailing_address": {
    "street": "123 Literary Lane",
    "city": "Bookville",
    "state": "MA",
    "zip_code": "02451",
    "country": "USA"
  },
  "phone_number": "555-123-4567",
  "books": [
    {
      "title": "The Great Gatsby",
      "author": "F. Scott Fitzgerald",
      "isbn": "978-0743273565",
      "publication_year": 1925,
      "country": "USA"
    },
    {
      "title": "Moby Dick",
      "author": "Herman Melville",
      "isbn": "978-1503280786",
      "publication_year": 1851,
      "country": "USA"
    },
    {
      "title": "To the Lighthouse",
      "author": "Virginia Woolf",
      "isbn": "978-0156907392",
      "publication_year": 1927,
      "country": "UK"
    },
    {
      "title": "The Catcher in the Rye",
      "author": "J.D. Salinger",
      "isbn": "978-0316769174",
      "publication_year": 1951,
      "country": "USA"
    }
  ]
}
```

| `$.bookstore_name`           | Find the `bookstore_name` field at the top level of the JSON.       |
| ---------------------------- | ------------------------------------------------------------------- |
| `$.mailing_address.zip_code` | Find the `zip_code` field in the mailing address.                   |
| `$.books[*].isbn`            | Find the <kbd>isbn</kbd> field in each entry in the array of books. |
| `$..country`                 | Find all `country` fields in the JSON.                              |

## Determining the priority order for the path generators

When Structural looks for matching fields, it checks the path generators in the order that they are displayed on the **Path Generators** panel.

For each field, it uses the first matching configuration.

To change the order of the path generators, drag and drop each configuration to the appropriate location in the list.

## Identifying matching fields on Collection View and Document View

On **Collection View** and **Document View**, when the assigned generator comes from a path generator, the generator assignment is marked with an icon.

<figure><img src="/files/SjVac5CLZWzULhQdIkmC" alt=""><figcaption><p>Field with a generator assigned by a path generator</p></figcaption></figure>

When you click the generator, the configuration panel indicates that the generator is assigned based on a path generator.

<figure><img src="/files/vxPch5ZL1qc0xxH0GftQ" alt=""><figcaption><p>Configuration panel for a field that matches a path generator</p></figcaption></figure>

## Overriding the path generator assignment

When you change the configuration for the field, the icon is removed. The override tooltip indicates that the path generator was overridden.

<figure><img src="/files/BBkDS2qcgyYtPZSohAzm" alt=""><figcaption><p>Field where the path generator is overridden by a different generator configuration</p></figcaption></figure>

If you set the generator to Passthrough, then the field reverts to the path generator.


# Identifying sensitive data

Tonic Structural uses its sensitivity scan to identify source data columns that contain sensitive information. The scan ignores truncated tables.

The sensitivity scan identifies Structural's built-in sensitivity types. It also looks for custom types that you define.

You can also manually mark a column as sensitive or not sensitive.

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Run the Structural sensitivity scan</strong></td><td>Run, configure, and get the results of the sensitivity scan.</td><td></td><td><a href="/pages/VUx4E8FZwKWHyDx6JJAC">/pages/VUx4E8FZwKWHyDx6JJAC</a></td></tr><tr><td><strong>Set column sensitivity manually</strong></td><td>Options to override the results of the sensitivity scan.</td><td></td><td><a href="/pages/tiWfa6RLfCj6UdEmEtGi">/pages/tiWfa6RLfCj6UdEmEtGi</a></td></tr><tr><td><strong>Built-in sensitivity types</strong></td><td>Types of sensitive data that the sensitivity scan can identify.</td><td></td><td><a href="/pages/hJzR1ON0UB155TpJMRaS">/pages/hJzR1ON0UB155TpJMRaS</a></td></tr><tr><td><strong>Configure custom sensitivity rules</strong></td><td>Set up rules to enable the scan to identify other sensitive columns based on the column data types and names.</td><td></td><td><a href="/pages/QLbWXt57pqVFLN2hzOAb">/pages/QLbWXt57pqVFLN2hzOAb</a></td></tr></tbody></table>


# Running the Structural sensitivity scan

The Structural sensitivity scan identifies sensitive columns in source data. The scan ignores truncated tables.

## When sensitivity scans run <a href="#sensitivity-scan-about" id="sensitivity-scan-about"></a>

For most data connectors, Structural runs sensitivity scans automatically based on specific events. You can also run manual sensitivity scans on demand.

Sensitivity scans can also run automatically at the same time each week.

### Event-based sensitivity scans <a href="#sensitivity-scan-automatic" id="sensitivity-scan-automatic"></a>

{% hint style="info" %}
Structural does not run automatic sensitivity scans for Databricks and Spark SDK workspaces.

By default, Structural does not run automatic sensitivity scans for file connector workspaces. To enable automatic sensitivity scans for the file connector, set the [environment setting](/app/admin/environment-variables-setting) `TONIC_ENABLE_FILES_PRIVACY_SCAN_AUTORUN` to `true`.
{% endhint %}

Structural automatically runs a sensitivity scan when you:

* Create a completely new workspace and connect a data source.
* Change the data connection details for the source database.
* Add a file group to a file connector workspace.

A child workspace always inherits the sensitivity designations from its parent workspace.

When you copy a workspace, Structural runs a new sensitivity scan on the copy to identify sensitive columns. However, it keeps the sensitivity designation for columns that you specifically marked as sensitive or not sensitive.

To not automatically run event-based sensitivity scans, set the [environment setting](/app/admin/environment-variables-setting) `TONIC_DISABLE_PRIVACY_SCAN_AUTORUN` to `true`. You can set this setting from the **Environment Settings** list on **Structural Settings**. You can also [override the setting](/app/workspace/workspace-configuration-settings/advanced-overrides) in individual workspaces.

### Manual sensitivity scans <a href="#sensitivity-scan-manual" id="sensitivity-scan-manual"></a>

In addition to the automatic scans, from **Privacy Hub**, you can [start a sensitivity scan manually](/app/generation/privacy-hub#privacy-hub-run-sensitivity-scan).

### Weekly sensitivity scans <a href="#sensitivity-scan-schedule" id="sensitivity-scan-schedule"></a>

Structural can also runs scheduled weekly sensitivity scans in the background.

By default, Structural Cloud does not run scheduled scans.

#### Default schedule and coverage for weekly scans

By default, on self-hosted instances, Structural runs sensitivity scans each Sunday at midnight.

The weekly scans run only on the workspaces that had the most recent activity.

* For self-hosted instances, the weekly scans run on the 10 workspaces that had the most recent activity.
* On Structural Cloud, for each organization, the weekly scans run on the 2 workspaces that had the most recent activity.

Activity includes:

* User-initiated updates that are included in the **Version History**.
* Data generation jobs.

#### Configuring weekly scans (self-hosted)

On self-hosted instances, you can configure whether to run the weekly scans, and set the schedule.

To manage the scheduled scans, use the following [environment settings](/app/admin/environment-variables-setting). You can add these settings to the **Environment Settings** list on **Structural Settings**.

* `TONIC_ENABLE_SCHEDULED_SENSITIVITY_SCAN` - Boolean to indicate whether to enable the scheduled weekly sensitivity scans.\
  \
  The default value is `true`. To disable the scheduled weekly scan, set this to `false`.
* `TONIC_SENSITIVITY_SCAN_DAY` - When scheduled scans are enabled, the day of the week on which to run the scans.\
  \
  The value is an integer between 0 and 6, where 0 is Sunday and 6 is Saturday.\
  \
  The default value is 0.
* `TONIC_SENSITIVITY_SCAN_HOUR` - When scheduled scans are enabled, the hour at which to run the scans. The setting uses the local time zone.\
  \
  The value is an integer between 0 and 23, where 0 is midnight and 23 is 11:00 PM.\
  \
  For example, a value of 14 indicates to run the job at 2:00 PM.\
  \
  The default value is 0.
* `TONIC_PII_SCAN_MAX_TIMEOUT_IN_MINUTES_IF_AUTOMATIC` - The number of minutes after which a scheduled scan times out.\
  \
  By default, the scan times out after 3 minutes.\
  \
  You can override this setting in an individual workspace.

## Configuring parallel processing for sensitivity scans <a href="#sensitivity-scan-parallelism" id="sensitivity-scan-parallelism"></a>

For improved performance, sensitivity scans can use parallel processing.

For relational databases such as PostgreSQL and SQL Server, to configure parallel processing, you use the [environment setting](/app/admin/environment-variables-setting) `TONIC_PII_SCAN_PARALLELISM_RDBMS`. The default value is 4.

For document-based databases such as MongoDB, you use the environment setting `TONIC_PII_SCAN_PARALLELISM_DOCUMENTDB`. The default value is 1.

## How Structural identifies sensitive values <a href="#sensitive-data-how-identified" id="sensitive-data-how-identified"></a>

The Structural sensitivity scan uses the following rules and processes to:

* Identify sensitive columns.
* Recommend generators for those columns. For information about applying recommended generators to columns, go to [Reviewing and applying recommended generators](/app/generation/generators-assign-config/generators-review-apply-recommended).
* Indicate its confidence that an identified column is sensitive and is of the detected sensitivity type.

Note that this process cannot guarantee perfect precision and recall. We strongly recommend that a human reviews the sensitivity scan results and the broader dataset to ensure that nothing sensitive was missed.

### **Rule-based data type, column name, and value analysis - High, medium, or low confidence** <a href="#scan-process-rule-based" id="scan-process-rule-based"></a>

To identify that a column contains sensitive information for a [built-in sensitivity type](/app/generation/identify-sensitive-data/sensitivity-types-built-in), Structural looks at the data type, column name, and column values. For the values, Structural uses a random sampling of rows from the data.

This part of the sensitivity scan uses regular expression matching and dictionary lookups. It produces high, medium, or low confidence detections.

When this part of the sensitivity scan determines that a column contains sensitive data, it:

* Marks the column as sensitive
* Assigns the sensitivity type to the column
* Recommends the generator configuration for the identified sensitivity type. Note that if the recommended generator is not compatible with the column, then Structural discards the recommendation.
* Marks the sensitivity detection as high, medium, or low confidence. The confidence level is based on a calculation of how well the column matched the applicable rules.

### **Custom sensitivity rules - Full confidence** <a href="#scan-process-custom-rules" id="scan-process-custom-rules"></a>

The sensitivity scan also looks for any columns that match custom sensitivity types that you define in your custom sensitivity rules.

Custom sensitivity rules are based on the column data type and column name. For more information about custom sensitivity rules, go to [Creating and managing custom sensitivity rules](/app/generation/identify-sensitive-data/custom-sensitivity-rules).

Custom sensitivity rules always produce full confidence detections.

When a column matches a custom sensitivity rule, Structural:

* Marks the column as sensitive.
* Assigns the sensitivity rule name as the sensitivity type.
* Recommends the generator preset from the sensitivity rule.
* Marks the sensitivity detection as full confidence.

### **Model-based analysis - Medium and low confidence** <a href="#scan-process-ai-model" id="scan-process-ai-model"></a>

Note that if LLM-based sensitivity detection is enabled, then model-based analysis is not used.

To identify additional sensitive columns that might not be captured by the other parts of the scan, the sensitivity scan uses an artificial intelligence (AI) model. Note that the model is pre-trained. Structural does not use customer data to train the model, and it does not send any customer data externally.

This part of the scan produces medium or low confidence detections for built-in entity types.

The model considers the table and column name. If the combination of table and column name is similar in meaning to a sensitivity type that Structural has a recommended generator for, then Structural:

* Marks the column as sensitive.
* Assigns the sensitivity type to the column.
* Recommends the generator configuration for that sensitivity type.
* Uses AI to compare the table name and column name combination to the sensitivity type, and produces a semantic similarity score.
* Based on the semantic similarity score, marks the sensitivity detection as either medium or low confidence.

### LLM-based sensitivity detection - Medium confidence

The sensitivity scan can use LLM-based sensitivity detection instead of the model-based analysis.

The LLM-based scan uses the database schema and, optionally, contextual source data, to determine whether a column contains a sensitive value and, if so, the sensitivity type.

This part of the scan produces medium confidence detections for built-in entity types.

The LLM-based sensitivity scan also processes LLM-based sensitivity rules. If LLM-based sensitivity detection is not enabled, then any LLM-based sensitivity rules are not used.

For information about the general flow for LLM-based sensitivity detection, go to [Data flow and privacy in Structural AI](/app/admin/structural-ai-use/structural-ai-data-flow-privacy).

For information about how to configure the use of LLM-based sensitivity detection, go to [Structural Cloud LLM configuration](/app/admin/structural-ai-use/structural-cloud-llm-configuration) or [Self-hosted LLM configuration](/app/admin/structural-ai-use/self-hosted-llm-configuration).

## Downloading the sensitivity scan log <a href="#sensitivity-scan-download-log" id="sensitivity-scan-download-log"></a>

To download the log of the most recent sensitivity scan, either:

* On the workspace management view, from the download menu, select **Download Sensitivity Scan Log**.
* On **Privacy Hub**, click **Reports and Logs**, then select **Scan Log**.

The log tracks the progress of the scan.

If a sensitivity scan fails, you can [ask for AI assistance to troubleshoot the failure](/app/workspace/jobs#using-ai-to-troubleshoot-failed-jobs).


# Manually indicating whether a column is sensitive

You can also manually indicate that a column is sensitive or not sensitive.

For example, the sensitivity scan might incorrectly identify a column as sensitive. Or a column might contain data that you consider sensitive but that does not match a detected sensitivity type.

When you manually change a column from not sensitive to sensitive, Structural marks the sensitivity detection as full confidence.

To change the sensitivity, you can prompt the Structural Agent. For example:

* `Change all datetime columns to be not sensitive.`
* `Change the Occupation column to be sensitive.`

For information on other ways to change whether a column is sensitive:

* For **Privacy Hub**, go to [Privacy Hub](/app/generation/privacy-hub#privacy-hub-protection-status-flag-sensitive).
* For **Database View**, go to:
  * For a single column, [Configuring an individual column](/app/generation/database-view/database-view-configure-column#database-view-column-config-single-sensitivity)
  * For multiple selected columns, [Configuring multiple columns](/app/generation/database-view/database-view-configure-bulk#database-view-column-config-multi-sensitivity)
* For **Table View**, go to [Table View](/app/generation/table-view#table-view-configure-column-sensitivity).

The Structural API also provides [endpoints to designate columns as sensitive or not sensitive](/app/api/quick-start-guide/tonic-api-column-sensitivity).


# Built-in sensitivity types that Structural detects

Structural identifies the following types of sensitive values. These include some information types that are considered by many privacy standards and frameworks such as HIPAA, GDPR, CCPA, and PCI.

For more information about the HIPAA and Safe Harbor information types that Structural detects, go to the Tonic.ai guide [Using Tonic Structural and the Safe Harbor method to de-identify PHI](https://www.tonic.ai/guides/using-tonic-structural-and-the-safe-harbor-method-to-de-identify-phi).

**Names**

* First&#x20;
* Last&#x20;
* Full

**Organization**

**Location**

* Street address
* ZIP
* PO Box
* City
* State and two-letter abbreviation
* Country
* Postal code&#x20;
* GPS coordinates

**Contact information**

* Email address
* Telephone number

**User credentials**

* Username
* Password

**Financial information**

* Credit card number
* International bank account number (IBAN)
* SWIFT code for bank transfers
* Money amount
* BTC (Bitcoin) address

**Identification**

* Social Security Number
* Passport number
* Driver's license number
* Birth date
* Gender
* Biometric identifier, such as a fingerprint or voiceprint
* Full face photographic images and similar images

**Medical information**

* ICD-9 and ICD-10 codes (Used to identify diseases)
* Medical record number
* Health plan beneficiary number
* Admission date
* Discharge date
* Date of death

**Other personal information**

* Marital status

**Accounts and licenses**

* Account number
* Certificate or license number

**Network and web location**

* IP address
* IPv6 address
* MAC address
* Web URL

**International Mobile Equipment Identity (IMEI)**

**Vehicle information**

* Vehicle identification number (VIN)
* License plate number


# Creating and managing custom sensitivity rules

{% hint style="info" %}
**Required global permission:** Create and manage sensitivity rules
{% endhint %}

By default, when a Structural security scan runs on a workspace, it looks for the [built-in sensitivity types](/app/generation/identify-sensitive-data/sensitivity-types-built-in).

You can also define custom sensitivity rules to identify other values and the corresponding recommended generator. Your data might include values that are specific to your organization.

To identify the columns that the sensitivity rule applies to, you can either:

* Match text in the column name.
* Provide a description to tell the LLM what to look for. The LLM-based sensitivity scan then matches the description against the source data columns.

For both types of custom sensitivity rule, you identify the applicable column data type, and select the generator preset to apply to matching columns.

## Enabling LLM-based sensitivity rules

LLM-based sensitivity rules are processed as part of the LLM-based sensitivity detection. When LLM-based sensitivity detection is not used, then the LLM-based sensitivity rules also are not used.

On self-hosted instances, even if LLM-based sensitivity detection is enabled, an additional environment setting specifically enables LLM-based sensitivity rules.

For more information, go to [Structural Cloud LLM configuration](/app/admin/structural-ai-use/structural-cloud-llm-configuration) and [Self-hosted LLM configuration](/app/admin/structural-ai-use/self-hosted-llm-configuration).

## Displaying the list of custom sensitivity rules <a href="#sensitivity-rules-list" id="sensitivity-rules-list"></a>

To display the current list of sensitivity rules, in the Structural navigation menu, click **Sensitivity Rules**.

<figure><img src="/files/zq7FU7BMJQ7C2zTAS5PT" alt=""><figcaption><p>Sensitivity Rules view with the lists of custom sensitivity rules</p></figcaption></figure>

On the **Sensitivity Rules** page:

* The **Column Name rules** list contains the list of sensitivity rules that check for a text or regular expression match in the column name.
* If LLM-based sensitivity rules are enabled, then the **LLM-based rules** list is displayed and contains the list of sensitivity rules that use the rule description as input for the LLM-based security scan. The LLM-based scan uses the description to check for matching columns.

The lists contain sensitivity rules for a self-hosted Structural instance or a Structural Cloud organization.

For each rule, the list includes:

* The rule name and description
* The recommended generator preset
* When the rule was most recently modified

## Filtering the rules <a href="#sensitivity-rules-filter" id="sensitivity-rules-filter"></a>

You can filter each rule list by the following:

* Rule name
* Rule description
* Generator preset name
* Name of the user who most recently updated the rule

In the filter field, start to type text from any of those values. As you type, the list is filtered to only include matching rules.

Note that when the list is filtered, you cannot change the display sequence of the rules.

## Setting the rule sequence <a href="#sensitivity-rules-sequence" id="sensitivity-rules-sequence"></a>

For each type of rule, Structural applies the rules based on their display order in the list. If a column matches more than one rule, Structural applies the first matching rule.

Column name rules take precedence over LLM-based rules. When a column matches both a column name rule and an LLM-based rule, the column name rule is used.

To change the display order of a rule, drag and drop it to the new location in the list.

Note that you cannot change the rule sequence when the list is filtered.

## Creating and editing a sensitivity rule <a href="#sensitivity-rule-create-edit" id="sensitivity-rule-create-edit"></a>

### Creating a sensitivity rule <a href="#sensitivity-rule-create" id="sensitivity-rule-create"></a>

To create a sensitivity rule:

1. On the **Sensitivity Rules** view, click **New Custom Rule**.
2. On the **Create Custom Rule** view, [configure the new rule](#sensitivity-rule-config).
3. Click **Save**.

### Editing a sensitivity rule <a href="#sensitivity-rule-edit" id="sensitivity-rule-edit"></a>

To change the configuration of a sensitivity rule:

1. On the **Sensitivity Rules** view, click the edit icon for the rule.
2. On the **Edit Custom Rule** view, [update the configuration](#sensitivity-rule-config).
3. Click **Save**.

Note that any changes to a sensitivity rule do not take effect until the next sensitivity scan.

## Sensitivity rule configuration <a href="#sensitivity-rule-config" id="sensitivity-rule-config"></a>

<figure><img src="/files/yNt2oUCN0BBfgnYg7HYg" alt=""><figcaption><p>Details view for a custom sensitivity rule</p></figcaption></figure>

### Rule name <a href="#sensitivity-rule-name-description" id="sensitivity-rule-name-description"></a>

In the **Name** field, type the name of the sensitivity rule. The rule name becomes the sensitivity type for matching columns.

The rule name:

* Must be unique.
* Cannot match the name of a built-in sensitivity type.

### Rule description

For column name sensitivity rules, use the **Description** field to provide an optional longer description of the sensitivity rule and how it is used.

For LLM-based sensitivity rules, the content of the **Description** field is what Structural sends to the LLM during LLM-based sensitivity detection. When providing the description for the LLM, to ensure the most accurate matches, be as specific as possible.

Note that if Structural is configured to not send sample data to the LLM, and the description refers to the column value and not the column name, the LLM cannot identify matching columns.

### Rule type

When both column name and LLM-based sensitivity rules are enabled, then under **Match Type**, click the type of rule.

<figure><img src="/files/kiFc46LCDTMDg5qHvKrH" alt=""><figcaption><p>Match Type setting to determine the type of custom sensitivity rule</p></figcaption></figure>

* To create a column name rule, click **Column Name**.
* To create an LLM-based rule, click **Description (LLM)**.

After you save the rule, you cannot change the rule type.

### Data type <a href="#sensitivity-rule-data-type" id="sensitivity-rule-data-type"></a>

From the **Data Type** dropdown list, select the data type for matching columns. For example, a rule might only be used for columns that contain text.

The available data types are general types that map to specific data types in a given database. The available types are:

* Array
* Binary
* Boolean
* Continuous Numerical
* Date Range
* Datetime
* Integer
* JSON
* MAC Address
* Network Address
* Text
* UUID
* XML

### Column name criteria <a href="#sensitivity-rule-column-name-conditions" id="sensitivity-rule-column-name-conditions"></a>

Under **Column Name Match**, provide the criteria to identify matching columns based on the column name.

Note that a matching column must match both the data type and the column name criteria.

#### Configuring text matching conditions <a href="#column-name-criteria-text-match" id="column-name-criteria-text-match"></a>

When you provide a list of text matching conditions, a matching column must match all of the conditions. In other words, the conditions are joined by `AND`.

To apply the same generator preset to columns that have completely different names, you must create separate sensitivity rules.

To create a list of text matching conditions:

<figure><img src="/files/bm0zmP1G2hPQVQ3q7t8J" alt=""><figcaption><p>Column name text match rules for a custom sensitivity rule</p></figcaption></figure>

1. Click **Text Match**.
2. To add a column name condition, click **Add String Match**.
3. For each condition:
   1. From the comparison type dropdown list, select the type of comparison. For example, **Contains**, **Starts with**, **Ends with**.
   2. In the comparison text field, provide the text to check for.\
      \
      The comparison text is case insensitive. For example, if you set a condition to match column names that contain the text `term`, it also matches column names that contain `TERM` or `Term` or `tErM`.
4. To remove a column name condition, click its delete icon.

#### Providing a regular expression <a href="#column-name-criteria-regex" id="column-name-criteria-regex"></a>

To use a regular expression to identify matching columns based on the column name:

<figure><img src="/files/YGTYklPjJgw1ncd5s6Oc" alt=""><figcaption><p>Column name regular expression field for a custom sensitivity rule</p></figcaption></figure>

1. Click **Regular Expression**.
2. In the field, provide the regular expression.

### Generator preset to apply <a href="#sensitivity-rule-generator-preset" id="sensitivity-rule-generator-preset"></a>

From the **Recommended Generator Preset** dropdown list, select the generator preset that is the recommended generator for matching columns.

To search for a specific preset, begin to type the generator preset name.

## Managing generator preset configuration <a href="#sensitivity-rule-preset-config" id="sensitivity-rule-preset-config"></a>

{% hint style="info" %}
**Required global permission:** Create and manage generator presets
{% endhint %}

When you configure a sensitivity rule, you can also create a new generator preset or update the configuration of the selected generator preset.

To create a new generator preset, click **Create Preset**. On the generator preset details panel, provide the generator preset configuration, then click **Create**.

To edit the selected generator preset, click **Edit Current Preset**. On the generator preset details panel, update the generator preset configuration, then click **Save and Apply**.&#x20;

For more information about generator preset configuration, go to [Managing generator presets](/app/generation/generators-assign-config/generator-presets#generator-presets-configure).

## Previewing the rule results

You cannot preview the results of LLM-based sensitivity rules. You can only test column name rules.

If you have access to a workspace, then you can use the workspace to preview the sensitivity rule results.

Under **Test Results**, from the workspace dropdown list, select the workspace to use.

Structural searches the workspace schema for matching columns based on the sensitivity rule configuration.

It displays any matching columns. You can filter the matching columns based on the table or column name.

<figure><img src="/files/Gi0TZdeWDvRYVnn4XecU" alt=""><figcaption><p>Test Results section to preview the results for a sensitivity rule</p></figcaption></figure>

For each matching column, the list includes:

* The column name and table
* A sample value from the source data. The sample source value is only present if you have the **Preview source data** permission for the workspace.
* A sample replacement value, based on the selected generator preset for the sensitivity rule. The sample replacement value is only present if you have the **Preview destination data** permission for the workspace.

## Deleting a sensitivity rule <a href="#sensitivity-rule-delete" id="sensitivity-rule-delete"></a>

To delete a sensitivity rule, on the **Sensitivity Rules** view, click the delete icon for the rule.

Note that existing generator recommendations for the rule remain in place until the next sensitivity scan.


# Table modes

Each table is assigned a table mode. The table mode determines at a high level how the table is populated in the destination database.

## Selecting the table mode for a table <a href="#table-mode-selection" id="table-mode-selection"></a>

{% hint style="info" %}
**Required workspace permission:** Assign table modes
{% endhint %}

Both **Database View** and **Table View** allow you to view and update the selected table mode for a table.

For **Database View**, go to [Viewing and configuring tables](/app/generation/database-view/database-view-tables#database-view-tables-assign-mode).

For **Table View**, go to [Table View](/app/generation/table-view#table-view-table-mode).

## Available table modes <a href="#table-mode-types" id="table-mode-types"></a>

### De-Identify

This is the default table mode for new tables.

In this mode, Tonic Structural copies over all of the rows to the destination database.

For columns that have the generator set to Passthrough, Structural copies the original source data to the destination database.

For columns that are assigned a generator other than Passthrough, Structural uses the generator to replace the column data in the destination database.

### Truncate

This mode drops all data for the table in the destination database. Sensitivity scans ignore truncated tables.

For data connectors other than Spark-based data connectors, the table schema and any constraints associated with the table are included in the destination database.

For Spark-based data connectors ([Databricks](/app/setting-up-your-database/databricks), [Spark SDK](/app/setting-up-your-database/spark-sdk)), the table is ignored completely.

For the [file connector](/app/setting-up-your-database/file-connector), file groups are treated as tables. When a file group is assigned Truncate mode, the data generation process ignores the files that are in that file group.

Any existing data in the destination database is removed. For example, if you change the table mode to Truncate after an initial data generation, the next data generation clears the table data. For Spark-based data connectors, the table is removed.

If you assign Truncate mode to a table that has a foreign key constraint, it fails during data generation. If this is a requirement, contact <support@tonic.ai> for assistance.

When [upsert](/app/workspace/workspace-configuration-settings/workspace-config-upsert) is enabled, the Truncate table mode does not actually truncate the destination table. Instead, it works more like Preserve Destination table mode, which preserves existing records in the destination table.

### Preserve Destination

This mode preserves the data in the destination database for this table. It does not add or update any records.

This feature is primarily used for very large tables that don't need to be de-identified during subsequent runs after the data exists in the destination database.

When you assign Preserve Destination mode to a table, Structural locks the generator configuration for the table columns.

The destination database must have the same schema as the source database.

You cannot use Preserve Destination mode when you:

* Enable upsert for a workspace.
* Write destination data to a container repository.

### Incremental

Incremental mode only processes the changes that occurred to the source table since the most recent data generation or other changes in the destination. This can greatly reduce generation time for large tables that don't have a lot of changes.

For Incremental mode to work, the following conditions must be satisfied:&#x20;

* The table must exist in the destination database. Either Structural created the table during data generation, or the table was created and populated in some other way.
* A reliable date updated column must be present. When you select Incremental mode for a table, Structural prompts you to select the date updated column to use.
* The table must have a primary key.

To maximize performance, we recommend that you have an index on the date updated field.

For tables that use Incremental mode, Structural checks the source database for records that have an updated date that that is greater than the maximum date in that column in the destination database.

When identifying records to update, Structural only checks the updated date. It does not check for other updates. Records where the generator configuration is changed are not updated if they do not meet the updated date requirement.

For the identified records, Structural checks for primary key matches between the source and destination databases, then does one of the following:

* If the primary key value exists in the destination database, then Structural overwrites the record in the destination database.
* If the primary key value does not exist in the destination database, then Structural adds a new record to the destination database.

This mode currently only updates and adds records. Rows that are deleted from the source database remain in the destination database.

To ensure accurate incremental processing of records, we recommend that you do not directly modify the destination database. A direct modification might cause the maximum updated date in the destination database to be after the date of the last data generation. This could prevent records from being identified for incremental processing.

Incremental mode is currently supported on PostgreSQL, MySQL, and SQL Server. If you want to use this table mode with another database type, contact <support@tonic.ai>.

You cannot use Incremental mode when you:

* Enable upsert for a workspace.
* Write destination data to a container repository.

## Indicating whether to return an error when destination data already exists (Databricks only) <a href="#table-modes-error-on-overwrite" id="table-modes-error-on-overwrite"></a>

For the Databricks data connector, the table mode configuration includes an **Error on Overwrite** setting. The setting indicates whether to return an error when Structural attempts to write data to a destination table that already contains data. The option is not available when you write destination data to Databricks Delta tables.

To return the error, toggle the setting to the on position.

To not return the error, toggle the setting to the off position.

## Applying a filter to tables <a href="#table-mode-filter-tables" id="table-mode-filter-tables"></a>

For workspaces that use following data connectors, the table mode configuration for De-Identify mode includes an option to apply a filter to the table:

* [Amazon Redshift](/app/setting-up-your-database/amazon-redshift)
* [Databricks](/app/setting-up-your-database/databricks)
* [Google BigQuery](/app/setting-up-your-database/google-bigquery)
* [Snowflake](/app/setting-up-your-database/snowflake)

Table filters provide a way to generate a smaller set of data when a data connector does not support subsetting. For more information, go to [Using table filtering for data warehouses and Spark-based data connectors](/app/generation/subsetting/table-filtering).

## Configuring partitioning for the destination database <a href="#table-mode-partition-config" id="table-mode-partition-config"></a>

This option is only available for workspaces that use the following data connectors:

* [Databricks](/app/setting-up-your-database/databricks)

On the table mode configuration panel, you can use the **Repartition** or **Coalesce** option to indicate a number of partitions to generate.

<figure><img src="/files/iU6FX6kWfUrk0bF2XJ0g" alt=""><figcaption><p>Table mode configuration panel<br>for a Spark-based workspace</p></figcaption></figure>

By default, the destination database uses the same partitioning as the source database. The partition option is set to **Neither**.

### Using the Repartition option <a href="#partition-repartition" id="partition-repartition"></a>

The **Repartition** option allows you to provide a specific number of partitions to generate.

To use the **Repartition** option:

1. Click **Repartition**.
2. In the field, enter the number of partitions.

### Using the Coalesce option <a href="#partition-coalesce" id="partition-coalesce"></a>

The **Coalesce** option allows you to provide a maximum number of partitions to generate. If the source data has fewer partitions than the number you specify, then Structural only generates that number.

The **Coalesce** option should be more efficient than the **Repartition** option.

To use the **Coalesce** option:

1. Click **Coalesce**.
2. In the field, enter the number of partitions.


# Generator information

Generators transform the data in a source database column. You assign the generators to use. Tonic Structural offers a variety of generators to transform different types of data.&#x20;

For details about how to assign and configure generators, and manage generator presets, go to [Generator assignment and configuration](/app/generation/generators-assign-config).

You can also view this [video overview of generators and how they work](https://youtu.be/UNngC2a6q94).

## About the available generators

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Generator summary</strong></td><td>Summary list of generators.</td><td></td><td><a href="/pages/oP5GxRAyF8UTY3LfMMwM">/pages/oP5GxRAyF8UTY3LfMMwM</a></td></tr><tr><td><strong>Generator reference</strong></td><td>Details about the characteristics and configuration options for each generator.</td><td></td><td><a href="/pages/0Ut5IaDYLB4ZhtomxgZ9">/pages/0Ut5IaDYLB4ZhtomxgZ9</a></td></tr><tr><td><strong>Generator API reference</strong></td><td>Details about the structure of each generator assignment in the API.</td><td></td><td><a href="/pages/Y8OgcfldJLoRKSFvnndi">/pages/Y8OgcfldJLoRKSFvnndi</a></td></tr></tbody></table>

## Generator characteristics and types

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Generator characteristics</strong></td><td>Common generator characteristics to be aware of, such as consistency and linking.</td><td></td><td><a href="/pages/Wd98iS6iDUPhKKNUklIq">/pages/Wd98iS6iDUPhKKNUklIq</a></td></tr><tr><td><strong>Composite generators</strong></td><td>Composite generators apply a generator to a specific data element or based on a condition.</td><td></td><td><a href="/pages/QAB0LpK9pEwKvEiG26py">/pages/QAB0LpK9pEwKvEiG26py</a></td></tr><tr><td><strong>Primary key generators</strong></td><td>Learn about generators that you can apply to primary key columns.</td><td></td><td><a href="/pages/-M4tFOe2vTNcjSI5hjUq">/pages/-M4tFOe2vTNcjSI5hjUq</a></td></tr></tbody></table>


# Generator summary

The following table summarizes the available generators. The table includes generator characteristics that you might take into account when you select the generator to use for a column.

[Generator hints and tips](/app/generation/generators-assign-config/common-usage) also provides some suggestions for generators to use for specific use cases.

<details>

<summary>Information in the table</summary>

The generator summary includes the following columns:

* **Generator** - The name of the generator, linked to the entry in the [generator reference](/app/generation/generators/generator-reference).
* **Description** - An overview description of the generator.&#x20;
* **Supported features -** Includes the following information:
  * The [generator characteristics](/app/generation/generators/generator-characteristics) that the generator supports
  * Whether the generator is a [composite generator](/app/generation/generators/generator-types/generators-composite) or a [primary key generator](/app/generation/generators/generator-types/primary-key-generators)
  * The generator [privacy ranking](/app/generation/privacy-report#privacy-report-privacy-ranking-about)

</details>

<table data-full-width="true"><thead><tr><th valign="top">Generator</th><th valign="top">Description</th><th valign="top">Supported features</th></tr></thead><tbody><tr><td valign="top"><a href="/pages/dVMNKoVkz5zWaDTMFAHA">Address</a><br><br>API: <a href="/pages/VDHXxQdFxO7jEDI5Px8m">AddressGenerator</a></td><td valign="top"><p>Generates replacement values for U.S. mailing addresses.</p><p>You select the address component or format for the replacement values.</p><p>For example, the column might only contain a street address or a postal code, or it might contain a full address.</p></td><td valign="top"><p>Consistency - Self and other<br>Linkable</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/Z8PccqJ9Svy9sV0QODZ6">Algebraic</a><br><br>API: <a href="/pages/IdQez0vTDCbeKxt5qWg8">AlgebraicGenerator</a></td><td valign="top"><p>Identifies the algebraic relationship between 3 or more numeric values, including at least one non-integer.</p><p>Based on the relationship, generates new values to match. If there is no relationship, uses the Categorical generator.</p></td><td valign="top">Linkable - linking is required<br><br>Privacy ranking: 3</td></tr><tr><td valign="top"><a href="/pages/CfuJagRPVBZmPM0oxBhU">Alphanumeric String Key</a><br><br>API: <a href="/pages/y0f5voM6MoBGv6EhtxZ5">AlphaNumericPkGenerator</a></td><td valign="top"><p>Generates unique alphanumeric strings of the same length as the input.</p><p>For example, for the origin value <code>ABC123</code>, the output value is a six-character alphanumeric string such as <code>D24N05</code>.</p></td><td valign="top"><p>Consistency - Self only</p><p>Primary key generator</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/SdRljlLIRwX5LBDUObJG">Array Character Scramble</a><br><br>API: <a href="/pages/XNn6JUVYLWaK5ss6vSSm">ArrayTextMaskGenerator</a></td><td valign="top"><p>Within an array, replaces letters with random other letters, and numbers with random other numbers.</p><p>Preserves punctuation and whitespace.</p></td><td valign="top"><p>Consistency - Self only<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/aqu6M4o7eOUf4BxCxw4m">Array JSON Mask</a><br><br>API: <a href="/pages/u7EpGuc14cDWkxhIwDur">ArrayJsonMaskGenerator</a></td><td valign="top"><p>Used to transform array values in JSON.</p><p>To identify values to transform, you provide a list of JSONPaths.</p><p>For each JSONPath, you assign a sub-generator to apply to matching values.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><a href="/pages/h4RCRnTH4yeX6hIDtIiV">Array Regex Mask</a><br><br>API: <a href="/pages/u95DTjy0veeI9D3Jh05U">ArrayRegexMaskGenerator</a></td><td valign="top"><p>Used to transform values in an array.</p><p>To identify values to transform, you provide a regular expression.</p><p>For each capture group in an expression, you assign a sub-generator to apply to matching values.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><a href="/pages/qeq2VIVXiMjO54X2J9Ez">ASCII Key</a><br><br>API: <a href="/pages/clFBetMsKpmmJTIuSUSN">AsciiPkGenerator</a></td><td valign="top"><p>Generates unique alpha-numeric strings based on any printable ASCII characters.</p><p>You can optionally exclude lowercase letters from the generated values.</p><p>The replacement value does not preserve the length of the original value.</p></td><td valign="top"><p>Consistency - Self only</p><p>Primary key generator</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/V2gL7zC9OepFyB5Fvwcj">Business Name</a><br><br>API: <a href="/pages/qhEUntD9aZcNPS4CsMPz">BusinessNameGenerator</a></td><td valign="top">Generates a random company name-like string.</td><td valign="top"><p>Consistency - Self or other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/JRbQS9Aj4nQn2To8ncff">Categorical</a><br><br>API: <a href="/pages/guL2oBjIWsY1KUKFWzNe">CategoricalGenerator</a></td><td valign="top"><p>Shuffles the original values for a column to different rows. Maintains the overall frequency of each value.</p><p>For example, a column contains the values <code>Small</code> (3 times), <code>Medium</code> (4 times), and <code>Large</code> (5 times).</p><p>In the transformed data, each value appears the same number of times, but the values are shuffled to different rows.</p></td><td valign="top"><p>Linkable</p><p>Differential privacy is configurable<br><br>Privacy ranking:</p><ul><li>2 with differential privac</li><li>3 without differential privacy</li></ul></td></tr><tr><td valign="top"><a href="/pages/YkyIbAnIucBzXXgehrmT">Character Scramble</a><br><br>API: <a href="/pages/rkl8AFhgEjfzTZnuNTya">TextMaskGenerator</a></td><td valign="top"><p>Replaces letters with random other letters and numbers with random other numbers.</p><p>Preserves punctuation, whitespace, and mathematical symbols.</p></td><td valign="top"><p>Consistency - Self only<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/cUTLku4sXzcOo8oyXfZV">Character Substitution</a><br><br>API: <a href="/pages/fy8n3fsQgTSDMuuKuSIt">StringMaskGenerator</a></td><td valign="top"><p>Replaces characters with other random characters. Preserves punctuation, capitalization, and whitespace.</p><p>A replacement character is always from within the same Unicode Block as the source character.</p><p>A source character is always mapped to the same destination character. For example, <code>M</code> might always map to <code>V</code>.</p></td><td valign="top"><p>Always self-consistent </p><p>Unique columns allowed<br><br>Privacy ranking: 4</p></td></tr><tr><td valign="top"><a href="/pages/OtQuLWmqqEkyWMBuQMt4">Company Name</a> (Deprecated)<br><br>API: <a href="/pages/QDHR5WXFUdZvlhWJHA08">CompanyNameGenerator</a></td><td valign="top"><p>This generator is deprecated. Use the <a href="/pages/0Ut5IaDYLB4ZhtomxgZ9#business-name">Business Name</a> generator instead.</p><p>Generates a random company name-like string.</p></td><td valign="top"><p>Consistency - Self or other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/I2CiiM0fByBwEjZSE6sI">Conditional</a><br><br>API: <a href="/pages/MVk7kwA8JmL2rTTe90xI">ConditionalGenerator</a></td><td valign="top"><p>Applies different generators to rows conditionally based on the column value.</p><p>For example, apply the Character Scramble generator for values other than Test.</p><p>You configure a list of conditions. Each condition performs a check against the column value.</p><p>For each condition, you assign a sub-generator to apply to matching values.</p></td><td valign="top"><p>Unique columns allowed</p><p>Composite generator. Other feature support is based on the sub-generators.</p><p></p><p>Privacy ranking:</p><ul><li>If a fallback generator is selected, then the lower of 5 or the fallback generator.</li><li>5 if no fallback generator is selected.</li></ul></td></tr><tr><td valign="top"><a href="/pages/KH9F697KstcRePbhBe1V">Constant</a><br><br>API: <a href="/pages/7YhXuxNZqKjSPVkhcF0a">ConstantGenerator</a></td><td valign="top"><p>Uses a single specified value to replace all of the values in the column.</p><p>The replacement value must be compatible with the column data type.</p></td><td valign="top"><p>Differential privacy</p><p>Data-free<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/WcDrd9HqklGLQliXDxIE">Continuous</a><br><br>API: <a href="/pages/ibZpgFFKaeYJgeBahlkY">GaussianGenerator</a></td><td valign="top"><p>Generates a continuous distribution to fit the underlying data.</p><p>Can link to other columns to create multivariate distributions.</p><p>Can also be partitioned by other columns.</p></td><td valign="top"><p>Linkable</p><p>Differential privacy is configurable<br><br>Privacy ranking:</p><ul><li>2 with differential privacy</li><li>3 without differential privacy</li></ul></td></tr><tr><td valign="top"><a href="/pages/tf7X7fg4sfL5kitEO8Le">Cross Table Sum</a><br><br>API: <a href="/pages/OnB96XREUMxGohHqnbuN">CrossTableAggregateGenerator</a></td><td valign="top"><p>Populates the column using the sum of values from a column in another table.</p><p>To select the rows to use, uses a foreign key value that matches the primary key value for the current row.</p><p>For example, to transform the <strong>Total_Sales</strong> column in the <strong>Customers</strong> table, from the <strong>Transactions</strong> table, use the sum of the <strong>Amount</strong> values for rows where the <strong>Customer_ID</strong> value matches the primary key value for the current customer.</p></td><td valign="top">Privacy ranking: 3</td></tr><tr><td valign="top"><a href="/pages/Z55blETTPvkFK2j5g3f3">CSV Mask</a><br><br>API: <a href="/pages/OFKSQZQ52dE7G3mLzOkC">CsvMaskGenerator</a></td><td valign="top"><p>Used to mask text in a delimited format.</p><p>Parses the text as a row where the columns are delimited by a specified character.</p><p>For each index, you assign a sub-generator to apply to the index value.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><a href="/pages/fRNZZPOEPGICF4gQsL1j">Custom Categorical</a><br><br>API: <a href="/pages/AsyoJDj5DdRAZoieTjp5">CustomCategoricalGenerator</a></td><td valign="top">Replaces the original column value with a value from list of values that you provide.</td><td valign="top"><p>Consistency - Self and other</p><p>Linkable</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/zFcGQfMbzYUhu2FxIBsP">Date Truncation</a><br><br>API: <a href="/pages/sIv7iAJxM0EqwLdkBhVc">DateTruncationGenerator</a></td><td valign="top"><p>Truncates dates or timestamps to a specific date or time component.</p><p>For example, you might truncate a date value to the month or a timestamp to the hour.<br>You can specify a fallback generator to use for values that the generator cannot process.</p></td><td valign="top">Privacy ranking: 5</td></tr><tr><td valign="top"><a href="/pages/AaJCKfvi0RThh0Fold19">Email</a><br><br>API: <a href="/pages/cSiodmqjXSXTjm030plR">EmailGenerator</a></td><td valign="top"><p>Scrambles characters in an email address.</p><p>Preserves the formatting and keeps the <code>@</code> and <code>.</code>. </p><p>You can identify specific email domains to not scramble.</p></td><td valign="top"><p>Consistency - Self only<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/c2QaKf5WNMB0eI9S8EX1">Event Timestamps</a><br><br>API: <a href="/pages/zbWO545ha1u50UdO0L13">EventGenerator</a></td><td valign="top"><p>Generates timestamps that fit an event distribution.</p><p>You can link columns to create a sequence of events across multiple columns.</p><p>You can also partition the generator by other columns.</p></td><td valign="top">Linkable<br><br>Privacy ranking: 3</td></tr><tr><td valign="top"><a href="/pages/aguF2E3lGT2yslfJLJTa">File Name</a><br><br>API: <a href="/pages/mRQCOU8teo52JgDaEG0t">FileNameGenerator</a></td><td valign="top"><p>Scrambles characters in a file name.</p><p>Preserves the formatting and the file extension.</p></td><td valign="top"><p>Consistency - Self only<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/0jbJZHtHpLka5FMSiH4s">Find and Replace</a><br><br>API: <a href="/pages/TepUJR3tP7VCvvztI8mZ">FindAndReplaceGenerator</a></td><td valign="top"><p>Replaces all instances of the find string with the replace string.</p><p>For the find string, you can optionally provide a regular expression.</p></td><td valign="top">Privacy ranking: 5</td></tr><tr><td valign="top"><p><a href="/pages/qRUy6GHhWvlWeTHJeRNv">Finnish Personal Identity Code</a><br></p><p>API: <a href="/pages/7J0bQTHx2gdp1DQKiwIC">FinnishPicGenerator</a></p></td><td valign="top"><p>Generates a valid Finnish Personal Identity Code (PIC).</p><p>You configure the date range during which the PIC was issued.</p></td><td valign="top"><p>Consistency - Self only</p><p>Data-free if not consistent</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br></p><p>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/j6T24MpHmdjNqUbomdiL">FNR</a><br><br>API: <a href="/pages/YCgMnGPO7rJsflQ8v92J">FnrGenerator</a></td><td valign="top"><p>Transforms Norwegian national identity numbers.</p><p>You can optionally preserve the gender and birthdate portions of the identifier values.</p></td><td valign="top"><p>Consistency - Self and other</p><p>Unique columns allowed<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/Jw6LyuQf8dAPbhVvghXt">Geo</a><br><br>API: <a href="/pages/Yrd7JmcySjDcAI9aHAkh">GeoGenerator</a></td><td valign="top">Used to transform columns that contain latitude and longitude values.</td><td valign="top"><p>Linkable</p><p>Unique columns allowed<br><br>Privacy ranking: 3</p></td></tr><tr><td valign="top"><a href="/pages/odSiwZYTNVWoHsIqkxWu">HIPAA Address</a><br><br>API: <a href="/pages/Myr5CNhdSaHmsjdbfwOQ">HipaaAddressGenerator</a></td><td valign="top">Can be used to generate cities, states, zip codes, and latitude/longitude values that follow HIPAA guidelines for safe harbor.</td><td valign="top"><p>Consistency - Self only<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/yOaXi1zpRvqtKPIi8LV9">Hostname</a><br><br>API: <a href="/pages/9QUF3jFy70V79ZS4JBiw">HostnameGenerator</a></td><td valign="top">Generates random host names, based on the English language.</td><td valign="top"><p>Consistency - Self and other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/VTTM6x7sMxY6mOYl1nLS">HStore Mask</a><br><br>API: <a href="/pages/G07RJPVa3OJP9RFhquty">HStoreMaskGenerator</a></td><td valign="top"><p>Used to transform values in an HStore column in a PostgreSQL database.</p><p>You specify a list of keys for which to transform the values.</p><p>For each key, you assign a generator to apply to the key value.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><a href="/pages/EJBjN3p71wRjD4x5ja7c">HTML Mask</a><br><br>API: <a href="/pages/xGkzP2rFfjHAPN0MmnOb">HtmlMaskGenerator</a></td><td valign="top"><p>Used to transform columns that contain HTML content.</p><p>To identify the values to transform, you provide a list of path expressions.</p><p>For each path expression, you assign a generator to apply to the matching value.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><p><a href="/pages/3mz5Zu4bCMPQ8JZCAd2F">IBAN</a><br></p><p>API: <a href="/pages/Ser2WaJyXVOe8eu9mVCz">IbanGenerator</a></p></td><td valign="top"><p>Generates an International Bank Account Number (IBAN).</p><p>You determine whether to preserve the banking code and country code from the original value.</p><p>You also determine whether to generate replacement values for invalid source values.</p></td><td valign="top"><p>Consistency - Self only</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE) when either:</p><ul><li>Assigned to a primary key column.</li><li>Consistency is enabled.</li></ul><p></p><p>Privacy ranking - </p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/2KDUZbkl4mGozBdm7vog">Integer Key</a><br><br>API: <a href="/pages/G9868jXaFcVADJJxYJhz">IntegerPkGenerator</a></td><td valign="top"><p>Generates unique integer values.</p><p>By default, the generated values are within the range of the column’s data type.</p><p>You can also specify a range for the generated values. The source values must be within that range.</p></td><td valign="top"><p>Consistency - Self only</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent</p><p>Primary key generator</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/30Xu7urpP01uo3UeX1PQ">International Address</a><br><br>API: <a href="/pages/6jFugteG83ssc6xo4XJH">InternationalAddressGenerator</a></td><td valign="top"><p>For Canadian mailing addresses, can generate:</p><ul><li>Street name</li><li>Postal code</li></ul><p>For United Kingdom (UK) mailing addresses, can generate:</p><ul><li>City</li><li>County</li><li>District</li><li>Country</li><li>Postal code</li></ul></td><td valign="top"><p>Consistency - Self only</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/KduWaTvOoFwua7IXNaWx">IP Address</a><br><br>API: <a href="/pages/cqhOghPXmzBZ6imSfMeL">IPAddressGenerator</a></td><td valign="top"><p>Generates a random IP address-formatted string.</p><p>You specify the percentage of IPv4 addresses.</p><p>The remaining addresses are IPv6.</p></td><td valign="top"><p>Consistency - Self or other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/DuCeR6mVkXvBqA8p34h5">JSON Mask</a><br><br>API: <a href="/pages/WWUdcfsgep5Uw4BWhmMP">JsonMaskGenerator</a></td><td valign="top"><p>Used to transform values in JSON columns.</p><p>To identify values to transform, you provide a list of JSONPaths.</p><p>For each JSONPath, you assign a sub-generator to apply to matching values.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><a href="/pages/DCIwJMeIf96Pkx82inIz">MAC Address</a><br><br>API: <a href="/pages/luQk5Opzxqq2KtkqVOH2">MACAddressGenerator</a></td><td valign="top">Generates a random MAC address formatted string.</td><td valign="top"><p>Consistency - Self only</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/FxvV01IUZ1tTht1c2SF7">Mongo ObjectId Key</a><br><br>API: <a href="/pages/iPkO3meIe0zGhQR0pwbS">ObjectIdPkGenerator</a></td><td valign="top"><p>Generates unique MongoDB objectId values.</p><p>Can be assigned to text columns that contain MongoDB ObjectId values.</p><p>The column value must be 12 bytes long.</p></td><td valign="top"><p>Consistency - Self only<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/yTj2MFU4g8qp7BfSsE1g">Name</a><br><br>API: <a href="/pages/BvNQ4vvWJYxZLuIjFD86">NameGenerator</a></td><td valign="top"><p>Generates a random name string from a dictionary of first and last names.</p><p>You specify the name format.</p><p>For example, a column might contain only a first name, or a full name that is last name first.</p></td><td valign="top"><p>Consistency - Self or other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/BcUxKABStDhe0Qy2jTdQ">Noise Generator</a><br><br>API: <a href="/pages/tTt0z7vATigzXDyV544z">NoiseGenerator</a></td><td valign="top"><p>Masks values in numeric columns.</p><p>Either adds or multiplies the original value by random noise.</p></td><td valign="top"><p>Consistency - Self or other<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/FZ1Tccf9nHDUbfjY9dJG">Null</a><br><br>API: <a href="/pages/sWuS54GoRM4wKlN7RfsW">NullGenerator</a></td><td valign="top">Replaces all of the column values with <code>NULL</code> values.</td><td valign="top"><p>Differential privacy</p><p>Data-free</p><p>Unique columns allowed<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/dV9aZtcRZdnNEXx7CJEr">Numeric String Key</a><br><br>API: <a href="/pages/CkEpHn6DoMovaqNFFDBQ">NumericStringPkGenerator</a></td><td valign="top">Generates unique numeric strings of the same length as the input numeric string.</td><td valign="top"><p>Consistency - Self only</p><p>Primary key generator</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/9UJhLJtuDqxViipYrauB">Passthrough</a><br><br>API: <a href="/pages/Uj0K7yWLvWPfXPjKF8aJ">PassthroughGenerator</a></td><td valign="top"><p>Default generator.</p><p>Does not perform any transformation on the source data.</p></td><td valign="top">Unique columns allowed<br><br>Privacy ranking: 6</td></tr><tr><td valign="top"><a href="/pages/muBo0FasU044YZ9WcPRG">Phone</a><br><br>API: <a href="/pages/EFPhZCzc60GKwBShPl0P">USPhoneNumberGenerator</a></td><td valign="top"><p>Generates a random telephone number that matches the country or region and format of the input telephone number.</p><p>For invalid telephone numbers, either replaces individual numbers or generates a valid replacement number.</p></td><td valign="top">Consistency - Self only<br><br>Privacy ranking: 3</td></tr><tr><td valign="top"><a href="/pages/00UvmmnzkwyY2aVyrRkZ">Random Boolean</a><br><br>API: <a href="/pages/6olQz5JyHUFbxOB2zq0I">RandomBooleanGenerator</a></td><td valign="top"><p>Generates a random boolean value.</p><p>You specify the percentage of true values. The remaining values are false.</p></td><td valign="top"><p>Differential privacy</p><p>Data-free<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/XlbTgf3f5IPlg1ePNLuX">Random Double</a><br><br>API: <a href="/pages/i90AqIzn1GHBFxyMZYd9">RandomDoubleGenerator</a></td><td valign="top">Generates a random double number that is between the specified minimum (inclusive) and maximum (exclusive) values.</td><td valign="top"><p>Differential privacy</p><p>Data-free<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/ZRoWdtHnVThdgDJAFtnq">Random Hash</a><br><br>API: <a href="/pages/yZqDz5NPdIOWClt9Ri4u">RandomStringGenerator</a></td><td valign="top">Generates a random hash string.</td><td valign="top"><p>Differential privacy</p><p>Data-free<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/J0no3tPglJjbX7pcB3Rj">Random Integer</a><br><br>API: <a href="/pages/IdAYVVk6vTAfbhkGPHxA">RandomIntegerGenerator</a></td><td valign="top">Returns a random integer that is between the specified minimum (inclusive) and maximum (exclusive) values.</td><td valign="top"><p>Differential privacy</p><p>Data-free<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/y3Ko1JJbu1073EdEk87c">Random Timestamp</a><br><br>API: <a href="/pages/ugiVelMleB3JVEtMhujX">RandomTimestampGenerator</a></td><td valign="top">Generates random dates, times, and timestamps that fall within a specified range.</td><td valign="top"><p>Differential privacy</p><p>Data-free<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/6mcOp7hMjQG64xzgQmHX">Random UUID</a><br><br>API: <a href="/pages/mKhHMxcwVZ4173P5jABj">UUIDGenerator</a></td><td valign="top">Generates a random new UUID string.</td><td valign="top"><p>Differential privacy</p><p>Data-free</p><p>Unique columns allowed<br><br>Privacy ranking: 1</p></td></tr><tr><td valign="top"><a href="/pages/wnst1bPkt24cDGEvXQFz">Regex Mask</a><br><br>API: <a href="/pages/EL2vffWYSJWjXB1myN6b">RegexMaskGenerator</a></td><td valign="top"><p>To identify values to transform, you provide a regular expression.</p><p>For each capture group in an expression, you assign a sub-generator to apply to matching values.</p></td><td valign="top"><p>Unique columns allowed</p><p>Composite generator. Other feature support is based on the sub-generators.<br><br>Privacy ranking: 5</p></td></tr><tr><td valign="top"><a href="/pages/dxgdfnALkeVYgwAI41Mu">Sequential Integer</a><br><br>API: <a href="/pages/4uupe7QNPCKaBotobqNI">UniqueIntegerGenerator</a></td><td valign="top">Generates a column of unique integer values that start with specified value, and then increment by 1 for each processed row.</td><td valign="top"><p>Linkable</p><p>Unique columns allowed<br><br>Privacy ranking: 3</p></td></tr><tr><td valign="top"><a href="/pages/lUlMa86LMU4TQaN6BOGY">Shipping Container</a><br><br>API: <a href="/pages/4IM8nJsEYw6HCSAxKTXW">ShippingContainerGenerator</a></td><td valign="top"><p>Generates values of ISO 6346 compliant shipping container codes.</p><p>The codes are all in the freight ("U") category.</p></td><td valign="top"><p>Consistency - Self or other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/oUZCRJ9qVIAT5OGhOmwH">SIN</a><br><br>API: <a href="/pages/Mz9mS42FRkHd2RYwn7vn">SINGenerator</a></td><td valign="top"><p>Generates a new valid Canadian Social Insurance Number.</p><p>Preserves the formatting from the original value.</p></td><td valign="top"><p>Consistency - Self only</p><p>Data-free if not consistent</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/V5Fv4f0Y7nCPYXNRZySY">SSN</a><br><br>API: <a href="/pages/n11fQqYw33VmaZFiRRLD">SsnGenerator</a></td><td valign="top"><p>Generates a new valid United States Social Security Number.</p><p>For numeric columns, the dashes (xxx-xx-xxxx) are always excluded.</p><p>Otherwise, you can specify the percentage of values for which to include the dashes.</p></td><td valign="top"><p>Consistency - Self or other</p><p>Differential privacy if not consistent</p><p>Data-free if not consistent<br><br>Privacy ranking:</p><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/O93C5heA0jclbt3E0m1a">Struct Mask</a><br><br>API: <a href="/pages/CcbHoYTZYfS9D0p4UYGi">StructMaskGenerator</a></td><td valign="top"><p>Used to transform StructFields within a StructType in Databricks data.</p><p>To identify the StructField value to transform, you provide a path expression.</p><p>For each path expression, you assign a sub-generator to apply to the matching values.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr><tr><td valign="top"><p><a href="/pages/oXYh3vmtTBe5lqcELOgW">Text Composition</a></p><p></p><p>API: <a href="/pages/qbwsuDb0Oq2kjLATvu90">TextCompositionGenerator</a></p></td><td valign="top">Generates a replacement text value that includes values from other columns in the same row.</td><td valign="top"><p>Consistent by default with the included columns.</p><p></p><p>Privacy ranking: 5</p></td></tr><tr><td valign="top"><a href="/pages/fk3ME8SWpUNErj5wHXU9">Timestamp Shift</a><br><br>API: <a href="/pages/RcKnFROcA6kW0PwapX7B">TimestampShiftGenerator</a></td><td valign="top"><p>Shifts timestamps by a random amount of a specific unit of time, within a set range.</p><p>The range can start before the original value.<br>You can specify a fallback generator to use for values that the generator cannot process.</p></td><td valign="top"><p>Consistency - Self or other<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/ImAAXAtbv51r0U5locyg">Unique Email</a><br><br>API: <a href="/pages/Y7Ji3rcAcGWys1sk8o3D">UniqueEmailGenerator</a></td><td valign="top"><p>Generates unique email addresses.</p><p>Replaces the username with a randomly generated GUID, and masks the domain with a character scramble.</p></td><td valign="top"><p>Consistency - Self only</p><p>Unique columns allowed<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/5QIAu2bGdTAJWMLcvhpP">URL</a><br><br>API: <a href="/pages/sPGiEl1VwSnZiWVoMla0">UrlGenerator</a></td><td valign="top"><p>Used to transform URLs.</p><p>Preserves the formatting. Keeps the URL scheme and top-level domain intact.</p></td><td valign="top">Unique columns allowed<br><br>Privacy ranking: 3</td></tr><tr><td valign="top"><a href="/pages/ltEMEZ8lClXIOLV3EWrk">UUID Key</a><br><br>API: <a href="/pages/svNCkXvq7BfySjKLJjKZ">UuidPkGenerator</a></td><td valign="top">Generates UUIDs.</td><td valign="top"><p>Consistency - Self only</p><p>Primary key generator</p><p>Unique columns allowed</p><p>Format-preserving encryption (FPE)<br><br>Privacy ranking:</p><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><a href="/pages/ApVlvOeIW4OwiEmivdhr">XML Mask</a><br><br>API: <a href="/pages/dIttKiW7ByeVYtoSkosK">XmlMaskGenerator</a></td><td valign="top"><p>Used to transform values in XML columns.</p><p>To identify the values to transform, you provide XPaths.</p><p>For each XPath, you assign a sub-generator to apply to the matching values.</p></td><td valign="top">Composite generator. Feature support is based on the sub-generators.<br><br>Privacy ranking: 5</td></tr></tbody></table>


# Generator reference

This generator reference provides the details for each of the the supported generators in Tonic Structural.

<details>

<summary>Information provided for each generator</summary>

For each generator, the reference provides:

* Overview description
* A table that contains:
  * Generator characteristics that you might want to take into account when you select the generator.
  * The generator [privacy ranking](/app/generation/privacy-report#privacy-report-privacy-ranking-about), which indicates the level of protection that the generator provides.
  * The generator ID to use in the Structural API. The generator ID is linked to the API details for the generator.
* Instructions for how to configure the generator

The generator characteristics include:

* [Consistency](/app/generation/generators/generator-characteristics/consistency) - Whether you configure the generator to base the the destination values on the source values.
* [Linking](/app/generation/generators/generator-characteristics/linking-generators) - Whether you can link columns that use the generator to indicate that there is a relationship between them.
* [Differential privacy](/app/generation/generators/generator-characteristics/differential-privacy) - Whether the generator supports differential privacy, which ensures that the source value cannot be reverse engineered from the output value.
* [Data-free](/app/generation/generators/generator-characteristics/data-free-generators) - Whether the generator is data-free, meaning that the output data is completely unrelated to the source data.
* [Allowed for primary keys](/app/generation/generators/generator-types/primary-key-generators) - Whether you can assign the generator to primary key columns.
* [Allowed for unique columns](/app/generation/generators/generator-characteristics/generators-uniqueness-constraints) - Whether you can assign the generator to columns that require unique values.
* [Uses format-preserving encryption (FPE)](/app/generation/generators/generator-characteristics/generators-fpe) - Whether the generator uses FPE to encrypt the values.

</details>

The generators are in alphabetical order by the generator name.

Here are some groupings to help to identify generators that are used for different types of values. [Generator hints and tips](/app/generation/generators-assign-config/common-usage) also provides some suggestions for generators to use for specific uses cases.

<details>

<summary>Composite generators</summary>

Transform data that uses complex formats or based on a condition. For more information, go to [Composite generators](/app/generation/generators/generator-types/generators-composite).

* [Array JSON Mask](/app/generation/generators/generator-reference/array-json-mask)
* [Array Regex Mask](/app/generation/generators/generator-reference/array-regex-mask)
* [Conditional](/app/generation/generators/generator-reference/conditional)
* [CSV Mask](/app/generation/generators/generator-reference/csv-mask)
* [HStore Mask](/app/generation/generators/generator-reference/hstore-mask)
* [HTML Mask](/app/generation/generators/generator-reference/html-mask)
* [JSON Mask](/app/generation/generators/generator-reference/json-mask)
* [Regex Mask](/app/generation/generators/generator-reference/regex-mask)
* [Struct Mask](/app/generation/generators/generator-reference/struct-mask)
* [XML Mask](/app/generation/generators/generator-reference/xml-mask)

</details>

<details>

<summary>Information type generators</summary>

These generators produce specific types of values.

* [Address](/app/generation/generators/generator-reference/address)
* [Business Name](/app/generation/generators/generator-reference/business-name) (and the deprecated [Company Name](/app/generation/generators/generator-reference/company-name))
* [Email](/app/generation/generators/generator-reference/email)
* [File Name](/app/generation/generators/generator-reference/file-name)
* [Finnish Personal Identity Code](/app/generation/generators/generator-reference/finnish-personal-identity-code)
* [FNR](/app/generation/generators/generator-reference/fnr)
* [Geo](/app/generation/generators/generator-reference/geo)
* [HIPAA Address](/app/generation/generators/generator-reference/hipaa-address)
* [Hostname](/app/generation/generators/generator-reference/hostname)
* [IBAN](/app/generation/generators/generator-reference/iban)
* [International Address](/app/generation/generators/generator-reference/international-address)
* [IP Address](/app/generation/generators/generator-reference/ip-address)
* [MAC Address](/app/generation/generators/generator-reference/mac-address)
* [Name](/app/generation/generators/generator-reference/name)
* [Phone](/app/generation/generators/generator-reference/phone)
* [Shipping Container](/app/generation/generators/generator-reference/shipping-container)
* [SIN](/app/generation/generators/generator-reference/sin)
* [SSN](/app/generation/generators/generator-reference/ssn)
* [Unique Email](/app/generation/generators/generator-reference/unique-email)
* [URL](/app/generation/generators/generator-reference/url)

</details>

<details>

<summary>Datetime value generators</summary>

These generators are used to specifically transform datetime values.

* [Date Truncation](/app/generation/generators/generator-reference/date-truncation)
* [Event Timestamps](/app/generation/generators/generator-reference/event-timestamps)
* [Random Timestamp](/app/generation/generators/generator-reference/random-timestamp)
* [Timestamp Shift Generator](/app/generation/generators/generator-reference/timestamp-shift-generator)

</details>

<details>

<summary>Key generators</summary>

Intended for use with primary key columns. For more information, go to [Primary key generators](/app/generation/generators/generator-types/primary-key-generators).

* [Alphanumeric String Key](/app/generation/generators/generator-reference/alphanumeric-string-key)
* [ASCII Key](/app/generation/generators/generator-reference/ascii-key)
* [Integer Key](/app/generation/generators/generator-reference/integer-key)
* [Numeric String Key](/app/generation/generators/generator-reference/numeric-string-key)
* [UUID Key](/app/generation/generators/generator-reference/uuid-key)

</details>

<details>

<summary>Numeric value generators</summary>

These generators are specifically intended to work with numeric values.

* [Algebraic](/app/generation/generators/generator-reference/algebraic)
* [Continuous](/app/generation/generators/generator-reference/continuous)
* [Cross Table Sum](/app/generation/generators/generator-reference/cross-table-sum)
* [Noise Generator](/app/generation/generators/generator-reference/noise-generator)
* [Random Double](/app/generation/generators/generator-reference/random-double)
* [Random Integer](/app/generation/generators/generator-reference/random-integer)
* [Sequential Integer](/app/generation/generators/generator-reference/sequential-integer)

</details>

<details>

<summary>String value generators</summary>

These generators are useful for transforming string values that aren't covered by a specific information type generator.

* [Categorical](/app/generation/generators/generator-reference/categorical)
* [Character Scramble](/app/generation/generators/generator-reference/character-scramble)
* [Character Substitution](/app/generation/generators/generator-reference/character-substitution)
* [Constant](/app/generation/generators/generator-reference/constant) - Also useable for numeric columns.
* [Custom Categorical](/app/generation/generators/generator-reference/custom-categorical) - Also useable for numeric columns.
* [Find and Replace](/app/generation/generators/generator-reference/find-and-replace)
* [Regex Mask](/app/generation/generators/generator-reference/regex-mask)
* [Text Composition](/app/generation/generators/generator-reference/text-composition)

</details>

<details>

<summary>Other value substitution and replacement generators</summary>

These generators perform other types of transformation on column values.

* [Array Character Scramble](/app/generation/generators/generator-reference/array-character-scramble)
* [Null](/app/generation/generators/generator-reference/null)
* [Random Boolean](/app/generation/generators/generator-reference/random-boolean)
* [Random Hash](/app/generation/generators/generator-reference/random-hash)
* [Random UUID](/app/generation/generators/generator-reference/random-uuid)

</details>


# Address

Generates a random mailing address-like string.

You can indicate which part of an address string that the column contains. For example, the column might contain only the street address or the city, or it might contain the full address.

## Characteristics <a href="#address-characteristics" id="address-characteristics"></a>

<table data-header-hidden><thead><tr><th width="297.09302325581393" valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent or consistent with another column.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/VDHXxQdFxO7jEDI5Px8m"><code>AddressGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#address-configure" id="address-configure"></a>

### Linking to other columns

From the **Link To** dropdown list, select the columns to link this column to.

You can link columns that use the Address generator to mask one of the following address components:

* City
* City State
* Country
* Country Code
* State
* State Abbreviation
* Zip Code
* Latitude
* Longitude

Note that when linked to another address column, a country or country code is always the United States.

### Identifying the address component

From the address component dropdown list, select the address component that this column contains.

#### Supported address components (connectors other than Databricks and Spark SDK)

For most data connectors, the available address components are:

* Building Number
* Cardinal Direction (North, South, East, West)
* City
* City Prefix (Examples: North, South, East, West, Port, New)
* City Suffix (Examples: land, ville, furt, town)
* City with State (Example: Spokane, Washington)
* City with State Abbr (Example: Houston, TX)
* Country (Examples: Spain, Canada)
* Country Code (Uses the 2-character country code. Examples: ES, CA)
* County
* Direction (Examples: North, Northeast, Southwest, East)
* Full Address
* Latitude (Examples: 33.51, 41.32)
* Longitude (Examples: -84.05, -74.21)
* Ordinal Direction (Examples: Northeast, Southwest)
* Secondary Address (Examples: Apt 123, Suite 530)
* State (Examples: Alabama, Wisconsin)
* State Abbr (Examples: AL, WI)
* Street Address (Example: 123 Main Street)
* Street Name (Examples: Broad, Elm)
* Street Suffix (Examples: Way, Hill, Drive)
* US Address
* US Address with Country
* Zip Code (Example: 12345)

#### Supported address components (Databricks and Spark SDK)

Databricks and Spark SDK workspaces only support the following address components:

* Building Number
* City
* Country
* Country Code
* Full Address
* Latitude
* Longitude
* State
* State Abbr
* Street Address
* Street Name
* Street Suffix
* US Address
* US Address with Country
* Zip Code

### Enabling and disabling consistency

Toggle the **Consistency** setting to indicate whether to make the column consistent.

By default, the consistency is disabled.

### Setting the type of consistency

If consistency is enabled, then by default, the generator is self-consistent.

To make the generator consistent with another column, from the **Consistent to** dropdown list, select the column.

When the Address generator is consistent with itself, then the same value in the source database is always mapped to the same destination value. For example, for a column that contains a state name, Alabama is always mapped to Illinois.

When the Address generator is consistent with another column, then the same value in the other column always results in the same destination value for the address column. For example, if the address column is consistent with a name column, then every instance of John Smith in the name column in the source database has the same address value in the destination database.

### Setting case sensitivity for consistency

When consistency is enabled, use the **Case-sensitive** toggle to indicate whether the consistency is case-sensitive.

By default, consistency is case-sensitive. For example, the values `Anytown` and `ANYTOWN` are considered different values. `Anytown` might always be replaced with `Chicago`, and `ANYTOWN` might be replaced with `Atlanta`.

To make the consistency case-insensitive, toggle **Case-sensitive** to the off position. When the consistency is case-insensitive, `Anytown` and `ANYTOWN` are considered the same value and have the same replacement.

### Enabling Structural data encryption

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Algebraic

The algebraic generator identifies the algebraic relationship between three or more numeric values and generates new values to match. At least one of the values must be a non-integer.

If a relationship cannot be found, then the generator defaults to the [Categorical](/app/generation/generators/generator-reference/categorical) generator.

This generator can be linked with other Algebraic generators.

## Characteristics <a href="#algebraic-characteristics" id="algebraic-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">3</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/IdQez0vTDCbeKxt5qWg8"><code>AlgebraicGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#algebraic-configure" id="algebraic-configure"></a>

To configure the generator, from the **Link To** dropdown list, select the columns to link this column to. You can select other columns that are assigned the Algebraic generator.

You must select at least three columns.

The column values must be numeric. At least one of the columns must contain a value other than an integer.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Alphanumeric String Key

Generates unique alphanumeric strings of the same length as the input.

For example, for the origin value `ABC123`, the output value is a six-character alphanumeric string such as `D24N05`.

You can configure the generator to:

* Preserve a value prefix. For example, if the origin value always starts with `acct_`, you might want to preserve that in the output value, and only mask the unique part of the identifier.
* Preserve a specified number of characters from the end of each source value. For example, you might want the last 3 characters of each destination value to be the same as the last 3 characters of the source value.&#x20;
* Only replace numerical characters, and keep other types of characters as is.

## Characteristics <a href="#alphanumeric-string-key-characteristics" id="alphanumeric-string-key-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/y0f5voM6MoBGv6EhtxZ5"><code>AlphaNumericPkGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#alphanumeric-string-key-configure" id="alphanumeric-string-key-configure"></a>

To configure the generator:

1. To preserve a prefix value, in the **Preserve prefix** field, type the prefix string to preserve in the output values.\
   \
   For example, if you set **Preserve prefix** to `acct_`, then any source values that start with `acct_` will also start with `acct_` in the destination data.
2. To preserve a set of characters from the end of the original value, in the **Preserve suffix length** field, type the the number of characters to preserve at the end of each value.\
   \
   For example, if you set **Preserve suffix** length to 3, then the destination values all have the same final 3 characters as the source values. For example, for the source value `abc123efg`, the destination value ends with `efg`.
3. To only replace the numerical characters, toggle **Mask digits only** to the on position.
4. Toggle the **Consistency** setting to indicate whether to make the generator self-consistent.\
   \
   By default, the generator is not consistent.
5. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Array Character Scramble

A version of the [Character Scramble](/app/generation/generators/generator-reference/character-scramble) generator that can be used for array values.

This generator replaces letters with random other letters, and numbers with random other numbers. Punctuation and whitespace are preserved.

For example, for the following array value:

`["ABC.123", 3, "last week"]`

The output might be something like:

`["KFR.860", 7, "sdrw mwoc"]`

This generator securely masks letters and numbers. There is no way to recover the original data.

## Characteristics <a href="#array-character-scramble-characteristics" id="array-character-scramble-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/XNn6JUVYLWaK5ss6vSSm"><code>ArrayTextMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#array-character-scramble-configure" id="array-character-scramble-configure"></a>

To configure the generator, toggle the **Consistency** setting to indicate whether to make the generator self-consistent.

By default, the generator is not consistent.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Array JSON Mask

This is a [composite generator](/app/generation/generators/generator-types/generators-composite).

A version of the [JSON Mask](/app/generation/generators/generator-reference/json-mask) generator that can be used for array values.

Runs a selected generator on values that match a user-specified [JSONPath](https://goessner.net/articles/JsonPath/index.html#e2).

## Characteristics <a href="#array-json-mask-characteristics" id="array-json-mask-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Determined by the specified sub-generators.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Determined by the specified sub-generators.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Determined by the specified sub-generators.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Determined by the specified sub-generators.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/u7EpGuc14cDWkxhIwDur"><code>ArrayJsonMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#array-json-mask-configure" id="array-json-mask-configure"></a>

### Adding a sub-generator <a href="#array-json-mask-add-sub-generator" id="array-json-mask-add-sub-generator"></a>

To assign a generator to a path expression:

1. Under **Sub-generators**, click **Add Generator**.\
   \
   On the sub-generator configuration panel, the **Cell JSON** field contains a sample value from the source database. You can use the previous and next icons to page through different values.
2. In the **Path Expression** field, type the JSONPath expression to identify the value to apply the generator to.\
   \
   To populate a path expression, you can also click a value in the **Cell JSON** field.\
   \
   **Matched JSON Values** shows the result from the value in **Cell JSON**.
3. By default, the selected generator is applied to any value that matches the expression.\
   \
   To limit the types of values to apply the generator to, from the **Type Filter**, specify the applicable types. You can select **Any**, or you can select any combination of **String**, **Number**, and **Null**.
4. From the **Generator Configuration** dropdown list, select the generator to apply to the path expression.\
   \
   You cannot select another composite generator.
5. Configure the selected generator.\
   \
   You cannot configure the selected generator to be consistent with another column.
6. To save the configuration and immediately add a generator for another path expression, click **Save and Add Another**.\
   \
   To save the configuration and close the add generator panel, click **Save**.

### Managing the sub-generator list <a href="#array-json-mask-manage-sub-generators" id="array-json-mask-manage-sub-generators"></a>

From the **Sub-Generators** list:

* To edit a generator assignment, click the edit icon.
* To remove a generator assignment, click the delete icon.
* To move a generator assignment up or down in the list, click the up or down arrow.

### Enabling data encryption <a href="#array-json-mask-data-encryption" id="array-json-mask-data-encryption"></a>

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Array Regex Mask

This is a [composite generator](/app/generation/generators/generator-types/generators-composite).

A version of the [Regex Mask](/app/generation/generators/generator-reference/regex-mask) generator that can be used for array values.

Uses regular expressions to parse strings and replace specified substrings with the output of specified generators. The parts of the string to replace are specified inside unnamed top-level capture groups.

## Characteristics <a href="#array-regex-mask-characteristics" id="array-regex-mask-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/u95DTjy0veeI9D3Jh05U"><code>ArrayRegexMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#array-regex-mask-configure" id="array-regex-mask-configure"></a>

### Adding a regular expression <a href="#array-regex-mask-add-regex" id="array-regex-mask-add-regex"></a>

To add a regular expression:

1. Click **Add Regex**.\
   \
   On the configuration panel, **Cell Value** shows a sample value from the source database. You can use the previous and next options to navigate through the values.
2. By default, **Replace all matches** is enabled.\
   \
   To only match the first occurrence of a pattern, toggle **Replace all matches** to the off position.
3. In the **Pattern** field, enter a regular expression.\
   \
   If the expression is valid, then Structural displays the capture groups for the expression.
4. For each capture group, to select and configure the generator to apply, click the selected generator.\
   \
   You cannot select another composite generator.
5. To save the configuration and immediately add a generator for another path expression, click **Save and Add Another**.\
   \
   To save the configuration and close the add generator panel, click **Save**.

### Managing the regular expressions list <a href="#array-regex-mask-manage-regex" id="array-regex-mask-manage-regex"></a>

From the **Regexes** list:

* To edit a regular expression, click the edit icon.
* To remove a regular expression, click the delete icon.


# ASCII Key

Generates unique alphanumeric strings based on any printable ASCII characters. The length of the source string is not preserved. You can choose to exclude lowercase letters from the generated values.

## Characteristics <a href="#ascii-key-characteristics" id="ascii-key-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent<strong>.</strong></td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/clFBetMsKpmmJTIuSUSN"><code>AsciiPkGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#ascii-key-configure" id="ascii-key-configure"></a>

To configure the generator:

1. To exclude lowercase letters from the generated values, toggle **Exclude Lowercase Alphabet** to the on position.
2. Toggle the **Consistency** setting to indicate whether to make the generator consistent.\
   \
   By default, the generator is not consistent.
3. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Business Name

Generates a random company name-like string.

## Characteristics <a href="#business-name-characteristics" id="business-name-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent or consistent with another column.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/qhEUntD9aZcNPS4CsMPz"><code>BusinessNameGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#business-name-configure" id="business-name-configure"></a>

### Enabling and disabling consistency

To configure the generator, toggle the **Consistency** setting to indicate whether to make the generator consistent.

By default, the generator is not consistent.

### Setting the type of consistency

If consistency is enabled, then by default it is self-consistent. To make the generator consistent with another column, from the **Consistent to** dropdown list, select the column.

* When the generator is consistent with itself, then a given source value is always mapped to the same destination value. For example, My Business is always mapped to New Business.
* When the generator is consistent with another column, then a given source value in that other column always results in the same destination value for the company name column.\
  \
  For example, if the company name column is consistent with a name column, then every instance of John Smith in the name column in the source database has the same company name in the destination database.

### Setting case sensitivity for consistency

When consistency is enabled, use the **Case-sensitive** toggle to indicate whether the consistency is case-sensitive.

By default, it is case-sensitive. For example, the values `Business1` and `BUSINESS1` are considered different values. `Business1` might always be replaced with `Example Co`, and `BUSINESS1` might be replaced with `MyCo`.

To make the consistency case-insensitive, toggle **Case-sensitive** to the off position. When the consistency is case-insensitive, `Business1` and `BUSINESS1` are considered the same value and have the same replacement.

### Enabling Structural data encryption

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Categorical

The Categorical generator shuffles the existing values within a field while maintaining the overall frequency of the values. It disassociates the values from other pieces of data. Note that NULL is considered a separate value.

For example, a column contains the values `Small`, `Medium`, and `Large`. `Small` appears 3 times, `Medium` appears 4 times, and `Large` appears 5 times. In the output data, each value still appears the same number of times, but the values are shuffled to different rows.

This generator is optimized for categories with fewer than 10,000 unique values. If your underlying data has more unique values (for example, your field is populated by freeform text entry), we recommend that you use the [Character Scramble](/app/generation/generators/generator-reference/character-scramble) or [Custom Categorical](/app/generation/generators/generator-reference/custom-categorical) generator.

## Characteristics <a href="#categorical-characteristics" id="categorical-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Configurable</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>2 if differential privacy enabled</li><li>3 if differential privacy not enabled</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/guL2oBjIWsY1KUKFWzNe"><code>CategoricalGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#categorical-configure" id="categorical-configure"></a>

To configure the generator:

1. From the **Link To** dropdown, select the columns to link to the current column.\
   \
   You can select from other columns that use the Categorical generator.
2. Toggle the **Differential Privacy** setting to indicate whether to make the output data differentially private.\
   \
   By default, differential privacy is disabled.
3. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Character Scramble

This generator replaces letters with random other letters and numbers with random other numbers. Punctuation, whitespace, and mathematical symbols are preserved.

For example, for the following input string:

`ABC.123 123-456-789 Go!`

The output would be something like:

`PRX.804 296-915-378 Ab!`

This generator securely masks letters and numbers. There is no way to recover the original data.

Character Scramble is similar to [Character Substitution](/app/generation/generators/generator-reference/character-substitution), with a couple of key differences.

While you can enable consistency for the entire value, Character Scramble does not always replace the same source character with the same destination character. Because there is no guarantee of unique values, you cannot use Character Scramble on unique columns.

Character Substitution, however, does always map the same source character to the same destination character. Character Substitution is always consistent, which makes it less secure than Character Scramble. You can use Character Substitution on unique columns.

## Characteristics <a href="#character-scramble-characteristics" id="character-scramble-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/rkl8AFhgEjfzTZnuNTya"><code>TextMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#character-scramble-how-to-configure" id="character-scramble-how-to-configure"></a>

To configure the generator, toggle the **Consistency** setting to indicate whether to make the generator self-consistent.

By default, the generator is not consistent.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Character Substitution

Performs a random character replacement that preserves formatting (spaces, capitalization, and punctuation).

Characters are replaced with other characters from within the same Unicode Block. A given source character is always mapped to the same destination character. For example, `M` might always map to `V`.

For example, for the following input string:

`Miami Store #162`

The output would be something like:

`Vgkjg Gmlvf #681`

Note that for a numeric column, when a generated number starts with a 0, the starting 0 is removed. This could result in matching output values in different columns. For example, one column is changed to 113 and the other to 0113, which also becomes 113.

Character Substitution is similar to [Character Scramble](/app/generation/generators/generator-reference/character-scramble), with a couple of key differences. Because Character Substitution always maps the same source character to the same destination character, it is always consistent. It also can be used for unique columns.

In Character Scramble, the character mapping is random, which makes Character Scramble slightly more secure. However, Character Scramble cannot be used for unique columns.

## Characteristics <a href="#character-substitution-characteristics" id="character-substitution-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">This generator is implicitly self-consistent. You do not specify whether the generator is consistent.<br><br>Every occurrence of a character always maps to the same substitute character.<br><br>Because of this, it can be used to preserve a join between two text columns, such as a join on a name or email.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">4</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/fy8n3fsQgTSDMuuKuSIt"><code>StringMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#character-substitution-configure" id="character-substitution-configure"></a>

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Company Name

{% hint style="info" %}
This generator is deprecated. Use the [Business Name](/app/generation/generators/generator-reference/business-name) generator instead.
{% endhint %}

Generates a random company name-like string.

## Characteristics <a href="#company-name-characteristics" id="company-name-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent or consistent with another column.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/QDHR5WXFUdZvlhWJHA08"><code>CompanyNameGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#company-name-configure" id="company-name-configure"></a>

To configure the generator, toggle the **Consistency** setting to indicate whether to make the generator consistent.

By default, the generator is not consistent.

If consistency is enabled, then by default it is self-consistent. To make the generator consistent with another column, from the **Consistent to** dropdown list, select the column.

When the generator is consistent with itself, then a given source value is always mapped to the same destination value. For example, My Company is always mapped to New Company.

When the generator is consistent with another column, then a given source value in that other column always results in the same destination value for the company name column. For example, if the company name column is consistent with a name column, then every instance of John Smith in the name column in the source database has the same company name in the destination database.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Conditional

This is a [composite generator](/app/generation/generators/generator-types/generators-composite).

Applies different generators to the value conditionally based on any value in the table.

For example, a Users table contains Name, Username, and Role columns. For the Username column, you can use a conditional generator to indicate that if the value of Role is something other than Test, then use the Character Scramble generator for the Username value. For Test users, the name is not masked.

You can also create conditions against the current column. For example, if the value is less than 4, use the Random Integer generator to generate a replacement value between 0 and 4. If the value is greater than or equal to 5, use the Random Integer generator to generate a value between 5 and 10.

## Characteristics <a href="#conditional-characteristics" id="conditional-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Determined by the selected generators.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Determined by the selected generators.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Determined by the selected generators.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Determined by the selected generators.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top"><p>Yes, but:</p><ul><li>Make sure that the configuration preserves uniqueness.</li><li>Do not use on primary key columns that are used for subsetting.</li></ul></td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>If a fallback generator is selected, then the lower of either 5 or the fallback generator.</li><li>5 if no fallback generator is selected</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/MVk7kwA8JmL2rTTe90xI"><code>ConditionalGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#conditional-configure" id="conditional-configure"></a>

The generator consists of a list of options. Each option includes the required conditions and the generator to use if those conditions are met.

### Setting the default generator <a href="#conditional-default-generator" id="conditional-default-generator"></a>

The generator always contains a **Default** option. The **Default** option is used if the value does not meet any of the conditions. To configure the **Default** option:

1. From the **Default** dropdown list, select the generator to use by default.
2. Configure the selected generator.

### Adding a condition option <a href="#conditional-add-condition" id="conditional-add-condition"></a>

To add a condition option:

1. Click **+ Conditional Generator**.
2. To add a condition:

   1. Click **+ Condition**.
   2. From the column list, select the column for which to check the value.\
      \
      To check the value of the current column, select **This field**.
   3. Select the comparison type.
   4. Enter the column value to check for.

   To remove a condition, click the delete icon for the condition.
3. From the **Generator** dropdown list, select the generator to run on the current column if the conditions are met.\
   \
   You cannot select another composite generator.
4. Choose the configuration options for the selected generator.

### Viewing and editing condition options <a href="#conditional-view-edit-condition" id="conditional-view-edit-condition"></a>

To view details for and edit a condition option, click the expand icon for that option.

### Removing a condition option <a href="#conditional-remove-condition" id="conditional-remove-condition"></a>

To remove a condition option, click the delete icon for the option.


# Constant

Uses a single value to mask all of the values in the column.

For example, you can replace every value in a string column with the value `String1`. Or you can replace every value in a numeric column with the value `12345`.

## Characteristics <a href="#constant-characteristics" id="constant-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">1</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/7YhXuxNZqKjSPVkhcF0a"><code>ConstantGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#constant-configure" id="constant-configure"></a>

To configure the generator, in the **Constant Value** field, provide the value to use.

The value must be compatible with the field type. For example, you cannot provide a string value for an integer column.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Continuous

Generates a continuous distribution to fit the underlying data.

This generator can be linked to other Continuous generators to create multivariate distributions and can be partitioned by other columns.

## Characteristics <a href="#continuous-characteristics" id="continuous-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Configurable</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>2 if differential privacy enabled</li><li>3 if differential privacy not enabled</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/ibZpgFFKaeYJgeBahlkY"><code>GaussianGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#continuous-configure" id="continuous-configure"></a>

To configure the generator:

1. From the **Link To** drop-down list, select the other Continuous generator columns to link to.\
   \
   The linking creates a multivariate distribution.
2. From the **Partition By** drop-down list, select one or more columns to use to partition the data.\
   \
   The selected columns must have the generator set to either Passthrough or Categorical.\
   \
   For more information about partitioning and how it works, go to [Partitioning a column](/app/generation/generators/generator-characteristics/partitioning).
3. Toggle the **Differential Privacy** setting to indicate whether to make the output data differentially private.\
   \
   By default, the generator is not differentially private.
4. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Cross Table Sum

Links columns in two tables. This column value is the sum of the values in a column in another table.

This generator does not provide a preview. The sums are not computed until the other table is generated.

For example, a **Customers** table contains a **Total\_Sales** column. The **Transactions** table uses a foreign key **Customer\_ID** column to identify the customer who made the transaction, and an **Amount** column that contains the amount of the sale. The **Customer\_ID** value in the **Transactions** table is a value from the **ID** primary key column in the **Customers** table.

You assign the Cross Table Sum generator to the **Total\_Sales** column. In the generator configuration, you indicate that the value is the sum of the **Amount** values for the **Customer\_ID** value that matches the primary key **ID** value for the current row.

For the **Customers** row for ID `123`, the **Total\_Sales** column contains the sum of the **Amount** column for **Transactions** rows where **Customer\_ID** is `123`.

## Characteristics <a href="#cross-table-sum-characteristics" id="cross-table-sum-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">3</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/OnB96XREUMxGohHqnbuN"><code>CrossTableAggregateGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#cross-table-sum-configure" id="cross-table-sum-configure"></a>

To configure the generator:

1. From the **Foreign Table** dropdown list, select the table that contains the column for which to sum the values.
2. From the **Foreign Key** dropdown list, select the foreign key.\
   \
   The foreign key identifies the row from the current table that is referred to in the foreign table.
3. From the **Sum Over** dropdown list, select the column for which to sum the values.
4. From the **Primary Key** dropdown list, select the primary key for the current table.
5. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# CSV Mask

This is a [composite generator](/app/generation/generators/generator-types/generators-composite).

Masks text columns by parsing the values as rows whose columns are delimited by a specified character.

You can assign specific generators to specific indexes. You can also use the generator that is assigned to a specific index as the default. This applies the generator to every index that does not have an assigned generator.

The output value maintains the quotes around the index values.

For example, a column contains the following value:

`"first","second","third"`

You assign the Character Scramble generator to index 0 and assign Passthrough to index 2. You select index 0 as the index to use for the default generator.

In the output, the first and second values are masked by the Character Scramble generator. The third value is not masked. The output looks something like:

`"wmcop", "xjorsl", "third"`

## Characteristics <a href="#csv-mask-characteristics" id="csv-mask-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/OFKSQZQ52dE7G3mLzOkC"><code>CsvMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#csv-mask-configure" id="csv-mask-configure"></a>

### Setting the delimiter <a href="#csv-mask-delimiter" id="csv-mask-delimiter"></a>

In the **Delimiter** field, type the delimiter that is used as a separator in the value.

For example, for the value `"first","second","third"`, the delimiter is a comma.

### Adding a sub-generator <a href="#csv-mask-add-sub-generator" id="csv-mask-add-sub-generator"></a>

You can configure a generator for any or all of the indexes. To add a sub-generator for an index:

1. Under **Sub-Generators**, click **Add Generator**.\
   \
   On the add generator dialog, the **Cell CSV** field contains a sample value from the source data. You can use the navigation icons to page through the values.
2. In the **CSV Index** field, type the index to assign a generator to.\
   \
   The index numbers start with 0.\
   \
   You cannot use an index that already has an assigned generator.\
   \
   **Matched CSV values** shows the value at that index for the current sample column value.
3. Under **Generator Configuration**, from the **Select a Generator** dropdown list, select the generator to use for the selected index.\
   \
   You cannot select another composite generator.\
   \
   To remove the selection, click the delete icon.
4. Configure the selected generator.\
   \
   You cannot configure the selected generator to be consistent with another column.
5. To save the configuration and immediately add a generator for another index, click **Save and Add Another**.\
   \
   To save the configuration and close the add generator panel, click **Save**.

### Managing the sub-generator list <a href="#csv-mask-manage-sub-generators" id="csv-mask-manage-sub-generators"></a>

From the **Sub-Generators** list:

* To edit a generator assignment, click the edit icon.
* To remove a generator assignment, click the delete icon.
* To move a generator assignment up or down in the list, click the up or down arrow.

### Setting the default for indexes without a generator <a href="#csv-mask-default-generator-index" id="csv-mask-default-generator-index"></a>

After you configure a generator for at least one index, the **Default Link** dropdown list is displayed.

From the **Default Link** dropdown list, select the index to use to determine how to mask values for indexes that do not have an assigned generator.

For example, you assign the Character Scramble generator to index 2. If you set **Default Link** to 2, then all indexes that do not have an assigned generator use the Character Scramble generator.


# Custom Categorical

A version of the [Categorical](/app/generation/generators/generator-reference/categorical) generator that selects from values that you provide instead of shuffling the original values.

## Characteristics <a href="#custom-categorical-characteristics" id="custom-categorical-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent or consistent with another column.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/AsyoJDj5DdRAZoieTjp5"><code>CustomCategoricalGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#custom-categorical-configure" id="custom-categorical-configure"></a>

### Linking the column

From the **Link To** dropdown list, select the columns to link this column to.

You can only select other columns that use the Custom Categorical generator.

### **Providing the values to use**

In the **Custom Categories** text area, provide the list of values that the generator can choose from.

To provide the values, you can either:

* Enter the values manually
* Provide an AI prompt to populate the values (Structural Cloud only)

#### **Entering the values manually**

When you enter the values manually, put each value on a separate line.

To add a NULL value to the list, use the keyword `{NULL}`.

#### **Providing an AI prompt**

{% hint style="info" %}
Only available on the Structural Cloud instance that is hosted in the United States, and on self-hosted instances that [enable Structural AI features](/app/admin/structural-ai-use/self-hosted-llm-configuration). Not available on the European instance of Structural Cloud.
{% endhint %}

To use an AI prompt to create the values:

1. In the AI prompt field below the **Custom Categories** text area, type the prompt to use to create the values.\
   \
   The prompt can include the number of values to create. For example, `10 names of flowers` or `20 cities in California`.\
   \
   If the prompt does not include a number, then Structural determines a reasonable set of values to generate based on the prompt. If there is a very limited set of values, then Structural often generates the full set of values. Otherwise it attempts to generate a reasonable number of values, usually between 10 and 20.
2. Press **Enter** or click the add values icon.

The values replace any existing values in the list.

After you use the prompt to create a set of values, you can edit the list manually.

For information about how Structural uses AI, go to [AI in Structural](/app/admin/structural-ai-use).

### **Configuring consistency**

Toggle the **Consistency** setting to indicate whether to make the column consistent.

By default, consistency is disabled.

If you enable consistency, then by default the generator is self-consistent.

To make the generator consistent with another column, from the **Consistent to** dropdown list, select the column.

When a generator is self-consistent, then a given value in the source database is always mapped to the same value in the destination database.

When a generator is consistent with another column, then a given source value in that column always results in the same value for the current column in the destination database. For example, a department column is consistent with a username column. For each instance of User1 in the source database, the value in the department column is the same.

### Enabling Structural data encryption

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Date Truncation

Truncates a date value or a timestamp to a specific part.

For a date or a timestamp, you can truncate to the year, month, or day.

For a timestamp, you can also truncate to the hour, minute, or second.

## Characteristics <a href="#date-truncation-characteristics" id="date-truncation-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/sIv7iAJxM0EqwLdkBhVc"><code>DateTruncationGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#date-truncation-configure" id="date-truncation-configure"></a>

To configure the generator:

1. From the dropdown list, select the part of the date or timestamp to truncate to.\
   \
   For both date and timestamp values, you can truncate to the year, month, or day. When you select one of these options, the time portion of a timestamp is set to 00:00:00. For the date, the values below the selected truncation value are set to 01. For example, when you truncate to month, the day value is set to 01, and the timestamp is set to 00:00:00.\
   \
   For a timestamp value, you also can truncate to the hour, minute, or second. The date values remain the same as the original data. The time values below the selected truncation value are set to 00. For example, when you truncate to minute, the seconds value is set to 00.
2. Toggle the **Birth Date** option.\
   \
   When you enable **Birth Date**, the generator shifts dates that are more than 90 years before the generation date to the date exactly 90 years before the generation date.\
   \
   For example, data generation occurs on January 1, 2023. Any date that occurs before January 1, 1933 is changed to January 1, 1933.<br>

   This is mostly intended for birthdate values, to group birthdates for everyone who is older than 89 into a single year. This is used to comply with HIPAA Safe Harbor.
3. From the **Fallback Generator** dropdown list, select how to handle values that the generator cannot process.\
   \
   By default, this is set to **Fail on error**, meaning that the generation fails when a value cannot be processed.\
   \
   You can instead assign a fallback generator, either:
   1. Passthrough, to pass through the value without changing it.
   2. Constant, to use a replacement value that you provide.
   3. Null, to null out the value.&#x20;
4. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.

## Truncation examples <a href="#date-truncation-examples" id="date-truncation-examples"></a>

Here are examples of date and time values and how the selected truncation affects the output:

<table><thead><tr><th valign="top">Option</th><th valign="top">Date value</th><th valign="top">Timestamp value</th></tr></thead><tbody><tr><td valign="top">Original value</td><td valign="top">2021-12-20</td><td valign="top">2021-12-20 13:42:55</td></tr><tr><td valign="top">Truncate to year</td><td valign="top">2021-01-01</td><td valign="top">2021-01-01 00:00:00</td></tr><tr><td valign="top">Truncate to month</td><td valign="top">2021-12-01</td><td valign="top">2021-12-01 00:00:00</td></tr><tr><td valign="top">Truncate to day</td><td valign="top">2021-12-20</td><td valign="top">2021-12-20 00:00:00</td></tr><tr><td valign="top">Truncate to hour</td><td valign="top">Not applicable</td><td valign="top">2021-12-20 13:00:00</td></tr><tr><td valign="top">Truncate to minute</td><td valign="top">Not applicable</td><td valign="top">2021-12-20 13:42:00</td></tr><tr><td valign="top">Truncate to second</td><td valign="top">Not applicable</td><td valign="top">2021-12-20 13:42:55</td></tr></tbody></table>


# Email

Scrambles the characters in an email address. It preserves formatting and keeps the `@` and `.` characters.

For example, for the following input value:

`johndoe@company.com`

The output value would be something like:

`brwomse@xorwxlt.slt`

By default, the generator scrambles the domain. You can configure the generator to not mask specific domains. You can also specify a domain to use for all of the output email addresses.

For example, if you configure the generator to not scramble the domain `company.com`, then the output for `johndoe@company.com` would look something like:

`brwomse@company.com`

This generator securely masks letters and numbers. There is no way to recover the original data.

If your email addresses include name values - for example, <John.Smith@mycompany.com> - then you can use the Regex Mask generator to produce email addresses that are tied to name values in the same table. For information on how to do this, go to [Generator hints and tips](/app/generation/generators-assign-config/common-usage#generator-tips-email-name-alignment).

## Characteristics <a href="#email-characteristics" id="email-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/cSiodmqjXSXTjm030plR"><code>EmailGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#email-configure" id="email-configure"></a>

To configure the generator:

1. In the **Email Domain** field, enter a domain to use for all of the output values.\
   \
   For example, use `@mycompany.com` for all of the generated values. The generator scrambles the content before the `@`.
2. In the **Excluded Email Domains** field, enter a comma-separated list of domains for which email addresses are not masked in the output values.\
   \
   This allows you, for example, to maintain internal or testing email addresses that are not considered sensitive.
3. Toggle the **Replace invalid emails** setting to indicate whether to replace an invalid email address with a generated valid email address.\
   \
   By default, invalid email addresses are not replaced.\
   \
   In the replacement values, the username is generated. If you specify a value for **Email Domain**, then the email addresses use that domain. Otherwise, the domain is generated.
4. Toggle the **Consistency** setting to indicate whether to make the column self-consistent.\
   \
   By default, consistency is disabled.
5. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Event Timestamps

Generates timestamps that fit an event distribution. The source timestamp must include a date. It cannot be a time-only value.

Link columns to create a sequence of events across multiple columns. This generator can be partitioned by other columns.

## Characteristics <a href="#event-timestamps-characteristics" id="event-timestamps-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">3</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/zbWO545ha1u50UdO0L13"><code>EventGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#event-timestamps-configure" id="event-timestamps-configure"></a>

To configure the generator:

1. From the **Link To** dropdown list, select the other Event Timestamps generator columns to link this column to.\
   \
   Linking creates a sequence across multiple columns.
2. From the **Partition** drop-down list, select one or more columns to use to partition the data.\
   \
   The selected columns must have their generator set to either Passthrough or Categorical.\
   \
   For more information about partitioning and how it works, go to [Partitioning a column](/app/generation/generators/generator-characteristics/partitioning).
3. The **Options** list displays the current column and linked columns.\
   \
   Use the **Up** and **Down** buttons to configure the column sequence.
4. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# File Name

This generator scrambles characters, but preserves formatting and keeps the file extension intact.

For example, for the following input value:

`DataSummary1.pdf`

The output value would look something like:

`RsnoPwcsrtv5.pdf`

This generator securely masks letters and numbers. There is no way to recover the original data.

## Characteristics <a href="#file-name-characteristics" id="file-name-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/mRQCOU8teo52JgDaEG0t"><code>FileNameGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#file-name-configure" id="file-name-configure"></a>

To configure the generator, toggle the **Consistency** setting to indicate whether to make the generator self-consistent.

By default, the generator is not consistent.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Find and Replace

This generator replaces all instances of the find string with the replace string.

For example, you can indicate to replace all instances of `abc` with `123`.

## Characteristics <a href="#find-and-replace-characteristics" id="find-and-replace-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/TepUJR3tP7VCvvztI8mZ"><code>FindAndReplaceGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#find-and-replace-configure" id="find-and-replace-configure"></a>

To configure the generator:

1. In the **Find** field, type the string to look for in the source column value.\
   \
   To use a regular expression to identify the source value, check the **Use Regex** checkbox.\
   \
   If you use a regular expression, use backslash ( `\` ) as the escape character.
2. In the **Replace** field, type the string to replace the matching string with.
3. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Finnish Personal Identity Code

Generates a valid Finnish Personal Identity Code (PIC) that would have been issued during a specific date range.

## Characteristics

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No, cannot be made differentially private.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/7J0bQTHx2gdp1DQKiwIC">FinnishPicGenerator</a></td></tr></tbody></table>

## How to configure

To configure the generator:

1. Under **Date Range**, set the start and date for the date range to generate the PICs for.
2. Toggle the **Consistency** setting to indicate whether to make the generator self-consistent.\
   \
   By default, the generator is not consistent.
3. If Structural data encryption is enabled, then to use it for this column, toggle **Use data encryption process** to the on position.


# FNR

The FNR generator transforms Norwegian national identity numbers. In Norwegian, the term for national identity number abbreviates to FNR.

The first six digits of an FNR reflects the person's birthdate. You can choose to preserve the birthdates from the source values in the destination values. If you do not preserve the source values, the destination values are still within the same date range as the source values.

Another digit in an FNR indicates whether the person is male or female. You can specify whether to preserve in the generated value the gender indicated in the source value.

The last digits in an FNR are a checksum value. The last digits in the destination value are not a checksum - the values are random.

## Characteristics <a href="#fnr-characteristics" id="fnr-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent or consistent with another column.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/YCgMnGPO7rJsflQ8v92J"><code>FnrGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#fnr-configure" id="fnr-configure"></a>

To configure the generator:

1. To preserve the gender from the source value in the destination value, toggle **Preserve Gender** to the on position.
2. To preserve the birthdate from the source value in the destination value, toggle **Preserve Birthdate** to the on position.
3. Toggle the **Consistency** setting to indicate whether to make the generator consistent.\
   \
   By default, consistency is disabled.
4. If you enable consistency, then by default the generator is self-consistent.\
   \
   To make the generator consistent with another column, from the **Consistent to** dropdown list, select the column.\
   \
   When a generator is self-consistent, then a given value in the source database is always mapped to the same value in the destination database.\
   \
   When a generator is consistent with another column, then a given value for that other column in the source database results in the same value in the destination database. For example, if the FNR column is consistent with a Name column, then every instance of John Smith in the source database results in the same FNR in the destination database.
5. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# Geo

This generator can be used to mask columns of latitude and longitude.

The Geo generator divides the globe into grids that are approximately 4.9 x 4.9 km. It then counts the number of points within each grid.

During data generation, each (latitude, longitude) pair is mapped to its grid.

* If the grid contains a sufficient number of points to preserve privacy, then the generator returns a randomly chosen point in that grid.
* If the grid does not contain enough points to preserve privacy, then the generator returns a random coordinate from the nearest grid that contains enough points.

## Characteristics <a href="#geo-characteristics" id="geo-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">No, cannot be made consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">Yes</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">3</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/Yrd7JmcySjDcAI9aHAkh"><code>GeoGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#geo-configure" id="geo-configure"></a>

To configure the generator:

1. From the **Link To** dropdown list, select the column to link to this one.\
   \
   You typically assign the Geo generator to both the latitude and longitude column, then link those columns.
2. From the value type dropdown, select whether this column contains a latitude value or a longitude value.
3. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# HIPAA Address

This generator can be used to generate cities, states, and zip codes that follow HIPAA guidelines for safe harbor.

## Handling of address parts <a href="#hipaa-address-parts" id="hipaa-address-parts"></a>

### **Zip codes** <a href="#hipaa-address-zip-codes" id="hipaa-address-zip-codes"></a>

How the HIPAA Address generator handles zip codes is based on whether the **Replace zeros in truncated Zip Code** toggle in the generator configuration is off or on.

By default, the setting is off. In this case, the last two digits of the zip code in the column are replaced with zeros, unless the zip code is a *low population* area as designated by the current census. For a low population area, all of the digits in the zip code are replaced with zeros.

If the setting is on, then the generator selects a real zip code that starts with the same three digits as the original zip code. For a low population area, if a state is linked, then the generator selects a random zip code from within that state. Otherwise the generator selects a random zip code from the United States.

### **Cities** <a href="#hipaa-address-cities" id="hipaa-address-cities"></a>

When a zip code column is not linke&#x64;*,* a random city is chosen in the United States. When a zip code is already added to the link, a city is chosen at random that has at least some overlap with the zip code.

If the original zip code is designated as a *low population* area, then a random city is chosen within the state. This is done only if the user has linked a State column. If they have not, a random city within the United States is chosen.

For example, if the original city and zip code are (Atlanta, 30305), the zip code would be replaced with 30300. Many cities contain zip codes that begin in 303, such as Atlanta, Decatur, Chamblee, Hapeville, Dunwoody, and College Park. One of these cities is chosen at random so that, for example, the final value is (Chamblee, 30300).

### **States** <a href="#hipaa-address-states" id="hipaa-address-states"></a>

HIPAA guidelines allow for information at the state level to be kept. Therefore, these values are passed through.

### **Latitude and longitude (GPS) coordinates** <a href="#hipaa-address-lat-long" id="hipaa-address-lat-long"></a>

GPS coordinates are randomly generated in descending order of dependence of the linked HIPAA address components:

1. If a zip code is linked, a random point within the same 3-digit zip code prefix is generated, if the 3-digit zip code prefix is not designated a low population area. If it is a low population area, use the linked state.
2. If a state is available and a zip code and city are not, or the zip code or city are in a 3-digit zip code prefix that is designated a low population area, then a random GPS coordinate is generated somewhere within the state.
3. If no zip code, city, or state is linked, or one or more of them were provided, but there was a problem generating a random GPS coordinate within the linked areas, then a GPS coordinate is generated at a random location within the United States.

**Note:** If the city component of the HIPAA address is linked with latitude and/or longitude, the GPS coordinate components are randomly generated independently of the city.

### **Other address parts** <a href="#hipaa-address-other-parts" id="hipaa-address-other-parts"></a>

All other address parts are generated randomly. The output value is not influenced at all by the underlying value in the column.

## Characteristics

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Yes, can be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>3 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/Myr5CNhdSaHmsjdbfwOQ"><code>HipaaAddressGenerator</code></a></td></tr></tbody></table>

## How to configure

To configure the generator:

1. From the **Link To** dropdown list, select the other columns to link to.\
   \
   You can only select columns that are also assigned the HIPAA Address generator.
2. From the address part dropdown list, select the type of address value that is in the column.
3. Toggle the **Replace zeros in truncated Zip Code** setting how to generate zip codes.\
   \
   If the setting is off, then the last two digits are replaced with zero. For low population areas, the entire zip code is populated with zeroes.\
   \
   If the setting is on, then a real zip code is selected that starts with the first three digits of the original zip code. For low population areas, if a state is linked, a random zip code from the state is used. Otherwise, a random zip code from the United States is used.
4. Toggle the **Consistency** setting to indicate whether to make the column self-consistent.\
   \
   By default, consistency is disabled.
5. When consistency is enabled, use the **Case-sensitive** toggle to indicate whether the consistency is case-sensitive.\
   \
   By default, consistency is case-sensitive. For example, the values `Anytown` and `ANYTOWN` are considered different values. `Anytown` might always be replaced with `Chicago`, and `ANYTOWN` might be replaced with `Atlanta`.\
   \
   To make the consistency case-insensitive, toggle **Case-sensitive** to the off position. When the consistency is case-insensitive, `Anytown` and `ANYTOWN` are considered the same value and have the same replacement.
6. If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.

## Spark supported address parts

For the HIPAA Address generator, Spark workspaces (Databricks and self-managed Spark clusters) only support the following address parts:

* City
* City with State
* City with State Abbr
* State
* State Abbr
* US Address
* US Address with Country
* Zip Code

The [Address generator](/app/generation/generators/generator-reference/address) provides support for additional address parts in Spark workspaces.


# Hostname

Generates random host names, based on the English language.

## Characteristics <a href="#hostname-characteristics" id="hostname-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Yes, can be made self-consistent or consistent with another column.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">No, cannot be linked.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Yes, if consistency is not enabled.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top"><ul><li>1 if not consistent</li><li>4 if consistent</li></ul></td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/9QUF3jFy70V79ZS4JBiw"><code>HostnameGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#hostname-configure" id="hostname-configure"></a>

To configure the generator, toggle the **Consistency** setting to indicate whether to make the generator consistent.

By default, the generator is not consistent.

If you enable consistency, then by default the generator is self-consistent. To make the generator consistent with another column, from **Consistent to**, select the column.

When the generator is consistent with itself, then a given value in the source database is mapped to the same value in the destination database. For example, Host123 in the source database always produces MyHostABC in the destination database.

When the generator is consistent with another column, then a given source value in the other column results in the same host name value in the destination database. For example, a host name column is consistent with a department column. Every instance of Sales in the source data is given the same host name in the destination database.

If [Structural data encryption](/app/generation/generators-assign-config/generators-data-encryption-config) is enabled, then to use it for this column, in the advanced options section, toggle **Use data encryption process** to the on position.


# HStore Mask

This is a [composite generator](/app/generation/generators/generator-types/generators-composite).

Runs selected generators on specified key values in an HStore column in a PostgreSQL database. HStore columns contain a set of key-value pairs.

## Characteristics <a href="#hstore-mask-characteristics" id="hstore-mask-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/G07RJPVa3OJP9RFhquty"><code>HStoreMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#hstore-mask-configure" id="hstore-mask-configure"></a>

### Adding a sub-generator <a href="#hstore-mask-add-subgenerator" id="hstore-mask-add-subgenerator"></a>

To assign a generator to a key:

1. Under **Sub-generators**, click **Add Generator**.\
   \
   On the sub-generator configuration panel, the **Cell HStore** field contains a sample value from the source database. You can use the previous and next icons to page through different values.
2. Under **Enter a key**, enter the name of a key from the column value.\
   \
   For example, for the column value:\
   \
   &#x20;`"pages"=>"446", "title"=>"The Iliad", "category"=>"mythology"`\
   \
   To apply a generator to the title, you would enter `title` as the key.\
   \
   **Matched HStore Values** shows the result from the value in **Cell HStore**.
3. From the **Generator Configuration** dropdown list, select the generator to apply to the key value.\
   \
   You cannot select another composite generator.
4. Configure the selected generator.\
   \
   You cannot configure the selected generator to be consistent with another column.
5. To save the configuration and immediately add a generator for another key, click **Save and Add Another**.\
   \
   To save the configuration and close the add generator panel, click **Save**.

### Managing the sub-generators list <a href="#hstore-mask-manage-sub-generators" id="hstore-mask-manage-sub-generators"></a>

From the **Sub-Generators** list:

* To edit a generator assignment, click the edit icon.
* To remove a generator assignment, click the delete icon.
* To move a generator assignment up or down in the list, click the up or down arrow.


# HTML Mask

This is a [composite generator](/app/generation/generators/generator-types/generators-composite).

Masks text columns by parsing the contents as HTML, and applying sub-generators to specified path expressions.

If applying a sub-generator fails because of an error, the generator selected as the fallback generator is applied instead.

Path expressions are defined using the [XPath syntax](https://www.w3schools.com/xml/xpath_syntax.asp).

For example, for the following HTML:

```html
<html>
<body>
  <div class="container">
    <h1>Title</h1>
    <p>Paragraph content</p>
    <ul>
      <li>Item 1</li>
      <li>Item 2</li>
      <li>Item 3</li>
    </ul>
  </div>
</body>
</html>
```

To get the value of `h1`, the expression is `//h1/text(`).

To get the value of the first list item, the expression is `//ul/li[1]/text()`.

## Characteristics <a href="#html-mask-characteristics" id="html-mask-characteristics"></a>

<table data-header-hidden><thead><tr><th valign="top"></th><th valign="top"></th></tr></thead><tbody><tr><td valign="top"><strong>Consistency</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Linking</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Differential privacy</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Data-free</strong></td><td valign="top">Determined by the selected sub-generators.</td></tr><tr><td valign="top"><strong>Allowed for primary keys</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Allowed for unique columns</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Uses format-preserving encryption (FPE)</strong></td><td valign="top">No</td></tr><tr><td valign="top"><strong>Privacy ranking</strong></td><td valign="top">5</td></tr><tr><td valign="top"><strong>Generator ID (for the API)</strong></td><td valign="top"><a href="/pages/xGkzP2rFfjHAPN0MmnOb"><code>HtmlMaskGenerator</code></a></td></tr></tbody></table>

## How to configure <a href="#html-mask-configure" id="html-mask-configure"></a>

### Adding a sub-generator <a href="#html-mask-add-sub-generator" id="html-mask-add-sub-generator"></a>

To assign a generator to a path expression:

1. Under **Sub-generators**, click **Add Generator**.\
   \
   On the sub-generator configuration panel, the **Cell HTML** field contains a sample value from the source database. You can use the previous and next icons to page through different values.
2. In the **Path Expression** field, type the path expression to identify the value to apply the generator to.\
   \
   **Matched HTML Values** shows the result from the value in **Cell HTML**.
3. From the **Generator Configuration** dropdown list, select the generator to apply to the path expression.\
   \
   You cannot select another composite generator.
4. Configure the selected generator.\
   \
   You cannot configure the selected generator to be consistent with another column.
5. To save the configuration and immediately add a generator for another path expression, click **Save and Add Another**.\
   \
   To save the configuration and close the add generator panel, click **Save**.

### Managing the sub-generators list <a href="#html-mask-manage-sub-generators" id="html-mask-manage-sub-generators"></a>

From the **Sub-Generators** list:

* To edit a generator assignment, click the edit icon.
* To remove a generator assignment, click the delete icon.
* To move a generator assignment up or down in the list, click the up or down arrow.

### Selecting the fallback generator <a href="#html-mask-fallback-generator" id="html-mask-fallback-generator"></a>

From the **Fallback Generator** dropdown list, select the generator to use if the assigned generator for a path expression fails.

The options are:

* [Passthrough](/app/generation/generators/generator-reference/passthrough)
* [Constant](/app/generation/generators/generator-reference/constant)
* [Null](/app/generation/generators/generator-reference/null)




---

[Next Page](/llms-full.txt/1)