Schema management settings

On Workspace Settings view for a workspace, the Schema Changes section contains the schema management settings..

Schema management settings for a workspace

Responding to schema changes

Schema changes include:

• Schema changes that could expose data, which if not addressed can result in data leakage. These changes include new tables and columns, and changes to data types.

• Notifications, which Structural can handle automatically during each data generation. These include removed tables and columns.

For more information, go to Viewing and resolving schema changes.

Selecting the handling option

On the Workspace Settings view, under Block Data Generation on Schema Changes, select how Structural responds when there are unaddressed changes to the database schema.

The options are:

• Do Not Block - With this option, schema changes never block data generation. When you select this option, then Structural automatically handles notifications during data generation.

  The Automatically apply generators toggle determines how Structural responds to changes that could expose data.

  • When this is enabled, then during data generation, Structural automatically applies generators to the affected columns.

  • When it is disabled, Structural does not change the column configuration.

• Block On Changes That Could Expose Data - Indicates to only block data generation if there are schema changes that might expose data, such as new columns. Structural automatically handles notifications during data generation. For this option, Structural does not block data generation for schema changes on truncated tables.

• Block On All Changes - For this option, if there are any unaddressed schema changes at all, either sensitive changes or notifications, then data generation fails.

How Structural selects generators to apply automatically

When Structural automatically applies generators to columns affected by a schema change:

• For new columns or document fields that are detected as sensitive, Structural applies the recommended generator for the sensitivity type.

• For new columns or document fields that are not sensitive, Structural applies an appropriate generator based on the data type.

• For changes to column data types, nullability, or uniqueness, Structural applies a new recommended generator or an appropriate generator for the data type.

Here is a summary of how Structural determines the generator to apply to columns that are not sensitive and do not match an existing sensitivity type or custom sensitivity rule:

Data type	Generator selection
Integer	If unique or a primary key, Integer Key. Otherwise Random Integer.
Boolean	Random Boolean
Datetime	Either Timestamp Shift, Date Truncation, or Random Timestamp
Text	If unique or a primary key, either Aphanumeric String Key, ASCII Key, or Numeric String Key. Otherwise either Character Scramble or Categorical.
Continuous	If unique or a primary key, Integer Key. Otherwise Random Double.
Network	IP Address
MAC address	MAC Address
UUID	If unique or a primary key, UUID Key. Otherwise Random UUID.
JSON	If Struct format, Struct Mask. Otherwise JSON Mask.
XML	Either XML Mask or HTML Mask
Other	If HStore format, HStore Mask
User-defined	Categorical

Indicating whether to cache the source schema

Schema caching is not available for document-based databases (MongoDB, DynamoDB).

About the schema cache

By default, every time you load a workspace, Structural queries the source database to retrieve the schema.

You can instead configure the workspace to cache the schema. Structural then updates the cache at a regular interval, and whenever a change to the workspace triggers a schema cache update.

You can also trigger a cache update manually.

By default, the schema cache is only used by calls from within Structural. To enable an external API request to use the cached schema, add the query parameter useSchemaCache=true to the request.

In the application, each update to the schema cache is represented by a schema retrieval job. Schema retrieval jobs are short-lived, and run on the Structural web server. You can view the schema retrieval jobs from the workspace Jobs view.

Note that the schema cache does not include the schema for JSON columns that use Document View. Those schemas are detected by a different scan.

Enabling and configuring schema caching

To enable and configure the caching:

1. On the Workspace Settings view, toggle Cache source schema for faster loading to the on position.

2. Under Schema Freshness, configure the maximum length of time between schema retrievals.

   1. In the field, provide the value.

   2. From the dropdown list, select the unit of time. You can configure the length of time in minutes, hours, or days.

   If the cached schema is older than that length of time, then the next time the application loads, it queries the source database for the current schema. The default value is 6 hours. Note that for some data connectors, schema retrievals run automatically in the background. This setting does not affect the frequency of those schema retrievals. For example, a schema retrieval runs automatically in the background every 2 hours. If you set the schema freshness to 6 hours, the background retrieval still runs every 2 hours. However, if you set the schema freshness to 1 hour, then schema retrieval occurs no more than 1 hour after the previous schema retrieval.

3. You can optionally enable diagnostic logging for the schema retrieval. Diagnostic logging adds additional diagnostic errors to help with troubleshooting. Note that this additional information might contain sensitive information such as schema identifiers. To enable diagnostic logging:

   1. Click Show advanced options.

   2. Toggle Enable diagnostic logging to the on position.