Viewing and resolving schema changes

A database schema can evolve over time. For example, a table is added, a column is removed, or a column data type changes.

It's important that you are aware of these changes and that you update your data generation configuration to address these changes.

In some cases, if you don't update the configuration, then sensitive data might be leaked. For example, when a new column is added, by default the generator is Passthrough. If you do not assign a different generator, then the next time you generate data, the source data is copied to the destination database without being masked.

Tonic Structural monitors your source database to look for changes to the data schema. It alerts you to those changes, and allows you to resolve the changes.

Types of schema changes

Sensitive schema changes

The following schema changes are addressed by configuration changes. If not addressed, there is a risk of leaking sensitive data.

When Structural detects an unresolved sensitive schema change, it always blocks data generation. Before you run data generation, you must resolve the issue.

Notifications

Notification schema changes are addressed by removing the configuration.

If you do not configure the workspace to block data generation for all schema changes, then Structural does this automatically for each data generation.

However, to permanently resolve the issue, you must dismiss the notification on Schema Changes view.

Schedule for schema change scans

When you navigate to a workspace in Structural, Structural always runs a scan to check for schema changes.

Structural can also run a periodic schema change detection scan in the background.

For databases other than Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, Structural by default runs a background scan every two hours.

For Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, Structural does not run any periodic scans. The data structure for these databases makes it expensive to run them. Instead, you can enable a daily schema change detection scan.

For information on how to configure whether and when Structural runs the periodic or daily detection scans, go to Configuring schema change detection.

Configuring schema change detection

For data connectors other than Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, you use the following environment settings to configure the periodic schema change detection. You configure the settings in the web server container:

• TONIC_ENABLE_QUICK_PERIODIC_SCHEMA_CHANGE_SCANS - Boolean to indicate whether to enable the periodic background schema change scan. Default is true.

• TONIC_PERIODIC_QUICK_SCHEMA_CHANGE_SCAN_INTERVAL_IN_MINUTES - If periodic background schema change detection is enabled, the number of minutes between scans. The default value is 120, which indicates to run the schema change detection every two hours.

For Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, use the following environment settings to enable and configure the daily schema change detection scan.

• TONIC_ENABLE_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS - Boolean to indicate whether to enable the daily schema change detection scan. Default is false.

• TONIC_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS_HOUR - If the daily schema change detection is enabled, this sets the hour at which to run the scan. The value is an integer between 0 and 23, where 0 is midnight and 23 is 11:00 PM. For example, a value of 14 indicates to run the job at 2:00 PM every day. Default is 0.

Viewing the unaddressed schema changes

Workspaces view provides a summary of the unaddressed schema changes for each workspace. Schema Changes view contains the complete list.

Schema change information on the Workspaces view

On Workspaces view, the Schema Changes column shows the number of sensitive and notification schema changes.

To display a more detailed summary of the schema change detection, hover over the column. The summary includes the timestamp of the last schema scan, and a link to Schema Changes view. It also indicates whether the workspace is configured to block data generation for all schema changes.

Displaying the Schema Changes view

To display Schema Changes view, either:

• On the workspace management view, in the workspace navigation bar, click Schema Changes.

• On Workspaces view, from the dropdown menu in the Name column, select Schema Changes.

Displaying Database View filtered to unresolved schema changes

To display Database View filtered to only display columns that have unresolved schema changes, click Open all in Database View.

Addressing sensitive schema changes

The Sensitive Schema Issues list contains the schema changes that could result in data leakage.

An issue is resolved when either:

• You resolve the issue from the Sensitive Schema Issues list.

• For columns that have nullability or data type changes, you change the assigned generator in Privacy Hub, Database View, or Table View.

If there are any unresolved sensitive schema changes, then data generation is blocked. If there are no sensitive schema changes, then the Sensitive Schema Issues section is not displayed.

Structural does not automatically resolve sensitive schema changes in a child workspace, even if the parent workspace configuration is updated. You always resolve the changes separately in the parent and child workspaces.

When you resolve a sensitive issue, Structural dismisses the issue and removes it from the list. For a column that changed data type or nullability, Structural resets the generator to Passthrough.

Displaying the column in Database View

To display Database View filtered to a column, click Open in Database View.

Resolving a single issue

To have Structural resolve the issue, either:

• Click the Resolve button.

• Click the Resolve dropdown, then click Resolve.

For a column-specific issue, to assign a different generator to the column:

1. Click the Resolve dropdown.

2. Click Apply New Generator.

For a child workspace, if the issue must also be resolved in the parent workspace, then the button changes to Go to Parent.

Resolving all of the issues

To resolve all of the issues:

1. Click Resolve All Issues.

2. On the confirmation dialog, click Resolve All.

For a child workspace, for issues that must also be resolved in the parent workspace, the button changes to Go to Parent.

Dismissing notification schema changes

The Notifications list contains schema changes that Structural can address automatically during data generation. These include removed schema items.

Notification schema changes only block data generation if you configure your workspace to block data generation for all unaddressed schema changes.

If there are no notification schema changes, then the Notifications list is not displayed.

For parent and child workspaces, when a child workspace overrides the parent workspace configuration for the table or column, you must resolve the change in the child workspace.

If there is a schema change for the removed table or column in the parent workspace configuration, then regardless of whether the configuration is inherited, you must resolve the change in the parent workspace before the change is resolved for the child workspace.

When you dismiss a notification, Structural:

• Removes the configuration for the removed table or column.

• Removes any links to the removed columns. The columns that were linked to the removed columns otherwise keep their current configuration.

Dismissing an individual notification

To dismiss a notification, click Dismiss.

Dismissing all of the notifications

To dismiss all of the notifications:

1. Click Dismiss All Notifications.

2. On the confirmation dialog, click Dismiss All.

Rerunning the sensitivity scan to check for new sensitive data

Whenever there are schema changes, especially new tables and columns, it is important to determine whether those new tables and columns contain sensitive data.

By default, Structural copies all rows from a table. The column generator is set to Passthrough, meaning that the source data is copied as is to the destination database.

From Privacy Hub, you can run a new sensitivity scan. You can then use the updated results to guide the table and column configuration.