Viewing and resolving schema changes

Required workspace permission: Resolve schema change warnings

A database schema can evolve over time. For example, a table is added, a column is removed, or a column data type changes.

It's important that you are aware of these changes and that you update your data generation configuration to address these changes.

In some cases, if you don't update the configuration, then sensitive data might be leaked. For example, when a new column is added, by default the generator is Passthrough. If you do not assign a different generator, then the next time you generate data, the source data is copied to the destination database without being masked.

In other cases, the data generation fails if you don't update the configuration. For example, a column changes its data type from integer to string. If the column is assigned the Random Integer generator, the data generation fails.

Tonic Structural monitors your source database to look for changes to the data schema. It alerts you to those changes, and allows you to acknowledge or resolve the changes. You can also configure your workspace so that you cannot generate data when there are unacknowledged or unresolved schema changes.

Schema changes that Structural detects

Structural detects the following schema changes.

Conflicting schema issues

Conflicting schema issues can cause the data generation to fail if they are not resolved.

Structural detects the following conflicting schema issues:

• Table is removed from the schema

• Column is removed from the schema

• Column changes data type

• Column changes nullability, for columns that are assigned the NULL generator

• A column that has an assigned generator becomes a foreign key. Foreign key columns must inherit the generator from the primary key.

Non-conflicting schema changes

Required license: Professional or Enterprise

Non-conflicting schema changes do not cause the data generation to fail. However, to prevent leakage of sensitive data, you should address these changes before you generate data.

Structural detects the following non-conflicting schema changes:

• Table is added to the schema. This includes new file groups that you add to a file connector workspace.

• Column is added to the schema

Schedule for schema change scans

When you navigate to a workspace in Structural, Structural always runs a scan to check for schema changes.

Structural can also run a periodic schema change detection scan in the background.

For databases other than Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, Structural by default runs a background scan every two hours.

For Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, Structural does not run any periodic scans. The data structure for these databases makes it expensive to run them. Instead, you can enable a daily schema change detection scan.

For information on how to configure whether and when Structural runs the periodic or daily detection scans, go to Configuring schema change detection.

Configuring schema change detection

For data connectors other than Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, you use the following environment settings to configure the periodic schema change detection. You configure the settings in the web server container:

• TONIC_ENABLE_QUICK_PERIODIC_SCHEMA_CHANGE_SCANS - Boolean to indicate whether to enable the periodic background schema change scan. Default is true.

• TONIC_PERIODIC_QUICK_SCHEMA_CHANGE_SCAN_INTERVAL_IN_MINUTES - If periodic background schema change detection is enabled, the number of minutes between scans. The default value is 120, which indicates to run the schema change detection every two hours.

For Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, use the following environment settings to enable and configure the daily schema change detection scan.

• TONIC_ENABLE_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS - Boolean to indicate whether to enable the daily schema change detection scan. Default is false.

• TONIC_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS_HOUR - If the daily schema change detection is enabled, this sets the hour at which to run the scan. The value is an integer between 0 and 23, where 0 is midnight and 23 is 11:00 PM. For example, a value of 14 indicates to run the job at 2:00 PM every day. Default is 0.

Preventing data generation when there are schema changes

Conflicting schema issues always prevent data generation. By default, non-conflicting schema changes do not block data generation.

However, you can configure Structural to always prevent data generation whenever there are any unacknowledged or unresolved schema changes.

To block data generation for any schema changes, on the Edit Workspace page, under Source Settings, toggle the Block data generation if schema changes detected setting to the on position.

Workspace setting to block data generation if there are schema changes

Viewing the unaddressed schema changes

The Workspaces view provides a summary of the unaddressed schema changes for each workspace. The Schema Changes view contains the complete list.

Schema change information on the Workspaces view

On the Workspaces view, the Schema Changes column shows the number of conflicting and non-conflicting schema changes.

To display a more detailed summary of the schema change detection, hover over the column. The summary includes the timestamp of the last schema scan, and a link to the Schema Changes view.

Schema changes overview on the Workspaces view

Displaying the Schema Changes view

To display the Schema Changes view, either:

• On the workspace management view, in the workspace navigation bar, click Schema Changes.

• On Workspaces view, from the dropdown menu in the Name column, select Schema Changes.

Resolving conflicting schema issues

To resolve an issue, you must have permission to perform the associated action.

The Conflicting Schema Issues list contains the schema changes that make your current workspace configuration invalid and that you have not yet resolved.

Conflicting Schema Issues list on the Schema Changes view

An issue is resolved when either:

• You resolve the issue from the Conflicting Schema Issues list.

• For columns that have nullability or data type changes, you change the assigned generator in Privacy Hub, Database View, or Table View.

If there are any unresolved conflicting schema issues, then data generation is blocked. If there are no conflicting schema issues, then the Conflicting Schema Issues section is not displayed.

For parent and child workspaces, for removed tables and columns, when a child workspace overrides the parent workspace configuration for the table or column, you must resolve the change in the child workspace.

If there is a conflicting change for the removed table or column in the parent workspace configuration, then regardless of whether the configuration is inherited, you must resolve the change in the parent workspace before the change is resolved for the child workspace.

For changes to column nullability or data type, you resolve the change separately in the child and parent workspaces. Depending on the configuration, the conflict might only exist in one of the workspaces.

Information in the issues list

For each issue, the list includes:

• Table name

• Column name, if the change affects a specific column

• Description of the schema change

• For changes to columns data type or nullability, a link to Database View. The link filters Database View to display only that column.

• Resolve button or Select dropdown list. For changes to column data type or nullability, the Select dropdown list allows you to either resolve the issue or update the column configuration. For a child workspace, if the issue must be resolved in the parent workspace, the button is Go to Parent. If you do not have access to the parent workspace, then the button is disabled.

The list does not include changed or removed columns for which the assigned generator is Passthrough.

Resolving a single issue

For the following types of issues, you can only resolve the issue. Resolving the issues allows Structural to do the required cleanup to reflect the removal. For more information, go to How Structural resolves conflicting issues.

• Removed table

• Removed column

For these issues, to resolve the issue, click Resolve.

For a column that changed nullability or data type, you can use the Select dropdown list to either:

• Resolve the issue. For more information, go to How Structural resolves conflicting issues.

• Assign a different generator to the column and then resolve the issue.

For these issues:

1. Click Select.

2. To have Structural resolve the issue:

   1. Select Reset to Passthrough.

   2. On the confirmation dialog, click Resolve.

3. To select a different generator for the column:

   1. Select Apply New Generator.

   2. On the generator configuration panel, select and configure the generator. For detailed configuration options for each generator, go to the Generator reference.

   3. When you change the generator configuration, the Mark Resolved button is enabled. To close the panel and also resolve the issue, click Mark Resolved.

For a child workspace, if the issue must also be resolved in the parent workspace, then the button changes to Go to Parent.

Resolving all of the issues

To resolve all of the issues:

Confirmation dialog to resolve all issues

1. Click Resolve All Issues.

2. On the confirmation dialog, click Resolve All.

For more information, go to How Structural resolves conflicting issues.

For a child workspace, for issues that must also be resolved in the parent workspace, the button changes to Go to Parent.

How Structural resolves conflicting issues

To resolve conflicting issues, other than for the columns that you assign a new generator to, Structural takes the following actions:

• Removes the configuration for the affected table or column.

• For a column that has a changed data type or nullability, Structural resets the generator to Passthrough.

• Removes the links to the affected columns. The columns that were linked to the affected columns otherwise keep their current configuration.

Dismissing non-conflicting schema changes

Required license: Professional or Enterprise

For Basic license users, if you know that there are non-conflicting changes, you can run a new sensitivity scan to get the protection status of the new columns.

The Notifications list contains schema changes that do not make the current configuration invalid. These changes are new tables and new columns.

Notifications list on the Schema Changes view

If there are non-conflicting schema changes, then data generation is blocked only if you configured your workspace to block data generation for all unaddressed schema changes.

If there are no non-conflicting schema changes, then the Notifications list is not displayed.

Structural automatically dismisses a notification when:

• You assign Truncate or Preserve Destination table mode to a new table.

• You assign a generator other than Passthrough to a new column.

Structural does not automatically dismiss non-conflicting schema changes in a child workspace, even if the parent workspace configuration is updated. You always dismiss the changes separately in the parent and child workspaces.

Dismissed notifications are removed from the list. Dismissing a notification does not change your workspace configuration configuration.

Information in the notifications list

For each notification, the list includes:

• Table name

• Column name, for new columns

• Description of the schema change

• A link to Database View. The link automatically filters Database View to only display the affected table or column.

• Dismiss button or Select dropdown list. For new columns, the Select dropdown list allows you to either dismiss the notification or assign a generator to the column.

Dismissing an individual notification

For a new table, the only option is to dismiss the notification. To dismiss the notification, click Dismiss.

For a new column, you can use the Select dropdown list to either:

• Dismiss the notification.

• Assign a different generator to the column and then dismiss the notification.

For a new column:

1. Click Select.

2. To dismiss the notification, select Dismiss Notification.

3. To assign a generator for the column:

   1. Select Apply New Generator.

   2. On the generator configuration panel, select and configure the generator. For detailed configuration options for each generator, go to the Generator reference.

   3. When you change the generator configuration, the Dismiss button is enabled. To close the panel and also dismiss the notification, click Dismiss.

Dismissing all of the notifications

To dismiss all of the notifications:

Confirmation dialog to dismiss all notifications

1. Click Dismiss All Notifications.

2. On the confirmation dialog, click Dismiss All.

Rerunning the sensitivity scan to check for new sensitive data

Whenever there are schema changes, especially new tables and columns, it is important to determine whether those new tables and columns contain sensitive data.

By default, Structural copies all rows from a table. The column generator is set to Passthrough, meaning that the source data is copied as is to the destination database.

From Privacy Hub, you can run a new sensitivity scan. You can then use the updated results to guide the table and column configuration.