Viewing and resolving schema changes
A database schema can evolve over time. For example, a table is added, a column is removed, or a column data type changes.
It's important that you are aware of these changes and that you update your data generation configuration to address these changes.
In some cases, if you don't update the configuration, then sensitive data might be leaked. For example, when a new column is added, by default the generator is Passthrough. If you do not assign a different generator, then the next time you generate data, the source data is copied to the destination database without being masked.
In other cases, the data generation fails if you don't update the configuration. For example, a column data type changes from integer to string. If the column is assigned the Random Integer generator, the data generation fails.
Tonic Structural monitors your source database to look for changes to the data schema. It alerts you to those changes, and allows you to acknowledge or resolve the changes. You can also configure your workspace so that you cannot generate data when there are unacknowledged or unresolved schema changes.
Schema changes that Structural detects
Structural detects the following schema changes.
Conflicting schema issues
Conflicting schema issues can cause the data generation to fail if they are not resolved.
Structural detects the following conflicting schema issues:
Table that has configured generators is removed from the schema.
Column that has a configured generator is removed from the schema.
Column that has a configured generator changes data type.
Column that is assigned the NULL generator changes nullability.
Column that has an assigned generator becomes a foreign key. Foreign key columns must inherit the generator from the primary key.
Non-conflicting schema changes
Non-conflicting schema changes do not cause the data generation to fail. However, to prevent leakage of sensitive data, you should address these changes before you generate data.
Structural detects the following non-conflicting schema changes:
Column is added to the schema.
Table without generator configuration is removed from the schema.
Column without a configured generator is removed from the schema.
Column without a configured generator changes data type.
Schedule for schema change scans
When you navigate to a workspace in Structural, Structural always runs a scan to check for schema changes.
Structural can also run a periodic schema change detection scan in the background.
For databases other than Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, Structural by default runs a background scan every two hours.
For Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, Structural does not run any periodic scans. The data structure for these databases makes it expensive to run them. Instead, you can enable a daily schema change detection scan.
For information on how to configure whether and when Structural runs the periodic or daily detection scans, go to Configuring schema change detection.
Configuring schema change detection
TONIC_ENABLE_QUICK_PERIODIC_SCHEMA_CHANGE_SCANS- Boolean to indicate whether to enable the periodic background schema change scan. Default istrue.TONIC_PERIODIC_QUICK_SCHEMA_CHANGE_SCAN_INTERVAL_IN_MINUTES- If periodic background schema change detection is enabled, the number of minutes between scans. The default value is120, which indicates to run the schema change detection every two hours.
For Databricks, Snowflake on AWS, Snowflake on Azure, and MongoDB, use the following environment settings to enable and configure the daily schema change detection scan.
TONIC_ENABLE_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS- Boolean to indicate whether to enable the daily schema change detection scan. Default isfalse.TONIC_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS_HOUR- If the daily schema change detection is enabled, this sets the hour at which to run the scan. The value is an integer between 0 and 23, where 0 is midnight and 23 is 11:00 PM. For example, a value of 14 indicates to run the job at 2:00 PM every day. Default is0.
Preventing data generation when there are schema changes
Conflicting schema issues always prevent data generation. By default, non-conflicting schema changes do not block data generation.
However, you can configure Structural to always prevent data generation whenever there are any unacknowledged or unresolved schema changes.
To block data generation for any schema changes, on the Edit Workspace page, under Source Settings, toggle the Block data generation if schema changes detected setting to the on position.
Viewing the unaddressed schema changes
Workspaces view provides a summary of the unaddressed schema changes for each workspace. Schema Changes view contains the complete list.
Schema change information on the Workspaces view
On Workspaces view, the Schema Changes column shows the number of conflicting and non-conflicting schema changes.
To display a more detailed summary of the schema change detection, hover over the column. The summary includes the timestamp of the last schema scan, and a link to Schema Changes view.
Displaying the Schema Changes view
To display Schema Changes view, either:
On the workspace management view, in the workspace navigation bar, click Schema Changes.
On Workspaces view, from the dropdown menu in the Name column, select Schema Changes.
Displaying Database View filtered to unresolved schema changes
To display Database View filtered to only display columns with unresolved schema changes, click Open all in Database View.
Resolving conflicting schema issues
The Conflicting Schema Issues list contains the schema changes that make your current workspace configuration invalid and that you have not yet resolved.
An issue is resolved when either:
You resolve the issue from the Conflicting Schema Issues list.
For columns that have nullability or data type changes, you change the assigned generator in Privacy Hub, Database View, or Table View.
If there are any unresolved conflicting schema issues, then data generation is blocked. If there are no conflicting schema issues, then the Conflicting Schema Issues section is not displayed.
If there is a conflicting change for the removed table or column in the parent workspace configuration, then regardless of whether the configuration is inherited, you must resolve the change in the parent workspace before the change is resolved for the child workspace.
For changes to column nullability or data type, you resolve the change separately in the child and parent workspaces. Depending on the configuration, the conflict might only exist in one of the workspaces.
Information in the issues list
For each issue, the list includes:
Table name.
Column name, if the change affects a specific column.
Path, for JSON columns that use Document View.
Description of the schema change.
For changes to columns data type or nullability, a link to Database View. The link filters Database View to display only that column.
Resolve button or Select dropdown list. For changes to column data type or nullability, the Select dropdown list allows you to either resolve the issue or update the column configuration. For a child workspace, if the issue must be resolved in the parent workspace, the button is Go to Parent. If you do not have access to the parent workspace, then the button is disabled.
The list does not include changed or removed columns for which the assigned generator is Passthrough.
Resolving a single issue
For the following types of issues, you can only resolve the issue. Resolving the issues allows Structural to do the required cleanup to reflect the removal. For more information, go to How Structural resolves conflicting issues.
Removed table
Removed column
For these issues, to resolve the issue, click Resolve.
For a column that changed nullability or data type, you can use the Select dropdown list to either:
Resolve the issue. For more information, go to How Structural resolves conflicting issues.
Assign a different generator to the column and then resolve the issue.
For these issues:
Click Select.
To have Structural resolve the issue:
Select Reset to Passthrough.
On the confirmation dialog, click Resolve.
To select a different generator for the column:
Select Apply New Generator.
On the generator configuration panel, select and configure the generator. For detailed configuration options for each generator, go to the Generator reference.
When you change the generator configuration, the Mark Resolved button is enabled. To close the panel and also resolve the issue, click Mark Resolved.
For a child workspace, if the issue must also be resolved in the parent workspace, then the button changes to Go to Parent.
Resolving all of the issues
To resolve all of the issues:
Click Resolve All Issues.
On the confirmation dialog, click Resolve All.
For more information, go to How Structural resolves conflicting issues.
For a child workspace, for issues that must also be resolved in the parent workspace, the button changes to Go to Parent.
How Structural resolves conflicting issues
To resolve conflicting issues, other than for the columns that you assign a new generator to, Structural takes the following actions:
Removes the configuration for the affected table or column.
For a column that has a changed data type or nullability, Structural resets the generator to Passthrough.
Removes the links to the affected columns. The columns that were linked to the affected columns otherwise keep their current configuration.
Dismissing non-conflicting schema changes
The Notifications list contains schema changes that do not make the current configuration invalid. These changes are new tables and new columns.
If there are non-conflicting schema changes, then data generation is blocked only if you configured your workspace to block data generation for all unaddressed schema changes.
If there are no non-conflicting schema changes, then the Notifications list is not displayed.
Structural automatically dismisses a notification when:
You assign Truncate or Preserve Destination table mode to a new table.
You assign a generator other than Passthrough to a new column.
Dismissed notifications are removed from the list. Dismissing a notification does not change your workspace configuration.
Information in the notifications list
For each notification, the list includes:
Table name.
Column name, for new columns.
Description of the schema change.
Dismiss button or Select dropdown list. For new columns, the Select dropdown list allows you to either dismiss the notification or assign a generator to the column.
Open in Database View button, which automatically filters Database View to only display the affected table or column.
Dismissing an individual notification
For a new table, the only option is to dismiss the notification. To dismiss the notification, click Dismiss.
For a new column, you can use the Select dropdown list to either:
Dismiss the notification.
Assign a different generator to the column and then dismiss the notification.
For a new column:
Click Select.
To dismiss the notification, select Dismiss Notification.
To assign a generator for the column:
Select Apply New Generator.
On the generator configuration panel, select and configure the generator. For detailed configuration options for each generator, go to the Generator reference.
When you change the generator configuration, the Dismiss button is enabled. To close the panel and also dismiss the notification, click Dismiss.
Dismissing all of the notifications
To dismiss all of the notifications:
Click Dismiss All Notifications.
On the confirmation dialog, click Dismiss All.
Rerunning the sensitivity scan to check for new sensitive data
Whenever there are schema changes, especially new tables and columns, it is important to determine whether those new tables and columns contain sensitive data.
By default, Structural copies all rows from a table. The column generator is set to Passthrough, meaning that the source data is copied as is to the destination database.
Last updated
Was this helpful?
