Viewing and resolving schema changes
A database schema can evolve over time. For example, a table is added, a column is removed, or a column data type changes.
It's important that you are aware of these changes and that you update your data generation configuration to address these changes.
In some cases, if you don't update the configuration, then sensitive data might be leaked. For example, when a new column is added, by default the generator is Passthrough. If you do not assign a different generator, then the next time you generate data, the source data is copied to the destination database without being masked.
In other cases, the data generation fails if you don't update the configuration. For example, a column changes its data type from integer to string. If the column is assigned the Random Integer generator, the data generation fails.
Tonic monitors your source database to look for changes to the data schema. It alerts you to those changes, and allows you to acknowledge or resolve the changes. You can also configure your workspace so that you cannot generate data when there are unacknowledged or unresolved schema changes.
Tonic detects the following schema changes.
Conflicting schema issues can cause the data generation to fail if they are not resolved.
Tonic detects the following conflicting schema issues:
- Table is removed from the schema
- Column is removed from the schema
- Column changes data type
- Column changes nullability, for columns that are assigned the NULL generator
Non-conflicting schema changes do not cause the data generation to fail. However, to prevent leakage of sensitive data, you should address these changes before you generate data.
Tonic detects the following non-conflicting schema changes:
- Table is added to the schema
- Column is added to the schema
When you navigate to a workspace in Tonic, Tonic always runs a scan to check for schema changes.
Tonic can also run a periodic schema change detection scan in the background.
For databases other than Databricks, Snowflake, and MongoDB, Tonic by default runs a background scan every two hours.
For Databricks, Snowflake, and MongoDB, Tonic does not run any periodic scans. The data structure for these databases makes it expensive to run them. Instead, you can enable a daily schema change detection scan.
For data connectors other than Databricks, Snowflake, and MongoDB, you use the following environment variables to configure the periodic schema change detection:
TONIC_ENABLE_QUICK_PERIODIC_SCHEMA_CHANGE_SCANS- Boolean to indicate whether to enable the periodic background schema change scan. Default is
TONIC_PERIODIC_QUICK_SCHEMA_CHANGE_SCAN_INTERVAL_IN_MINUTES- If periodic background schema change detection is enabled, the number of minutes between scans. The default value is
120, which indicates to run the schema change detection every two hours.
For Databricks, Snowflake, and MongoDB, use the following environment variables to enable and configure the daily schema change detection scan.
TONIC_ENABLE_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS- Boolean to indicate whether to enable the daily schema change detection scan. Default is
TONIC_DAILY_EXPENSIVE_SCHEMA_CHANGE_SCANS_HOUR- If the daily schema change detection is enabled, this sets the hour at which to run the scan. The value is an integer between 0 and 23, where 0 is midnight and 23 is 11:00 PM. For example, a value of 14 indicates to run the job at 2:00 PM every day. Default is
Conflicting schema issues always prevent data generation. By default, non-conflicting schema changes do not block data generation.
However, you can configure Tonic to always prevent data generation whenever there are any unacknowledged or unresolved schema changes.
To block data generation for any schema changes, on the Edit Workspace page, under Source Server, toggle the Block data generation if schema changes detected setting to the on position.
Workspace setting to block data generation if there are schema changes
The Workspaces view provides a summary of the unaddressed schema changes for each workspace. The Schema Changes view contains the complete list.
On the Workspaces view, the Schema Changes column shows the number of conflicting and non-conflicting schema changes.
To display a more detailed summary of the schema change detection, hover over the column. The summary includes the timestamp of the last schema scan, and a link to the Schema Changes view.
Schema changes overview on the Workspaces view
To display the Schema Changes view, in the Tonic navigation pane, click Schema Changes.
The Conflicting Schema Issues list contains the schema changes that make your current workspace configuration invalid and that you have not yet resolved.
Conflicting Schema Issues list on the Schema Changes view
An issue is resolved when either:
- You resolve the issue from the Conflicting Schema Issues list.
- For columns that have nullability or data type changes, you change the assigned generator in Privacy Hub, Database View, or Table View.
If there are any unresolved conflicting schema issues, then data generation is blocked. If there are no conflicting schema issues, then the Conflicting Schema Issues section is not displayed.
For parent and child workspaces, for removed tables and columns, when a child workspace overrides the parent workspace configuration for the table or column, you must resolve the change in the child workspace.
If there is a conflicting change for the removed table or column in the parent workspace configuration, then regardless of whether the configuration is inherited, you must resolve the change in the parent workspace before the change is resolved for the child workspace.
For changes to column nullability or data type, you resolve the change separately in the child and parent workspaces. Depending on the configuration, the conflict might only exist in one of the workspaces.
For each issue, the list includes:
- Table name
- Column name, if the change affects a specific column
- Description of the schema change
- For changes to columns data type or nullability, a link to Database View. The link filters Database View to display only that column.
- Resolve button or Select dropdown list. For changes to column data type or nullability, the Select dropdown list allows you to either resolve the issue or update the column configuration. For a child workspace, if the issue must be resolved in the parent workspace, the button is Go to Parent. If you do not have access to the parent workspace, then the button is disabled.
The list does not include changed or removed columns for which the assigned generator is Passthrough.
- Removed table
- Removed column
For these issues, to resolve the issue, click Resolve.
For a column that changed nullability or data type, you can use the Select dropdown list to either:
- Assign a different generator to the column and then resolve the issue.
For these issues:
- 1.Click Select.
- 2.To have Tonic resolve the issue:
- 1.Select Reset to Passthrough.
- 2.On the confirmation dialog, click Resolve.
- 3.To select a different generator for the column:
- 1.Select Apply New Generator.
- 3.When you change the generator configuration, the Mark Resolved button is enabled. To close the panel and also resolve the issue, click Mark Resolved.
For a child workspace, if the issue must also be resolved in the parent workspace, then the button changes to Go to Parent.
To resolve all of the issues:
Confirmation dialog to resolve all issues
- 1.Click Resolve All Issues.
- 2.On the confirmation dialog, click Resolve All.
For a child workspace, for issues that must also be resolved in the parent workspace, the button changes to Go to Parent.
To resolve conflicting issues, other than for the columns that you assign a new generator to, Tonic takes the following actions:
- Removes the configuration for the affected table or column.
- For a column that has a changed data type or nullability, Tonic resets the generator to Passthrough.
- Removes the links to the affected columns. The columns that were linked to the affected columns otherwise keep their current configuration.
The Notifications list contains schema changes that do not make the current configuration invalid. These changes are new tables and new columns.
Notifications list on the Schema Changes view
If there are non-conflicting schema changes, then data generation is blocked only if you configured your workspace to block data generation for all unaddressed schema changes.
If there are no non-conflicting schema changes, then the Notifications list is not displayed.
Tonic automatically dismisses a notification when:
- You assign Truncate or Preserve Destination table mode to a new table.
- You assign a generator other than Passthrough to a new column.
Tonic does not automatically dismiss non-conflicting schema changes in a child workspace, even if the parent workspace configuration is updated. You always dismiss the changes separately in the parent and child workspaces.
Dismissed notifications are removed from the list. Dismissing a notification does not change your workspace configuration configuration.
For each notification, the list includes:
- Table name
- Column name, for new columns
- Description of the schema change
- A link to Database View. The link automatically filters Database View to only display the affected table or column.
- Dismiss button or Select dropdown list. For new columns, the Select dropdown list allows you to either dismiss the notification or assign a generator to the column.
For a new table, the only option is to dismiss the notification. To dismiss the notification, click Dismiss.
For a new column, you can use the Select dropdown list to either:
- Dismiss the notification.
- Assign a different generator to the column and then dismiss the notification.
For a new column:
- 1.Click Select.
- 2.To dismiss the notification, select Dismiss Notification.
- 3.To assign a generator for the column:
- 1.Select Apply New Generator.
- 3.When you change the generator configuration, the Dismiss button is enabled. To close the panel and also dismiss the notification, click Dismiss.
To dismiss all of the notifications:
Confirmation dialog to dismiss all notifications
- 1.Click Dismiss All Notifications.
- 2.On the confirmation dialog, click Dismiss All.
Whenever there are schema changes, especially new tables and columns, it is important to determine whether those new tables and columns contain sensitive data.
By default, Tonic copies all rows from a table. The column generator is set to Passthrough, meaning that the source data is copied as is to the destination database.