V1004 - V1053

A new entry is added each week, and contains the release notes for all of the Tonic versions that were released during that week.

Releases V1051 - V1053 were removed from quay because of a regression.

V1046 - V1053

December 15, 2023

Enable administration functions in Tonic Cloud - For Tonic Cloud customers, the new Account Admin permission set provides access to Tonic administration functions for their organization. The Account Admin can reset passwords, delete users, copy and share all workspaces, and download the usage report. The Account Admin permission set is initially granted to the first user in the organization.

Databricks

• For the Test Cluster Connection option, added an error message when the specified cluster cannot describe all the tables in the specified catalog + database.

• Improved error messaging for the Test Cluster Connection option.

File connector

• When you use the API to add files to a file group, Tonic now validates that the file exists and does not duplicate a file that is already in the group.

MySQL

• Added a new environment setting, TONIC_MYSQL_MAX_CONCURRENT_INDEX_CREATION, to limit the number of concurrent indexes that are created. The default value is 0, which indicates that there is no limit.

SQL Server

• External tables are no longer displayed on the application or carried over to the destination database.

V1039 - V1045

December 8, 2023

When you select the option to write destination data to container artifacts, you can now use Google Artifact Registry (GAR) authorization using Google Cloud Platform (GCP) service account keys.

For the JSON Mask and XML Mask generators, fixed the data preview for JSON or XML field samples that are larger than 120MB by generating a smaller subset of the field.

The Name generator now supports consistency with other columns.

Added new API endpoints to retrieve and set table replacements. These new endpoints are compatible with workspaces for data connectors that do not have schemas, such as Spark-based databases and the file connector. The existing endpoints, which require you to provide a schema, eventually will be deprecated.

Amazon EMR

• You can now write data generation output either to a single, static location or to a job-specific folder. Previously, all data generation output was exported to a job-specific folder.

File connector

• You can now use the Continuous generator in file connector workspaces.

MySQL

• When writing destination data to a container repository, you can now provide the content of a my.cnf file that contains custom configuration.

• Fixed an issue where indexes that use expressions in their definition caused an error during data generation.

Oracle

• To fix an issue where turning off database links caused triggers to break, we now run the ALTER TRIGGER statements after the TRIGGERS are validated.

PostgreSQL

• For Azure PostgreSQL, fixed an issue with the destination database connection test.

• For Azure PostgreSQL, fixed the connection test check for schemas that are not admin owned.

SQL Server

• Data generation jobs that are canceled during the schema gathering phase now cancel more promptly.

Snowflake

• On Tonic Cloud, the Snowflake with Azure option is now enabled correctly.

• Free trial users can now use Snowflake on AWS and Snowflake on Azure.

V1034 - V1038

December 1, 2023

Added an environment setting TONIC_DELETE_COLUMN_SCHEMA_ON_WORKSPACE_DELETE. If the setting is true, then when a workspace is deleted, Tonic also deletes the associated rows from the ColumnSchemas table in the Tonic application database.

The new environment setting TONIC_NOTIFICATION_SMTP_TRUST_CERTIFICATE indicates whether to allow the SMTP server certificate to be trusted.

Improved the performance of previewing data in Privacy Hub.

Fixed an issue where SSO groups were not removed when the value of TONIC_SSO_GROUP_FILTER_REGEX changed in a way that excluded previously imported groups. The removed groups are removed from any workspaces that they were granted access to.

For the Timestamp Shift Generator, added Month and Year as options for the date part to use to set the allowed range.

When writing data to container artifacts, Tonic now first shuts down the temporary database before it begins to write data to the container.

Amazon EMR

• Fixed an issue where, when the Athena workgroup used engine version 3, you could not preview data for tables that contained struct data types. Tonic now supports both Athena engine versions 2 and 3.

• Fixed an issue where Professional plan customers could not save workspaces.

• Fixed an issue where the Struct Mask generator editor did not display a preview of the value.

• Enabled support for the Email generator as a sub-generator within the Struct Mask generator.

Databricks

• Fixed an issue where the Struct Mask generator editor did not display a preview of the value.

• The FNR generator is now supported.

• Enabled support for the Email generator as a sub-generator within the Struct Mask generator.

MongoDB

• Fixed an issue where erroneous schema change items were reported when the frequency of a field changed.

Oracle

• Fixed an issue where when Oracle database links were not enabled (TONIC_ORACLE_DBLINK_ENABLED is false), privileges were not copied from the source to the destination.

V1031 - V1033

November 22, 2023

Amazon EMR

• Fixed an issue where jobs failed if Tonic could not retrieve the tracking link for the EMR Step logs. To ensure maximum compatibility with all AWS regions, tracking links are now valid for 7 days instead of 14 days.

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

Databricks

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

MySQL

• On destination databases, Tonic no longer checks for REPLICATION CLIENT and REPLICATION SLAVE grants.

Spark SDK

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

Spark with Livy

• Fixed an issue with assigning the Passthrough generator as a sub-generator for the Regex Mask generator.

V1027 - V1030

November 17, 2023

When an email notification contains a link to a comment, clicking the link now correctly navigates to and displays the comment.

Added a flag to mark log files that might contain sensitive information.

You can now assign the Timestamp Shift generator to primary key columns, including columns that are part of a composite primary key.

When writing destination data to a container repository, based on the size of the source database, Tonic attaches resource requests to the datapacker pod to verify that the cluster has sufficient resources for the data.

Tonic now supports writing destination data to container registries on Amazon Elastic Container Registry (Amazon ECR).

Fixed an issue where users could not assign access to permission sets when they did not have access to create and manage custom permission sets.

For foreign keys that have multiple primary keys, Tonic now prevents data generation if the assigned generation configuration for the primary key columns is inconsistent.

Standardized the display format of the heading area of the workspace management views.

File connector

• FIxed an issue with deleting multiple files from a file group.

MongoDB

• From Privacy Hub, you can now use the option to review and apply recommended generators to all of the detected sensitive fields.

• When there are schema changes that are not yet scanned, such as a new collection or new fields in a collection, they are now handled as Passthrough. The schema changes no longer cause data generation to fail.

MySQL

• You can now configure a MySQL workspace to write the destination data to a container repository.

PostgreSQL

• Fixed an issue where when an excluded schema did not exist in the source database, the test connection option returned an error.

• Customers are now by default automatically enrolled in Data Pipeline V2 mode, and do not see the option to enable or disable it. The ability to enable or disable Data Pipeline V2 is granted to individual customers as needed.

Snowflake

• Fixed an issue where when an excluded schema did not exist in the source database, the test connection option returned an error.

V1024 - V1026

November 10, 2023

API endpoints for subset configuration - The Tonic API now includes endpoints for subsetting configuration. You can use the endpoints to retrieve the subsetting configurations for a workspace, update subsetting configuration, and remove subsetting configuration. A subsetting configuration identifies a table as either a target table (percentage or WHERE clause) or a lookup table.

Improved how Tonic identifies values as names, to reduce false positives.

For upsert data generation, fixed an issue that caused failures on tables that contain foreign keys but no primary keys.

File connector

• Sensitivity scans no longer discard the most recently generated set of downloadable generated files, which are generated from files uploaded from a local file system.

MongoDB

• Fixed an issue that caused duplicate or inaccurate schema issues to be displayed on Schema Changes view.

Oracle

• Tonic now validates destination database objects, such as views and packages, that were invalidated by the data generation. A warning is issued for objects that fail to validate. Validation failure can be caused by insufficient permissions for the Tonic user on the destination schema.

PostgreSQL

• For Data Pipeline V2 data generation, fixed a race condition that could cause data generation to fail.

V1018 - V1023

November 3, 2023

Toggle between database server and container repository - On the workspace configuration view, if the data connector supports writing to a container repository, you can now switch between writing to a database server and writing to a container repository. Tonic saves the information you provide for each option.

Workspace override for statistics seed - From the workspace configuration view, you now can override the Tonic-wide statistics seed that is set as the value of the TONIC_STATISTICS_SEED environment setting. You can either provide a custom seed value for the workspace, or disable consistency across data generation runs for the workspace. Consistency also applies across workspaces that have the same custom seed value. For more information, go to Enabling consistency across runs or multiple databases.

New telemetry URL - Tonic telemetry now routes through https://telemetry.tonic.ai/ instead of https://api2.amplitude.com/. The following IP addresses must be allowed:

• 75.2.74.76

• 99.83.246.105

The following IP addresses no longer need to be allowed: 52.43.241.47, 54.186.140.101, 54.203.75.164, 44.236.122.176, 34.215.78.194, 54.149.61.206, 54.191.147.220, 52.24.22.222, 52.37.168.36, 54.213.191.53, 54.68.108.104, 52.10.121.164, 52.27.184.186, 44.239.225.209, 54.148.216.233, 52.88.224.247

Other updates

Fixed an issue in the Kubernetes client configuration that caused Tonic to reject the SSL certificate of a Kubernetes context.

Fixed an issue where, during subset configuration, an error was returned incorrectly if the user had permission to edit subsetting but not to view source data.

Improved the error message that displays when an uploaded virtual foreign key file is invalid.

Fixed an issue that prevented the Character Substitution generator from being used on primary and foreign key columns when subsetting.

When a workspace import encounters an unhandled exception, the error now displays correctly.

For the Address generator, the Country and Country Code options can now be linked. When linked, the country and country code are either “United States” or “US” to match the other linkable components, which are locations in the United States.

Fixed an issue where SSO login using Okta did not work when a custom authorization server is used.

Fixed an issue where, when using data science mode on Tonic Cloud, users could not download CSV files that contained synthetic data.

Data generation jobs that write to a container now include the datapacker logs, if the worker has permissions to read the pods and logs.

Fixed an issue with uploading container output to Harbor.

For string data type columns:

• Added yyyy/MM/dd as a valid format for the Timestamp Shift generator.

• Added yyyy/MM/dd and MMddyyyy as available output formats for the Random Timestamp generator.

Databricks

• For Databricks 11.3 and later, the Databricks data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

Spark SDK

• The Spark SDK data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

Spark with Livy

• For Spark 2.3.x and 2.4.2, the Spark with Livy data connector now supports the Regex Mask generator. The regular expressions might work slightly differently than on non-Spark data connectors.

SQL Server

• Added a new worker environment setting, SQL_SERVER_SCRIPT_CROSS_DATABASE_REFERENCES. The default is true, which preserves the existing behavior. To prevent scripting of objects that are defined in other databases that the source database references, change the setting to false.

• Fixed issues where SQL Server connections did not honor the TONIC_SQL_COMMAND_TIMEOUT environment setting. The default is 0, which indicates an infinite timeout.

V1011 - V1017

October 27, 2023

Apply recommended generators to multiple detected sensitive columns - On Privacy Hub, a new banner displays the number of detected (not manually designated) sensitive columns that are not protected. From the banner, you can display the list of columns, grouped by sensitivity type. For each sensitivity type, for the selected columns, you can:

• Apply the recommended generator

• Ignore the generator recommendation

• Mark the columns as not sensitive

There is also an option to apply the recommended generators to all of the selected columns across all of the sensitivity types. For more information, go to Reviewing and applying recommended generators.

Writing destination data to container registries (beta feature) - For data connectors that support it, you can now configure a workspace to write destination data to container artifacts on a container registry instead of to a database server. The Job History view provides access to the generated artifacts for each job. For more information, go to Writing data generation output to a container repository and Viewing and downloading container artifacts.

Other updates

Fixed an issue where generated API documentation did not exclude endpoints and schemas from API versions other than the version being viewed.

Fixed an issue where the Protection Audit Trail displayed the incorrect override status of columns in child workspaces.

On the System Status tab of Tonic Settings, fixed an issue where the Data Sharing section incorrectly showed logging as unable to connect.

Databricks

• On the table mode selection panel, for workspaces that write to Databricks Delta tables, disabled the Error on Overwrite table configuration for tables. The setting is not used in this case.

File connector

• In the file group configuration, added options to skip lines from the beginning or end of CSV files.

MongoDB

• On Collection View, fixed an issue that prevented array fields from displaying in the Hybrid Document View of a collection.

Oracle

• For new workspaces, the Preserve source database file storage preferences setting is now off by default instead of on. Existing workspaces are not affected by this change.

V1004 - V1010

October 20, 2023

Configure Tonic environment settings from Tonic Settings - On the Tonic Settings view, a new Environment Settings tab allows you to configure a subset of Tonic environment settings (previously referred to as Tonic environment variables). To use the Environment Settings tab to configure settings, you must have the new Manage environment settings global permission. When you change the values from Tonic Settings, you do not need to restart Tonic. For details, see Configuring environment settings.

Other updates

For the Regex Mask generator, fixed an issue where quotes in regular expressions were malformed.

Fixed an issue where the Test Webhook option failed when it shouldn’t.

Fixed issue where the Confirm Data Generation panel erroneously indicated that Tonic could not connect to the destination database when it actually was only unable to connect to the source database.

For the Address generator, removed an inappropriate value from the possible street names.

Improved error messages when Tonic containers fail to start because of missing or incorrect environment variable values.

Databricks

• Fixed source catalog workspace handling when the Databricks Unity source catalog contains a table with a key constraint.

• For Databricks 10.4 and earlier, generations to output Databricks tables now correctly create a new destination Databricks database if the specified one is not found.

File connector

• For file connector workspaces, the post-job scripts option is now hidden. The file connector does not support post-job scripts.

• For Amazon S3, fixed an issue where Tonic did not delete a temporary file that was created to test permissions.

Google BigQuery

• External tables are now supported with some restrictions. Destination tables must be native BigQuery tables and cannot be external tables, whether masked or passthrough. Performance might be affected because of Google’s implementation of external tables.

MongoDB

• Fixed an issue where an error was thrown when generators were applied.

Snowflake on AWS and Azure

• Added an option to use key pair authentication to connect to the source and destination databases.