Release notes


July 1, 2022
Refreshed the Audit Trail user interface on Privacy Hub. The new Protection Audit Trail provides a paginated list of the updates to the sensitivity designation and generator assignments.
Deep links now work correctly when you use Google SSO to authenticate.
Error messages from Oracle are now displayed in response to invalid where clauses in subset configuration.
Made minor memory improvements to the Address generator.
  • Reduced the frequency of schema change detection on Snowflake databases. This can result in cost savings on Snowflake clusters, because the clusters can sleep more often.
  • Snowflake generation now works correctly when there are foreign key constraints.
  • Added support for security policies, sequences, check constraints, and system versioned temporal tables.


June 24, 2022
Improved cross-tab support for automatic logouts when you configure an inactivity period.
The Update option in the actions menu now takes you directly to the System tab on the Admin Panel instead of the Users tab.
Corrected the password length requirement to be 12 characters or greater instead of greater than 12 characters.
Improved the estimated row progress for scaled tables.
Eliminated a race condition that occurred when applying constraints.
SQL Server:
  • The Categorical generator can now support more than 2 billion rows in a category.
  • Can now run concurrent jobs that use different versions of Databricks.
  • Tonic now supports ORC and Hive tables in Databricks.


June 17, 2022
When users create a new password, Tonic now displays a panel with the password requirements, and indicates whether the password meets those requirements.
Improved the parallelization and concurrency for processing foreign key constraints.
Databricks and Spark EMR
  • Improved the performance of the Noise generator.
  • Improved display of longer key values in the Key column. Widened the column and added truncation.
  • Optimized queries against the Tonic database.


June 10, 2022
  • For Postgres databases, Tonic now supports name and char data types.
  • For Tonic single sign-on, Tonic now supports Azure Active Directory.
  • From the administration screen, administrators for customers that run Tonic on Kubernetes and Docker can now download logs from all containers that run Tonic.
  • For Spark-powered integrations, Tonic now supports the Address generator as a sub-generator.
  • Improved performance for:
    • Random Hash generator on Spark and Databricks
    • UUID Generator on Spark
    • Subsetting
  • Improved the browsing experience on low resolution displays.


May 27, 2022
  • Skip batch instead of failing generations on Postgres in some cases when values fail to be inserted
  • Improvement to Synthesis Reports for AI Synthesizer generated data
  • Changing Lambda deployment from ECR to ZIP for integrations using Lambda functions. Images no longer need to be manually deployed to ECR.


May 25, 2022
  • Support for arrays with Struct Mask generator on EMR/Spark and Databricks
  • Fix issue causing privacy scans on Dremio to fail
  • Minor UI fixes to workspace page
  • Fix for data generation issue on Db2 iSeries when destination is empty
  • More robust handing of Databricks host URLs


May 24, 2022
  • Adds data types to all keys in Mongo Collection view and tooltip popovers to key names
  • Updates to data generation UI
  • Fix shaky dialog when table mode resizes


May 23, 2022
  • Add data type advanced search for Oracle, Redshift, and BigQuery
  • Performance improvements for constraint restoration
  • Improved performance of Random Integer generator on Spark
  • Fix issue on SQL Server preserve destination mode when table names contain a ".".
  • Postgres: Fix issue copying arrays with trailing slashes


May 20, 2022
  • Allow the console log level to be configured on the Web Server and Worker
  • User passwords can now be reset by an Admin in the Admin Panel


May 19, 2022
  • Added compatibility with SQL Server 2012
  • Spark SDK: Support for validation of workspace when processing a dataframe
  • Minor fixes for MongoDB integration


May 17, 2022
  • Support graceful cancellation of privacy metric computation for AI Synthesizer
  • Java SDK documentation available from the SDK Setup dialog
  • Only show SSO and Enterprise licensing information to users when appropriate
  • Minor UI fixes for scrolling on the workspace view table
  • Minor UI fixes for the Edit Workspace page


May 17, 2022
  • For S3/EMR data source, support separately testing each component of the connection (Glue catalog, EMR cluster, and S3 bucket)
  • The Additive Noise Generator is renamed to "Noise Generator" with two noise options "Additive" (existing) and "Multiplicative" (new)
  • Improve Javadocs for Spark SDK and other enhancements
  • Fixes for the SQL beautifier on post-job scripts, including scenarios which may have crashed the app on Safari
  • Skip processing Temporal Types when not supported (SQL Server 2012) to prevent job failures


May 16, 2022
  • Add hourly heartbeat/status log message for web server and worker
  • Group database/integrations on the create workspace screen by type


May 13, 2022
  • Add support for Azure Datalake Storage Gen 2 as an output destination for Databricks
  • Don't include statistics for Oracle schema copy
  • MongoDB: Support uploading foreign key file, support deleting configured foreign keys
  • Do not stop generation in SQL Server if schema pre-fetch fails
  • Improved deletion of old logs to prevent timeouts


May 13, 2022
  • Adds support for Spark 3.1
  • Improvements to Dremio for data type searching, Struct support, and a new getVersion method on SDK
  • Fix workspace name not updating in dropdown after rename
  • Don't allow non-editor/owner to open table mode selector


May 12, 2022
  • Update subsetting tooltip help text
  • Don't check destination recovery mode on Azure SQL
  • Improve performance of table constraint application when subsetting by applying single and cross table constraints in parallel
  • Fix logo navigation redirect when signed in as a user with no workspaces
  • Fix issue with setting a partition filter on Spark
  • Fix issue with collection picker on MongoDB showing removed collections
  • Fix excessive memory usage of Hostname generator


May 10, 2022
  • Improves performance of Custom Categorical generator on Spark and Databricks
  • Add support for Azure Databricks
  • Display SQL Server schema generation progress as a percentage
  • Do not fetch column data on configuration popover open to reduce queries against the source database
  • Reduce the number of queries to the source database for table/column details during generation


May 10, 2022
  • Spark 2.4 and Databricks Library Improvements
  • Add ability to remove users in the Admin Panel
  • Fixed issue with the displayed port when connecting to MongoDB via connection string
  • Fix for data table column resizing not rendering properly
  • Support random latitude and longitudes with HIPAA Address generator


May 9, 2022
  • New design for the workspace dropdown selector
  • Add warning when previewing AI Synthesizer on text which does not appear to be categorical
  • MongoDB: Collections no longer uncollapse when a generator is applied in hybrid view


May 6, 2022
  • Add support for Spark SDK on Spark v2.4
  • Fix UI crash when selecting incremental as Table Mode
  • Fix AI synthesizer when no categorical columns present
  • Ensure AI Synthesizer model is not trained if only one column is selected
  • Render HTML Mask in the generator model panel on table view
  • Fix issue with not being able to set incremental table mode column if you refresh after setting it to incremental


May 5 2022
  • Added a pre-job check to guard against cycles in applied generators
  • Better support for environments where websockets are not supported
    • Add environment flag to disable websockets on API
    • Gracefully fail if websockets can't connect


May 3, 2022
  • Fixed issue preventing connecting to Postgres databases using an SSH tunnel
  • Fixed nullability error in US Phone generator


May 3, 2022
  • Fix issue where collection loads were attempted for non-Mongo DB's in some instances
  • Write pg_dump results directly to file instead of reading them via stdout to prevent process locks
  • Fixed isssue where copying a workspace on the Workspace View doesn't update the table of workspaces


May 2, 2022
  • Support applying generator values for PK fields in MongoDB to their FK fields
  • Reduce Frequency of Upstream table filtering logs
  • Remove double scroll bar on foreign key page when viewing foreign keys
  • Improved database introspection queries on IBM Db2 iSeries


April 29, 2022
  • Consistently clear local storage on authorization failures
  • Aggregate dropped row exceptions for more efficient logging


April 28, 2022
  • Improved performance for schema scans on MongoDB
  • Minor fixes to workspace view
  • Fix for concurrent lambda function update issues
  • Fix brief non-ideal state while loading collections on MongoDB


April 27, 2022
  • Allow skipping of privacy scan on MongoDB using TONIC_TABLE_SKIP_REGEX environment variable


April 27, 2022
  • Improve usability of processing with Java SDK
  • Close connection to application database when not being used
  • Removed excess querying of database version


April 26, 2022
  • Improves performance of Name Generator on Spark / Databricks
  • Change Dremio schema input to Tag Input and add an informational popover
  • Do not null FKs between reference tables
  • Allow foreign keys to be added by editors


April 26, 2022
  • Improves performance of the SSN Generator on EMR Spark / Databricks
  • Keep the password field empty when copying from source database settings to destination database
  • Improved exception handling when masking and preserving destination tables
  • More quickly alert user when a cycle has been found in their relationships
  • Handle Postgres interval data types with a null boundary
  • Disable Add Foreign Keys tab for non-workspace owner users


April 25, 2022
  • The Workspaces view now supports bulk actions. Share, transfer, delete, and leave multiple workspaces by checking the boxes at the right end of each row, and choosing the desired action from the Actions menu.
  • Fixed an issue where Safari would fail to load the application
  • Fixes to constraint application parallelism
Breaking Changes
  • The response for the Workspace deletion endpoint (/api/workspace/{workspaceId}) has been modified slightly


April 22, 2022
  • Enable Timestamp Generator To Work Without Statistics For Spark Object Masks
  • When a mongo foreign key field has a primary key with a generator applied, replace the example data with a link to the collection with the primary key
  • Fix for MySql generation hanging with empty password


April 21, 2022
  • Improves performance of Name Generator on EMR and Databricks
  • Fix Display Bug in AI Synthesizer Configuration Panel
  • Fix application of constraints in Redshift


April 20, 2022
  • Add the version of the worker that ran a job to the job details
  • On the Mongo Collection View for foreign/primary keys: Added a hover tooltip for key icons, replaced the generator with a label
  • Refresh the last collection visited when a new foreign key file is uploaded and optimize single_doc foreign key
  • Disable date truncation generator as a subgenerator for Spark
  • Fixes Notification service consuming all disk space when crashing
  • Show the appropriate Conditional sub-generator label instead of passthrough
  • Fixed in memory table query for SQL Server


April 19, 2022
  • Teardown and database creation performance improved for SQL Server
  • Skip preserve destination tables (just like truncated) for privacy scan
  • Fix not allowing generators allowed on arrays that were scanned prior to v439


April 18, 2022
  • Adds support for events (dependent rows) to AI Synthesizer (formerly known as Smart Linking)
  • Allow wildcard (%) in schema name for Dremio
  • Support Date Truncation and Timestamp Shift on Snowflake TIMESTAMP_TZ and Redshift TIMESTAMPTZ columns
  • Disable the generator dropdown in the UI for columns with both primary and foreign keys
  • MongoDB - Fix errors when linking Categorical Generators
  • Bug fixes for synthesis report
  • Fix data type mappings for Dremio Integer and Varchar


April 14, 2022
  • Display whether webhooks were used in job details
  • Terminology changes
    • Masked is changing to De-identify
    • Synthesized is changing to Scale
    • Truncated is changing to Truncate
    • Generator name change: Smart Linking is changing to AI Synthesizer
  • Fixes to SQL script beautifier


April 14, 2022
  • Move single connections on connection pools behind a feature flag
  • Do not drop indexes on auto increment columns in MySQL to improve performance
  • Add type filter to JSON Mask generator
  • Remove unique constraint on workspace names and change default workspace name to "Untitled Workspace"
  • Display key icons for user uploaded foreign/primary key fields in Mongo Single document view
  • Fix issue where 0 results in Workspaces table sometimes let you navigate to a negative page number
  • Fix issue where workspace permissions didn't update in the UI until a refresh
  • Fixes authentication error on Postgres when username has special characters
  • Fixed reopening open connections when TONIC_ENABLE_SINGLE_CONNECTION is false
  • Remove owner section from exported workspace body
  • Improve readability of workspace tags in small-width window sizes


April 13, 2022
  • Apply foreign key constraints serially on a different thread to avoid deadlocks & improve performance
  • Clear bulk column search query on filters reset
  • Memory improvements to Mongo schema serialization
  • Better handling of schema name in Java SDK


April 12, 2022
  • Allow changes to order of JSON Mask sub-generators
  • Running a job is now disabled when the license expiration date has passed
  • Separate Workspace Sharing and Workspace Role-based Workspace access into different license features
  • Improve SQL Server performance by prefetching all tables & views
  • When using Java SDK only show Java supported generators


April 11, 2022
  • Improves performance of Email Generator on EMR and Databricks
  • Add support from Dremio with Spark
  • Allowing Kubernetes ImagePullBackOff for up to 5 minutes before throwing error when updating through UI
  • Fixes inability to run a data generator on Spark with default database as the source


April 8, 2022
  • Display estimated time remaining on row-based tasks in job details page
  • Display key icons for user uploaded foreign/primary key fields in Mongo hybrid documents
  • Copy Generator in Mongo: Fix bug where if generator was copying a field that wasn't in the document it would throw exception
  • Add exception handling for SQL Server datetime columns to drop invalid records, better SQL Server XML type exception handling
  • Create source pool just before processing tables to avoid any timeouts in SQL Server


April 8, 2022
  • Display row based progress on jobs whenever possible
  • Statistics Jobs for the Java Spark SDK now appear in Jobs Table/Jobs Detail Page
  • Skip pre-job health check for PyML service when not needed


April 7, 2022
  • Added database metric sharing for AWS users
  • Don't null self-referential foreign keys on reference tables when subsetting


April 7, 2022
  • Improves performance of Email Generator on EMR and Databricks
  • Better small-width responsiveness of tables in UI


April 6, 2022
  • Add synthesis report
  • Support for update via UI with Docker
  • Prevent unexpected behavior of progress tracker when the system clock shifts
  • Fixes error when creating partition function with null range on SQL Server
  • Fix issues with loading screen showing when not needed


April 5, 2022
  • Add Remove Whitespace Transformer to Copy generator
  • Add ability to generate JSONPath expression for JSON Mask generator by clicking on preview JSON
  • Improved support for large decimal numeric types in Postgres


April 4, 2022
  • Add exception handling for invalid SSH Tunnel private key format and list valid formats in tooltip


April 4, 2022
  • Apply constraints in parallel to data generation in Postgres
  • Ensure workspace tag input closes when focus is lost from outside click
  • Remove foreign key checks when applying constraints in MySQL


April 1, 2022
  • Invalidate refresh tokens on logout, log in or out all open sessions (tabs) on login or logout, cross-tab inactivity timeout support
  • Improve load times on workspace pages
  • Fix linking of non-consistent generators in Spark


March 31, 2022
  • Include workspace ID in URLs
  • Parallelize writes on MySQL passthrough tables
  • Allow decimals in subsetting percentages
  • Don't allow duplicate entries of tables in table mode settings for workspace
  • Handle null values with Copy generator on MongoDB
  • Database query performance improvements


March 30, 2022
  • Improve MySQL generation performance by applying constraints in parallel
  • Fix issue where indexes fail to be applied to converted columns on SQL Server
  • Fix issue with SSO account creation failing


March 29, 2022
  • Support Copy generator on MongoDB
  • Support parallelism of MongoDB schema scan
  • Support truncation mode on MongoDB
  • Set Command Timeout in Postgres to prevent timeouts


March 25, 2022
  • Change password requirements
    • Length >= 12 characters
    • Must include number
    • Must include lowercase letter
    • Must include uppercase letter
    • Must include non-alphanumeric character
  • Fixes to transfer workspace text
  • Allow opening Admin Panel when user has no workspaces


March 25, 2022
  • Improve constraint restoration performance on MySQL
  • Allow incremental tables in MySQL where all columns are keys
  • Prevent app from crashing when admin status of user is unknown


March 24, 2022
  • Add login rate limitin3
  • Fix issue with functions and procedures not being created in the destination on Oracle databases


March 23, 2022
  • On EMR and Databricks, write JSON files to JSON instead of Parquet
  • Improved handling of Latitude/Longitude with Smart Linking Generator
  • Add support for enum arrays columns in Postgres
  • Fix issue with EMR cross account data generation failing to resolve the Glue database


March 17, 2022
  • Workspace Updates - Admin View All, Leave, Share & Transfer V2, Owner Filter, and more
  • Parse as bigint if decimal fails
  • Oracle Whitelist support in BuildDb
  • Initialize after log sink reset
  • Replace slashes with underscores in build versioning
  • Add 'No generators' filter for bulk table view
  • Add progress tracker for SQL server post-data script generation
  • Allow support for views without fully qualified tables in Redshift
  • Improve performance of listing workspaces
  • Improve performance of the Constant Generator on EMR and Databricks
  • Sensitive columns now appear in the privacy hub for Mongo workspaces
  • Allow processing of decimals with a scale of 1 up to 9223372036854775807 in Postgres
  • Check for cloudsqlsuperuser permission on GCP Cloud SQL destination postgres databases
  • Address Lambda Function Setup bug
  • Improve Regex Mask Display of Empty Matches and values with new lines
  • Do not fail completely when SSO groups do not resolve
  • Misc Redshift View fixes
  • Fix issue where column can't be in schema changes and show as sensitive
  • Fix issue where some downstream tables are missing their foreign keys in the destination database when using subsetting
  • Fix issue with shifting table size on data preview in database view
  • Prevent error from occurring when MongoDB collection size is too large


March 10, 2022
  • Added support for user defined enum types as keys in subsetting
  • Simplify schema diff changes error toast on generation
  • Show more unprotected columns button mongo privacy hub
  • Added filtering on custom value generator dropdown
  • Add ConsistentOn to AdditiveNoiseGenerator
  • Enabling Timeshift Generator on Compound Primary Keys
  • Regex Mask Generator supports matching empty capture groups
  • Accommodate OFFSET in where clauses in SQL Server subsetting
  • Miscellaneous workspace config improvements
  • Add support Azure Sql with SqlServerDatabaseCreator
  • Add progress tracking for mongo schema scans
  • Add syntax highlighting for where clauses in subsetting
  • Add support for integer primary keys on smallint
Bug fixes
  • Fixed error message for MySQL data type not found
  • Fix Postgres Test Connection bug
  • Fix Workspace patches after rename or modify tags
  • Fix Unix Timestamp empty column bug
  • Throw OperationCanceledException instead of CancelTonicJobException on cancellation request
  • Fix Redshift view replication missing schema


February 24, 2022
  • Oracle perf improvements
  • Add new range options for integer pk generator
  • CNPJ and CPF generator performance improvements
  • Updates Regex Mask Generator to support replacing all matches
  • Start the progress token for passthrough tables when there are no passthrough tables with references
  • Add support for Timestampshift for spark
  • Add warning system to Privacy Scan
  • Add ability to update via UI for Admin users for Kubernetes
  • Regex mask generator with Custom Value Processor
  • Check for cancellation and report progress on text copy in Postgresql
  • Miscellaneous perf improvements
Bug fixes
  • Start the progress token for passthrough tables when there are no passthrough tables with references
  • Treat non-standard oracle error code 5 not as an error
  • Fix grabbing workspace users for comments
  • Dispose the source connection instead of just closing in Postgres
  • Fix for test discovery exception
  • Fix postgres sequences behavior
  • Object mask bulk view fix
  • Miscellaneous typos fixed
  • Fixed database type for BigQuery test connection


February 15, 2022
  • Relax unique index constraint on username
  • Added tonic version in containers as an environment variable
  • Adding logic to bypass Fallback generators for Spark databases
  • Support case insensitive name consistency
  • Disable dropping replacements on missing columns for mongo
  • Update Oracle driver
  • Enable additive noise generation for strings in mongo
  • Add support for unsigned 64-bit integer columns in integer key generator
  • Add random timestamp and timestamp shift generators as options for integer columns (for unix timestamp columns)
  • Add checks for min/maxTime on timestamp and min/maxDate on dates
  • Adding null ref check for fallback generation option check during preview
  • Revamped workspace configuration screen
  • Support for Decimal on Spark (when using C# UDFs)
  • Allow Regex Generator on PKs
  • Added new environment variable to filter tables ingested by UI for Oracle
  • Misc perf improvements
Bug fixes
  • Fix mongo view more values issues
  • Disable dropping replacements on missing columns for mongo
  • Fix Workspace specified via API without DBs can be modified in UI
  • Fixing Random Timestamp generator on Java
  • Date truncation generator bug fixes


February 8, 2022
  • Better error exception handling for number conversions in Mongo
  • Added loading indicator when viewing single Mongo document
  • Improved Mongo performance and logging
  • Added table filter validation for EMR
  • Handling numbers as doubles instead of integers in Snowflake
  • Added system schema to Postgres
  • Added the Geo Generator for latitude and longitude data
  • Added support for materialized view in Redshift
Bug fixes
  • Fixed array regex mask generator with correct subgenerator options
  • Fixed conditional generator by adding binding


February 4, 2022
  • Updated to dotnet 6
  • Moved FK import to FK page
  • Added support for tinyint in MySQL as a primary key
  • Added table partition filtering for EMR
  • Enabled Regex Mask generator on EMR
  • Added support decimals with precision and scale for EMR
  • Moved Tonic version number to hamburger menu
  • Added job sorting by start time
  • Ability to retrieve Kubernetes logs via API
  • Improved behavior of upstream reference tables for subsetting
  • Improved logging for subset filtering
  • Updated sensitivity scan to better reflect job status
  • Improved privacy scan performance for Mongo
  • Introduced automated schema checks (not for Mongo)
  • Added HTML Mask generator
  • Added Additive Noise Generator
  • Added CNPJ generator
  • Added Array Regex Mask generator
Deprecated endpoints
Updates to the foreign key feature impact the API endpoints used to fetch, upload, and delete the Foreign Key file. Two previous API endpoints are now marked as deprecated:
  • DELETE /api/datasource/delete_fkupload has been marked as deprecated and replaced by DELETE /api/workspace/{workspaceId}/foreign-key/
  • GET /api/datasource/download_fkupload/ has been marked as deprecated and replaced by GET /api/workspace/{workspaceId}/foreign-key/
The main change between both APIs is the switch to using workspaceId to fetch the Foreign Key file instead of the datasourceId.
Both are slated to be removed after 2022-06-01.
Bug fixes
  • Fix number parsing issues in JSON mask generator
  • Fix MySQL parsing of auto_increment columns
  • Eliminated re-querying of data type on EMR Spark