Release Notes


September 27, 2021
    Jobs now auto-stop if the worker crashes
    Support for ENUM Arrays in Postgres.
    Categorical generator can now be used on non-string types in MongoDB.
    Allow Conditional Generator on JSON, JSONB, and Boolean columns.
    CustomCategorical and Categorical Generator support for MongoDB arrays.
    Add optional 'use compression' flag for MySQL.
Bug Fixes
    Fixed issue when swapping between MongoDB workspaces
    Fix TimestampShift Generator for mongo.
    Subsetting enhancements
    EMR partitioned table UI bug fix.
    Fixed an issue where metadata for the conditional generator did not always save correctly
    Fix EMR Test Connection issue.


September 20, 2021
    JSON Mask for Spark.
    Make bulk applying generators more friendly on columns with disparate types.
    Generated columns are now supported for Postgres.
    EMR will output CSV files if the input file is a CSV.
    Added support for unsigned integer columns to the Integer Key Generator.
    Performance improvements for MySQL.
Bug Fixes
    Improved iSeries UI response time.
    Fixed copy workspace issue where some fields were empty until a page refresh was performed.
    EMR Hive DDL fix for Csv.
    Fixed issue with droping indexes on auto-incrementing composite keys in Mysql
    Improve DB2 post data order.
    Improved MySql performance for CopyTable.


September 10, 2021
    Add Pagination to ECR Image search.
    Workspace Tags.
    Optimized the Privacy Scan for Oracle DBs.
    Add custom categorical generator to Mongo DB.
    Support for EMR cross account.
Bug Fixes
    Fix issues with resizable panes in Database and Table view.
    Allow non-null self references in a table in subsetting only if the table is a reference table.
    Security enhancements
    Fixed issues in MongoDB workspaces with dismissing all schema diffs.
    Better handling of Check constraints in MySQL


September 3, 2021
    Added TLS support for iSeries Db2.
    Cloudwatch logs are available inside Tonic for AWS Lambda based jobs.
    Support for 'consistent on column' in Regex Mask Generator.
Bug Fixes
    Fix SIN generator being applied to non-sin columns.
    Optimized Schema construction for Destination DB in DB2 LUW.
    Truncated tables now handled properly for AWS Lambda based jobs.
    Fix rendering of tables with double digit replacements on bulk view.
    "Consistent On" drop down is now alphabetized.


August 30, 2021
    Added destination database connection summary to job start confirmation.
    Subsetting and post-job action indicators have been added to the workspaces view.
    Added workspace name as a webhook content option.
Bug Fixes
    Fixed issue with Postgres index restoration and error handling.
    Fixes for Tonic UI when connected to MongoDB


August 23, 2021
    Added option to Preserve Lambda S3 files
    Added view support for Db2 LUW.
    Improved RegexMaskGenerator performance.
Bug Fixes
    Various fixes and improvements for Lambda generation.
    Add additional logging to Lambda.
    Change Lambda default config options (timeouts, memory).


August 19, 2021
    Added support for incremental table mode on Postgres.
    Enable TimeStampShiftGenerator for text columns.
    Updates to the Privacy Hub for MongoDB
    Add support for object, variant types in snowflake.
    HIPAA Address Generator now handles extended zip codes that don't contain a hyphen.
    Inactivity timeout and authorization/refresh token timeout can now be configured.
    MySQL now copies over routines and events (if permissions are set) to the destination database.
Bug Fixes
    Empty schemas in Db2 output database are now handled properly.
    Http now forwards to https on the healthcheck endpoint of the PII Scanner container.
    Resolved issue in the subset preview when the estimated row count of the source is unavailable.


August 11, 2021
    New UI to view all workspaces.
    Updates to Privacy Hub to track progress in protecting sensitive fields.
    Db2 support for triggers and stored procedures.
    Prevent Db2 summary tables and views from showing in the UI.
    Support for Regex, Json, and XML generators on Snowflake and Redshift.
    Allow Content-Type Header in webhook requests.
    Ability to cancel data generation jobs from the job details page.
    Schema differences can now be detected on a Mongo collection.
Bug Fixes
    Memory usage reduction on MySQL and Oracle.
    ErrorOnOverwrite for Databricks Table Mode.
    Usability improvements with MongoDB workspaces.


August 2, 2021
    Added ability to process Redshift and Snowflake using AWS Lambda.
    Update Databricks assets in parallel.
    Updated Sandbox Terms, analytics, licensed features.
Bug Fixes
    Updated connection to Google Cloud PostgreSQL databases.


July 29, 2021
    User information has been added to the Privacy Hub Audit Trail.
Bug Fixes
    Improve usability when working with Db2.
    Refactor queries to restore indexes and add timeout logic in MySQL.
    Add batch size logic for subsetter when handling larger rows of data.


July 22, 2021
    Users can now download job logs directly through the Tonic application.
    Integer primary key generator allows more control.
    Add index restoration parallelism for PostgreSQL.
    Add name case consistency.
Bug Fixes
    Prevent Safari from crashing.
    No longer print environment variables when launching the PII detection or machine learning containers.
    Improve API performance for endpoint api/schemadiff.
    Mongo RegexMaskEditor now gets column values.


July 15th, 2021
    Smart Linking generators added - train neural networks to mimic the implicit relationships across columns.
    Webhooks now support sending JSON object literals.
    MongoDB expanded generators on allowed data types and consistency for multiple types of paths.
    Clarified error messages in the UI.
    Constant generator will now show a true/false drop down when added to a boolean field.
Bug Fixes
    Improved performance of schema change detection on large workspaces.
    Handle long generator names in the UI.
    Allow notifications container to start up without TONIC_URL env variable.
    Updated TLS cipher negotiation between Tonic and AWS Aurora Mysql.
    Full name will now be consistent with first and last name.
    Null values in Sql Server xml fields are now handled properly.
    Optimized memory usage on very large Sql Server rows.
    Reduced the number of times statistics have to be calculated for Json Mask Generator, Xml Mask Generator, and Regex Mask Generator.
    Fixed issue where S3 + Spark jobs would not start.


June 21st, 2021
    Preserve N bytes for MAC address generation
    Modeling panel in table view can now be resized
    Repartition and coalescing added for Databricks
    Subset preview on small screen improved
    Speed up PII Scan for MySQL
    Updated subsetting logs
    Increment default fingerprint schema version
Bug Fixes
    Disable Generate Data during page load
    Fix constant generator timestamp issue
    Handle case when Spark not installed on EMR cluster
    Increase Big Query limit of 10GB
    Memory optimized tables fixed when tearing down SQL Server
    Shift + Select in Bulk in MySQL fixed


June 11th, 2021
    Webhooks can now be created to alert external systems when a job has finished.
Add Webhooks to Tonic
    Magnifying glass now appears in Collection View in Mongo to show additional values for a given path.
    Post Job Scripts can now be put in a specific order by user.
    MongoDB expanded to support comments, notifications, current date generator, collection search.
    Generators can be found by metadata now as well as name.
    Copy workspace action limited to owner only.
    Remove tutorial video.
    Show UI warnings for Oracle when rows are rejected by sqlldr.
Bug Fixes
    Data in identity columns is now properly inserted into Redshift databases.
    User defined types are now dropped in the correct order when tearing down a SQL Server database.
    Job status API no longer returns information about other jobs.
    Make the polling for jobs resilient to failed checks.
    Even better upstream null handling.


May 27th, 2021
    Added support for Db2 iSeries
    Added support for Delta Table on Databricks
    Added support for reading IAM roles off Databricks machines instead of providing IAM credentials
    Added a UI notification when your Tonic version is 10+ versions behind
    Added support for MongoDB 2.4 and 3.4
    Added the ability to automatically skip tables that match a regular expression via an environment variable
    Added the ability to obfuscate values inside an array in Mongo
    Added the ability for the JSON Mask Generator to parse json objects containing escaped characters and surrounded by quotes
    Added the ability to automatically cast for mismatched types during downstream subsetting
    Added a Regex Generator
    Added a Generative Adversarial Network Generator
Bug Fixes
    Refreshes subset preview when table mode is changed
    Improved handling of IAM credentials on Databricks when an IAM instance profile is present
    Hide "Other" in generator dropdown when there are no additional generators


April 29th, 2021
    Added support for MongoDB
    Added support for Amazon Redshift
    Delta Table support on Databricks
    Upgrades the subset preview with UI improvements
Bug Fixes
    Better handling of adding foreign keys pointing to non-primary keys
    Checks for Postgres version mismatch
    Remove generator button now works in synthesis mode and on key columns
    Added support to break subsetting cycle when both Foreign Key and Primary Key are nullable


April 20th, 2021
    Subset Preview
    Support for Google SSO, including ability to read group membership
    Java UDFs on Spark for Character Scramble and Mac Address generators
    Added custom value processor extension framework
Bug Fixes
    Auto-increment on MySql bug fix
    No longer remote auto-increment on primary keys or on preserved tables
    Fixes composite foreign key issue with one nullable column in key
    Incremental mode now works when a rowversion or timestamp column is on the table
    Fixed collation issue for MySQL 8


April 8th, 2021
    Add environment variable for altering MySql batch size during CopyRows
    Ability to search label on Foreign Key viewer
    CSV Generator support for tab/multi delimiters
    Better performance tracking in the UI for upstream and downstream tables
    Sunset Classic Subsetting in favor of the now default Full algorithm
Bug Fixes
    Fix auto_increment issue on bigint columns
    Don't fail if foreign key column has been removed
    Fix Auto Increment on MySql when dropping indexes
    Better support for JSON during MySql generation


March 31st, 2021
    Support added for non-primary key auto-incrementing columns
    MySQL no longer requires locks on the source DB during generations
    Generators are now easier to remove in the UI
    Reference tables can now be defined in subsetting configurations
    Enhancements to the Foreign Key UI
    Added a clickable link to the Job Start notification
    New Character Separated Value generator added
Bug Fixes
    Performance optimizations for the Address generator
    Improved UI experience for workspaces with thousands of tables
    Optimized memory usage for large workspaces during data generation jobs


March 12th, 2021
    Subsetting upstream has better handling for table relations with multiple constraint groups where one of the constraint groups is often null
    Differential Privacy is now available for Continuous generator
    Added new Tonic logos
    Generators are not allowed on foreign key columns that are also primary keys
    Subsetting is now functional when Primary Key Generators are applied to primary keys
Bug Fixes
    Resolved data upload for MySQL
    Oracle 19 helper improvements
    HIPAA address generator fix for zip codes
    Refresh data table after Foreign Keys update


March 3rd, 2021
    Password reset functionality added
    Search and Sort columns when adding Foreign keys in Tonic
    Nullable Foreign keys are now an option when adding them in Tonic
    New Date Shift generator added
    Foreign Keys are now sorted alphabetically by default
    Ability to set a starting point in the sequential integer generator
Bug Fixes
    Fix in how we handle multi-column primary keys in subsetting
    @ mentioning style improvements for Commenting
    UI fixes for Foreign Keys section
    Enhancements to statistics generation with conditional + categorical generators
    Copying Databricks workspace works as expected


February 17th, 2021
    Support for Db2 LUW added
    New option to set strictness for schema changes
    Added support for pg_repack extension
    Added support for more Key generators in Spark
    Added support for tinyint and smallint data types in subsetting
Bug Fixes
    Better cache handling for subsetting
    Improvements for NULL checking in Consistency On
    Fixed issues in Subsetting Preview
    Reduced collisions in Unique Email generator
    Workers no longer crash when unable to obtain a queued job
    Empty Post Job Scripts now throw a warning rather than failing the job
    Better log messages with primary key generators fail
    Fixed arithmetic overflow error when calculating SQL Server database size
    Fixed upstream exhaustion in subsetting


February 3rd, 2021
    Protection Audit trail now logs enable/disable of Differential Privacy toggle
    Support for Memory Optimized tables in SQL Server
    Ability to add new relationships in the Foreign Keys section
    NULL generator can now be used on columns with uniqueness constraints
    New Subsetting option: Full Algorithm (default is still Classic Algorithm)

Bug Fixes

    Fixed issue with retrieving column names when adding FK in Foreign Keys section
    resolved conflict when setting generator to consistent on a column with a constant generator
    Switching between email generator and unique email generator now clears state as expected
    MySQL point columns no longer halt generation


January 22nd, 2021
    New UUID Key generator
    Support for PostgreSQL Client keys
    Options to preserve OUIs and Uniqueness in MAC address generator
    New HIPAA Address generator
    Email generator can be used on synthesized tables
    Custom Categorical generators can now be linked
    Workspace ID copy button added
    Column output data can now be made consistent on other column's data
    Spark filtering of tables in Databricks
    Improved logging
Bug Fixes
    Better handling of times without timezone values in Postgres
    UI fixes
    Subscriptions and Publications no longer copied in Postgres


December 23rd, 2020
    Improved performance of synthesized tables
    Added new Foreign Key Viewer + Remover
Bug Fixes
    Fixed issue with display of password reset errors
    Fixed rendering issue with boundary of some popovers


December 11th, 2020
    API endpoint test_destination_db_connection added
    User can now filter tables by schema in the Database View
    Support for Hive + Spark datasource added
    Support for synthesis on datetime primary keys
Improvements/Bug Fixes
    Improved error messages displayed in the UI
    Logging improvements
    Synthesis mode improvements
    Fixed UI issue with First connection wizard


November 25th, 2020
    Better logging during job execution
    Support for Spark + AVRO files
    Various Spark improvements


November 11th, 2020
    Undo/Redo capabilities for workspace changes
    More information in Job Details
    New Strict Mode available when generating data via the API
    Now supports for AVRO files
    Now supports EMR Steps API
Bug Fixes
    Error logging enhancements for Spark
    Various UI improvements


October 28th, 2020
    Added support for Spark/S3 as a data source and destination
    Improved Company Name generator
    Added support for text and ntext types in SQL Server
    Added support for Google Big Query as a data source and destination
    Workspace ownership can now be transferred from user to user
Bug Fixes
    AWS Commons extension no longer breaks data generations
    JSON and XML mask generator configuration fixes
    Minor display fixes


October 14th, 2020
    Find and Replace Generator introduced
    Tonic saves a copy of the workspace configuration to job history every time a generation is started
    You can now share workspaces with your SSO defined groups
Bug Fixes
    Corrected rending issue for column sort option in the Database View


September 29th, 2020
    Support for PingID SSO
    Option to download foreign key file
    Social Insurance Generator added
Bug Fixes
    Test Connection errors now show why the connection failed


September 17th, 2020
Post Generate Script
    Added the ability to create post job scripts that will be executed against the destination server at the end of the generation
    Added Cross Table Sum generator. Allows summing of rows from another table by partition
    Added single sign on support
    Added Enterprise license key support
    Added a check to validate the foreign key file before a generation executes
    Allow maximum for Integer Key generator
    Removed dependency on RabbitMQ
    Added ability to have SQL Server and Postgres trust server certificates in workspace editor
    Additional logging was added around relationship integrity during subsetting
    Added adjustable batch sizes for Oracle
Bug Fixes
    Fixed several rendering issues in the table view
    Tries multiple connection methods for Oracle
    Supports Extended VarChar2s for Oracle
    No longer hides foreign key column headings for synthesize mode
    Job descriptions for jobs cancelled before they started running are more accurate
    Testing the source database connection now works for viewers and auditors
    Workspaces are now sorted alphabetically in the workspace drop down
    Changing a linked generator now properly breaks the link
    Foreign key columns now show up properly without needing a refresh after setting the primary key table to synthesized mode
    Expand long column names on hover in the Privacy Hubs's audit trail
    Workspaces with broken connections to their database can once again be edited
    Views no longer show in the list of tables for Oracle databases.


August 20th, 2020
    Workspace Sharing - allows sharing workspaces between multiple users with different access controls (This is an Enterprise Plan feature)
Workspace Sharing
    Added 'In' operator to the Conditional Generator.
    Added environment variable to bypass certificate validation when connecting to databases with self signed certificates.
    Schema change detection will now flag columns with null generators that have been made not-null.
    Significantly improved the speed that information is gathered about the source database affecting overall UI performance and schema change detection.
    Added additional diagnostic information around query execution times to the logs.
Bug Fixes
    Fixed issue where switching table mode to synthesized mode in Database view would not work.
    Increase accuracy of error messages on Sql Server when tables fail to get created.
    Integer Key Generator in Oracle for Number columns now base off of the precision, stopping overflow
    Resolved name collision when subsetting with a table that has more than one foreign key to the same primary key.
    Fixed preview on tables that use the sequential integer generator.
    Changing synthesized row counts from database view now saves properly.
    Popover of column name on table view no longer flashes.
    Large constant values no longer cause display issues in Database view.
    Fixed issue with connection test sometimes failing when using MySql.
    Character Scramble now preserves null values when operation on JSON.
    Fixed OID collision issue on Oracle.
    Preview no longer fails when using complicated partitioning strategies.
    Constant Generator no longer loses preview when you click away.


July 31st, 2020
    ASCII Character Primary Key Generator which supports a wider range of characters in the column.
    New API Endpoint for getting information on a single data generation job
    Key Generators now work on Unique Columns
    Improvements to PII Detection:
      Added detection for columns containing passwords, postal codes, and VINs.
      Reduced false positives for SWIFT codes, ICD9, ICD10, US Cities
    Switched to a new MySQL driver
    Make XML generator more human readable
    Allow synthesize mode on Conditional Generator and Unique Email Generator
    Add event generator in Oracle
Performance Improvements
    UI performance for resolving schema difference
Bug Fixes
    Better checking of uniqueness requirements for columns
    Fixed constraints on large tables timing out in SqlServer
    Fixed conversion issue with XML columns outside of subsets during subsetting
    Block subsetting with synthesized tables
    Do not allow source and destination to be the same in MySQL
    Make preview stay up to date when generator is removed from different view
    Fixed vulnerability with lodash


July 15th, 2020
    Tables on Sql Server can now be updated in an incremental fashion where only the changes since the last generation are processed. More details here.
Incremental Updates
    Added support for customer categorical generator in synthesize mode.
    Date truncation generator has been added so you can truncate dates to a specified date part.
    Added support for timestamp ranges in Postgres.
    Phone number generator now supports multinational phone numbers. The output phone number will match the country/region of the input phone number.
    Integer key generator can now operate on an Int64/Long column.
    Fixed issue where a specifically sized payload could cause the API request to fail.
    Geo generator properly passes through null values.
    Added back remove table API.
    Tables containing unicode characters now work in preserve destination mode on Sql Server.
    Improved handling of transaction scopes for Sql Server.
    Fixed issue where generators applied through the conditional generator sometimes used their default options during generation.
    Generators added through autodetect are now able to be marked consistent/deferentially private.
    Resolved multithreading issue using the key generators on tables with linked foreign keys.
    Sql server date columns no longer show time in the table view.


July 1st, 2020
    Added ability for full name generators to be consistent with partial name generators.
    Generator popovers now scroll after reaching a certain height.
    Add Random Double generator.
    Add ability to delete Tonic account.
    Improved job progress tracking for Oracle.
    Allow workspaces to be imported and exported.
    Added support for conditional generator on Sql Server image columns.
    Negative numbers can now be used in the constant generator with numeric columns.
    Added support for timezone arrays in Postgres.
Performance Improvements
    Significant performance increase when using the continuous generator.
Bug Fixes
    No longer displaying out of date conditional generator configuration in the collapsed view.
    Fixed issue with not being able to backspace the constant value on a constant generator applied to a numeric column.
    User's can no longer duplicate subset targets on the subset configuration page.
    Removed conditional generator as a sub-generator choice when using the xml or json mask generators.
    Fixed issue where consistency could not be applied to a generator inside the conditional generator.
    Loading indicators on the table view are now more consistent.
    Continuous generator now works with smaller partitions when generating statistics.
    Job details page now scrolls when number of steps is too large for the screen area.
    Fixed issue where adding a custom categorical generator inside the JSON mask generator could cause an error.
    Removing a table from the Schema Changes page now properly removes it from the subsetting configuration.
    Disabled including tables out of subset was enabled when subsetting was not.
    Fixed broken link for API documents on Job Details page.
    Fixed issue where sequential integer generator didn't reset itself between generations.
    Forced browser to get new version of assets after each release.
    Add sql injection safety to generators that partition.
    Fixed issue with switching between workspaces on the jobs view.
    Moved popovers in database view to not be blocking other columns.
    Fix issue with attribute info generator failing on empty attributes.
    Stopped job progress from updating after job is complete.
    Fixed issue with canceled jobs still running if they were cancelled before they could run.


June 17th, 2020
    Added the ability to specify a domain and enable consistency on the unique email generator
    Conditional generator can now be used with the unique email generator
    Added confirmation step for canceling a generation.
    Added support for JSON and JSONB arrays in Postgres.
Performance Improvements
    Updates to your workspace are now done through Json Patch.
Bug Fixes
    Fixed issue where a table was unable to be switched to synthesized mode.
    Fixed issue where min and max on the random integer generator were not editable.
    Fixed issue where multiple changes might not be saved properly if executed within a few milliseconds of each other.
    Fixed issue where constraints failed to apply to a large Sql server database.
    Fixed display issue where the password input appeared to be filled in on the destination database connection screen.
    Swagger docs no longer report enum fields as integers.
    Fixed issue where edit, copy, and delete workspace buttons where still clickable even when disabled.
    Decreased clickable area on consistency and differential privacy switches.
    Fixed issue with NaN values in Postgres double fields.


May 29th, 2020
    A new subsetting option allows you to process tables that are not included in your subset.
    Several Conditional Generator improvements:
      Conditional Generator can now filter rows by a regular expression.
      Conditional Generator now supports the following additional generators: Categorical, Custom Categorical, Alphanumeric Key, Numeric String Key, Integer Key.
      'Is null' and 'is not null' operators added to conditional generator
    Custom Categorical Generator now supports numeric types.
    Random timestamp generator can now be added to text columns.
    Password managers are now prevented from interfering with the database connection form.
    Improved logging for constraint application on Sql server.
    Renamed 'private' to 'sensitive' when referring to a column with data that needs to be protected.
    Column headers are now red when a column is sensitive but not protected.
Performance Improvements
    Workspace updates no longer happen on keystroke and will now wait until you exit the field or popup.
    Changed the way the application loads workspaces to increase performance for large workspaces.
Bug Fixes
    Fix display issue with Custom Categorical Generator when there were no categories.
    Fixed error message when applying random timestamp generator.
    Fixed issue with column editor size changing when marking a column as sensitive.
    Fixed an issue with xml columns on Sql server


May 12th, 2020
    Tables can now be filtered by their current mode in the database view.
    Added support for the 'contains' operator on the conditional generator.
    Company name generator now supports consistency.
    Added version check when editing a workspace to better support multitab/multibrowser use. Multiple users can now edit the same workspace without worrying about race conditions. Note: this is a first step, more multiuser features are in development.
Performance Improvements
    Improved speed of tables in mask mode by fixing issue introduced in v76.
Bug Fixes
    Fixed issue with applying a large number of constraints with Sql Server.
    Random boolean, IP address, and random integer generators now work correctly with the conditional generator
    Tasks that complete immediately now display correctly in the job details page.
    Fixed display issue with data preview button on privacy hub.


May 8th, 2020
    Added a generator for shipping container codes.
    Custom Categorical Generator now works on json fields.
    Destination database names can now differ from the source for Sql Server.
    Job completion time has been added to the Job Details page.
    Additional diagnostic tools added to docker containers.
    Column search performance in bulk editor significantly improved.
    Removed Intercom
Bug Fixes
    Preserving a partitioned table on Sql Server no longer causes an error.
    Fixed issue with the Categorical Generator when the table is empty.
    Fixed issue with the last line break sometimes being removed from the category list on the Custom Category Generator.
    Fix preserve tables for non-Oracle databases.
    Styling fixes for Conditional Generator when used in Firefox.
    Fixed issue where insert into an XML column can fail on Sql Server when the payload has a large attribute, too many nodes, or too much nesting.