Release Notes

V196

March 31st, 2021

Features

  • Support added for non-primary key auto-incrementing columns

  • MySQL no longer requires locks on the source DB during generations

  • Generators are now easier to remove in the UI

  • Reference tables can now be defined in subsetting configurations

  • Enhancements to the Foreign Key UI

  • Added a clickable link to the Job Start notification

  • New Character Separated Value generator added

Bug Fixes

  • Performance optimizations for the Address generator

  • Improved UI experience for workspaces with thousands of tables

  • Optimized memory usage for large workspaces during data generation jobs

V189

March 12th, 2021

Features

  • Subsetting upstream has better handling for table relations with multiple constraint groups where one of the constraint groups is often null

  • Differential Privacy is now available for Continuous generator

  • Added new Tonic logos

  • Generators are not allowed on foreign key columns that are also primary keys

  • Subsetting is now functional when Primary Key Generators are applied to primary keys

Bug Fixes

  • Resolved data upload for MySQL

  • Oracle 19 helper improvements

  • HIPAA address generator fix for zip codes

  • Refresh data table after Foreign Keys update

V185

March 3rd, 2021

Features

  • Password reset functionality added

  • Search and Sort columns when adding Foreign keys in Tonic

  • Nullable Foreign keys are now an option when adding them in Tonic

  • New Date Shift generator added

  • Foreign Keys are now sorted alphabetically by default

  • Ability to set a starting point in the sequential integer generator

Bug Fixes

  • Fix in how we handle multi-column primary keys in subsetting

  • @ mentioning style improvements for Commenting

  • UI fixes for Foreign Keys section

  • Enhancements to statistics generation with conditional + categorical generators

  • Copying Databricks workspace works as expected

V178

February 17th, 2021

Features

  • New option to set strictness for schema changes

  • Added support for pg_repack extension

  • Added support for more Key generators in Spark

  • Added support for tinyint and smallint data types in subsetting

Bug Fixes

  • Better cache handling for subsetting

  • Improvements for NULL checking in Consistency On

  • Fixed issues in Subsetting Preview

  • Reduced collisions in Unique Email generator

  • Workers no longer crash when unable to obtain a queued job

  • Empty Post Job Scripts now throw a warning rather than failing the job

  • Better log messages with primary key generators fail

  • Fixed arithmetic overflow error when calculating SQL Server database size

  • Fixed upstream exhaustion in subsetting

v165

February 3rd, 2021

Features

  • Protection Audit trail now logs enable/disable of Differential Privacy toggle

  • Support for Memory Optimized tables in SQL Server

  • Ability to add new relationships in the Foreign Keys section

  • NULL generator can now be used on columns with uniqueness constraints

  • New Subsetting option: Full Algorithm (default is still Classic Algorithm)

Bug Fixes

  • Fixed issue with retrieving column names when adding FK in Foreign Keys section

  • resolved conflict when setting generator to consistent on a column with a constant generator

  • Switching between email generator and unique email generator now clears state as expected

  • MySQL point columns no longer halt generation

V159

January 22nd, 2021

Features

  • New UUID Key generator

  • Support for PostgreSQL Client keys

  • Options to preserve OUIs and Uniqueness in MAC address generator

  • New HIPAA Address generator

  • Email generator can be used on synthesized tables

  • Custom Categorical generators can now be linked

  • Workspace ID copy button added

  • Column output data can now be made consistent on other column's data

  • Spark filtering of tables in Databricks

  • Improved logging

Bug Fixes

  • Better handling of times without timezone values in Postgres

  • UI fixes

  • Subscriptions and Publications no longer copied in Postgres

V154

December 23rd, 2020

Features

  • Improved performance of synthesized tables

  • Added new Foreign Key Viewer + Remover

Bug Fixes

  • Fixed issue with display of password reset errors

  • Fixed rendering issue with boundary of some popovers

V152

December 11th, 2020

Features

  • API endpoint test_destination_db_connection added

  • User can now filter tables by schema in the Database View

  • Support for Hive + Spark datasource added

  • Support for synthesis on datetime primary keys

Improvements/Bug Fixes

  • Improved error messages displayed in the UI

  • Logging improvements

  • Synthesis mode improvements

  • Fixed UI issue with First connection wizard

V145

November 25th, 2020

Features

  • Better logging during job execution

  • Support for Spark + AVRO files

  • Various Spark improvements

V142

November 11th, 2020

Features

  • Undo/Redo capabilities for workspace changes

  • More information in Job Details

  • New Strict Mode available when generating data via the API

  • Now supports for AVRO files

  • Now supports EMR Steps API

Bug Fixes

  • Error logging enhancements for Spark

  • Various UI improvements

V132

October 28th, 2020

Features

  • Added support for Spark/S3 as a data source and destination

  • Improved Company Name generator

  • Added support for text and ntext types in SQL Server

  • Added support for Google Big Query as a data source and destination

  • Workspace ownership can now be transferred from user to user

Bug Fixes

  • AWS Commons extension no longer breaks data generations

  • JSON and XML mask generator configuration fixes

  • Minor display fixes

V129

October 14th, 2020

Features

  • Find and Replace Generator introduced

  • Tonic saves a copy of the workspace configuration to job history every time a generation is started

  • You can now share workspaces with your SSO defined groups

Bug Fixes

  • Corrected rending issue for column sort option in the Database View

V123

September 29th, 2020

Features

  • Support for PingID SSO

  • Option to download foreign key file

  • Social Insurance Generator added

Bug Fixes

  • Test Connection errors now show why the connection failed

V119

September 17th, 2020

Features

Post Generate Script
  • Added the ability to create post job scripts that will be executed against the destination server at the end of the generation

  • Added Cross Table Sum generator. Allows summing of rows from another table by partition

  • Added single sign on support

  • Added Enterprise license key support

  • Added a check to validate the foreign key file before a generation executes

  • Allow maximum for Integer Key generator

  • Removed dependency on RabbitMQ

  • Added ability to have SQL Server and Postgres trust server certificates in workspace editor

  • Additional logging was added around relationship integrity during subsetting

  • Added adjustable batch sizes for Oracle

Bug Fixes

  • Fixed several rendering issues in the table view

  • Tries multiple connection methods for Oracle

  • Supports Extended VarChar2s for Oracle

  • No longer hides foreign key column headings for synthesize mode

  • Job descriptions for jobs cancelled before they started running are more accurate

  • Testing the source database connection now works for viewers and auditors

  • Workspaces are now sorted alphabetically in the workspace drop down

  • Changing a linked generator now properly breaks the link

  • Foreign key columns now show up properly without needing a refresh after setting the primary key table to synthesized mode

  • Expand long column names on hover in the Privacy Hubs's audit trail

  • Workspaces with broken connections to their database can once again be edited

  • Views no longer show in the list of tables for Oracle databases.

V107

August 20th, 2020

Features

  • Workspace Sharing - allows sharing workspaces between multiple users with different access controls (This is an Enterprise Plan feature)

Workspace Sharing
  • Added 'In' operator to the Conditional Generator.

  • Added environment variable to bypass certificate validation when connecting to databases with self signed certificates.

  • Schema change detection will now flag columns with null generators that have been made not-null.

Performance

  • Significantly improved the speed that information is gathered about the source database affecting overall UI performance and schema change detection.

  • Added additional diagnostic information around query execution times to the logs.

Bug Fixes

  • Fixed issue where switching table mode to synthesized mode in Database view would not work.

  • Increase accuracy of error messages on Sql Server when tables fail to get created.

  • Integer Key Generator in Oracle for Number columns now base off of the precision, stopping overflow

  • Resolved name collision when subsetting with a table that has more than one foreign key to the same primary key.

  • Fixed preview on tables that use the sequential integer generator.

  • Changing synthesized row counts from database view now saves properly.

  • Popover of column name on table view no longer flashes.

  • Large constant values no longer cause display issues in Database view.

  • Fixed issue with connection test sometimes failing when using MySql.

  • Character Scramble now preserves null values when operation on JSON.

  • Fixed OID collision issue on Oracle.

  • Preview no longer fails when using complicated partitioning strategies.

  • Constant Generator no longer loses preview when you click away.

V101

July 31st, 2020

Features

  • ASCII Character Primary Key Generator which supports a wider range of characters in the column.

  • New API Endpoint for getting information on a single data generation job

  • Key Generators now work on Unique Columns

  • Improvements to PII Detection:

    • Added detection for columns containing passwords, postal codes, and VINs.

    • Reduced false positives for SWIFT codes, ICD9, ICD10, US Cities

  • Switched to a new MySQL driver

  • Make XML generator more human readable

  • Allow synthesize mode on Conditional Generator and Unique Email Generator

  • Add event generator in Oracle

Performance Improvements

  • UI performance for resolving schema difference

Bug Fixes

  • Better checking of uniqueness requirements for columns

  • Fixed constraints on large tables timing out in SqlServer

  • Fixed conversion issue with XML columns outside of subsets during subsetting

  • Block subsetting with synthesized tables

  • Do not allow source and destination to be the same in MySQL

  • Make preview stay up to date when generator is removed from different view

  • Fixed vulnerability with lodash

V97

July 15th, 2020

Features

  • Tables on Sql Server can now be updated in an incremental fashion where only the changes since the last generation are processed. More details here.

Incremental Updates
  • Added support for customer categorical generator in synthesize mode.

  • Date truncation generator has been added so you can truncate dates to a specified date part.

  • Added support for timestamp ranges in Postgres.

  • Phone number generator now supports multinational phone numbers. The output phone number will match the country/region of the input phone number.

  • Integer key generator can now operate on an Int64/Long column.

Bugs

  • Fixed issue where a specifically sized payload could cause the API request to fail.

  • Geo generator properly passes through null values.

  • Added back remove table API.

  • Tables containing unicode characters now work in preserve destination mode on Sql Server.

  • Improved handling of transaction scopes for Sql Server.

  • Fixed issue where generators applied through the conditional generator sometimes used their default options during generation.

  • Generators added through autodetect are now able to be marked consistent/deferentially private.

  • Resolved multithreading issue using the key generators on tables with linked foreign keys.

  • Sql server date columns no longer show time in the table view.

V93

July 1st, 2020

Features

  • Added ability for full name generators to be consistent with partial name generators.

  • Generator popovers now scroll after reaching a certain height.

  • Add Random Double generator.

  • Add ability to delete Tonic account.

  • Improved job progress tracking for Oracle.

  • Allow workspaces to be imported and exported.

  • Added support for conditional generator on Sql Server image columns.

  • Negative numbers can now be used in the constant generator with numeric columns.

  • Added support for timezone arrays in Postgres.

Performance Improvements

  • Significant performance increase when using the continuous generator.

Bug Fixes

  • No longer displaying out of date conditional generator configuration in the collapsed view.

  • Fixed issue with not being able to backspace the constant value on a constant generator applied to a numeric column.

  • User's can no longer duplicate subset targets on the subset configuration page.

  • Removed conditional generator as a sub-generator choice when using the xml or json mask generators.

  • Fixed issue where consistency could not be applied to a generator inside the conditional generator.

  • Loading indicators on the table view are now more consistent.

  • Continuous generator now works with smaller partitions when generating statistics.

  • Job details page now scrolls when number of steps is too large for the screen area.

  • Fixed issue where adding a custom categorical generator inside the JSON mask generator could cause an error.

  • Removing a table from the Schema Changes page now properly removes it from the subsetting configuration.

  • Disabled including tables out of subset was enabled when subsetting was not.

  • Fixed broken link for API documents on Job Details page.

  • Fixed issue where sequential integer generator didn't reset itself between generations.

  • Forced browser to get new version of assets after each release.

  • Add sql injection safety to generators that partition.

  • Fixed issue with switching between workspaces on the jobs view.

  • Moved popovers in database view to not be blocking other columns.

  • Fix issue with attribute info generator failing on empty attributes.

  • Stopped job progress from updating after job is complete.

  • Fixed issue with canceled jobs still running if they were cancelled before they could run.

V90

June 17th, 2020

Features

  • Added the ability to specify a domain and enable consistency on the unique email generator

  • Conditional generator can now be used with the unique email generator

  • Added confirmation step for canceling a generation.

  • Added support for JSON and JSONB arrays in Postgres.

Performance Improvements

  • Updates to your workspace are now done through Json Patch.

Bug Fixes

  • Fixed issue where a table was unable to be switched to synthesized mode.

  • Fixed issue where min and max on the random integer generator were not editable.

  • Fixed issue where multiple changes might not be saved properly if executed within a few milliseconds of each other.

  • Fixed issue where constraints failed to apply to a large Sql server database.

  • Fixed display issue where the password input appeared to be filled in on the destination database connection screen.

  • Swagger docs no longer report enum fields as integers.

  • Fixed issue where edit, copy, and delete workspace buttons where still clickable even when disabled.

  • Decreased clickable area on consistency and differential privacy switches.

  • Fixed issue with NaN values in Postgres double fields.

V87

May 29th, 2020

Features

  • A new subsetting option allows you to process tables that are not included in your subset.

  • Several Conditional Generator improvements:

    • Conditional Generator can now filter rows by a regular expression.

    • Conditional Generator now supports the following additional generators: Categorical, Custom Categorical, Alphanumeric Key, Numeric String Key, Integer Key.

    • 'Is null' and 'is not null' operators added to conditional generator

  • Custom Categorical Generator now supports numeric types.

  • Random timestamp generator can now be added to text columns.

  • Password managers are now prevented from interfering with the database connection form.

  • Improved logging for constraint application on Sql server.

  • Renamed 'private' to 'sensitive' when referring to a column with data that needs to be protected.

  • Column headers are now red when a column is sensitive but not protected.

Performance Improvements

  • Workspace updates no longer happen on keystroke and will now wait until you exit the field or popup.

  • Changed the way the application loads workspaces to increase performance for large workspaces.

Bug Fixes

  • Fix display issue with Custom Categorical Generator when there were no categories.

  • Fixed error message when applying random timestamp generator.

  • Fixed issue with column editor size changing when marking a column as sensitive.

  • Fixed an issue with xml columns on Sql server

V84

May 12th, 2020

Features

  • Tables can now be filtered by their current mode in the database view.

  • Added support for the 'contains' operator on the conditional generator.

  • Company name generator now supports consistency.

  • Added version check when editing a workspace to better support multitab/multibrowser use. Multiple users can now edit the same workspace without worrying about race conditions. Note: this is a first step, more multiuser features are in development.

Performance Improvements

  • Improved speed of tables in mask mode by fixing issue introduced in v76.

Bug Fixes

  • Fixed issue with applying a large number of constraints with Sql Server.

  • Random boolean, IP address, and random integer generators now work correctly with the conditional generator

  • Tasks that complete immediately now display correctly in the job details page.

  • Fixed display issue with data preview button on privacy hub.

V83

May 8th, 2020

Features

  • Added a generator for shipping container codes.

  • Custom Categorical Generator now works on json fields.

  • Destination database names can now differ from the source for Sql Server.

  • Job completion time has been added to the Job Details page.

  • Additional diagnostic tools added to docker containers.

Performance

  • Column search performance in bulk editor significantly improved.

  • Removed Intercom

Bug Fixes

  • Preserving a partitioned table on Sql Server no longer causes an error.

  • Fixed issue with the Categorical Generator when the table is empty.

  • Fixed issue with the last line break sometimes being removed from the category list on the Custom Category Generator.

  • Fix preserve tables for non-Oracle databases.

  • Styling fixes for Conditional Generator when used in Firefox.

  • Fixed issue where insert into an XML column can fail on Sql Server when the payload has a large attribute, too many nodes, or too much nesting.

V80

Apr 28, 2020

Features

  • You can now apply generators conditionally based on the values within the column.

Conditional Generator
  • Added support for array types in Postgres

  • Added a custom categorical generator, which allows you to specify a list of categories for a given column

Custom Categorical Generator
  • Starting a data generation via the API will now return the ID of the job

  • Primary key generators can now be applied in the bulk edit view.

  • Added additional information on the job details page and in the logging, especially around subsetting.

  • Return existing refresh tokens if they exist to better enable multiple open sessions for the same account.

Performance

  • Improved the speed of the integer primary key generator

  • Improved subsetting performance when subset targets are referenced by large tables

Bug Fixes

  • Tonic will now ignore tables created by unsupported database add-ons that previously would require the user to truncate.

    • You may see these tables show up as a schema change alert. Most common ones are spatial_ref_sys and sysdiagrams.

  • Skip some the relationship checks needed for synthesis when table mode is not set to synthesize.

  • Fixed display issue where primary key would show as needing a generator for synthesis mode.

  • Fixed issue where the table drop down on the subset configuration view sometimes rendered in the wrong location.

  • Json path generators will no longer overwrite objects or arrays.

  • Fixed issue where identity sequence was not being set correctly on the destination table.

  • Fix issue when empty strings are present in a column with the numeric string or alphanumeric string primary key generators attached

V74

Apr 14, 2020

Features

Bug Fixes

  • Replaced browser alert with proper Toast notification when a job fails to be scheduled

  • Oracle now returns columns in the order of their ordinal position

  • Adjusted sizing on subset dropdown

  • Fixed issue where the progress tracker could fail to write events near the end of a generation.

V73

Apr 6, 2020

Features

Security

  • Upgraded Java version on docker images to resolve security issues.

Bug Fixes

  • Fixed issue where consistency did not work across multiple workers.

  • Ensure that subset configuration is validated when the subset configuration page has not been loaded yet.

  • Fixed issue where Privacy Hub could continue to refresh even after a scan was complete.

  • Incorrect generators are no longer showing in the MS Sql Server image data type.

  • Fixed magnifying glass placement on Privacy Hub on field hover.

  • Fixed various issues with toast notifications. Reset scrollbar position on table selection change in bulk editor.

V72

Mar 25, 2020

Features

  • Improved logging when testing a connection

  • Added MySql 8 compatibility for our collated queries.

  • Complete overhaul of progress tracking and the job details page. Jobs Details now has progress bars, spinners, and various icons to present the current status of a job.

Performance

  • Index restoration parallelized on MySql.

  • Foreign keys added to a MySql table are now added with 'foreign_key_checks=0'.

  • Tables marked as 'Preserve Destination' now complete significantly faster on SQL Server.

Security

Bug Fixes

  • Fixed issue in MySql Restore Foreign Key query

  • Fixed issue when user changes a JsonMask or XmlMask generator to other generator types

  • Tiny Int not treated as Boolean in MySQL

  • Fixed MySql issue with preserved tables having the same name in different schemas

  • PII Scanner now picks up on zip code data by column name.

  • PII detection using lookup tables now removes leading and trailing whitespaces and converts all values to lowercase. This fixes our PII detection for cities and states.

  • Test Connection button no longer holds on to table locks on MySql databases.

  • Workspace Name field now receives focus when workspace editor is opened.

  • Enter key can now be used to save Workspace Editor.

  • Allow UUID primary keys as downstream type in subsetting mode.

  • Fix issue with GUID primary keys when subsetting is enabled.

  • Long columns names now render properly on Privacy Hub.

  • Fixed issue with bulk adding consistent generators.

  • Account creation no longer hangs when server has no internet connection.

  • Fixed spacing issue with JSON Mask Generator interface.

V66

Mar 10, 2020

Features

  • Bulk operations on multiple columns in the Database View:

    • Add and remove generators

    • Change privacy status

Bulk Column Operations
  • UI now distinguishes between primary (gold) and foreign keyed (black) columns

  • Advanced search for columns in the Database view.

Advanced Search
  • Generator select popover on database view is now at feature parity with same popover in Table View.

  • Ability to manually mark columns as private

  • Columns with private data that are not protected are flagged as red (Database View only) and have a warning in the generator selector

  • tonic_worker now has a /health endpoint

  • Phone Number Generator supports integer columns

  • Differential Privacy warnings now show for Categorical Generator when being used on JSON and XML columns

  • Ability for user to provide their own secret to be used for encrypting both JWTs and database connection info

  • Ability to change password (under User Settings in the hamburger menu located on the right of the menu bar)

Performance

  • Preserved tables in MySQL are no longer copied into the preserve schema via SELECT INTO. Instead they are RENAME'd

  • No longer drop and restore primary keys and indexes in MySQL

Bug Fixes

  • Better support for 3 byte unicode characters when writing data to masked tables in Postgres

  • Fixed rare concurrency exception being thrown in progress tracker when jobs are running

  • Fixed malformed query in Sql Server which is used when subsetting

  • MySQL handling of character

  • Additional logging of MySQL index handling

V62

Feb 19, 2020

Features

  • Bulk operations on multiple tables in the Database View

  • Where clause in subset - in addition to a target %, you can now specify a custom where clause as the target for subsetting

  • URL Generator is more private

  • PII type is shown in Privacy Hub

Bugs

  • Styling issues for Chrome and Firefox

  • Event Ordering defaults corrected

  • Columns with nothing put nulls gave bad data previews

V61

Feb 11, 2020

Features

  • Health Endpoints /health added for web server and PII detection server

  • Non-null values are no longer shown during data preview

  • Sort by column name in privacy hub

  • Ability to delete a workspace

  • Columns part of truncated tables are now labelled as such in the Database view

  • PII Detection now uses the Character Scramble on suspected phone numbers

  • Support for Noda time in PostgresSQL and throughout the product

  • Other columns no longer shows in the database view

  • Email Generator now has an excluded domain filter

  • Subset with Foreign Key Upload File

Performance

  • Database View UI rendering improvements

  • UI improvements for MySQL databases with 1000s of tables

  • Ability to process tables in parallel across all databases. This setting is controlled by the TONIC_TABLE_PARALLELISM environmental variable

  • Faster reads from source database. This affects SQL Server, MySQL, and Postgres and should see improvements for reads from the source database across the board

  • Optimization for SQL Server to improve writes

  • Better distribution of load to workers

Bug Fixes

  • Fixed bug in XML generation for Postgres

  • Reduce rate of false positive when detecting zip codes

  • Subsetting now properly supports self-referential foreign keys

  • SQL Server uniqueidentifier columns now properly work with the UUID generator

  • Privacy hub has better styling for long column names

  • Fixed issues related to change of focus on Target % input on subset configuration page

V57

Jan 2, 2020

Features

  • Support consistency for the email generator on a unique column

  • Rename and consolidation of docker images. Please contact [email protected] before upgrading for a new docker-compose file

  • Massive refactor of subsetting, mostly behind the scenes. User facing improvements:

    • Support for all databases (not just Postgres and MySQL)

    • Heroku no longer requires a temporary database

    • Performance boost

  • Categorical generator now supports differential privacy (https://www.tonic.ai/post/differential-privacy-comes-to-tonic/)

Bug Fixes

  • Fix for certain unicode characters in table/schema names

  • Fix for character scramble on numerical columns

  • Several fixes for air gapped deployments

  • Significant performance improvements for the privacy scan

V55

Dec 27, 2019

Features

  • Suggested Generators now show privacy scan suggested generator when applicable

Bug Fixes

  • Better support for SSH Tunneling over private IPs

V52

Nov 15, 2019

Features

  • Generator improvements:

    • Email Generator can be used on columns with a unique constraint

    • Character scramble now supports consistency

    • Filename and Email generators make use of character scramble as opposed to character substitution (more secure). They both also now support consistency.

  • More improvements to PII Detection, specifically speed, resilience, and better name detection

  • Ability to duplicate workspaces (see screenshot below)

  • Show generators applied on column in bulk view when table is truncated

  • Support for SSH Bastion when connecting to source and output database

  • Added new ENV variable for controlling parallelism for MySQL generation

Duplicate Workspace

Bug Fixes

  • Fixed bad description on FK Relationships upload help text

  • Fixed layout issue in Replacement Panel

  • Fixed issue with ICD10 PII detection

  • Fixed routing issues on Heroku

V50

Oct 29, 2019

Features

  • Support for JSON datatype in MySQL

  • More easily remove a generator from a column in the bulk view

  • Improved PII detection

  • Ability to change table modifiers on Database View

  • Subsetting preview of included and excluded tables

Subsetting Preview

Performance

  • Increase speed of restoring indexes for MySQL

Bug Fixes

  • Fixed bad timestamp for jobs when still in queued state

  • Fixed bug found when using some versions of MySQL where dropping an index on a FK column results in an error

  • Fixed description for algebraic generator

  • Resolves an issue that was causing the job details page to fail to load.

  • Fixed MySql failing to create foreign keys because the db/schema wasn't specified in all cases

  • More logging

V42

Oct 14, 2019

Features

  • XML generator - use XPath to target one or more values for a generator. This implementation is very similar to the current JSON generator

  • Partitioning is out of beta. The Continuous and Event generators can be partitioned by either a Categorical or Passthrough generator

  • Support for computed columns in SQL Server

  • Re-sizing of autodetect and bulk view table trees, for long table names

  • Schema change notifications now use small icons

  • Active page is highlighted on sidebar and remember sidebar state

  • Check for correct user permissions on source database for SQL Server when first connecting (MySQL and Postgres already have this)

  • Gzip or br compress api responses

Performance

  • Faster Schema generation for SQL Server

  • Only SELECT required columns when gathering rows for data generation

  • Improved UI perf for SQL Server databases with a large number of objects

Bug Fixes

  • Properly send ALL logs to the log viewer in the Jobs UI

  • Continue when (most) errors occur while running pre-data script on SQL Server

  • Fixed issue blocking successful CSV data generation

  • No longer incorrectly filter out tables in the DBO schema in sql server

  • Improved logging around some SMO tasks for Sql Server

V36

Sep 20, 2019

Features

  • New UI and Schema View

  • Ability to generate in place (MySQL only)

  • Support consistency across databases and runs via the new TONIC_STATISTICS_SEED env variable

Performance

  • Many MySQL performance improvements

Bug Fixes

  • Fixed potential deadlock issue in MySQL

  • No longer re-loads session when fetching a new refresh token

  • Better handling of comments inside JSON blobs that are being masked

V32

Sep 5, 2019

Features

  • Auto-detect now supports booleans

  • Users can now click on data cells to see large text strings

  • PII reports (see screenshot below)

Example PII Report

Performance

  • Additional multi-threading added in various places for data generation

  • Process generated tables before passthrough to encourage fast failure

  • New environmental variables so users can change the connection timeout, and min and max pool size

Bug Fixes

  • Support for Constant Generator on MySQL Blob columns

  • Support MySQL set datatype

  • Improved logging for PII detection

  • Multiple fixes involving switching between synthesize, excluded, and masked

  • Progress tracking added to BigQuery

  • POST to /api/autodetect/config will generate a default config first if one does not exist.

  • Multiple other small fixes

V25

Features

  • PII Detection is now available locally

  • Column widths on the table are now saved between sessions

  • Truncating tables is now prevented when it will lead to FK violations when generating data

  • Ability to more quickly add JSONPaths

  • Schema Diff now auto-fixes issues with model due to schema changes

  • Performance improvements to data generation, and JSON Generator in particular which is now multi-threaded

Schema Changes

Bug Fixes

  • Ability to handle NULL database values in the JSON Mask Generator

  • JSON Paths that start with the same prefix, e.g. $.value and $.value1 no longer cause an error

  • No longer cutting off list of tables in Auto Detect dialog

  • Better clean up of excluded_tables and scaled_tables in Fingerprint when tables are dropped from DB

  • Fixed issue preventing data generation on Postgres Standby DBs (i.e. Read replicas in RDS nomenclature)

V18

Features

  • Can now hit 'Enter' on any input in the Database connection form and it will submit the form

Bug Fixes

  • Better layout for JSON Paths with long names

  • Fixed a 'missing key' prop issue in front end

  • Adding too many escapes to MySQL data when writing to CSV (prior to upload).

  • We now automatically truncate mysql tables before data generation begins

V17

Features

  • JSON generator now supports consistency

  • Better JSON Mask UI

  • Name Generator now has consistency option

  • Can run auto detect without sending logs

  • API Documentation

  • Better tracking of Allos Console logs

  • Better progress tracking

Bug Fixes

  • Intermittent failures of data table are fixed

  • GetAllTablesAndColumns no longer fails when table has columns with multiple constraints on them

  • No longer front end crash when subset target table no longer exists

  • Use Custom Data button location has been fixed

  • Performance improvements

  • Fixed a myriad of issues in the First Connection Experience when using a custom data source

  • Additional fixes for the JSON Mask Generator

  • Fixed issue created in v16 where changing a table's mode to 'TRUNCATE' wasn't being saved properly

V14

Features

  • Insight into whether database generation job is running or queued (was not distinguished in prior versions)

  • Ability to cancel a currently running database generation job

  • Ability to assign generators by jsonpath for the JSON Mask Generator

  • Tonic now prevents users from entering identical source and destination database information

  • When connecting to a database Tonic now defaults the Port to the standard port for the database selected.

  • Tonic now checks if the db account has necessary permissions and warns if it does not.

  • Subset button in header now reflects the state of your subset configuration

  • Google BigQuery now allows user to specify an input and output Dataset for generation.

Bug Fixes

  • Better escaping of schema and table names in mysql

  • Workspace edit dialog file upload inputs (Foreign Key upload and BigQuery Service Account upload) were out of sync

  • No longer show stale data in the table UI when switching between tables quickly

  • We were not properly handling queries to pg_catalog tables where columns had recently been dropped from tables

  • Fix to allow synthesizing on mysql tables that have weird characters in table name

  • Fix error message which occurs when user checks permissions on a Postgres database with 0 tables

  • Fix to properly keep in sync the source and destination database names in Sql Server

  • Constant no longer appears twice in the generator list for JSON columns in Postgres

  • Table search dropdown now clears search query once user hits 'enter' or closes dialog

  • CreatedDate in allos_db tables now has correct values by default

V10

  • Ability to Filter data table based on SQL WHERE clauses

Filter table view with a WHERE clause
  • Support for Google Big Query

  • Continuous generator now supports nullable columns

  • Remember last visited Workspace and Table and navigate directly to them when reloading page

  • Added on-premise installation option of one Docker image

  • FK Columns are greyed out in UI

  • And several additional bug fixes

V3

note: we've transitioned from major.minor.hotfix to simple integer based versioning

Features

  • Autodetect Generators (beta) - this feature that scans the source dataset and by analyzing both the structural properties (data types, column names, foreign key constraints) and the content of each data field it take a first pass at picking generators and linking generators.

Autodetect Generators
  • Ability to define custom foreign key relationships, this addresses the issue where a database doesn't have any FK constraints or it's missing some

Upload a custom definition of foreign key relationships
  • Algebraic Generator - when you link 3 or more columns (A, B, C, ...) with the algebraic generator applied it will search the space of functions (A + B = C, A / B = C, ... ) to find the function that best describes the algebraic relationship between the linked columns.

Algebraic generator applied to 3 related numeric columns
  • We now support ability to specify via an ENV variable which schemas to include

  • Email generator now supports a custom email domain

  • Renamed Gaussian generator to Continuous

  • Added TIN generator

    • First two digits are always 00

    • Guaranteed uniqueness. We use each of the 10M possibilities once and only once except when needing to be consistent (see above). This generator cannot be used in tables with more than 10M unique rows

    • Format preserving. If the cell value uses a hyphen we add a hyphen, otherwise we do not.

  • Upgraded pg_dump to v11

2.5.0

Features

  • A handful of generators now support a notion of consistency across the database. In short, when consistency is turned on for a specific generator, the same input column will map to the same output column across an entire database (where it's turned on). Consistency can also be used to preserve the carnality of the source dataset in the generated data.

Consistency applied to the Address generator
  • We now use row count instead of scale factor for synthesizing data

  • Synthesize mode now supports starting with a table that's empty

  • Better handling of DateTimeOffset data type in Sql Server

  • Added hostname generator

  • Re-factored of how generators are executed during data generation

  • We now show a tutorial video the first time a user logs in

  • Renamed Table Mode 'Excluded' to 'Truncated'

  • Build scripts now build win10-x64 assets

  • Changed the (!) failure icon in the jobs dropdown to a button so it's more obvious

  • UI and server versions now show when you click on the Tonic logo

  • CSVs with headers now map to column names

Bug Fixes

  • No long require a file to exist for the google_application_credentials env variable

  • Couldn't generate data if we used the MAC address generator with colons

  • Support for MySQL datetimes with 0000-00-00 00:00:00

2.4.0

  • Improved onboarding experience for first time users

  • Dropped walkthrough tutorial from 2.3.0

  • Renamed platform from Allos to Tonic

  • Additional support for date fields

    • Added Random Timestamp generator

    • Added Event generator

  • Partitioning for select generators (behind a feature flag) - contact [email protected] if you want to try this feature

Event generator and Gaussian with partitioning by city

2.3.0

  • Added walkthrough tutorial and demo dataset for new users

  • IP Generator now uses 100% IPv4 by default

  • Fixed bad UX with Min/Max in Random Integer generator

  • Improved Gaussian Generator performance

  • Added user menu

  • The distribution of nulls in the source dataset is now persisted in transformed columns in the generated dataset

  • Added SSN Generator

  • Improved Address Generator (Zip-> City->State hierarchy preserved when columns using the address generator are linked)

2.2.0

  • Support for CSV files as a datasource

  • Support for arrays in Postgres

  • Improved PII detection

  • Support for Postgres databases that don’t have passwords

  • Renamed several generators

    • Categorical —> Shuffle

    • Random String —> Hash

    • String Mask —> Character Substitution

    • Text Mask —> Text Scramble

  • Added generator description callouts

Generator Description

2.1.0

  • Support for SQL Server as a datasource

  • JSON masking generator

  • Subsetting (Postgres only) - we integrated our open source subsetting tool

Subsetting

2.0.0

  • Support for MySQL as a datasource

  • Synthesized Mode (beta) - in addition to masking, you can now synthesize any number of rows in a table while preserving foreign key relationships

Synthesized Mode