Enabling consistency

Consistency is an option for some generators that when turned on, maps the same input to the same output across an entire database.

Consistency can also be maintained across multiple databases of varying types. For example, if consistency is turned on for a name generator, it always maps the same input name (for example, Albert Einstein) to the same output (for example, Richard Feynman).

You can also view this video overview of consistency.

Why use consistency?

The primary reasons for using consistency are to:

Enable joining on columns that don't have explicit database constraints in the schema. This is often seen with values such as email addresses. With consistency, you can completely anonymize an email address and still use it in a join.
Preserve the approximate cardinality of a column. For example, a city column contains 50 different cities. To randomize this column but still have ~50 cities, you can use consistency to maintain the approximate cardinality. Because consistency does not guarantee uniqueness, the cardinality might change. However, it is guaranteed to not increase. If unique 1-to-1 mappings are required, a Key generator should be used.
Match duplicated data across 1 or more databases. For example, you have a user database that contains a username in both a column and a JSON blob, and another database that contains their website activity, identified by the same username values. To anonymize the username, but still have the username be the same in all locations/databases, use consistency.

Types of consistency

Self-consistency

Self-consistency indicates that the value in the destination database is consistent with the value of the same column in the source database.

For example, a column contains a first name. You make the assigned generator self-consistent. A given first name in the source database is always replaced by the same first name in the destination database. For example, the first name value John is always replaced by the value Michael.

Consistency with another column

Consistency with another column indicates that the value in the destination database is consistent with the value of a different column in the source database.

For example, a column contains an IP address. You make the assigned generator consistent with the username column. Every row that has the username User1 in the input database has the same IP address in the destination database.

When you select a generator as the sub-generator for a composite generator, in most cases you cannot configure the generator to be consistent with another column. Only the Conditional generator and the Regex Mask generator allow a sub-generator to be consistent with another column.

Note that consistency with another column cannot be configured in a generator preset. You can only configure it when you configure an individual column.

Enabling consistency

To enable consistency, on the generator configuration panel, toggle the Consistency switch.

Not all generators support consistency.

Consistency is a function of the both the data type and the value.

For example, a numeric field contains the value 123. A string/varchar field contains the value "123".

Both fields have consistent generators applied.

The output is not consistent between the two fields.

Consistency example

To demonstrate the effect of consistency on the output, we'll use a column that contains a first name, and that uses the Name generator.

Here is the sample input and output when consistency is not enabled:

In this sample data, the first name Melissa appears twice, but is mapped to Walton the first time and Linn the second time.

Here is the sample input and output when consistency is enabled:

In this case, the first name Melissa is mapped to Rosella both times.

Consistency considerations

Consistency does not imply uniqueness

A consistent generator ensures that the same input value always produces the same output value.

It does not guarantee that two different input values produce two different output values.

Consistent generators are not 1:1 mappings.

Consistency can reduce data privacy

Consistency can reduce the privacy of your data, because it reveals something about the frequency of the data values.

For example, if someone is familiar with the source data values and frequency, they might be able to connect the source and destination values. For example, they know that Jane appears 20 times and Michael appears 3 times in the source. When they see 20 instances of Susan and 3 instances of John, they might infer that Susan is mapped from Jane and John from Michael.

However, this risk does require some knowledge of the source data. Tonic Structural does not store mappings of the source data to the destination data. In other words, someone can see that in the destination data the name Susan appears 20 times and the name John appears 3 times. But without any knowledge of the source data, they cannot determine that Susan is mapped from Jane and John is mapped from Michael.

Also, the mapping of source to destination values is not guaranteed to be unique. Both Jane and Michael could be mapped to John. In that case there would be 23 instances of John, which would not match the frequency of a specific source value. To guarantee unique values, use a primary key generator.

Consistency is across an entire database

Any column, regardless of which table it resides in, is consistent with any other column that uses the same consistent generator.

For example, your database includes a Customers table and an Employees table. Each table contains a column for the first name of the customer or employee. You assign the Name generator to both columns to generate a first name, and make the generators consistent. The same first name value in either column is mapped to the same destination value. For example, the first name John is always mapped to Michael, whether the name John appears in the Customers table or the Employees table.

However, by default, consistency is not guaranteed between data generation runs, even if the run is on the same database.

Enabling consistency across runs or multiple databases

By default, consistency is only guaranteed across a single data generation for a single workspace.

For example, for a column that contains a first name value, you assign the Name generator and configure the generator to be consistent. The first time you run data generation, all instances of the name John might be replaced with Michael. The next time you run data generation, all instances of the name John might instead be replaced with Gregory.

You can enable consistency across runs and workspaces so that, for example, every time you run a data generation, John is always replaced with Michael.

To do this, you configure a seed value. You can either:

Configure the Structural environment setting TONIC_STATISTICS_SEED. This ensures consistency across all workspaces and data generation runs.
Configure a seed value for a workspace. This ensures consistency across all data generation runs for that workspace, as well as across other workspaces that have the same seed value.
Disable cross-data generation consistency for a workspace. This indicates to not have consistency across data generation runs or with other workspaces.

Configuring a Structural seed value

To ensure consistency across all data generations and workspaces, add the following environment setting to the Structural worker and web server containers:

TONIC_STATISTICS_SEED: <ANY 32-BIT SIGNED INTEGER>

When you configure a value for this environment setting, then consistency is across all data generations for all workspaces that do not either:

Have a workspace seed value configured.
Have disabled consistency across data generations.

Overriding the Structural seed value for a workspace

For an individual workspace, you can override the Structural seed value. When you override the Structural seed value, you can either:

Disable consistency across data generation runs for the workspace.
Provide a seed value for the workspace.

When a workspace has a configured seed value, then consistency is across the data generation runs for that workspace.

Consistency is also across all of the data generations for all of the workspaces that have the same seed value.

On the workspace details view, to override the Structural seed value:

Toggle Override Statistics Seed to the on position.
To disable consistency across data generations, click Don't use consistency.
To provide a seed value for the workspace:
1. Click Consistency value.
2. In the field, enter the seed value. It must be a 32-bit signed integer. The value defaults to the current value of TONIC_STATISTICS_SEED.

Generators that can only be made self-consistent

The following generators can be made consistent to themselves. This means that the same input value in the column always produces the same output value.

Generators that can be made self-consistent and to other columns

The following generators can be made consistent either to themselves or to other columns.

When a column is consistent to another column, the output value is based on the other column.

For example, a column contains a company name. You assign the Company Name generator, and make it consistent with the username column. Every row that has the username User1 in the input database has the same company name in the destination database.

Last updated 2 months ago