Enabling consistency
Consistency is an option for some generators that when turned on, maps the same input to the same output across an entire database.
Consistency can also be maintained across multiple databases of varying types. For example, if consistency is turned on for a name generator, it always maps the same input name (for example, Albert Einstein) to the same output (for example, Richard Feynman).
The primary reasons for using consistency are to:
- Enable joining on columns that don't have explicit database constraints in the schema. This is often seen with values such as email addresses. With consistency, you can completely anonymize an email address and still use it in a join.
- Preserve the approximate cardinality of a column. For example, a city column contains 50 different cities. To randomize this column but still have ~50 cities, you can use consistency to maintain the approximate cardinality. Because consistency does not guarantee uniqueness, the cardinality might change. However, it is guaranteed to not increase. If unique 1-to-1 mappings are required, a Key generator should be used.
- Match duplicated data across 1 or more databases. For example, you have a user database that contains a username in both a column and a JSON blob, and another database that contains their website activity, identified by the same username values. To anonymize the username, but still have the username be the same in all locations/databases, use consistency.
Self-consistency indicates that the value in the destination database is consistent with the value of the same column in the source database.
For example, a column contains a first name. You make the assigned generator self-consistent. A given first name in the source database is always replaced by the same first name in the destination database. For example, the first name value
John is always replaced by the value Michael.Consistency with another column indicates that the value in the destination database is consistent with the value of a different column in the source database.
For example, a column contains an IP address. You make the assigned generator consistent with the username column. Every row that has the username
User1 in the input database has the same IP address in the destination database.When you select a generator as the sub-generator for a composite generator, in most cases you cannot configure the generator to be consistent with another column. Only the Conditional generator and the Regex Mask generator allow a sub-generator to be consistent with another column.
Note that consistency with another column cannot be configured in a generator preset. You can only configure it when you configure an individual column.
To enable consistency, on the generator configuration panel, toggle the Consistency switch.
Not all generators support consistency.
Address Generator with Consistency Switch
Consistency is a function of the both the data type and the value.
For example, a numeric field contains the value 123. A string/varchar field contains the value "123".
Both fields have consistent generators applied.
The output is not consistent between the two fields.
To demonstrate the effect of consistency on the output, we'll use a column that contains a first name, and that uses the Name generator.
Here is the sample input and output when consistency is not enabled:
Sample input and output for a first name field with consistency disabled for the Name generator
In this sample data, the first name Melissa appears twice, but is mapped to Walton the first time and Linn the second time.
Here is the sample input and output when consistency is enabled:
Sample input and output for a first name field with consistency enabled for the Name generator
In this case, the first name Melissa is mapped to Rosella both times.
A consistent generator ensures that the same input value always produces the same output value.
It does not guarantee that two different input values produce two different output values.
Consistent generators are not 1:1 mappings.
Consistency reduces the privacy of your data, because it reveals something about the frequency of the data values.
However, Tonic does not store mappings of the source data to the destination data. In other words, someone can see that in the destination data the name Susan appears 20 times and the name John appears 3 times. But they cannot determine that Susan is mapped from Jane and John is mapped from Michael.
Any column, regardless of which table it resides in, is consistent with any other column that uses the same consistent generator.
For example, your database includes a Customers table and an Employees table. Each table contains a column for the first name of the customer or employee. You assign the Name generator to both columns to generate a first name, and make the generators consistent. The same first name value in either column is mapped to the same destination value. For example, the first name John is always mapped to Michael, whether the name John appears in the Customers table or the Employees table.
However, by default, consistency is not guaranteed between data generation runs, even if the run is on the same database.
To ensure consistency across data generations, add the following environment variable to the Tonic worker and web server containers:
TONIC_STATISTICS_SEED: <ANY 32-BIT SIGNED INTEGER>
The following generators can be made consistent to themselves. This means that the same input value in the column always produces the same output value.
The following generators can be made consistent either to themselves or to other columns.
When a column is consistent to another column, the output value is based on the other column.
For example, a column contains a company name. You assign the Company Name generator, and make it consistent with the username column. Every row that has the username User1 in the input database has the same company name in the destination database.
Last modified 1mo ago