Datasets workflow for text redaction and synthesis

You can use Textual to generate versions of files where sensitive values are synthesized or redacted.

To only generate redacted files, you use a Tonic Textual dataset.

You can also optionally configure a Textual pipeline to generate redacted files in addition to the JSON output.

At a high level, to use Textual to create synthesized or redacted data:

Create and populate a dataset or pipeline

Create a Textual dataset or pipeline. A dataset is a set of files to redact. A pipeline is used to generate JSON output that can be used to populate an LLM system. Pipelines also provide an option to generate redacted versions of the selected files.
Add files to the dataset or pipeline.
Textual supports almost any free-text file, PDF files, .docx files, and .xlsx files. For images, Textual supports PNG, JPG (both .jpg and .jpeg), and TIF (both .tif and .tiff) files.
For a dataset or an uploaded files pipeline, as you add the files, Textual automatically uses its built-in models to identify entities in the files and generate the pipeline output. For a cloud storage pipeline, to identify the entities and generate the output, you run the pipeline.

Review the redaction results

For a dataset, review the types of entities that were detected across all of the files. For pipeline files, the file details include the entities that were detected in that file.

Configure entity type handling

At any time, including before you upload files and after you review the detection results, you can configure how Textual handles the detected values for each entity type.

For datasets, you can also provide added and excluded values for each entity type.

Select the handling option for each entity type

By default, Textual redacts the entity values, which means to replace the values with a placeholder that identifies the type of sensitive value. For example, PERSON, LOCATION. For PDF files and image files, redaction means to cover the value with a black box. For a given entity type, you can instead choose to synthesize the values, which means to replace the original value with a realistic replacement. You can also choose to ignore the values, and not replace them. For a dataset, Textual automatically updates the file previews and downloadable files to reflect the updated configuration.

For a pipeline, the updated configuration is applied the next time you run the pipeline, and only applies to new files.

Define added and excluded values for entity types

Optionally, in a dataset, you can create lists of values to add or exclude as a specific entity type. You might do this to reflect values that are not detected or that are detected incorrectly.

Pipelines do not allow you to add or exclude individual values.

Manually update PDF files

Datasets also provide additional options for PDF files. These options are not available in pipelines.

You can add manual overrides to a PDF file. When you add a manual override, you draw a box to identify the affected portion of the file.

You can use manual overrides either to ignore the automatically detected redactions in the selected area, or to redact the selected area.

To make it easier to process multiple files that have a similar format, such as a form, you can create templates that you can apply to PDF files in the dataset.

Download the redacted and synthesized files

After you complete the redaction configuration and manual updates, you download the redacted and synthesized versions of the files to use as needed.

Last updated 1 month ago

Create and populate a dataset or pipeline

Create a Textual dataset or pipeline. A dataset is a set of files to redact. A pipeline is used to generate JSON output that can be used to populate an LLM system. Pipelines also provide an option to generate redacted versions of the selected files.

Add files to the dataset or pipeline.

Textual supports almost any free-text file, PDF files, .docx files, and .xlsx files. For images, Textual supports PNG, JPG (both .jpg and .jpeg), and TIF (both .tif and .tiff) files.

For a dataset or an uploaded files pipeline, as you add the files, Textual automatically uses its built-in models to identify entities in the files and generate the pipeline output. For a cloud storage pipeline, to identify the entities and generate the output, you run the pipeline.

Configure entity type handling

At any time, including before you upload files and after you review the detection results, you can configure how Textual handles the detected values for each entity type.

For datasets, you can also provide added and excluded values for each entity type.

Select the handling option for each entity type

For a pipeline, the updated configuration is applied the next time you run the pipeline, and only applies to new files.

Define added and excluded values for entity types

Optionally, in a dataset, you can create lists of values to add or exclude as a specific entity type. You might do this to reflect values that are not detected or that are detected incorrectly.

Pipelines do not allow you to add or exclude individual values.

Manually update PDF files

Datasets also provide additional options for PDF files. These options are not available in pipelines.

You can add manual overrides to a PDF file. When you add a manual override, you draw a box to identify the affected portion of the file.

You can use manual overrides either to ignore the automatically detected redactions in the selected area, or to redact the selected area.

To make it easier to process multiple files that have a similar format, such as a form, you can create templates that you can apply to PDF files in the dataset.