What is the Textual Agent?

The Textual Agent is a chat-based tool to help you to explore the content of dataset files and configure the file processing.

When does the Textual Agent start?

The Textual Agent starts when you create or edit a Textual dataset.

Each combination of dataset and user has its own Textual Agent chat.

Enabling the Textual Agent on a self-hosted instance

The Textual Agent is always enabled on Textual Cloud.

For information on how to enable the Textual Agent on a self-hosted instance, go to .

Tonic Textual allows you to put your text-based data to work for you.

A Textual dataset is a collection of files from a local file system or cloud storage. Textual scans the dataset files to identify sensitive values. You can then choose to redact or replace those sensitive values, to produce output files in the same format that you can safely use for development and training.

A guided redaction project identifies blocks out sensitive values in files. For example, you might use guided redaction to prepare documents in response to a Freedom of Information Act request. Textual scans the files to identify values. You can then review and adjust the results before you download the output files.

You can use the Textual SDK or the Textual REST API to manage datasets or to remove sensitive values from individual text strings and files.

Want to know what's in the latest Textual releases? Go to the .

The User Profile page displays a summary of information about your Textual account.

From the user profile page, you can copy your organization identifier, configure a team name, and manage your Textual API keys. For more information about managing your Textual API keys, go to .

Displaying the User Profile page

To display the User Profile page:

Note that these instructions are for setting up a new account on Textual Cloud. For a self-hosted instance, depending on how it is set up, you might either create an account manually or use single sign-on (SSO).

Signing up for Textual

To get started with a new Textual account:

1. Go to .

2. Click Sign up.

3. Enter your email address.

4. Create and confirm a password for your Textual account.

5. Click Sign Up.

Textual creates your account and sends you an email message to activate the account.

After you activate your account and log in, Textual displays the Textual home page, which you can use to preview how Textual detects and replaces values. For more information, go to .

Using the Textual free trial

When you set up an account on Textual Cloud, you start a Textual free trial.

Using the Getting Started options

On the home page, the Getting Started section provides links to tasks and information related to:

• Using the

• Working with

• Working with

• Working with

To view the available options for a panel, hover the mouse over it. Hover over an option to display a tooltip to identify the action that the option performs.

Word count limit

During the free trial Textual scans up to 100,000 words for free. Note that Textual counts actual words, not tokens. For example, "Hello, my name is John Smith." counts as six words.

After the 100,000 words, Textual disables scanning for your account. Until you purchase a pay-as-you-go subscription, you cannot add files to a dataset.

Viewing your current usage

To display your current usage, in the navigation menu, click the usage icon.

Next steps - pay-as-you-go or product demo

Textual also prompts you to purchase a , which allows an unlimited number of words scanned for a flat rate per 1,000 words.

You can also request a Textual product demo.

Expanding and collapsing the Textual Agent

The Textual Agent displays at the right side of the application.

When the Textual Agent is collapsed, to expand it, click the agent icon.

When the Textual Agent is expanded, to collapse it, click the agent icon.

An entity type is a category of entity value. For example, the entity value John might be an example of the Given Name entity type.

Tonic Textual comes with a built-in set of entity types that it always detects.

You can also configure custom entity types, which you can use to detect values that are not covered by the built-in entity types.

When you create a custom entity type, you can either:

• Use regular expressions to identify matching values. You might create this type of custom entity when there are a limited number of values or the values follow specific formats that can easily be identified with a regular expression.

• Training a model is an iterative process that can take hours or days, depending on your data. You might create this type of custom entity when there are a large number of values that do not follow a specific format. The values need to be identified more by context.

You can also view this .

To start a new custom entity type:

1. On the Custom Entity Types page, click Create Custom Entity Type, then select Model-based entity type.

1. On the next panel, in the Custom entity type name field, provide a name for the custom entity type.

2. In the Annotation guidelines text area, provide instructions for how the model should identify values that you want to find. This is the first version of the model guidelines.

1. Click Next.

Textual displays the Test data setup page.

Before you can use a model-based entity type, you must select the model to use.

On the model list page, the active model is marked as Active.

To change the active version:

1. Hover the mouse over the model to make the active model for the entity type.

2. Click Activate.

Renaming the entity type

To change the entity type name, on the entity type details page:

1. Click the actions menu next to the entity type name.

1. In the menu, click Edit entity name.

2. On the Edit Name panel, provide the new name for the entity type.

3. Click Save.

Deleting the entity type

You cannot delete an entity type that is enabled for any datasets. For information on how to enable and disable an entity type in datasets, go to .

To delete a model-based custom entity type, in the entity types list, click the delete icon.

On the confirmation panel, click Delete.

You can ask the Textual Agent to provide information about the content of the dataset files and to update the entity handling configuration.

The following are just a few examples:

To delete a dataset:

1. Either:

The dataset name displays at the top left of the dataset details page.

To change the dataset name:

1. On the dataset details page, click

To display the synthesis options for an entity type, click the icon next to the handling option dropdown list.

Tonic Textual can process the following types of files:

• .txt

• .csv

• .tsv

For a dataset that generates JSON output, the file preview displays the original file content.

Text files

For a text file, you can switch between Markdown and the JSON output.

To generate original format output files for a cloud storage dataset, on the dataset details page, click Generate to <cloud storage type>.

Tonic Textual generates the output files to the . If the output location is not configured, then the generate option is disabled.

For datasets that produce JSON output, Textual generates the output files automatically as soon as the output location is configured.

For the Given Name and Family Name entity types, you can configure:

• Whether to treat the same name with different casing as a different value.

• Whether to replicate the gender of the original value.

The Project files and Entity settings page display the following summary results for the dataset::

• The number of detected entities

• The percentage of dataset content that is sensitive

Before you create guided redaction projects, configure the following options.

A detected value might match multiple entity types.

For example, a telephone number might match both the Phone Number and Numeric Value entity types.

On most dataset details views, each value is only counted once, for the entity type that it is assigned in the output file. The Analytics page under Entities analysis allows you to choose whether to include a value in the counts for all of the types that it matches.

By default, a detected value is assigned the entity type that it most closely matches. For our example, the telephone number value most closely matches the Phone Number entity type, and so by default is included in the Phone Number count and values list.

If the entity type is ignored, or the value is excluded, then Textual moves the value to the next matching type.

In our example, if you set the handling option for Phone Number to Ignore, then the telephone number value is added to the count and values list for the Numeric Value entity type.

You can assign tags to each guided redaction project, to help to further identify and link projects. For example, you might use a tag to indicate when a project is for a FOIA request.

On the Guided Redaction page, the Tags column contains the assigned tags for the project.

On the project settings panel, the Tags field lists the project tags. To display the settings panel, either:

• On the Guided Redaction page, click the settings icon for the project.

• On the project details page, click More, then click Settings.

To change the assigned tags:

1. Click the Tags column or field.

2. To add a tag, type the tag text, then press enter.

3. To remove a tag, click its delete icon.

4. To remove all of the tags, click the delete icon at the right of the tags field.

To delete a guided redaction project:

1. Either:

   1. On the Guided Redaction page, click the options menu for the project, then click Delete project.

   2. On the project details page, click More, then click Delete project.

2. On the confirmation panel, click Delete.

From the project settings panel, you can change the project name and an optional description.

To display the settings panel, either:

• On the Guided Redaction page, click the settings icon for the project.

• On the project details page, click More, then click Settings.

On the project settings panel:

1. In the Name field, provide the new name.

2. In the Description field, provide the description.

3. Click Save.

Tonic Textual uses dataset permission sets for role-based access (RBAC) of each dataset.

A dataset permission set is a set of dataset permissions. Each permission provides access to a specific dataset feature or function.

Textual provides built-in dataset permission sets. Organizations can also configure custom permission sets.

To share dataset access, you assign dataset permission sets to users and to SSO groups, if you use SSO to manage Textual users. Before you assign a dataset permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group.

To change the current access to the dataset:

1. Either:

   • On the Datasets page, click the share icon.

   • On the Dataset settings page of the dataset details, click Share.

1. The dataset access panel contains the current list of users and groups who have access to the dataset, and displays their assigned dataset permission sets. To add a user or group to the list of users and groups:

   1. In the search field, begin to type the user email address or group name.

   2. From the list of matching users or groups, select the user or group to add.

The Entities Catalog page displays the list of detected entity values for the dataset. To display the Catalog, in the left menu on the dataset details page, click Catalog.

Information in the Entities Catalog

The Entities Catalog lists each instance of an entity value separately. For example, the given name John is detected twice in one file and 3 times in another file. The Entities Catalog then contains 5 entries for John.

The Entities Catalog does not include entities for entity types that have the entity type handling option set to Ignore.

For each value instance, the Entities Catalog includes:

• The entity value.

• How the value appears in the output, based on the selected handling option for the value's entity type.

• The entity type.

• A confidence score to indicate how confident Textual is that the value is correctly detected and identified.

Filtering the Entities Catalog

Filtering by entity value

To filter the list by text in the entity value, in the search field, begin to type the text.

As you type, Textual filters the list to only include entity values that contain that text.

Filtering by entity type

By default, the Entities Catalog list includes all of the entity types. To filter the list to a specific entity type, click All types, then select the entity type. To remove the filter, select All types.

Filtering by file

By default, the Entities Catalog list includes values from all of the files. To filter the list to only include values detected in a specific file, click All files, then select the file. To remove the filter, select All files.

Sorting the Entities Catalog

You can sort the Entities Catalog by the value, transformation, entity type, and confidence score.

To sort by a column, click the column heading.

To reverse the sort order, click the column heading again.

To display the list of custom entity types, in the navigation menu, click the custom entity types icon.

Information in the list

For each custom entity type, the Custom Entity Types page includes the following information:

• Name of the custom entity type.

• Whether the custom entity type is regex-based or model-based.

• For model-based custom entity types, the status. The status indicates where the entity type is in the creation process.

• When the custom entity type was most recently updated.

• The number of projects where the custom entity type is active.

Filtering the list

Filtering by name

To filter the list by the entity type name, in the search field, begin to type text in the name. As you type, Textual filters the list to only include matching entity types.

Filtering by type

By default, the list includes both regex-based and model-based custom entity types.

To filter the list to only include one of the formats:

1. Click Filter by type.

2. In the filter dropdown list, select the format to include.

Filtering by creator

By default, the list includes all of the custom entity types that were created by users in your organization.

To filter the list to only include custom entity types that were created by specific users:

1. Click Filter by creator.

2. In the dropdown list, check the checkbox for each user to include.

Sorting the list

By default, the list is sorted alphabetically by the entity type name.

You can sort the list by the name, status, or update date.

To sort by a column, click the column heading.

To reverse the sort order, click the heading again.

When you add files to a local files dataset, or change the file selection for a cloud storage dataset, Tonic Textual automatically scans the files to identify the entities that they contain.

When you change the dataset configuration, Textual also prompts you to run a new scan. For example, a new scan is required when you:

• Configure added values

• Change the available custom entity types

The file list reflects the current scanning status for the file. A file is initially queued for scanning. When the scan starts, the status changes to scanning. When Textual finishes processing a file, it marks the file as scanned.

When a file needs to be rescanned, a warning icon displays in front of the file name.

As Textual processes each file, it updates the results.

Pausing the file processing

If needed, you can pause the file processing. To pause the processing, click the pause icon.

The information in the results only reflect the files that are scanned.

For a cloud storage dataset, when you generate output, Textual only includes files that are scanned.

Starting a scan on a paused file

After you pause the scan, you can start a scan on individual files.

To start a scan on a file, click the refresh icon for the file.

Downloading logs for files that fail to process

When Textual is unable to process a file, it displays an error for that file.

To download log files for the failed file:

1. Click the options menu for the file.

2. Click Download Logs.

When Tonic Textual generates replacement values, those values are always consistent within the dataset. Consistency means that the same original value always produces the same replacement value.

You can configure how Textual generates the replacement values for each entity type.

You can also configure synthesis options from the Textual Agent.

The Entities Analytics page displays a summary of the detected entity types. For each entity type, you can display the distribution across the dataset files.

It does not include entity types that have their entity type handling option set to Ignore.

You can also use the Textual Agent to ask questions about the detected entity types and counts.

Selecting the value count option

On the Entities Analytics page, you can choose how Textual determines the displayed value counts and entity types.

• Match counts to redacted files - Displays value counts based on the output files. For this view, the counts do not include entity types that are ignored. The counts also resolve entity values that match multiple types and entity values that share some text. Each value is counted as a single type.

• Show all detected entities - Displays the full detection value counts. For this view, the counts per entity type include all of the entities that Textual found during processing. This includes:

  • Values for ignored entity types

Summary counts

The panels at the top of the page provide summary information for the detected entity values. The displayed values are based on the selected value count option.

The summary information includes:

• The number of detected entity values.

• The number of detected entity types.

• The percentage of detected values that are redacted.

• The percentage of detected values that are synthesized.

Counts by entity type

The entity types list on the Entities Analytics page displays a summary of the detected value counts for the detected entity types. The displayed entity types and counts are based on the selected value count option.

For each entity type, the list includes:

• The count of detected values

• The percentage of detected values in the dataset that are of that type

By default, the entity types are listed in descending order based on the value count.

You can sort the list by the entity type, count, and percentage. To sort by a column, click the heading. To reverse the sort order, click the heading again.

Displaying the top 10 file list for an entity type

When you click an entity type, Textual displays a panel that lists the 10 files that contain the most detected values for that entity type.

The panel also allows you to change the handling option for the entity type.

For a local file dataset, you upload and remove new files directly.

On Tonic Textual Cloud, and by default for self-hosted instances, Textual stores the uploaded files in the application database.

On a self-hosted instance, you can instead configure an S3 bucket where Textual stores the files. In the S3 bucket, the files are stored in a folder that is named for the dataset identifier.

For more information, go to Setting the S3 bucket for file uploads and redactions.

For an example of an IAM role with the required permissions, go to .

Adding files to the dataset

From the dataset details page, to add files to the dataset:

1. In the left menu, click Project files.

2. On the dataset files page, click Upload Files.

1. Search for and select the files.

Textual uploads and then processes the files. For more information about file processing, go to .

On a self-hosted instance, when a file fails to upload, you can download the associated logs. To download the logs, click the options menu for the file, then select Download Logs.

Removing files from the dataset

To remove a file from the dataset:

1. In the file list, click the options menu for the file.

2. In the options menu, click Delete File.

The following built-in entity types have additional type-specific synthesis options.

For all entity types, you can:

• Map original values to replacement values.

• Provide a constant value to use to replace all of the original values, except for mapped values.

Mapping original values to replacement values

You can map specific original values to specific replacement values.

For example, for the Given Name entity type, you might indicate to always replace John with Michael and Mary with Melissa.

For original values that are not in the mapping:

• If you also provid a constant value, then Textual uses that value.

• Otherwise, Textual selects the values.

Providing the mapping

In the text area, provide a JSON object that maps the original values to the replacement values. For example:

With the above configuration for the Language entity type:

• All instances of French are changed to German.

• All instances of English are changed to Japanese.

Values are case-insensitive

The values are case-insensitive.

For example, if you specify "John": "Michael", then Structural also replaces john with michael and JOHN with MICHAEL.

Leading and trailing punctuation is ignored

Structural ignores leading and trailing punctuation.

To continue the example of "John": "Michael", Structural also replaces 'John' with 'Michael'.

Providing a constant replacement value

You can provide a single constant value to use to replace all instances of the entity type. For example, you might replace all bank numbers with "0000000".

In the Constant Value field, provide the value to use.

When you provide a constant value, Textual ignores other .

However, Textual does respect any .

By default, when you select the Synthesize option for Date/Time values, Textual shifts the datetime values to a value that occurs within 7 days before or after the original value.

To customize how Textual sets the new values, you can:

• Set a different range within which Textual sets the new values.

• Indicate whether to scramble date values that Textual cannot parse.

• Indicate whether to shift all of the original values by the same amount and in the same direction.

• Add additional date formats for Textual to recognize.

Adjusting the range for the replacement values

By default, Textual adjusts the dates to values that are within 7 days before or after the original date.

To change the range:

1. In the Left bound on # of Days To Shift field, enter the number of days before the original date within which the replacement datetime value must occur. For example, if you enter 10, then the replacement datetime value cannot occur earlier than 10 days before the original value.

2. In the Right bound on # of Days To Shift field, enter the number of days after the original date within which the replacement datetime value must occur. For example, if you enter 6, then the replacement datetime value cannot occur later than 6 days after the original value.

Shifting all values in a file by the same amount

By default, Textual applies different shifts to the original values. Some replacement dates might be earlier, and some might be later. The amount of shift might also vary.

Within a given file, to shift all of the datetime values in the same way, check Apply same shift for entire document.

For example, if this is checked, Textual might shift all datetime values in a file 3 days into the future.

Replacing datetime values in unsupported formats

Textual can parse datetime values that use either a format in or a format that you add.

The Scramble Unrecognized Dates checkbox indicates how Textual should handle datetime values that it does not recognize.

By default, the checkbox is checked, and Textual scrambles those values.

To instead pass through the values without changing them, uncheck Scramble Unrecognized Dates.

Adding datetime formats

By default, Textual is able to recognize datetime values that use a format from .

Under Additional Date Formats, you can add other datetime formats that you know are present in your data.

The formats must use a .

To add a format, type the format in the field, then click +.

To remove a format, click its delete icon.

The Entities Catalog lists the values that Textual detected in the dataset files.

The Dataset search allows you to search for specific values to determine whether Textual detected them and assigned the correct entity type.

You can also use the Textual Agent to ask whether the dataset includes specific values.

Starting a search

To start a search:

1. On the dataset details page, click Dataset search.

2. In the search field, provide the text to search for.

When the text matches a full word in a file, Textual displays the results.

Textual only searches for full words. For example, if you type "nu", it does not find the word "number". You must type the full word "number".

Viewing the search results

The search results include a separate row for each found instance of the search text.

For each instance, the results include:

• The instance itself, with the immediate context. The context includes the few words before and after the search text.

• The assigned entity type, if Textual detected the search text as an entity. If it did not, then the entity type is None.

• The name of the file where the instance was found.

At the top of the list, Textual displays the approximate number of matches and number of files that contain matches.

Depending on the number of matches, the list might not initially include all of the results. If there are additional results, then the Load more link displays. When you click Load more, Textual adds the next batch of results to the end of the list.

Filtering the search results

You can filter the results to only include matches in selected files.

To filter the results:

1. Click Filter by file.

2. In the file list, check the checkbox for each file to include the results for.

The PDF Settings section of the Dataset settings page provides options to configure how to work with PDF files.

You can also configure the signature, synthesis, and LLM classification mode options from the Textual Agent.

Configuring whether to redact PDF signatures

By default, Textual redacts scanned-in signatures in PDF files. You can configure the dataset to instead ignore the signatures.

Under PDF Settings:

• To redact PDF signatures, toggle Detect and redact signatures in PDFs to the on position. This is the default configuration.

• To ignore PDF signatures, toggle Detect and redact signatures in PDFs to the off position.

Using the new synthesis process

Textual has developed an updated synthesis process that is currently implemented for the following entity types:

• URLs

• Names

• Custom entity types

In particular, the new synthesis process improves the display of the synthesized values in PDF files. The values better match the available space and the original font.

Under PDF Settings, the New PDF synthesis mode (experimental) determines which process to use.

To use the new process, toggle the setting to the on position.

Using an LLM to analyze structured components

To improve the detection accuracy for sensitive values, you can optionally use an LLM to analyze structured components such as tables and form fields.

To enable this, under PDF Settings, toggle Use LLM to classify structured data for PII detection to the on position.

Selecting the OCR model to use (self-hosted only)

For PDFs as well as images, if multiple optical character recognition (OCR) models are available, you can select the specific model to use in the dataset. For information on how to enable specific models, go to .

Under PDF Settings, from the OCR Engine dropdown list, select the model to use.

For each file in a dataset, you can download the output file.

Downloading a single output file

From the file list, to download a single output file, click the options menu for the file, then select the download option.

The dataset file preview also provides options to download the file.

Datasets that generate redacted output files

For datasets that generate redacted versions of the source files, the option is Download File.

Datasets that generate JSON output

For datasets that generate JSON output, you can download either the JSON output or the redacted version of the Markdown content.

For RTF files, you can also download the redacted content in HTML format.

Downloading all of the output files

To download all of the output files, click the download icon that is next to the file filter field.

For datasets that generate redacted versions of the source files, the download happens immediately.

For datasets that generate JSON output:

• To download the JSON output for the files, select Download JSON.

• To download the redacted Markdown content for all of the files, select Download Markdown.

• If the dataset contains any RTF files, then to download only the redacted HTML content for all of the RTF files in the dataset, select Download HTML (RTF only). If you select this option, then the download does not include other types of files.

From a text, PDF, and image file preview, you can add manual redactions. When you add a manual redaction, you select the entity type to assign to it.

Adding manual redactions

Text files

To add a manual redaction to a text file:

1. In the source text panel on the left, select the text to redact.

2. On the redaction panel, from the dropdown list, select the entity type for the redaction.

PDF and image files

To add a manual redaction to a PDF or image file:

1. In Original view, to change to redaction mode, click Add Redaction.

1. While in redaction mode, to add a redaction:

   1. Draw a box around the content to redact.

   2. On the redaction details panel, from the dropdown list, select the entity type for the redaction.

1. To exit redaction mode, click Done.

Textual displays the count of manual redactions on the current page.

Removing a manual redaction

Text files

To remove a manual redaction from a text file:

1. In either Redacted or Original view, click the redaction.

2. Click Delete.

PDF and image files

To remove a manual redaction from a PDF or image file:

1. In either Redacted or Original view, click the redaction.

2. Click Delete Redaction.

Location values include the following types:

• Location

• Location Address

• Location State

• Location Zip

For location types that include zip codes, you can also specify how to generate the new zip code values.

Selecting the type of address generator to use

Under Address generator type, select the type of address generator to use:

• HIPAA-compliant address generator. This option generates values similar to those generated by the .

• Non-HIPAA address generator. This option generates values similar to those generated by the .

If you configured a Textual statistics seed that matches a Structural statistics seed, then the generated address values are consistent with values generated in Structural. A given address value produces the same output value in both applications.

For example, in both Textual and Structural, a source address value 123 Main Street might be replaced with 234 Oak Avenue.

Using realistic replacement values

For Location State, based on HIPAA guidelines, both the Synthesize option and the Ignore option pass through the value.

For location types other than Location State, by default, Textual replaces a location value with a realistic corresponding value. For example, "Main Street" might be replaced with "Fourth Avenue".

To instead scramble the values, uncheck Replace with realistic values.

Generating replacement zip codes

Using zeroes for the last two digits

By default, to generate a new zip code, Textual selects a real zip code that starts with the same three digits as the original zip code. For a low population area, Textual instead selects a random zip code from within the United States.

To instead replace the last two digits of the zip code with zeros, check Replace zeroes for zip codes. For a low population area, Textual instead replaces all of the digits in the zip code with zeros.

This option is also used when you truncate zip codes to the first 3 digits. When you truncate zip codes, and enable this option, Textual adds zeroes to fill out the zip code to 5 digits.

Truncating zip codes to 3 digits

To truncate zip codes to the first 3 digits, check Use 3-digit zip codes.

If you also check Replace zeroes for zip codes, then Textual replaces the truncated digits with zeroes.

Replacing foreign zip codes with zeroes

For zip codes from outside of the United States, then to replace those zip codes with zeroes, check Replace foreign zip codes with zeroes.

For a dataset that generates output files of the same type as the original file, the file preview displays the source and output versions of the file, and highlights the detected entity values.

From a file preview, you can change the entity type handling. For some file types, you can also add manual redactions.

Tonic Textual tracks each change to the dataset configuration.

You can view the list of changes to the dataset configuration, and revert the dataset configuration to a selected version.

Displaying the dataset version history

To display the version history, click the clock icon at the top right of the dataset details page.

The Version history panel lists the versions in descending order, with the current version at the top of the list.

Displaying details for a version

For each version, the Version history list initially displays:

• A summary description of the change.

• The timestamp when the change occurred.

• The name of the user who made the change.

To display additional details about the change, expand the version description. For example, for a change to the handling option for an entity type, the details include the entity type, the previous handling option, and the new handling option.

Restoring a version

You can restore an earlier version of the dataset. When you restore a version, Textual reverts the changes that occurred since that version.

On the expanded description of a version, to restore that version, click the Restore to <version number> button.

Textual adds a version to the history. The new version is marked as Restored, and the description indicates the version that you restored.

The expanded details for the version list the changes that Textual reverted when you restored the earlier version.

When you select the Synthesis option for the Age entity type, you can either:

• Shift the age within a specific range.

• Pass through ages younger than 90, and group ages 90 and older into a single value.

Shifting the age within a range

To shift the age within a specified range:

1. From the Age generator type dropdown list, select Age shift.

2. In the Range of Years +/- for the Shifted Age field, enter the number of years before and after the original value to use as the range for the synthesized value.

By default, Textual shifts the age value to a value that is within 7 years before or after the original value.

Passing through or grouping the age value

Based on the original age value, to either pass through or group the value, from the Age generator type dropdown list, select Passthrough or group age.

With this option:

• If the original age value is younger than 90, then the value is left as is in the output.

• If the original age is 90 or older, then the value is replaced with 90+.

Indicating whether to scramble values that Textual cannot parse

By default, Textual scrambles age values that it cannot parse.

To instead pass through the value unchanged, uncheck Scramble Unrecognized Ages.

Tags can help you to organize your datasets. For example, you can use tags to indicate datasets that belong to different groups, or that deal with specific areas of your data.

You can manage tags from the Datasets page or from the Project Settings tab on the dataset details.

Datasets page

On the Datasets page, the Tags column displays the currently assigned tags.

On the dataset

To change the tag assignment for a dataset:

1. Click Tags.

2. On the dataset tags panel, to create a new tag, type the tag text, then press Enter. You can also select an existing tag from the dropdown list.

1. To remove a tag, click its delete icon.

2. To remove all of the tags, click the delete all icon.

Dataset details

On the Dataset settings page of the dataset details, the Tags section lists the currently assigned tags.

To change the tag assignment for the dataset:

1. Click the Tags field.

2. To create a new tag, type the tag text, then press Enter. You can also select an existing tag from the dropdown list.

1. To remove a tag, click its delete icon.

2. To remove all of the tags, click the delete all icon.

To display the list of files for the dataset, on the dataset details page, click Project files.

For cloud storage dataset, the file list displays the full path to each file.

Information in the file list

For each file, the file list includes the following information:

• The name of the file. For cloud storage datasets, the file name includes the full path to the file. When the file needs to be rescanned, a warning icon displays in front of the name.

• The number of detected entity values in the file. Note that Textual does not display an entity value count for CSV files.

• The total number of words in the file. Note that Textual does not display the word count for CSV files.

• When the file was added to the dataset.

Filtering the file list

To filter the list based on the file name, in the search field, begin to type text from the file name.

As you type, Textual updates the file list to only include matching files.

Sorting the file list

You can also sort the list by any of the columns.

• To sort by a column, click the column heading.

• To reverse the sort order, click the column heading again.

When you select the Synthesis option for a Date of Birth, you can either:

• Shift the date within a specific range.

• Truncate all dates to January 1. The year is set based on whether the original year is more than 89 years ago.

You can also configure how to handle dates that Textual does not recognize, and add additional date formats for Textual to recognize.

Shifting the date within a range

To shift the date within a specified range:

1. From the Generator Type dropdown list, select Date shift generator.

2. In the Left bound on # of Days To Shift field, enter the number of days before the original date within which the replacement datetime value must occur. For example, if you enter 10, then the replacement datetime value cannot occur earlier than 10 days before the original value.

3. In the Right bound on # of Days To Shift field, enter the number of days after the original date within which the replacement datetime value must occur. For example, if you enter

Truncating the date

To truncate the date, from the Generator Type dropdown list, select Date Truncation Generator.

With this option, in the output:

• The month and day are always January 1.

• If the original year is less than 90 years ago, then the year remains as is in the output.

• If the original year is 90 years or more ago, then the year is set to the current year minus 89.

Selecting how to handle invalid values

Textual can parse datetime values that use either a format that Textual supports by default or a format that you add.

The Scramble Unrecognized Dates checkbox indicates how Textual should handle datetime values that it does not recognize.

By default, the checkbox is checked, and Textual scrambles those values.

To instead pass through the values without changing them, uncheck Scramble Unrecognized Dates.

Adding datetime formats

By default, Textual is able to recognize a specific list of .

Under Additional Date Formats, you can add other datetime formats that you know are present in your data.

The formats must use a .

To add a format, type the format in the field, then click +.

To remove a format, click its delete icon.

From the dataset file list, to display the preview, either:

• Click the file name.

• Click the options menu, then click Preview.

From a PDF file preview, you can choose to ignore a specific value:

1. In the source or results panel, click the value.

2. On the details panel, to ignore the value, toggle Ignore this instance to the on position.

On the Guidelines refinement page, you prepare the guidelines that define a model.

To test each version of the guidelines, Textual uses the guidelines to detect entity values in the test data. It then generates scores to indicate how closely those detection results match the values that you established during your initial review.

Textual also generates recommendations to improve the guidelines.

Viewing the initial version of the guidelines

For a custom model entity type, the overall process is as follows:

Select and annotate test files

The first step is to identify the entity values that are in a small set of test files. The test files and established values are used both to iterate over the model guidelines and to assess how well your trained models perform.

You can configure how Textual works with PDFs. For an individual PDF file, you can add manual overrides to selected areas of a file. Manual overrides can ignore detected values from Tonic Textual, or add redactions.

For Phone Number values, you can choose whether to generate a realistic phone number. If you do, then the generated values can be consistent with values generated in Structural.

Selecting the generator type

From the Phone number generator type dropdown list:

Available column content types

When Textual processes a CSV file, it identifies the content of each column as one of the following:

• Sensitive - The column values are of a single type and format, and the values match an entity type.

In .docx and .xslx files, as long as the URL entity type handling option is not set to Off, Textual automatically changes the destination of hyperlinks to google.com.

On the Dataset settings page, the Word Document Settings section contains settings to determine how to manage .docx images, tables, and comments.

To display the Dataset settings page, on the dataset details page, click Project settings

The entity types settings for a project determine the Textual entity types that Textual detects in new project files. You can enable or disable both . You cannot enable model-based custom entity types.

For example, you can tell Textual to ignore all Given Name values in new project files.

For built-in entity types, you can also configure specific values to add to or exclude from the detection. For example, you can keep the Given Name entity type active, but indicate to ignore the value "Mark".

By default, all of the entity types are active, and there are no added or excluded values.

Any changes to the project entity type settings only affect files that are added after the change. Files that were already scanned are not affected.

You can override the entity type settings in individual files. For more information, go to .

The project status indicates where the project is in the redaction and review process.

For information about how to configure the available status values, go to .

Each new project is assigned the built-in Not started status.

You can set the status from either the Guided Redaction page or the project details page.

The project details page contains the list of files in the project. To display the project details, on the Guided Redaction page, click the project name.

Information in the file list

For each file in the project, the list includes the following information:

Setting up projects

Configure entity types and access

For Email Address values, you can choose to preserve the original email address domains. For example, source email addresses for example.com remain example.com email addresses in the output.

Before you start training models, on the Model data setup page, you select the training data to use.

About training data

The training data is a much larger set of files than the test data, and can include hundreds of files or more. The data should ideally contain at least 1,000 values for the entity type. For example, for an entity type to identify health conditions, you might use 5 medical appointment reports in your test data, but several hundred medical reports for your training data.

Similar to the test files, the training files should be relatively small - no more than 5,000 words.

For training data, there is no option to paste in text. Training data files are either uploaded from a local file system or selected from a cloud storage solution.

If you selected the test data from a cloud storage solution, then you must use the same cloud storage solution for the training data. For example, if you selected the test data from Amazon S3, then you must select the training data from Amazon S3.

You can add files to the training data at any time. New files are only used for models that are trained after the files are added.

Uploading files from a local file system

If there are no training files, then on the Model data setup page, click Upload files, then search for and select the files to upload.

To add more uploaded files to the training data:

1. Click Add training data.

1. Click Upload files.

2. Search for and select the files to add to the training data.

Selecting files from cloud storage

If there are no training files, then on the Model data setup page, click the cloud storage solution to use, then select the files to add.

If the test data came from a cloud storage solution, then you must use the same cloud storage option for the training data.

To add more cloud storage files to the training data:

1. Click Add training data.

1. Select the cloud storage solution.

2. Select the files to add to the training data.

For training data, you can select entire folders. Textual then adds all of the files in the folder.

Displaying the content of a training data file

On the Model data setup page, to display the content of an uploaded file, click the file name.

Training data file statuses

Each training data file goes through the following statuses:

• Queued for upload - The file is not yet uploaded.

• Uploading - Textual is uploading the file.

• Ready - The file is uploaded and is used for subsequent model training.

Model training cannot start until all of the currently uploaded files are Ready.

Deleting training files

On the Model data setup page, to delete a training file:

1. Click its delete icon.

2. On the confirmation panel, you can choose to skip the confirmation when you delete training files. If you select this option, then the next time you delete a training file, the file is deleted immediately, and the panel does not display.

3. Click Delete.

When you delete a training file:

• For existing models that annotated the file:

  • The entity counts continue to reflect the entities that were detected in the file

  • The file name remains in the list on the model details.

For PDF files, you can add manual overrides to the initial detections, which are based on the detected data types and handling configuration.

For each manual override, you select an area of the file.

For the selected area, you can either:

• Ignore any automatically detected values. For example, a scanned form might show an example or boilerplate content that doesn't actually contain sensitive values.

• Redact that area. The file might contain sensitive content that Tonic Textual is unable to detect. For example, a scanned form might contain handwritten notes.

You can also apply a template to the file.

You can also .

Selecting the manual override option for a file

To manage the manual overrides for a PDF file:

1. In the file list, click the options menu for the file.

2. In the options menu, click Edit Redactions.

The File Redactions panel displays the file content. The values that Textual detected are highlighted. The page also shows any manual overrides that were added to the file.

Applying a PDF template to a file

If a dataset contains multiple files that have the same format, then you can create a template to apply to those files. For more information, go to .

On the File Redactions panel, to apply a template to the file, select it from the template dropdown list.

When you apply a PDF template to a file, the manual overrides from that template are displayed on the file preview. The manual overrides are not included in the Redactions list.

Adding a manual override

On the File Redactions panel, to add a manual override to a file:

1. Select the type of override. To indicate to ignore any automatically detected values in the selected area, click Ignore Redactions. To indicate to redact the selected area, click Add Manual Redaction.

2. Use the mouse to draw a box around the area to select.

Textual adds the override to the Redactions list. The icon indicates the type of override.

In the file content:

• Overrides that ignore detected values within the selected area are outlined in red.

• Overrides that redact the selected area are outlined in green.

Navigating to a manual override

To select and highlight a manual override in the file content, in the Redactions list, click the navigate icon for the override.

Removing a manual override

To remove a manual override, in the Redactions list, click the delete icon for the override.

Saving the manual overrides

To save the current manual overrides, click Save.

For datasets that produce redacted files, for each entity type, you choose how to handle the detected values. This determines how each value displays in the output files.

For datasets that create JSON output, the entity type handling determines the display in downloaded Markdown or HTML files.

In addition to the dataset details page options, you can also use the Textual Agent to configure the entity type handling.

Available handling options

The available options are:

• Synthesize - Indicates to replace the value with another realistic value. For example, the first name value Michael might be replaced with the value John. The synthesized values are always consistent, meaning that a given entity value always has the same replacement value. For example, if the first name Michael appears multiple times in the text, it is always replaced with John. Textual does not synthesize any excluded values. For custom entity types, Textual scrambles the values.

• Redact - This is the default option, except for the Full Mailing Address entity type, which is ignored by default. For text files, Redact indicates to tokenize the value - to replace it with a token that identifies the entity type followed by a unique identifier. For example, the first name value Michael might be replaced with NAME_GIVEN_12m5s. The identifiers are consistent, which means that for a given original value, the replacement always has the same unique identifier. For example, the first name Michael might always be replaced with NAME_GIVEN_12m5sb

Selecting the handling option for a specific entity type

On the Entity settings page, to select the handling option for an entity type:

1. In the De-identification Setting column, click the dropdown.

2. In the dropdown list, click the option.

On the Analytics page, the also provides an option to set the handling option.

Selecting the handling option for all of the entity types

On the Entity settings page, to select the same handling option for all of the entity types, from the Bulk Edit dropdown above the data type list, select the option.

As you redact and review the project files, you set the status of each file and project. Projects and files use the same set of status values.

Each status is associated with a color.

Built-in statuses

Textual comes with a set of built-in statuses. The built-in statuses are:

• Not started - This is the default status. It is applied automatically to all new projects and files. You cannot delete this status.

• In progress

• Ready for review

• Review in progress

• Done - This is intended to be the final status for a project or file. You cannot delete this status.

You can change the name and assigned color of all of the built-in statuses. You can delete the built-in statuses that you do not need, except for the Not started status and the Done status.

You can also add custom statuses, to accommodate your particular redaction and review process.

Displaying the status list

On the Guided Redaction page, to display the status list, click Status settings.

The status list shows the status name and the associated color.

Adding a status

From the Status Settings panel, to add a status:

1. Click Add a status.

2. On the status configuration panel, in the field, provide a name for the status. Status names must be unique.

3. Click the color to assign to the status.

4. Click

Editing a status

From the Status Settings panel, to change the status configuration:

1. Click the status name.

2. On the status configuration panel, you can change the status name and color. Remember that the status name must be unique.

1. To save the changes, click Save.

Deleting a status

You cannot delete:

• The built-in Not started status.

• The built-in Done status.

• A status that is currently assigned to a project or file.

From the Status Settings panel, to delete a status:

1. Click the status name.

2. On the status configuration panel, click the delete icon.

The Textual guided redaction tool blocks out sensitive values in files. For example, you might use guided redaction to prepare documents to provide in response to a Freedom of Information Act request.

Guided redaction supports built-in and custom entity types. Textual completes an initial scan of the files to identify sensitive values in the files. You can then manually add and remove redactions.

The values are covered with a black or white box. You assign and display reference codes to identify the type of content for each redaction.

The overall workflow is as follows:

Pre-project setup

Before you create a guided redaction project:

1. Set up the list of statuses that can be assigned to a file or a project. Each status is associated with a color. Textual provides a set of built-in statuses, including a Not started status that is applied automatically to all new projects and files, and a Done status that indicates that the project or file is complete. You can configure custom statuses and change the names and colors of the built-in statuses. You can also delete the built-in statuses, except for Not started and Done.

2. If you assign reference codes to the redactions, configure the reference codes. Reference codes identify the type of information that is redacted. You can optionally link each code to up to 5 Textual entity types. For example, if you use a single code for all name values, then you could link that code to both the Given Name and Family Name entity types. Be sure to map reference codes to all of the entity types that are present in your files.

Project creation and population

1. .

2. . For new files, Textual uses its built-in models to scan each file for entity values.

File redaction

For each file, in Redaction mode, you can .

Every redaction must be assigned a reference code. Redactions that do not have reference codes are not redacted in the output.

As you work on the redaction, you can update the and statuses.

Use the to track the project activity.

Redaction review

In Review mode, .

As you work on the review, you can update the and statuses.

Use the to track the project activity.

Output preview and download

In Preview mode, based on the current redactions.

When the redaction and review is complete, . All downloaded output is in PDF format, with the individual PDF files bundled into a .zip file.

When you download files, you select whether the output is redacted or is to be used for review.

• Review output does not cover the redactions, and always displays the reference codes.

• For redacted output, you select the color of the box and whether to display the reference codes.

You use a Textual dataset to detect sensitive values in files. The dataset output can be either:

• Files in the same format as the original file, with the sensitive values replaced based on the dataset configuration.

• JSON files that contain a summary of the detected values and replacements.

You can use the to explore and configure a dataset.

You can also create and manage datasets from the or

A dataset might contain multiple files that have the same structure, such as a set of scanned-in forms.

Instead of adding the same manual overrides for each file, you can use a PDF file in the dataset to create a template that you can apply to other PDF files in the dataset.

When you , you can apply a template.

In a dataset, for each built-in entity type, you can configure additional values to detect, and values to exclude. You cannot define added and excluded values for custom entity types.

You might add values that Textual does not detect because, for example, they are specific to your organization or industry.

You might exclude a value because:

By default, when you select the Synthesize option for a custom entity type, Textual scrambles the original value.

From the generator dropdown list, select the generator to use to create the replacement value.

The available generators are:

The dataset details page provides information about the results of the sensitivity detection, including the overall results, the results per file, and the results per entity type.

You can also use the to explore the dataset results.

You can export a model-based entity type to an encrypted file. The export includes the entire configuration.

You can then import the entity type into another instance of Textual.

Exporting an entity type

To create a project:

1. On the Guided Redaction page, click New Project.

For each dataset file, the Project files page displays:

• The number of detected entity values in the file. The count does not include values for entity types for which the entity type handling is set to Ignore.

• The total number of words in the file.

Redacted view - output

For a redacted file, the preview by default shows the Redacted view, which reflects the output version of the file.

Sensitive values are redacted, synthesized, or ignored based on the dataset configuration.

For PDF and image files, for entity types that use the Redact handling option, the values are covered by black boxes.