1 of 100

Tonic Textual

Tonic Textual guide

Tonic Textual allows you to put your text-based data to work for you.

A Textual dataset is a collection of files from a local file system or cloud storage. Textual scans the dataset files to identify sensitive values. You can then choose to redact or replace those sensitive values, to produce output files in the same format that you can safely use for development and training.

A guided redaction project identifies blocks out sensitive values in files. For example, you might use guided redaction to prepare documents in response to a Freedom of Information Act request. Textual scans the files to identify values. You can then review and adjust the results before you download the output files.

You can use the Textual SDK or the Textual REST API to manage datasets or to remove sensitive values from individual text strings and files.

Startup and overview

Textual SDK, REST API, and other integrations

Need help with Textual? Contact [email protected].

Getting started with Textual

Note that these instructions are for setting up a new account on Textual Cloud. For a self-hosted instance, depending on how it is set up, you might either create an account manually or use single sign-on (SSO).

To get started with a new Textual account:

Go to .
Click Sign up.
Enter your email address.
Create and confirm a password for your Textual account.
Click Sign Up.

Textual creates your account. After you log in, Textual prompts you to provide some additional information about yourself and how you plan to use Textual.

After you fill out the information and click Get Started, Textual displays the Textual Home page, which you can use to preview how Textual detects and replaces values. For more information, go to .

Using the Textual free trial

When you set up an account on Textual Cloud, you start a Textual free trial.

Using the Getting Started checklist

When you start a free trial, Textual provides a checklist to guide you through initial steps to get started and learn more about Textual and what it can do.

The checklist displays automatically when you first log in. You can close and display it as needed. To display the checklist, in the Textual navigation menu, click Getting Started.

As you complete a step, Textual automatically marks it as completed.

The checklist includes:

When you click the step, you navigate to the Home page. Textual displays a popup panel that describes the task.
The checklist displays the installation command and an option to copy it. The step is marked as complete when you click the copy icon.
When you click the step, you are prompted to create an API key.
Creating an SDK request to redact a or a . When you click the step, you navigate to the Request Explorer. The step is marked as completed when you close the popup panel that describes the task.

Word count limit

During the free trial Textual scans up to 100,000 words for free. Note that Textual counts actual words, not tokens. For example, "Hello, my name is John Smith." counts as six words.

After the 100,000 words, Textual disables scanning for your account. Until you purchase a pay-as-you-go subscription, you cannot add files to a dataset.

Viewing your current usage

During your free trial, Textual displays the current usage in the following locations:

On the Home page
In the navigation menu

Next steps - pay-as-you-go or product demo

Textual also prompts you to purchase a , which allows an unlimited number of words scanned for a flat rate per 1,000 words.

You can also request a Textual product demo.

Managing your user profile

The User Profile page displays a summary of information about your Textual account.

From the user profile page, you can copy your organization identifier, configure a team name, and manage your Textual API keys. For more information about managing your Textual API keys, go to Creating and revoking Textual API keys.

Displaying the User Profile page

To display the User Profile page:

Click the user icon at the top right.
In the user menu, click User Profile.

Copying your organization identifier

The profile summary includes the identifier of your organization in Textual.

To copy the identifier, click the copy icon.

Adding a team name

From the User Profile page, you can specify a team name. For example, you might use the team name field to identify a specific department or project that you belong to.

To add a team name, type the name in the field.

Entity types

About entity types

An entity type is a category of entity value. For example, the entity value John might be an example of the Given Name entity type.

Tonic Textual comes with a built-in set of entity types that it always detects.

You can also configure custom entity types, which you can use to detect values that are not covered by the built-in entity types.

When you create a custom entity type, you can either:

Use regular expressions to identify matching values. You might create this type of custom entity when there are a limited number of values or the values follow specific formats that can easily be identified with a regular expression.
Define and train a model to identify matching values. Training a model is an iterative process that can take hours or days, depending on your data. You might create this type of custom entity when there are a large number of values that do not follow a specific format. The values need to be identified more by context.

You can also view this video overview of entity types and entity type handling.

Viewing the list of custom entity types

To display the list of custom entity types, in the Textual navigation bar, click Custom Entities.

Information in the list

For each custom entity type, the Custom Entity Types page includes the following information:

Name of the custom entity type.
Whether the custom entity type is regex-based or model-based.
For model-based custom entity types, the status. The status indicates where the entity type is in the creation process.
When the custom entity type was most recently updated.
The number of projects where the custom entity type is active.

Filtering the list

Filtering by name

To filter the list by the entity type name, in the search field, begin to type text in the name. As you type, Textual filters the list to only include matching entity types.

Filtering by type

By default, the list includes both regex-based and model-based custom entity types.

To filter the list to only include one of the formats:

Click Filter by type.
In the filter dropdown list, select the format to include.

Filtering by creator

By default, the list includes all of the custom entity types that were created by users in your organization.

To filter the list to only include custom entity types that were created by specific users:

Click Filter by creator.
In the dropdown list, check the checkbox for each user to include.

Sorting the list

By default, the list is sorted alphabetically by the entity type name.

You can sort the list by any of the data columns.

To sort by a column, click the column heading.

To reverse the sort order, click the heading again.

Managing model-based custom entity types

Required global permission - either:

Create custom entity types
Edit any custom entity type

Self-hosted instances must also .

While a regex-based entity type identifies values that match regular expressions, for a model-based custom entity type, you train a model to identify the entity values.

A model-based entity type is useful when the values are identified more by context than by format. For example, for an entity type that identifies the names of health conditions, it would not be possible to set up regular expressions that identify the values.

You iterate over text-based guidelines that identify the entity type values in a smaller set of data, then use a larger set of data to train one or more models.

Each trained model is based on a selected version of the guidelines.

You select the trained model to use for the custom entity type, and select the datasets to enable the custom entity type for.

Getting started

Defining the entity type

Activating and managing an entity type

Overview of the process to create a model-based custom entity type

For a custom model entity type, the overall process is as follows:

Select and annotate test files

The first step is to identify the entity values that are in a small set of test files. The test files and established values are used both to iterate over the model guidelines and to assess how well your trained models perform.

When you create the model-based custom entity type, you provide an initial description of the entity type. For example, "Scientific names of health conditions". The description is the first version of the model guidelines. The guidelines tell the model how to identify the entity type values.
You then select a small set of smaller test files that contain entity values. For example, if you typically use Textual to redact values in patient appointment reports, then you might upload a few of those reports to use as test files. The files should be no more than 5,000 words.
Textual uses your initial guidelines to identify values in the files.
You then review and correct the annotations to identify the definitive set of entity values that the test files contain.

Iterate over model guidelines

After you establish the entity values in your test files, you iterate over the guidelines for the model.

For each version of the guidelines, Textual uses the guidelines to detect entity values in the test files.

Textual then compares the values that the guidelines version detects against the values that you established when you annotated the test files.

Textual generates scores to identify how well that version of the guidelines performed. If you are not satisfied with the results, you can update the guidelines to create a new version.

Textual automatically generates suggestions to improve the guidelines, based on how well the current guidelines identified the values. For example, it might suggest more specific wording or additional text to describe exceptions.

Select training data

When you have guidelines that you are satisfied with, you select a larger set of data to use for model training.

The training data should contain at least 1,000 entity values. The files should still be relatively small - no more than 5,000 words.

For example, when setting up a custom entity type to identify health conditions, you might use 5 or 6 appointment reports for your test data, but several hundred reports for your training data.

Train models

When you create a model, you select the guidelines version to use for it.

The model uses the guidelines to annotate the training data - in other words, to detect entity values in the training files. You review the annotation results to determine whether you are satisfied with the detections.

If you are not satisfied, you can:

Return to the guidelines refinement to edit the guidelines.
Create a new guidelines version.
Create a model that uses the new version.

If you are satisfied, then you can start the model training. Model training can take a very long time - sometimes hours or days - depending on the data.

When the model finishes training, it scans and identifies values in the original test data. Each trained model receives a score to identify how well its detections matched the definitive values that you established.

Select a model to use

To make the entity type available to use, you select the trained model to use.

The custom entity type is then active and can be enabled or disabled within individual datasets.

Starting a new model-based custom entity type

To start a new custom entity type:

On the Custom Entity Types page, click Create Custom Entity Type, then select Model-based entity type.

On the next panel, in the Custom entity type name field, provide a name for the custom entity type.
In the Annotation guidelines text area, provide instructions for how the model should identify values that you want to find. This is the first version of the model guidelines.

Click Next.

Textual displays the Test data setup page.

Iterating over the guidelines to use for model training

On the Guidelines refinement page, you prepare the guidelines that define a model.

To test each version of the guidelines, Textual uses the guidelines to detect entity values in the test data. It then generates scores to indicate how closely those detection results match the values that you established during your initial review.

Textual also generates recommendations to improve the guidelines.

Viewing the initial version of the guidelines

To work on the guidelines, click Guidelines refinement. The Guidelines refinement option is enabled when you complete the review on the initial set of test files.

The first time you display the Guidelines refinement page, Textual uses the guidelines that you provided during the entity type creation to populate the Version 1 tab.

At the left are the guidelines.

At the right is the list of test data files.

File statuses for the guidelines refinement

For each version of the guidelines, Textual uses the guidelines to detect entity values in the test data.

The file statuses are:

Queued for annotation - Textual has not yet scanned the file.
Annotating - Textual is in the process of scanning the file.
Annotated - The scan is complete.

Reviewing the test scores for the guidelines

When Textual uses guidelines to detect entity values in the test files, it sets the number of detected entities and a set of scores. The scores reflect how well the detections match the entity values that you established in the test data setup. If you change the established values in the test data, Textual updates the scores for the guidelines.

The overall entity count and scores across all files are displayed across the top of the page. The file list displays the entity count and scores for each file.

The scores are:

Precision score - Measures the accuracy of positive predictions. Indicates how many of the detected entities were correctly identified. For example, the guidelines detect 10 values. If only 3 of those are correct, then the precision score is lower than if 7 of those are correct.
Recall score - Measures the model's ability to find all of the entities. Indicates how many of the actual entities it detected. For example, the guidelines detect 10 correct values. If the total number of correct values is 20, then the recall score is lower than if the total number of correct values is 12.
F1 score - The harmonic mean of precision and recall. The goal is to have a balance between precision and recall. The guidelines should produce annotations that are both accurate and complete. Detecting all of the correct values is not useful if the guidelines also detect a large number of incorrect values. And detecting only correct values is not useful if the guidelines only detect a fraction of the total number of correct values.

Reviewing the guideline detections

To review the entity values that Textual detected based on the current version of the guidelines, click the file name.

Editing the guidelines

Based on how accurately Textual detected the entity values, Textual generates suggested changes to the guidelines.

For example, it might suggest additional language to more specifically identify values that the previous version either missed or detected incorrectly.

To start a new version of the guidelines:

Click Edit. If there are suggestions, you can also click Review.

On the Annotation guidelines panel, the current guidelines are displayed in an editable text area on the left. On the right is a summary of the suggested updates to the guidelines. To display the proposed replacement guidelines, toggle Show diff to the on position.

To update the guidelines, you can either:
- Update the guidelines manually.
- Accept all of the suggestions, and replace the current guidelines. To do this, click Accept changes.
- Manually copy text from the suggestions and paste it into the guidelines.
To save the guidelines version, and start the detection and scoring, click Save new version.

Textual creates a new tab for the new version of the guidelines. The tab label is Version n, where n is incremented for each new version. The most recent version is at the left.

Selecting the training data for your models

Before you start training models, on the Model data setup page, you select the training data to use.

About training data

The training data is a much larger set of files than the test data, and can include hundreds of files or more. The data should ideally contain at least 1,000 values for the entity type. For example, for an entity type to identify health conditions, you might use 5 medical appointment reports in your test data, but several hundred medical reports for your training data.

Similar to the test files, the training files should be relatively small - no more than 5,000 words.

For training data, there is no option to paste in text. Training data files are either uploaded from a local file system or selected from a cloud storage solution.

If you selected the test data from a cloud storage solution, then you must use the same cloud storage solution for the training data. For example, if you selected the test data from Amazon S3, then you must select the training data from Amazon S3.

You can add files to the training data at any time. New files are only used for models that are trained after the files are added.

Uploading files from a local file system

If there are no training files, then on the Model data setup page, click Upload files, then search for and select the files to upload.

To add more uploaded files to the training data:

Click Add training data.

Click Upload files.
Search for and select the files to add to the training data.

Selecting files from cloud storage

If there are no training files, then on the Model data setup page, click the cloud storage solution to use, then select the files to add.

If the test data came from a cloud storage solution, then you must use the same cloud storage option for the training data.

To add more cloud storage files to the training data:

Click Add training data.

Select the cloud storage solution.
Select the files to add to the training data.

For training data, you can select entire folders. Textual then adds all of the files in the folder.

Displaying the content of a training data file

On the Model data setup page, to display the content of an uploaded file, click the file name.

Training data file statuses

Each training data file goes through the following statuses:

Queued for upload - The file is not yet uploaded.
Uploading - Textual is uploading the file.
Ready - The file is uploaded and is used for subsequent model training.

Model training cannot start until all of the currently uploaded files are Ready.

Creating and training models for a model-based entity type

After you select your training data, on the Model training page, you create one or more trained models.

For each model, you select the version of the guidelines to use. Textual first uses those guidelines to annotate the training data. Based on how well the guidelines identified the values in the training data, you decide whether to start the model training.

When the training is complete, the model scans the test data. The model is scored based on how well it detected the definitive values that you confirmed in the test data.

Information on the model list

For each model, the model list includes:

Model - The model name. Models are automatically named Model n, where n is the number of the model. For example, the first model you create is Model 1, the second is Model 2, and so on.
Status - The model status. The possible statuses are:
- Annotating - The model is using the selected guidelines to annotate the training data.
- Ready for training - The annotation is complete. For models with this status, Textual displays a Review option to allow you to review the annotations.
- Training - The training is in progress. Textual displays the percentage of training data that the model has trained on.
- Ready - The model is trained. You can select any trained model as the active model for the entity type.
Guideline version - The version of the guidelines used for the model. To view the guidelines text, click the view icon.
Benchmark score - A score that indicates how well the model performed when it annotated the test data after training.
Detected entities - The number of entity values that the model detected in the training data.
# of files - The number of training files that were used for the annotation and model training.

Starting a new model

To start a new model:

Click Create new model.

On the Create new model panel, from the Guideline version dropdown list, select the version of the guidelines to use for the model.
Click Save.

Textual adds the model to the list and uses the selected guidelines version to annotate the training data files.

Reviewing the annotations for a model

Before you train the model, you review the annotations to see how well the model performed.

To review the annotations, click the model name. Models that are ready to review also display a Review and Train link next to the model name.

On the model details page:

On the left is the list of training data files, with the number of entities detected in each file.
On the right is the list of the entities in the training files, in descending order by the number of occurrences.

To display the content of a file with the annotations highlighted, click the file name.

After you review the annotations, if you are not satisfied with the results, to return to the guidelines refinement:

In the model list, in the Guideline version column, click the view icon.
On the guidelines panel, click Go to guidelines refinement.

For a model that is not trained yet, the model details page also displays a Modify guidelines option.

Textual displays the Guidelines Refinement page, and selects that guidelines version. You can then , then that uses the new version.

Training the model

If you are satisfied with the annotation results, then on the model details page, to start the training, click Train model.

Downloading a data package for a model

To help troubleshoot issues with a trained model, you can download a model data package to send to Tonic.ai.

The data package is a .zip file that contains the following:

General information about the custom entity type and model. Includes the entity type name entity type identifier, and the model identifier.
The set of test files, including the established entity values that you identified.
The set of training files, including the entity values that the model identified.

To download the data package, either:

On the Model Training page, click the download icon for the model.
On the model details page, click Download Training Data.

Selecting the active model for the entity type

Before you can use a model-based entity type, you must select the model to use.

On the model list page, the active model is marked as Active.

To change the active version:

Hover the mouse over the model to make the active model for the entity type.
Click Activate.

Renaming or deleting a model-based custom entity type

Renaming the entity type

To change the entity type name, on the entity type details page:

Click the actions menu next to the entity type name.

In the menu, click Edit entity name.
On the Edit Name panel, provide the new name for the entity type.
Click Save.

Deleting the entity type

You cannot delete an entity type that is enabled for any datasets. For information on how to enable and disable an entity type in datasets, go to .

To delete a model-based custom entity type, in the entity types list, click the delete icon.

On the confirmation panel, click Delete.

Create and manage datasets

Viewing the dataset list and details

Viewing the list of datasets

Displaying the Datasets page

To display the Datasets page, in the navigation menu, click Datasets.

The datasets list only displays the datasets that you have access to.

Users who have the global permission View all datasets can see the complete list of datasets.

For each dataset, the Datasets page includes:

The name of the dataset
Any tags assigned to the dataset. For datasets that you can edit, there is also an option to assign tags. For more information, go to .
The user who most recently updated the dataset
When the dataset was created

Filtering the datasets by name

To filter the datasets by name, in the search field, begin to type text that is in the dataset name.

As you type, the list is filtered to only include datasets with names that contain the filter text.

Filtering the datasets by tag

You can assign tags to each dataset. Tags can help you to organize and provide a quick glance into the dataset configuration.

On the Datasets page, to filter the datasets by their assigned tags:

In the heading for the Tags column, click the filter icon.
On the tag list, check the checkbox for each tag to include.

To find a specific tag, in the search field, type the tag name.

Displaying details for a dataset

Required dataset permission: View dataset settings

To display the details page for a dataset, on the Datasets page, click the dataset name.

The dataset details page includes:

The tags assigned to the dataset, as well as an option to add tags. For more information, go to .
The list of files in the dataset. For a cloud storage dataset, where the files can be located across multiple folders, Textual navigates to the first folder that contains selected dataset files.
The results of the scan for entity values
The configured handling for each entity type

Creating a dataset

Required global permission: Create datasets

When you create a dataset, you specify:

The type of output to produce
The source location for the files.
If the files are in cloud storage, the connection credentials.

Setting the name, source type, and output type

To create a dataset:

On the Datasets page, click Create a Dataset.

In the Dataset Name field, provide a name for the dataset.
Under Output Format, select the type of output to generate.
Under File Source, select the source type. If the source type is a cloud storage option, then provide the required credentials.
Click Save.
For cloud storage datasets:
1. Textual prompts you to configure the initial file selection. For more information, go to .
2. After you select the files, it prompts you to select an output location. For more information, go to .

Providing credentials for Amazon S3

On self-hosted instances, we are deprecating the options to provide credentials on the dataset panel and read credentials from environment variables.

Instead, the credentials must be included in the configuration of an IAM role that has the correct permissions.

If the source type is Amazon S3, provide the credentials to use to connect to Amazon S3.

For a self-hosted instance, select the location of the credentials. You can either provide credentials manually, or use credentials that are configured in environment variables. Note that after you save the dataset, you cannot change the selection.
If you are not using environment variables, then in the Access Key field, provide an AWS access key that is associated with an IAM user or role. For an example of a role that has the required permissions for an Amazon S3 dataset, go to .
In the Access Secret field, provide the secret key that is associated with the access key.
From the Region dropdown list, select the AWS Region to send the authentication request to.
In the Session Token field, provide the session token to use for the authentication request.
To test the credentials, click Test AWS Connection.
By default, connections to Amazon S3 use Amazon S3 encryption. To instead use AWS KMS encryption:
1. Click Show Advanced Options.
2. From the Server-Side Encryption Type dropdown list, select AWS KMS.
3. In the Server-side Encryption AWS KMS ID field, provide the KMS key ID. Note that if the KMS key doesn't exist in the same account that issues the command, you must provide the full key ARN instead of the key ID.
Note that after you save the new dataset, you cannot change the encryption type.
Click Save. Textual prompts you to .

Providing Azure credentials

If the source type is Azure, provide the connection information:

In the Account Name field, provide the name of your Azure account.
In the Account Key field, provide the access key for your Azure account.
To test the connection, click Test Azure Connection.
Click Save. Textual prompts you to .

Providing SharePoint credentials

If the source type is SharePoint, provide the credentials for the Entra ID application.

The credentials must have the following application permissions (not delegated permissions):

Files.Read.All - To see the SharePoint files
Files.ReadWrite.All -To write redacted files and metadata back to SharePoint
Sites.ReadWrite.All - To view and modify the SharePoint sites

To provide the credentials:

In the Tenant ID field, provide the SharePoint tenant identifier for the SharePoint site.
In the Client ID field, provide the client identifier for the SharePoint site.
In the Client Secret field, provide the secret to use to connect to the SharePoint site.
To test the connection, click Test SharePoint Connection.
Click Save. Textual prompts you to .

Deleting datasets

Required dataset permission: Delete a dataset

To delete a dataset:

On the dataset details page, click Settings.
On the Dataset Settings page, click Delete Dataset.
Click Confirm Delete.

Assigning tags to datasets

Required dataset permission: Edit dataset settings

Tags can help you to organize your datasets. For example, you can use tags to indicate datasets that belong to different groups, or that deal with specific areas of your data.

You can manage tags from both the Datasets page and the dataset details.

Managing tags from the Datasets page

On the Datasets page, the Tags column displays the currently assigned tags.

To change the tag assignment for a dataset:

Click Tags.
On the dataset tags panel, to add a new tag, type the tag text, then press Enter.
To remove a tag, click its delete icon.
To remove all of the tags, click the delete all icon.

Managing tags from the dataset details page

On the dataset details page, the assigned tags display under the dataset name.

To change the tag assignment:

Click Tags.
On the dataset tags panel, to add a new tag, type the tag text, then press Enter.
To remove a tag, click its delete icon.
To remove all of the tags, click the delete all icon.

Changing the dataset name

Required dataset permission: Edit dataset settings

The dataset name displays in the panel at the top left of the dataset details page.

To change the dataset name:

On the dataset details page, click Settings.
On the Dataset Settings page, in the Dataset Name field, provide the new name for the dataset..

Click Save Dataset.

Changing cloud storage credentials and output location

Required dataset permission: Edit dataset settings

For a cloud storage dataset, you can:

Update the cloud storage credentials. Note that this option is only available if you provided the credentials manually. If you use the credentials set in environment variables, then you cannot change the credentials.
Change the output location for the generated output files

You configure the connection credentials and output location in the Connection Settings section of the Dataset Settings page.

To display the Dataset Settings page, on the dataset details page, click Settings.

After you update the configuration, click Save Dataset.

Changing cloud storage credentials

Amazon S3

To provide updated credentials for Amazon S3:

In the Access Key field, provide an AWS access key that is associated with an IAM user or role. For an example of a role that has the required permissions for an Amazon S3 dataset, go to .
In the Access Secret field, provide the secret key that is associated with the access key.
From the Region dropdown list, select the AWS Region to send the authentication request to.
In the Session Token field, provide the session token to use for the authentication request.
To test the credentials, click Test AWS Connection.

Azure

To provide updated credentials for Azure:

In the Account Name field, provide the name of your Azure account.
In the Account Key field, provide the access key for your Azure account.
To test the connection, click Test Azure Connection.

SharePoint

SharePoint credentials must have the following application permissions (not delegated permissions):

Files.Read.All - To see the SharePoint files
Files.ReadWrite.All -To write redacted files and metadata back to SharePoint
Sites.ReadWrite.All - To view and modify the SharePoint sites

To provide updated credentials for SharePoint:

In the Tenant ID field, provide the SharePoint tenant identifier for the SharePoint site.
In the Client ID field, provide the client identifier for the SharePoint site.
In the Client Secret field, provide the secret to use to connect to the SharePoint site.
To test the connection, click Test SharePoint Connection.

Setting the output location

The output location is where Textual writes the redacted files.

When you create a cloud storage database, after you select the initial set of files and folders, Textual prompts you to select the output location.

For an existing dataset, you set the output location from the Dataset Settings page.

Under Select Output Location, select the cloud storage folder where Textual writes the output files for the dataset.

When you generate output for a cloud storage dataset, Textual creates a folder in the output location. The folder name is the identifier of the job that generated the files.

Within the job folder, Textual recreates the folder structure for the original files.

Textual then writes the output files to the corresponding folders.

Sharing dataset access

Required permissions:

Global permission - View users and groups
Either:
- Global permission - Manage access to datasets
- Dataset permission - Share dataset access

Tonic Textual uses dataset permission sets for role-based access (RBAC) of each dataset.

A dataset permission set is a set of dataset permissions. Each permission provides access to a specific dataset feature or function.

Textual provides built-in dataset permission sets. Organizations can also configure custom permission sets.

To share dataset access, you assign dataset permission sets to users and to SSO groups, if you use SSO to manage Textual users. Before you assign a dataset permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group.

To change the current access to the dataset:

Either:
- On the Datasets page, click the share icon for the dataset to share.
- On the dataset details page, click Share.

The dataset access panel contains the current list of users and groups who have access to the dataset, and displays their assigned dataset permission sets. To add a user or group to the list of users and groups:
1. In the search field, begin to type the user email address or group name.
2. From the list of matching users or groups, select the user or group to add.
For a user or group, to change the assigned dataset permission sets:
1. Click Access. The dropdown list displays the list of custom and built-in dataset permission sets.
2. Under Custom Permission Sets, check the checkbox next to each dataset permission set to assign to the user or group. To remove an assigned dataset permission set, uncheck the checkbox.
3. Under Built-In Permission Sets, click the dataset permission set to assign to the user or group. You can only assign one built-in permission set. By default, for an added user or group, the Viewer permission set is selected. To not grant any built-in permission set, select None.

Manage dataset files

Supported file types

Tonic Textual can process the following types of files:

txt
csv
tsv
docx
xlsx
pdf
png
tif or tiff
jpg or jpeg

Uploading and deleting local files

For a local file dataset, you upload and remove new files directly.

On Tonic Textual Cloud, and by default for self-hosted instances, Textual stores the uploaded files in the application database.

On a self-hosted instance, you can instead configure an S3 bucket where Textual stores the files. In the S3 bucket, the files are stored in a folder that is named for the dataset identifier.

For more information, go to Setting the S3 bucket for file uploads and redactions.

For an example of an IAM role with the required permissions, go to .

Adding files to the dataset

Required dataset permission: Upload files to a dataset

From the dataset details page, to add files to the dataset:

In the panel at the top left, click Upload Files.

Search for and select the files.

Textual uploads and then processes the files. For more information about file processing, go to Tracking and managing file processing.

Do not leave the page while files are uploading. If you leave the page before the upload is complete, then the upload stops.

You can leave the page while Textual is processing the file.

On a self-hosted instance, when a file fails to upload, you can download the associated logs. To download the logs, click the options menu for the file, then select Download Logs.

Removing files from the dataset

Required dataset permission: Delete files from a dataset

To remove a file from the dataset:

In the file list, click the options menu for the file.
In the options menu, click Delete File.

Selecting cloud storage files

Required dataset permission: Edit dataset settings

For a cloud storage dataset, you manage files from the file selection panel.

When you create a dataset, after you provide the cloud storage credentials and save the dataset, Textual immediately prompts you to select dataset files. After you select the files and click Next, Textual prompts you to set the output location.

For an existing dataset, to display the File Selection panel, on the dataset details page, click Select Files.

The file selection includes:

Whether to restrict the dataset to specific file types
The files or folders to include in the dataset

When you change the file selection, Textual scans the files for entities. For more information, go to .

Filtering files by file extension

When you select files, you can filter the selectable files based on file extension.

To limit the file extensions to include:

Click File Extension Filter. By default, all file extensions are included, and none of the checkboxes are checked.
Check the checkbox for each file extension to include. As you select the file extensions to include, Textual updates the navigation pane so that you can only select files that have one of those file extensions. It hides files that have other file extensions and folders that do not contain files with the selected file extensions.

Selecting files and folders to include

In the file selection area, you navigate to and select the folders and files to add to the dataset.

Navigating through the folders

In the navigation area, to display the contents of a folder, click the Open link for the folder.

Selecting a file or folder

To add a folder or file to the dataset, check its checkbox.

Managing selected folders

In the navigation pane, when you check a folder checkbox, Textual adds it to the Prefix Patterns list.

Adding a folder manually

Instead of navigating to a folder and selecting it, you can add the path to the list manually

To add a folder path:

Click Add Prefix Pattern.
In the field, type the path to the folder, then click the save icon.

Removing folder paths

To remove a folder path from the dataset, either:

In the navigation pane, uncheck its checkbox.
In the Prefix Patterns list, click its delete icon.

For the selected folders, the dataset includes all of the applicable files in the folder that:

Are of a file type that Textual supports
Match the file extension filter

Managing selected files

In the navigation pane, when you select an individual file, Textual adds it to the Selected Files list.

To delete a file, either:

In the navigation pane, uncheck its checkbox.
In the Selected Files list, click its delete icon.

Navigating the file list

For a local files dataset, the file list is a single list of uploaded files.

For cloud storage dataset, the file list initially displays the first folder that contains dataset files. You can then navigate through the folders and files.

For a cloud storage dataset, you can search for folders in the currently displayed bucket or folder.

To search for a folder, in the search field, start to type the folder name.

Tracking and managing file processing

When you add files to a local files dataset, or change the file selection for a cloud storage dataset, Tonic Textual automatically scans the files to identify the entities that they contain.

When you change the dataset configuration, Textual also prompts you to run a new scan. For example, a new scan is required when you:

Configure added and excluded values
Change the available custom entity types

The file list reflects the current scanning status for the file. A file is initially queued for scanning. When the scan starts, the status changes to scanning. When Textual finishes processing a file, it marks the file as scanned.

As Textual processes each file, it updates the results in the dataset details heading and the entity types list.

Pausing the file processing

If needed, you can pause the file processing. To pause the processing, click Pause.

The information in the heading and entity types list only reflect the files that are scanned.

For a cloud storage dataset, when you generate output, Textual only includes files that are scanned.

Starting a scan on a paused file

Required dataset permission: Start a scan of dataset files

After you pause the scan, you can start a scan on individual files.

To start a scan on a file:

Click the options menu for the file.
Click Scan.

Downloading logs for files that fail to process

Required dataset permission: Start a scan of dataset files

When Textual is unable to process a file, it displays an error for that file.

To download log files for the failed file:

Click the options menu for the file.
Click Download Logs.

Configure the redaction

Working with custom entity types

From the entity types list, you can set whether each custom entity type is active. For regex-based custom entity types, you can edit the custom entity configuration.

Enabling and disabling custom entity types

Required dataset permission: Edit dataset settings

In the entity types list, custom entity types include a toggle to indicate whether the custom entity type is active for that dataset.

Active entity types are either in found types or not found types list, based on whether Textual detected values of that type in the dataset data.

To disable a custom entity type, set the toggle to the off position.

When a custom entity type is not active, it is listed under Inactive custom entity types. To enable the custom entity type, set the toggle to the on position.

Updating the configuration of a regex-based custom entity type

Required global permission - either:

Create custom entity types
Edit any custom entity type

From the dataset details, you can edit the configuration of a regex-based custom entity type. To edit a regex-based custom entity type, click the settings icon for that type.

Note that any changes to the custom entity type settings affect all of the datasets that use the custom entity type.

For information on how to configure a custom entity type, go to .

Displaying the Custom Entity Types page

Required global permission - either:

Create custom entity types
Edit any custom entity type

From the entity types list, to display the Custom Entity Types page, click Custom Entity Types.

The page opens in a new tab.

Running a new scan to reflect custom entity type changes

When you enable, disable, or edit custom entity types, the changes do not take effect until you run a new scan.

To run a new scan, click Scan.

Selecting the handling option for entity types

Required dataset permission: Edit dataset settings

For datasets that produce redacted files, for each entity type, you choose how to handle the detected values. This determines how each value displays in the output files.

Datasets that produce JSON output do not use entity type handling options.

Available handling options

The available options are:

Synthesis - Indicates to replace the value with another realistic value. For example, the first name value Michael might be replaced with the value John. The synthesized values are always consistent, meaning that a given entity value always has the same replacement value. For example, if the first name Michael appears multiple times in the text, it is always replaced with John. Textual does not synthesize any excluded values. For custom entity types, Textual scrambles the values.
Redaction - This is the default option, except for the Full Mailing Address entity type, which is Off by default. For text files, Redaction indicates to tokenize the value - to replace it with a token that identifies the entity type followed by a unique identifier. For example, the first name value Michael might be replaced with NAME_GIVEN_12m5s. The identifiers are consistent, which means that for a given original value, the replacement always has the same unique identifier. For example, the first name Michael might always be replaced with NAME_GIVEN_12m5sb, while the first name Helen might always be replaced with NAME_GIVEN_9ha3m2. For PDF files, Redaction indicates to either cover the value with a black box, or, if there is space, display the entity type and identifier. For image files, Redaction indicates to cover the value with a black box. Textual does not redact any excluded values.
Off - Indicates to not make any changes to the values. For example, the first name value Michael remains Michael. This this the default option for the Full Mailing Address entity type.

Selecting the handling option for a specific entity type

To select the handling option for an individual entity type, click the option for that type.

Selecting the handling option for all of the entity types

To select the same handling option for all of the entity types, from the Bulk Edit dropdown above the data type list, select the option.

Configuring handling of file components

Required dataset permission: Edit dataset settings

The Dataset Settings panel includes options for how Textual handles the following file components:

For .docx files, images and comments
For PDF files, scanned-in signatures

To display the Dataset Settings page, on the dataset details page, click Settings.

Configuring how to handle .docx images

For .docx images, including .svg files, you can configure the dataset to either:

Redact the image content. When you select this option, Textual looks for and blocks out sensitive values in the image.
Ignore the image.
Replace the images with black boxes.

On the Dataset Settings page, under Image settings for DOCX files:

To redact the image content, click Redact contents of images using OCR. This is the default selection.
To ignore the images entirely, click Ignore images during scan.
To replace the images with black boxes, click Replace images from the output file with black boxes.

Configuring how to handle .docx tables

For .docx tables, you can configure the dataset to either:

Redact the table content. When you select this option, Textual detects sensitive values and replaces them based on the entity type configuration.
Block out all of the table cells. When you select this option, Textual places a black box over each table cell.

On the Dataset Settings page, under Table settings for DOCX files:

To redact the table content, click Redact content using the entity type configuration. This is the default selection.
To block out the table content, click Block out all table cell content.

Configuring how to handle .docx comments

For comments in a .docx file, you can configure the dataset to either:

Remove the comments from the file.
Ignore the comments and leave them in the file.

On the Dataset Settings page, to remove the comments, toggle Remove comments from the output file to the on position. This is the default configuration.

To ignore the comments, toggle Remove comments from the output file to the off position.

Configuring whether to redact PDF signatures

By default, Textual redacts scanned-in signatures in PDF files. You can configure the dataset to instead ignore the signatures.

On the Dataset Settings page:

To redact PDF signatures, toggle Detect and redact signatures in PDFs to the on position. This is the default configuration.
To ignore PDF signatures, toggle Detect and redact signatures in PDFs to the off position.

Adding manual overrides to PDF files

Required dataset permission: Edit dataset settings

For a PDF file in a dataset, you can add manual overrides to selected areas of a file. Manual overrides can either ignore redactions from Tonic Textual, or add redactions.

Editing an individual PDF file

For PDF files, you can add manual overrides to the initial redactions, which are based on the detected data types and handling configuration.

For each manual override, you select an area of the file.

For the selected area, you can either:

Ignore any automatically detected redactions. For example, a scanned form might show an example or boilerplate content that doesn't actually contain sensitive values.
Redact that area. The file might contain sensitive content that Tonic Textual is unable to detect. For example, a scanned form might contain handwritten notes.

You can also apply a template to the file.

Selecting the manual override option for a file

To manage the manual overrides for a PDF file:

In the file list, click the options menu for the file.
In the options menu, click Edit Redactions.

The File Redactions panel displays the file content. The values that Textual detected are highlighted. The page also shows any manual overrides that were added to the file.

Applying a PDF template to a file

If a dataset contains multiple files that have the same format, then you can create a template to apply to those files. For more information, go to .

On the File Redactions panel, to apply a template to the file, select it from the template dropdown list.

When you apply a PDF template to a file, the manual overrides from that template are displayed on the file preview. The manual overrides are not included in the Redactions list.

Adding a manual override

On the File Redactions panel, to add a manual override to a file:

Select the type of override. To indicate to ignore any automatically detected values in the selected area, click Ignore Redactions. To indicate to redact the selected area, click Add Manual Redaction.
Use the mouse to draw a box around the area to select.

Textual adds the override to the Redactions list. The icon indicates the type of override.

In the file content:

Overrides that ignore detected values within the selected area are outlined in red.
Overrides that redact the selected area are outlined in green.

Navigating to a manual override

To select and highlight a manual override in the file content, in the Redactions list, click the navigate icon for the override.

Removing a manual override

To remove a manual override, in the Redactions list, click the delete icon for the override.

Saving the manual overrides

To save the current manual overrides, click Save.

Preview and obtain output

Downloading local output files

Required dataset permission: Download redacted dataset files

For each file in a dataset, you can download the output file.

Downloading a single output file

From the file list, to download a single output file:

Click the options menu for the file.
In the options menu, click Download File.

Downloading all of the output files

To download all of the output files, click Download All Files.

Generating cloud storage output files

To generate original format output files for a cloud storage dataset, on the dataset details page, click Generate to <cloud storage type>.

Tonic Textual generates the output files to the . If the output location is not configured, then the generate option is disabled.

For datasets that produce JSON output, Textual generates the output files automatically as soon as the output location is configured.

Guided redaction (Beta)

About guided redaction

The guided redaction feature is currently in beta.

The Textual guided redaction tool blocks out sensitive values in files. For example, you might use guided redaction to prepare documents to provide in response to a Freedom of Information Act request.

Guided redaction supports built-in and custom entity types. Textual completes an initial scan of the files to identify sensitive values in the files. You can then manually add and remove redactions.

The values are covered with a black or white box. You assign and display reference codes to identify the type of content for each redaction.

The overall workflow is as follows:

Pre-project setup

Before you create a guided redaction project:

Set up the list of statuses that can be assigned to a file or a project. Each status is associated with a color. Textual provides a set of built-in statuses, including a Not started status that is applied automatically to all new projects and files, and a Done status that indicates that the project or file is complete. You can configure custom statuses and change the names and colors of the built-in statuses. You can also delete the built-in statuses, except for Not started and Done.
If you assign reference codes to the redactions, configure the reference codes. Reference codes identify the type of information that is redacted. You can optionally link each code to up to 5 Textual entity types. For example, if you use a single code for all name values, then you could link that code to both the Given Name and Family Name entity types. Be sure to map reference codes to all of the entity types that are present in your files.

Project creation and population

.
. For new files, Textual uses its built-in models to scan each file for entity values.

File redaction

For each file, in Redaction mode, you can .

Every redaction must be assigned a reference code. Redactions that do not have reference codes are not redacted in the output.

As you work on the redaction, you can update the and statuses.

Use the to track the project activity.

Redaction review

In Review mode, .

As you work on the review, you can update the and statuses.

Use the to track the project activity.

Output preview and download

In Preview mode, based on the current redactions.

When the redaction and review is complete, . All downloaded output is in PDF format, with the individual PDF files bundled into a .zip file.

When you download files, you select whether the output is redacted or is to be used for review.

Review output does not cover the redactions, and always displays the reference codes.
For redacted output, you select the color of the box and whether to display the reference codes.

Configuring guided redaction options

Before you create guided redaction projects, configure the following options.

Configuring the available file and project statuses

As you redact and review the project files, you set the status of each file and project. Projects and files use the same set of status values.

Each status is associated with a color.

Built-in statuses

Textual comes with a set of built-in statuses. The built-in statuses are:

Not started - This is the default status. It is applied automatically to all new projects and files. You cannot delete this status.
In progress
Ready for review
Review in progress
Done - This is intended to be the final status for a project or file. You cannot delete this status.

You can change the name and assigned color of all of the built-in statuses. You can delete the built-in statuses that you do not need, except for the Not started status and the Done status.

You can also add custom statuses, to accommodate your particular redaction and review process.

Displaying the status list

On the Guided Redaction page, to display the status list, click Status Settings.

The status list shows the status name and the associated color.

Adding a status

Required global permission: Create guided redaction status values

From the Status Settings panel, to add a status:

Click Add a status.
On the status configuration panel, in the field, provide a name for the status. Status names must be unique.
Click the color to assign to the status.
Click Save.

Editing a status

Required global permission: Edit guided redaction status values

From the Status Settings panel, to change the status configuration:

Click the status name.
On the status configuration panel, you can change the status name and color. Remember that the status name must be unique.
To save the changes, click Save.

Deleting a status

Required global permission: Edit guided redaction status values

You cannot delete:

The built-in Not started status.
The built-in Done status.
A status that is currently assigned to a project or file.

From the Status Settings panel, to delete a status:

Click the status name.
On the status configuration panel, click the delete icon.

Configuring reference codes for guided redaction

Reference codes are used to indicate the type of value that is redacted.

Before you can start a guided redaction, you must set up the reference codes.

You can optionally link each reference code to one or more Textual entity types. Make sure that you map reference codes to all of the entity types that appear in your files.

Displaying the reference codes

To display the list of reference codes:

In the Textual navigation bar, click Guided Redaction.
On the Guided Redaction page, click Reference Codes Settings.

For each each reference code, the list includes:

Code value
Any Textual entity types that the code is mapped to
The user who most recently updated the code configuration
When the code was most recently updated

Creating a reference code

Required global permission: Create redaction reference codes

To create a reference code:

Click New Reference Code.
In the Reference Code field, type the code.
Optionally, check the checkbox next to each built-in entity type that applies to the reference code. You can link up to 5 entity types to a reference code. For example, a reference code is used for any name value. You would link the reference code to both the Given Name and Family Name entity types. The list indicates when an entity type is already linked to a reference code. You can link the same entity type to multiple reference codes. Linking entity types is optional. If the reference code represents a value that is not represented in the Textual entity types, then you do not link it.
Click Create.

Editing a reference code

Required global permission: Edit redaction reference codes

For a reference code, you can change the code value and the assigned built-in entity types.

To edit a reference code:

Click the settings icon for the code.
On the details panel, update the code. You can change the code and the assigned entity types.
Click Save.

Deleting a reference code

Required global permission: Edit redaction reference codes

You cannot delete a reference code that is currently assigned to a redaction.

To delete a reference code, click its delete icon.

Creating and managing guided redaction projects

Setting up projects

Configure entity types and access

Viewing the list of guided redaction projects

Required permissions

Either:

Global permission - either:
- Use guided redaction projects
- View all guided redaction projects
Guided redaction permission: Access to view or perform an action on one or more guided redaction projects

The Guided Redaction page contains the list of redaction projects.

To display the Guided Redaction page, in the Textual navigation bar, click Guided Redaction.

The list displays the projects that you have access to.

Information in the list

For each project, the list includes:

The name of the project
Any tags assigned to the project
The number of files in the project
The project status
When the project was created
The user who created the project

Filtering the project list

You can filter the list based on the project name.

In the search field, type text from the project name. As you type, Textual updates the list to only include matching projects.

Sorting the project list

By default, the list is sorted in descending order by the creation date. The most recently created projects are at the top of the list.

You can sort the list by any of the other columns, other than the options column.

To sort by a column, click the column heading. To reverse the sort order, click the column heading again.

Creating a guided redaction project

Required global permission: Create guided redaction projects

To create a project:

On the Guided Redaction page, click New Project.
On the new project panel, in the Project Name field, provide a name for the project.
Click Save.

Textual displays the project details page.

Assigning tags to a project

You can assign tags to each guided redaction project, to help to further identify and link projects. For example, you might use a tag to indicate when a project is for a FOIA request.

On the Guided Redaction page, the Tags column contains the assigned tags for the project.

On the project settings panel the Tags field lists the project tags. To display the settings panel, either:

On the Guided Redaction page, click the settings icon for the project.
On the project details page, click More, then click Settings.

To change the assigned tags:

Click the Tags column or field.
To add a tag, type the tag text, then press enter.
To remove a tag, click its delete icon.
To remove all of the tags, click the delete icon at the right of the tags field.

Enabling and disabling entity types and values

The entity types settings for a project determine the Textual entity types that Textual detects in new project files. You can enable or disable both . You cannot enable model-based custom entity types.

For example, you can tell Textual to ignore all Given Name values in new project files.

For built-in entity types, you can also configure specific values to add to or exclude from the detection. For example, you can keep the Given Name entity type active, but indicate to ignore the value "Mark".

By default, all of the entity types are active, and there are no added or excluded values.

Any changes to the project entity type settings only affect files that are added after the change. Files that were already scanned are not affected.

You can override the entity type settings in individual files. For more information, go to .

Displaying the project settings panel

The entity type configuration for the project is part of the project settings panel.

To display the project settings panel, either:

On the Guided Redaction page, click the settings icon.
On the project details panel, click More, then select Settings.

After you change the project settings, Textual automatically rescans the files to add or remove automatically detected redactions that are affected by the changes.

Determining the enabled entity types

By default, a project includes all of the built-in and custom entity types. When you add a file, Textual automatically scans the file to identify instances of those entity types.

From the settings panel, you can disable entity types. For example, if you disable the Occupation entity type, Textual does not scan for occupation values.

On the settings panel, to exclude an entity type from the initial file scan, set the entity type toggle to the off position.

Configuring added and excluded values for built-in entity types

For each built-in entity type, you can configure added and excluded values.

You might add values that Textual does not detect because, for example, they are specific to your organization or industry.

You might exclude values that Textual redacts incorrectly.

To display the panel to add and exclude values, click the add or exclude values icon.

On the panel:

Use the Add to detection tab to add values.
Use the Remove from detection tab to remove values.

Each value can be either a specific word or phrase to add or exclude, or a regular expression to identify the values to add or exclude. Regular expressions must be C# compatible.

Sharing access to a guided redaction project

Required permissions

Global permission - View users and groups

Either:

Global permission - Control access to all guided redaction projects
Guided redaction permission - Share guided redaction access

Textual uses guided redaction permission sets for role-based access control (RBAC) of each project.

A guided redaction permission set is a collection of guided redaction permissions.

Textual provides built-in guided redaction permission sets. Organizations can also configure custom permission sets.

To share project access, you assign guided redaction permission sets to users and to SSO groups, if you use SSO to manage Textual users. Before you assign a guided redaction permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group.

To change the current access to a project:

Either:
1. On the Guided Redaction page, click the share icon for the project.
2. On the project details page, click More, then click Share.
The project access panel contains the current list of users and groups who have access to the project, and displays their assigned guided redaction permission sets. To add a user or group to the list of users and groups:
1. In the search field, begin to type the user email address or group name.
2. From the list of matching users or groups, select the user or group to add.
For a user or group, to change the assigned guided redaction permission sets:
1. Click Access. The dropdown list displays the list of custom and built-in guided redaction permission sets.
2. Under Custom Permission Sets, check the checkbox next to each guided redaction permission set to assign to the user or group. To remove an assigned guided redaction permission set, uncheck the checkbox.
3. Under Built-In Permission Sets, click the guided redaction permission set to assign to the user or group. You can only assign one built-in permission set. By default, for an added user or group, the Viewer permission set is selected. To not grant any built-in permission set, select None.

Deleting a project

Required guided redaction permission: Delete a project

To delete a guided redaction project:

Either:
1. On the Guided Redaction page, click the options menu for the project, then click Delete project.
2. On the project details page, click More, then click Delete project.
On the confirmation panel, click Delete.

Managing the list of project files

The project details page contains the list of files in the project. To display the project details, on the Guided Redaction page, click the project name.

Information in the file list

For each file in the project, the list includes the following information:

Name of the file
Number of redactions in the file
Number of pages in the file
Number of comments in the file
Status of the file
Name of the user who most recently made changes in the file
When the most recent change occurred

Filtering the file list

You can filter the file list based on the file name.

To filter the list, in the search field, type text in the file name. As you type, Textual updates the list to only include matching files.

Sorting the file list

By default, the file list is sorted in descending order based on the update date. The most recently updated files are at the top of the list.

You can sort the file list by any column except for the options column. To sort the list by a column, click the column heading. To reverse the sort order, click the column heading again.

Supported file types

You can use guided redaction for the following types of files:

.jpg
.msg
.pdf
.png
.txt

Adding files to the project

Required guided redaction permission: Upload files to a project

When you add files to a project, Textual automatically assigns the Not started status to those files. It also scans the files for built-in entity types, based on the project configuration.

To add files to a project:

On the project details page, click Upload Files.
Search for and select the files to add.

Removing a file from a project

Required guided redaction permission: Delete files from a project

To delete a file from the project, either:

On the project details page, click the delete icon for the file.
On the file details page, click More, then click Delete File.

Navigating project file details

Required guided redaction permission

Either:

Edit file redactions
Preview project files
Manage comments
Download project files

To display the file details, on the project details page, click the file name.

On the file details page:

For a non-paginated file, such as plain text file, the file content displays as a single block.
For a paginated file, such as a PDF, the file displays one page at a time. The page list for the file displays at the left.

In the file content, the redactions are highlighted. The first time you view a file, these are the redactions from the initial Textual scan.

Searching for text

To search for a specific piece of text in a file, type the text into the search field, then press Enter.

If it finds the text, Textual displays the number of matches in the file, you can then navigate between the matches.

Displaying redaction details

When you click a redaction that was added by the Textual scan, the redaction panel displays.

For redactions that Textual added, the redaction panel displays the entity type.

Navigating between file pages

For a paginated file, to jump to a specific page, in the left-hand page navigation, click the page.

Viewing the redaction list

At the right of the file details is the file redaction list.

For a paginated file, the redaction list is grouped by page. For each redaction, the redaction list contains the redacted text, or an indicator that the redaction is for a selected area or the entire page.

The redaction list also lists the assigned reference codes.

To search for specific values in the file, in the search field, type the value text.

Processing a project file

Redacting, reviewing, and previewing a file

Configuring and commenting on files

Redacting a project file

Required guided redaction permission: Edit file redactions

In Redaction mode, you add and remove redactions.

To change to Redaction mode, in the file heading, click the edit icon.

Adding redactions

In a text file, you can select specific text to redact.

For non-text files such as a PDF or an image, you redact a selected area or an entire page.

Redacting selected text in a text file

To add a single redaction of selected text:

Select the text. Note that you can add a redaction of text within an existing redaction.
On the redaction panel, select the reference codes that apply.

Redacting a selected area

For files other than plain text files, you redact an area that you draw over the page. You can draw over a a single word, or you might use this option to redact a displayed image or an entire table.

To redact a selected area:

Click Draw Redaction or press d. This puts you in draw redaction mode.
Use the mouse to draw a box over the area to redact.
After you draw the box, identify the reference codes that apply.

The selected reference codes replace any individual redactions that were within the area.

However, Textual does save those individual redactions. If you remove the redaction for the selected area, the individual redactions are restored.

To exist draw redaction mode, click Draw Redaction or press d again.

Redacting an entire page

For paginated files such as a PDF, you can redact an entire page.

To redact the current page:

Click Redact Current Page.
On the panel, select the reference codes that apply.

The selected reference codes replace any individual redactions that were on the page.

However, Textual does save those individual redactions. If you remove the redaction for the entire page, the individual redactions are restored.

Identifying redactions that do not have reference codes

Every redaction must be assigned at least one reference code. To enable Textual to assign reference codes when it performs its scan, you must assign reference codes to the entity types that are present in the project files.

Redactions that do not have reference codes are not redacted in the downloaded output files. They also are not marked in Review mode or Preview mode.

In Redaction mode:

In the file content, redactions without a reference code are outlined in red.
The redactions list displays a separate list of redactions that do not have assigned reference codes. The list heading includes a link to the reference code settings.

Changing the assigned reference codes for a redaction

For each redaction, you can add and remove assigned reference codes.

From the file content

From the file content, to change the assigned reference codes for a redaction:

Click the redaction.
On the panel, to add a reference code, click Add, then select the reference code to add.
To remove a reference code, click the delete icon for the reference code.

From the redaction list

The redaction list at the right includes the assigned reference codes for each redaction. A + icon displays next to the assigned codes. To change the assigned reference codes:

Click the redaction.
On the panel, check the checkboxes for the reference codes to add.

From the list of entities that do not have reference codes to add reference codes:

Click the entity value, then click Assign reference codes.
On the panel, check the checkboxes for the reference codes to add.

Removing redactions

You can remove redactions from the file.

When you remove a redaction of a selected area or an entire page, Textual restores any individual redactions that were there previously.

Removing an individual redaction

From the file content, to remove a single redaction:

Click the redaction.
On the redaction panel, click the delete icon.

Removing multiple redactions

From the redaction list, to remove one or more redactions:

Check the checkbox for each redaction to remove.
Click Remove.

Adding and removing text highlights

Instead of redacting selected text, you can choose to highlight it.

Text highlights are not assigned reference codes. In the redaction list and output, the text is struck through instead of covered.

Adding a text highlight

To add a text highlight:

Select the text to highlight.
On the panel, click the strikethrough icon.

Removing a text highlight

To remove a highlight:

Click the highlight.
On the panel, click the delete icon.

Selecting multiple PDF redactions to update

In a PDF file, to update multiple redactions:

Either click Bulk Selection, or press b. This puts you into bulk selection mode.
Use the mouse to draw a box around the redactions to select.
On the panel, select the updates to apply.

When you are finished with the updates, to exit bulk selection mode, click Bulk Selection or press b again.

Undoing and redoing redaction changes

Textual maintains a history of your redaction changes.

To undo the most recent change, click the undo icon.

To restore the most recent undone change, click the redo icon.

Note that for changes applied in bulk selection mode, Fabricate only undoes or redoes one change at a time. For example, if you selected 3 redactions and then removed them, when you select the undo option, it only restores one of those redactions. To restore all of those redactions, you would select undo 3 times.

Reviewing redactions

In Review mode, you mark individual redactions as reviewed.

This indicates that you verified that the redaction is correctly detected and identified.

To change to Review mode, in the file heading, click the review icon.

Review mode does not highlight redactions that do not have an assigned reference code. That content is not redacted.

In the redaction list, all of the reference codes use the same highlighting. There is no distinction between automatically and manually assigned codes.

Navigating between redactions

At the top left is the summary of the reviewed redactions. It shows the number of reviewed redactions and the total number of redactions.

The redactions panel at the right displays the list of redactions. There are no options to change the configuration. The list does not contain redactions that do not have assigned reference codes.

To navigate directly to the first unreviewed redaction, click the icon next to the review summary.

When you click a redaction, the review panel also allows you to navigate to the previous and next redactions.

Marking a redaction as reviewed

To mark a redaction as reviewed, either:

In the file content:
1. Click the redaction.
2. On the panel, click Mark as reviewed.
In the redaction list, click the redaction.

When you mark a redaction as reviewed:

In the header, the count of reviewed redactions is updated.
In the file content, the redaction is highlighted in green
In the redaction list, a check mark is added in front of the redaction.

Unmarking a reviewed redaction

To mark a reviewed redaction as not reviewed, either:

In the file content:
1. Click the redaction.
2. On the panel, click Unmark as reviewed.
In the redaction list, click the redaction.

Previewing file output

Required guided redaction permission: Preview project files

Preview mode displays the redacted content as it appears in the downloaded file.

To change to Preview mode, in the file heading, click the preview icon.

In preview mode:

The boxes that cover the redacted content are always black.
The reference codes display on the boxes.

Preview mode does not display redactions that do not have an assigned reference code. That content is not redacted in the output.

Setting the file status

The file status reflects where the file is in the redaction and review process. The status is selected from the .

From the project file list, or the file details page, to set the status of a file:

Click the current status value.
In the dropdown list, select the new status.

Overriding the project entity type configuration

By default, each file in a project uses the project-wide settings for Textual entity types. The entity type settings include:

Whether each entity type is enabled. You can enable or disable both built-in and regex-based custom entity types. You cannot enable model-based custom entity types. When you disable an entity type, values of that type are ignored by the Textual scan.
For each built-in entity type, added and excluded values.

Within a file, you can override the project settings. When you save the new settings, Textual automatically rescans the file to change the detected entities based on the new settings.

Displaying the file-specific entity type settings

To display the file-specific entity type settings, either:

On the details panel for a redaction, click the settings icon.
In the file header, click More, then click File-specific entity detection settings.

Enabling and disabling entity types

To enable an entity type, set the entity type toggle to the on position.

To disable an entity type, set the toggle to the off position.

Adding and excluding values for a built-in entity type

To display the panel to add and exclude values, click the add or exclude values icon.

On the panel:

Use the Add to detection tab to add values.
Use the Remove from detection tab to remove values.

Each value can be either a specific word or phrase to add or exclude, or a regular expression to identify the values to add or exclude. Regular expressions must be C# compatible.

Saving and updating the detection based on the changes

When you finish the configuration changes, click Save.

Textual rescans the file and changes the detected entities based on the new settings.

For example, the initial scan detected a Given Name value. You change the file-specific settings to disable the Given Name entity type. After the new scan, the value is no longer highlighted.

Commenting on a project file

You can add and respond to comments on the redacted file.

For example, you might want to add a message to a future reviewer to explain why a value is redacted, or why you selected the reference codes that you did.

Or a reviewer might want to add a comment to indicate why they believe that a redaction is not correct.

Users can respond to comments, which starts a comment thread.

You can resolve or delete a comment thread. For example, a comment requests a change to a reference code assignment. After you make the change, you can resolve or delete the comment thread.

Viewing the comments list

To view the list of comments, in the toolbar, click Comment or press c. This puts you into comment mode. To exit comment mode, click Comment or press c again.

In comment mode, the redactions list is replaced with the Comments panel, which contains a list of comment threads.

The dropdown at the top right controls which comment threads to include. You can display either:

All comment threads, both open and resolved
Only open comment threads. This is the default.
Only resolved comment threads.

Starting a comment thread

To start a new comment thread in a file:

To start a new comment thread, you must be in comment mode. To enter comment mode, click Comment or press c.
For a PDF or image, click the location where you want to place the comment. For other text files, select the text to attach the comment to.
In the comment field, type the text of the comment.
Click Comment.

The comment thread is represented by a comment icon. The comment icons are always visible, even when the comments list is not displayed.

To exit comment mode, click Comment or press c again.

Adding a response to a comment thread

To add a response to a comment thread:

Click the comment icon for the thread. You do not need to change to comment mode.
In the text field, type the text of the response.
Click Reply.

Editing a comment in a thread

To edit a comment from within a thread:

Click the comment icon for the thread. You do not need to change to comment mode.
Click the edit icon for the comment to edit.
In the text field, edit the text of the comment.
Click Save.

Resolving a comment thread

To resolve a comment thread:

Click the comment icon for the thread. You do not need to change to comment mode.
Click the resolve icon.

Deleting a comment thread

To delete a comment thread.

Click the comment icon for the thread. You do not need to change to comment mode.
Click the delete icon.

Viewing the audit log for a project

The audit log tracks the following actions associated with the project:

Project is created
Files are added or removed
Project status changes
File status changes
Comments are added
Files are exported

To display the audit log:

On the project details page, click More.
In the menu, click Audit log.

For each entry, the audit log includes the following information:

When the action occurred.
The email address of the user who performed the action.
The type of action.
A description of the action.

Textual Python SDK

Installing the Textual SDK

The is a Python SDK that you can use to redact text and files.

It requires Python 3.9 or higher.

To install the Tonic Textual Python SDK, run:

pip install tonic-textual

Creating and revoking Textual API keys

Required global permission: Create an API key

To be able to use the Textual SDK, you must have an API key.

Alternatively, you can use the Textual API to to use for authentication.

Viewing the list of API keys

You manage keys from the User API Keys section of the User Profile page.

To display the User Profile page:

Click the user icon at the top right.
In the user menu, click User Profile.

Creating a Textual API key

To create a Textual API key:

From the User API Keys section, click Create New Key.
In the API Key Name field, type a name to use to identify the key.

Click the save icon.

Textual adds the key to the list.

Copying a Textual API key

To copy the text of an API key, so that you can use it in an SDK or API request, click its copy icon.

Revoking a Textual API key

To revoke a Textual API key, in the User API Keys list, click the Revoke option for the key to revoke.

Configuring the API key as an environment setting

You cannot instantiate the SDK client without an API key.

Instead of providing the key every time you call the Textual API, you can configure the API key as the value of the TONIC_TEXTUAL_API_KEY.

Obtaining JWT tokens for authentication

Instead of an API key, you can use the Textual API to obtain a JSON Web Token (JWT) to use for authentication.

Configuring the JWT and refresh token lifetimes

JWT lifetime

By default, a JWT is valid for 30 minutes.

On a self-hosted instance, to configure a different lifetime, set the SOLAR_JWT_EXPIRATION_IN_MINUTES.

Refresh token lifetime

You use a refresh token to obtain a new JWT. By default, a refresh token is valid for 10,000 minutes, which is roughly equivalent to 7 days.

On a self-hosted instance, to configure a different lifetime, set the environment variable SOLAR_REFRESH_TOKEN_EXPIRATION_IN_MINUTES.

Obtaining your first JWT and refresh token

To obtain your first JWT and refresh token, you make a login request to the Textual API. Before you can make this call, you must have a Textual account.

To make the call, perform a POST operation against:

The request payload is:

For example:

In the response:

The jwt property contains the JWT.
The refreshToken property contains the refresh token.

Obtaining a new JWT and refresh token

You use the refresh token to obtain both a new JWT and a new refresh token.

To obtain the new JWT and token, perform a POST operation against:

The request payload is:

In the response:

The jwt property contains the new JWT.
The refreshToken property contains the new refresh token.

Instantiating the SDK client

Whenever you call the Textual SDK, you first instantiate the SDK client.

To work with Textual datasets, or to redact individual files, you instantiate TonicTextual.

Instantiating when the API key is already configured

If the API key is configured as the value of the environment variable TONIC_TEXTUAL_API_KEY, then you do not need to provide the API key when you instantiate the SDK client.

For Textual datasets, or to use the redact method:

from tonic_textual.redact_api import TextualNer
# The default URL is https://textual.tonic.ai (Textual Cloud)
# If you host Textual, provide your Textual URL
textual = TextualNer()

Instantiating when the API key is not configured

If the API key is not configured as the value of the environment variable TONIC_TEXTUAL_API_KEY, then you must include the API key in the request.

For Textual datasets, or to use the redact method:

from tonic_textual.redact_api import TonicTextual
api_key = "your-tonic-textual-api-key"
# The default URL is https://textual.tonic.ai (Textual Cloud)
# If you host Textual, provide your Textual URL
textual = TonicTextual(api_key=api_key)

Datasets and redaction

You can use the Tonic Textual SDK to manage datasets and to redact individual strings and files.

Redact individual files

Required global permission: Use the API to parse or redact a text string

You can use the Textual SDK to redact and synthesize values in individual files.

Before you perform these tasks, remember to .

For a self-hosted instance, you can also configure the S3 bucket to use to store the files. For more information, go to . For an example of an IAM role with the required permissions, go to .

Sending a file to Textual

To send an individual file to Textual, you use .

You first open the file so that Textual can read it, then make the call for Textual to read the file.

The response includes:

The file name
The identifier of the job that processed the file. You use this identifier to retrieve a transformed version of the file.

Getting the file with redacted or synthesized values

After you use to send the file to Textual, you use to retrieve a transformed version of the file.

To identify the file, you use the job identifier that you received from textual.start_file_redaction. You can for the detected entity values.

Before you make the call to download the file, you specify the path to download the file content to.

Transcribe and redact an audio file

You can send an audio file to the Tonic Textual SDK. Textual creates a transcription of the audio file, and then redacts the transcription text as a string.

Audio file limitations

The file must be 25MB or smaller, and must be one of the following file types:

m4a
mp3
webm
mp4
mpga
wav

Sending the transcription and redaction request

To transcribe and redact an audio file, you use .

The request includes the .

The includes the redacted or synthesized content and details about the detected entity values.

Configure entity type handling for redaction

Required dataset permission: Edit dataset settings

By default, when you:

Configure a dataset
Redact a string
Retrieve a redacted file

Textual does the following:

For the string and file redaction, replaces detected values with tokens.
For LLM synthesis, generates realistic synthesized values.

When you make the request, you can:

Override the default behavior.
For individual files and text strings, specify custom entity types to include.

Specifying the handling option for entity types

For each entity type, you can choose to redact, synthesize, or ignore the value.

When you redact a value, Textual replaces the value with a token that consists of the entity type. For example, ORGANIZATION.
When you synthesize a value, Textual replaces the value with a different realistic value.
When you ignore a value, Textual passes through the original value.

To specify the handling option for entity types, you use the generator_config parameter.

Where:

<entity_type> is the identifier of the entity type. For example, ORGANIZATION. For the list of built-in entity types that Textual scans for, go to . For custom entity types, the identifier is the entity type name in all caps. Spaces are replaced with underscores, and the identifier is prefixed with CUSTOM_. For example, for a custom entity type named My New Type, the identifier is CUSTOM_MY_NEW_TYPE. From the Custom Entity Types page, to copy the identifier of a custom entity type, click its copy icon.

<handling_option> is the handling option to use for the specified entity type. The possible values are Redaction, Synthesis, GroupingSynthesis, ReplacementSynthesis, and Off.

For example, to synthesize organization values, and ignore languages:

Specifying a default handling option

For string and file redaction, you can specify a default handling option to use for entity types that are not specified in generator_config.

To do this, you use the generator_default parameter.

generator_default can be either Redaction, Synthesis, GroupingSynthesis, ReplacementSynthesis, or Off.

Providing added and excluded values for entity types

You can also configure added and excluded values for each entity type.

You add values that Textual does not detect for an entity type, but should. You exclude values that you do not want Textual to identify as that entity type.

To specify the added values, use label_allow_lists.
To specify the excluded values, use label_block_lists.

For each of these parameters, the value is a list of entity types to specify the added or excluded values for. To specify the values, you provide an array of regular expressions.

The following example uses label_allow_lists to add values:

For NAME_GIVEN, adds the values There and Here.
For NAME_FAMILY, adds values that match the regular expression ([a-z]{2}).

Including custom entity types

When you redact a string or download a redacted file, you can provide a comma-separated list of custom entity types to include. Textual then scans for and redacts those entity types based on the configuration in generator_config.

For example:

Textual REST API

About the Textual REST API

The Tonic Textual REST API allows you to more deeply integrate Textual functions into your existing workflows.

You can use the REST API as another tool alongside the Textual application and the Textual Python SDK. The Python SDK supports the same actions as the REST API. We recommend the Python SDK for customers who already use Python.

You can download the Textual OpenAPI specification from:

REST API authentication

Before you can use the API, you must create a Tonic Textual API key. For information on how to obtain a Textual API key, go to Creating and revoking Textual API keys.

When you call the API, you place your API key in the authorization header of the request, similar to the following curl request, which fetches the list of datasets for the current user.

curl --request GET \
--url "https://textual.tonic.ai/api/dataset" \
--header "Content-Type: application/json" \
--header "Authorization: API_KEY"

Most Textual API requests require authentication. For each request, the reference information indicates whether the request requires an API key.

For requests that require an API key, if you do not provide a valid API key, you receive a 401 Unauthorized response.

Redaction

Use the Tonic Textual REST API to redact text. Redaction means to detect and replace sensitive values.

Redact text strings

You can use the Tonic Textual REST API to redact text strings, including:

Plain text
JSON
XML
HTML

Textual provides a specific endpoint for each format. For JSON, XML, and HTML, Textual only redacts the text values. It preserves the underlying structure.

Datasets

Use the REST API to manage datasets.

Manage datasets

Use the REST API to create and manage datasets.

Entity linking

Use the REST API to link entity values.

Access management

Use the API to retrieve information about users and groups, and to manage access to datasets.

Install and administer Textual

Textual architecture

The following diagram shows how data and requests flow within the Tonic Textual application:

Textual application database

The Textual application database is a PostgreSQL database that stores the dataset configuration.

If you do not configure an S3 bucket, then it also stores uploaded files and files that you use the SDK to redact.

Textual datastore in Amazon S3

You can configure an S3 bucket to store uploaded files and individual files that you use the SDK to redact. For more information, go to .

If you do not configure an S3 bucket, then the files are stored in the Textual application database.

Textual components

Textual web server

Runs the Textual user interface.

Textual worker

A textual instance can have multiple workers.

The worker orchestrates jobs. A job is a longer running task such as the redaction of a single file.

If you redact a large number of files, you might deploy additional workers and machine learning containers to increase the number of files that you can process concurrently.

Textual machine learning

A textual installation can have 1 or more machine learning containers.

The machine learning container hosts the Textual models. It takes text from the worker or web server and returns any entities that it discovers.

Additional machine learning containers can increase the number of words per second that Textual can process.

OCR service

The OCR service converts PDFs and images to text that Textual can then scan for sensitive values.

For more information, go to .

LLM service

Textual only uses the LLM service for .

Setting up and managing a Textual Cloud pay-as-you-go subscription

The Tonic Textual pay-as-you-go plan allows you to automatically bill a credit card for your Textual usage.

The Textual subscription plan charges a flat rate for each 1000 words. You are billed each month based on when you started your subscription. For example, if you start your subscription on the 12th of the month, then you are billed every month on the 12th.

Tonic.ai integrates with a payment processing solution to manage the payments.

Setting up the subscription

To start a new subscription, from a usage pane or upgrade prompt, click Upgrade Plan.

You are sent to the payment processing solution to enter your payment information.

Tracking your usage

The panel on the Home page shows the usage for the current month.

To view additional usage details, click Manage Plan.

The Manage Plan page displays the details for your subscription.

Account summary

The summary at the top left contains an overview of the subscription payment information, as well as the total number of words scanned since you started your account.

From the summary, you can go to the payment processing solution to view and manage payment information.

30-day usage graph

The graph at the top of the page shows the words scanned per day for the previous 30 days.

Current billing period

The Current Billing Period panel summarizes your usage for the current month, and provides information about the next payment.

Next billing date

The Next billing date panel shows when the next billing period begins.

Payment history

The Payment History section shows the list of subscription payments.

For each payment, the list shows the date and amount, and whether the payment was successful.

To download the invoice for a payment, click its Invoice option.

Updating the payment information

You can update the payment information for your subscription. For example, you might need to choose a different credit card or update an expiration date.

To manage the payment information:

On the home page, in the usage panel, click Manage Plan.
On the Manage Plan page, from the account summary, click Manage Payment.

You are sent to the payment processing solution to update your payment information.

Canceling a subscription

To cancel a subscription, from the Manage Plan page:

Click Manage Payment.
In the payment processing solution, select the cancellation option.

The cancellation takes effect at the end of the current subscription month.

Structure of JSON output files

The JSON output provides access to Markdown content and identifies the entities that were detected in the file.

Common elements in the JSON output

Information about the entire file

All JSON output files contain the following elements that contain information for the entire file:

{
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

fileType

The type of the original file.

content

Details about the file content. It includes:

Hashed and Markdown content for the file
Entities in the file

schemaVersion

An integer that identifies the version of the JSON schema that was used for the JSON output.

Textual uses this to convert content from older schemas to the most recent schema.

For specific file types, the JSON output includes additional objects and properties to reflect the file structure.

Hashed and Markdown content

The JSON output contains hashed and Markdown content for the entire file and for individual file components.

hash

The hashed version of the file or component content.

text

The file or component content in Markdown notation.

Entities

The JSON output contains entities arrays for the entire file and for individual file components.

Each entity in the entities array has the following properties:

start

Within the file or component, the location where the entity value starts.

For example, in the following text:

My name is John.

John is an entity that starts at 11.

end

Within the file or component, the location where the entity value ends.

For example, in the following text:

My name is John.

John is an entity that ends at 14.

label

The type of entity.

For a list of the entity types that Textual detects, go to .

text

The text of the entity.

score

The confidence score for the entity.

Indicates how confident Textual is that the value is an entity of the specified type.

language

The language code to identify the language for the entity value. For example, en indicates that the value is in English.

Plain text files

For plain text files, the JSON output only contains the information for the entire file.

{
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown content>",
    "hash": "<hashed content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.csv files

For .csv files, the structure contains a tables array.

The tables array contains a table object that contains header and data arrays..

For each row in the file, the data array contains a row array.

For each value in a row, the row array contains a value object.

The value object contains the entities, hashed content, and Markdown content for the value.

{
  "tables": [
    {
      "tableName": "csv_table",
      "header": [//Columns that contain heading info (col_0, col_1, and so on)
        "<column identifier>"
      ],
      "data": [  //Entry for each row in the file
        [   //Entry for each value in the row
          {    
            "entities": [   //Entry for each entity in the value
              {
                "start": <start location>,,
                "end": <end location>,
                "label": "<value type>",
                "text": "<value text>",
                "score": <confidence score>,
                "language": "<language code>"
              }
            ],
            "hash": "<hashed value content>",
            "text": "<Markdown value content>"
          }
        ]
      ]
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   ///Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.xlsx files

For .xlsx files, the structure contains a tables array that provides details for each worksheet in the file.

For each worksheet, the tables array contains a worksheet object.

For each row in a worksheet, the worksheet object contains a header array and a data array. The data array contains a row array.

For each cell in a row, the row array contains a cell object.

Each cell object contains the entities, hashed content, and Markdown content for the cell.

{
  "tables": [   //Entry for each worksheet
    {
      "tableName": "<Name of the worksheet>",
      "header": [ //Columns that contain heading info (col_0, col_1, and so on)
        "<column identifier>"
      ],
      "data": [   //Entry for each row
        [   //Entry for each cell in the row
          {
            "entities": [   //Entry for each entity in the cell
              {
                "start": <start location>,
                "end": <end location>,
                "label": "<value type>",
                "text": "<value text>",
                "score": <confidence score>,
                "language": "<language code>"
              }
            ],
            "hash": "<hashed cell content>",
            "text": "<Markdown cell content>"
          }
        ]
      ]
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.docx files

For .docx files, the JSON output structure adds:

A footnotes array for content in footnotes.
An endnotes array for content in endnotes.
A header object for content in the page headers. Includes separate objects for the first page header, even page header, and odd page header.
A footer object for content in the page footers. Includes separate objects for the first page footer, even page footer, and odd page footer.

These arrays and objects contain the entities, hashed content, and Markdown content for the notes, headers, and footers.

{
  "footNotes": [   //Entry for each footnote
    {
      "entities": [   //Entry for each entity in the footnote
        {
          "start": <start location>,
          "end": <end location>,
          "pythonStart": <start location in Python>,
          "pythonEnd": <end location in Python>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
          "exampleRedaction": null
        }
      ],
      "hash": "<hashed footnote content>",
      "text": "<Markdown footnote content>"
    }
  ],
  "endNotes": [   //Entry for each endnote
    {
      "entities": [   //Entry for each entity in the endnote
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed endnote content>",
      "text": "<Markdown endnote content>"
    }
  ],
  "header": {
    "first": {
      "entities": [   //Entry for each entity in the first page header
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed first page header content>",
      "text": "<Markdown first page header content>"
    },
    "even": {
      "entities": [   //Entry for each entity in the even page header
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed even page header content>",
      "text": "<Markdown even page header content>"
    },
    "odd": {
      "entities": [   //Entry for each entity in the odd page header
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed odd page header content>",
      "text": "<Markdown odd page header content>"
    }
  },
  "footer": {
    "first": {
      "entities": [   //Entry for each entity in the first page footer
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed first page footer content>",
      "text": "<Markdown first page footer content>"
    },
    "even": {
      "entities": [   //Entry for each entity in the even page footer
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed even page footer content>",
      "text": "<Markdown even page footer content>"
    },
    "odd": {
      "entities": [   //Entry for each entity in the odd page footer
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed odd page footer content>",
      "text": "<Markdown odd page footer content>"
    }
  },
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

PDF and image files

PDF and image files use the same structure. Textual extracts and scans the text from the files.

For PDF and image files, the JSON output structure adds the following content.

`pages` array

The pages array contains all of the content on the pages. This includes content in tables and key-value pairs, which are also listed separately in the output.

For each page in the file, the pages array contains a page array.

For each component on the page - such as paragraphs, headings, headers, and footers - the page array contains a component object.

Each component object contains the component entities, hashed content, and Markdown content.

`tables` array

The tables array contains content that is in tables.

For each table in the file, the tables array contains a table array.

For each row in a table, the table array contains a row array.

For each cell in a row, the row array contains a cell object.

Each cell object identifies the type of cell (header or content). It also contains the entities, hashed content, and Markdown content for the cell.

`keyValuePairs` array

The keyValuePairs array contains key-value pair content. For example, for a PDF of a form with fields, a key-value pair might represent a field label and a field value.

For each key-value pair, the keyValuePairs array contains a key-value pair object.

The key-value pair object contains:

An automatically incremented identifier. For example, id for the first key-value pair is 1, for the second key-value pair is 2, and so on.
The start and end position of the key-value pair
The text of the key
The entities, hashed content, and Markdown content for the value

PDF and image JSON outline

{
  "pages": [   //Entry for each page in the file
    [   //Entry for each component on the page
      {
        "type": "<page component type>",
        "content": {
          "entities": [   //Entry for each entity in the component
            {
              "start": <start location>,
              "end": <end location>,
              "label": "<value type>",
              "text": "<value text>",
              "score": <confidence score>,
              "language": "<language code>"
            }
          ],
          "hash": "<hashed component content>",
          "text": "<Markdown component content>"
        }
      }
    ],
  "tables": [   //Entry for each table in the file
    [   //Entry for each row in the table
      [   //Entry for each cell in the row
        {
          "type": "<content type>",   //ColumnHeader or Content
          "content": {
            "entities": [  //Entry for each entity in the cell
              {
                "start": <start location>,
                "end": <end location>,
                "label": "<value type>",
                "text": "<value text>",
                "score": <confidence score>,
                "language": "<language code>"
              }
            ],
            "hash": "<hashed cell text>",
            "text": "<Markdown cell text>"
          }
        }
      ]
    ]
  ],
  "keyValuePairs": [   //Entry for each key-value pair in the file
    {
      "id": <incremented identifier>,
      "key": "<key text>",
      "value": {
        "entities": [  //Entry for each entity in the value
          {
            "start": <start location>,
            "end": <end location>,
            "label": "<value type>",
            "text": "<value text>",
            "score": <confidence score>,
            "language": "<language code>"
          }
        ],
        "hash": "<hashed value text>",
        "text": "<Markdown value text>"
      },
      "start": <start location of the key-value pair>,
      "end": <end location of the key-value pair>
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   ///Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.eml and .msg files

For email message files, the JSON output structure adds the following content.

Email message identifiers

The JSON output includes the following email message identifiers:

The identifier of the current message
If the message was a reply to another message, the identifier of that message
An array of related email messages. This includes the email message that the message replied to, as well as any other messages in an email message thread.

Recipients

The JSON output includes the email address and display name of the message recipients. It contains separate lists for the following:

Recipients in the To line
Recipients in the CC line
Recipients in the BCC line

Subject line

The subject object contains the message subject line. It includes:

Markdown and hashed versions of the message subject line.
The entities that were detected in the subject line.

Message timestamp

sentDate provides the timestamp when the message was sent.

Message body

The plainTextBodyContent object contains the body of the email message.

It contains:

Markdown and hashed versions of the message body.
The entities that were detected in the message body.

Message attachments

The attachments array provides information about any attachments to the email message. For each attached file, it includes:

The identifier of the message that the file is attached to.
The identifier of the attachment.
The JSON output for the file.
The count of words in the original file.
The count of words in the redacted version of the file.

Email message JSON outline

{
  "messageId": "<email message identifier>",
  "inReplyToMessageId": <message that this message replied to>,
  "messageIdReferences": [<related email messages>],
  "senderAddress": {
    "address": "<sender email address>",
    "displayName": "<sender display name>"
  },
  "toAddresses": [  //Entry for each recipient in the To list
    {
      "address": "<recipient email address>",
      "displayName": "<recipient display name>"
    }
  ],
  "ccAddresses": [ //Entry for each recipient in the CC list
    {
      "address": "<recipient email address>",
      "displayName": "<recipient display name>"
    }
  ],
  "bccAddresses": [ //Entry for each recipient in the BCC list
    {
      "address": "<recipient email address>",
      "displayName": "<recipient display name>"
    }
  ],
  "sentDate": "<timestamp when the message was sent>",
  "subject": {
    "text": "<Markdown version of the subject line>",
    "hash": "<hashed version of the subject line>",
    "entities": [   //Entry for each entity in the subject line
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "plainTextBodyContent": {
    "text": "<Markdown version of the message body>",
    "hash": "<hashed version of the message body>",
    "entities": [ //Entry for each entity in the message body
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "attachments": [ //Entry for each attached file
    {
      "parentMessageId": "<the message that the file is attached to>",
      "contentId": "<identifier of the attachment>",
      "fileName": "<name of the attachment file>",
      "document": {<JSON for the attached file>},
      "wordCount": <number of words in the attachment>,
      "redactedWordCount": <number of words in the redacted attachment>
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [ //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

Tonic Textual

Tonic Textual guide

Startup and overview

Textual SDK, REST API, and other integrations

Getting started with Textual

Signing up for Textual

Using the Textual free trial

Using the Getting Started checklist

Word count limit

Viewing your current usage

Next steps - pay-as-you-go or product demo

Managing your user profile

Displaying the User Profile page

Copying your organization identifier

Adding a team name

Entity types

About entity types

Viewing the list of custom entity types

Information in the list

Filtering the list

Filtering by name

Filtering by type

Filtering by creator

Sorting the list

Managing model-based custom entity types

Getting started

Defining the entity type

Activating and managing an entity type

Overview of the process to create a model-based custom entity type

Select and annotate test files

Iterate over model guidelines

Select training data

Train models

Select a model to use

Starting a new model-based custom entity type

Iterating over the guidelines to use for model training

Viewing the initial version of the guidelines

File statuses for the guidelines refinement

Reviewing the test scores for the guidelines

Reviewing the guideline detections

Editing the guidelines

Selecting the training data for your models

About training data

Uploading files from a local file system

Selecting files from cloud storage

Displaying the content of a training data file

Training data file statuses

Creating and training models for a model-based entity type

Information on the model list

Starting a new model

Reviewing the annotations for a model

Training the model

Downloading a data package for a model

Selecting the active model for the entity type

Renaming or deleting a model-based custom entity type

Renaming the entity type

Deleting the entity type

Create and manage datasets

Viewing the dataset list and details

Viewing the list of datasets

Displaying the Datasets page

Filtering the datasets by name

Filtering the datasets by tag

Displaying details for a dataset

Creating a dataset

Setting the name, source type, and output type

Providing credentials for Amazon S3

Providing Azure credentials

Providing SharePoint credentials

Deleting datasets

Assigning tags to datasets

Managing tags from the Datasets page

Managing tags from the dataset details page

Changing the dataset name

Changing cloud storage credentials and output location

Changing cloud storage credentials

Amazon S3

Azure

SharePoint

Setting the output location