1 of 100

Tonic Textual

Tonic Textual guide

Tonic Textual allows you to put your text-based data to work for you.

A textual dataset is a collection of files from a local file system or cloud storage.

Textual scans the data files to identify sensitive values. It redacts or replaces those sensitive values, to produce output files in the same format that you can safely use for development and training.

You can also instead use a dataset to prepare unstructured text for use in an LLM system. Textual can produce a JSON summary of the detected values, which includes Markdown-formatted output.

You can use the Textual SDK or the Textual REST API to manage datasets or to remove sensitive values from individual text strings and files.

Startup and overview

Textual SDK, REST API, and Snowflake Native App

Need help with Textual? Contact [email protected].

Getting started with Textual

When you sign up for a Tonic Textual account, you can immediately get started with a new pipeline.

Note that these instructions are for setting up a new account on Textual Cloud. For a self-hosted instance, depending on how it is set up, you might either create an account manually or use single sign-on (SSO).

To get started with a new Textual account:

Go to https://textual.tonic.ai/.
Click Sign up.
Enter your email address.
Create and confirm a password for your Textual account.
Click Sign Up.

Textual creates your account. After you log in, Textual prompts you to provide some additional information about yourself and how you plan to use Textual.

After you fill out the information and click Get Started, Textual displays the Textual Home page, which you can use to preview how Textual detects and replaces values. For more information, go to Previewing Textual detection and redaction.

Using the Textual free trial

When you set up an account on Textual Cloud, you start a Textual free trial.

Using the Getting Started checklist

When you start a free trial, Textual provides a checklist to guide you through initial steps to get started and learn more about Textual and what it can do.

The checklist displays automatically when you first log in. You can close and display it as needed. To display the checklist, in the Textual navigation menu, click Getting Started.

As you complete a step, Textual automatically marks it as completed.

The checklist includes:

Using the Home page to preview Textual redaction. When you click the step, you navigate to the Home page. Textual displays a popup panel that describes the task.
Installing the Textual SDK. The checklist displays the installation command and an option to copy it. The step is marked as complete when you click the copy icon.
Creating an API key. When you click the step, you are prompted to create an API key.
Creating an SDK request to redact a text string or a file. When you click the step, you navigate to the Request Explorer. The step is marked as completed when you close the popup panel that describes the task.

Word count limit

During the free trial Textual scans up to 100,000 words for free. Note that Textual counts actual words, not tokens. For example, "Hello, my name is John Smith." counts as six words.

After the 100,000 words, Textual disables scanning for your account. Until you purchase a pay-as-you-go subscription, you cannot:

Add files to a dataset or pipeline
Run a pipeline

Viewing your current usage

During your free trial, Textual displays the current usage in the following locations:

On the Home page
In the navigation menu

Next steps - pay-as-you-go or product demo

Textual also prompts you to purchase a pay-as-you-go subscription, which allows an unlimited number of words scanned for a flat rate per 1,000 words.

You can also request a Textual product demo.

Entity types that Textual detects

Tonic Textual comes with a built-in set of entity types that it detects. You can also configure custom entity types, which detect values based on regular expressions.

You can also view this video overview of entity types and entity type handling.

Creating and managing custom entity types

Required global permission - either:

Create custom entity types
Edit any custom entity type

In addition to the built-in entity types, you can also create custom entity types.

Custom entity types are based on regular expressions. If a value matches a configured regular expression for the custom entity type, then it is identified as that entity type.

You can control whether each dataset uses each custom entity type.

Viewing the list of custom entity types

To display the list of entity types, in the Textual navigation bar, click Custom Entity Types.

For each custom entity type, the list includes:

Entity type name and description.
Regular expressions to identify matching values.
The number of datasets that the entity type is active for.

Creating, editing, and deleting a custom entity type

Creating a custom entity type

Required global permission: Create custom entity types

To create a custom entity type, on the Custom Entity Types page, click Create Custom Entity Type.

The dataset details page also contains a Create Custom Entity Type option.

After you :

To save the new type, but not scan dataset files for the new type, click Save Without Scanning Files.
To both save the new type and scan for it, click Save and Scan Files.

To detect new custom entity types in a dataset, Textual needs to run a scan. If you do not run the scan when you save the custom entity type, then on the dataset details page, you are prompted to run a scan.

Editing a custom entity type

Required global permission: You can edit any custom entity type that you create.

Users with the global permission Edit any custom entity type can edit any custom entity type.

To edit a custom entity type, on the Custom Entity Types page, click the edit icon for the entity type.

You can also edit a custom entity type from the dataset details page.

For an existing entity type, you can change the description, the regular expressions, and the enabled datasets.

You cannot change the entity type name, which is used to produce the identifier to use to configure the entity type handling from the SDK.

After you update the configuration:

To save the changes, but not scan dataset files based on the updated configuration, click Save Without Scanning Files.
To both save the new type and scan based on the updated configuration, click Save and Scan Files.

To reflect the changes to custom entity types in a dataset, Textual needs to run a scan. If you do not run the scan when you save the changes, then on the dataset details page, you are prompted to run a scan.

Deleting a custom entity type

When you delete a custom entity type, it is removed from the datasets that it was active for.

To delete a custom entity type:

On the Custom Entity Types page, click the delete icon for the entity type.
On the confirmation panel, click Delete Entity Type.

Custom entity type configuration settings

The custom entity type configuration includes:

Name and description
Regular expressions to identify matching values. From the configuration panel, you can test the expressions against text that you provide.
Datasets to make the entity type active for. You can also enable and disable custom entity types from the dataset details pages.

Name and description

In the Name field, provide a name for the entity type. Each custom entity type name:

Must be unique within an organization.
Can only contain alphanumeric characters and spaces. Custom entity type names cannot contain punctuation or other special characters.

After you save the entity type, you cannot change the name. Textual uses the name as the basis for the identifier that you use to refer to the entity type in the SDK.

In the Description field, provide a longer description of the custom entity type.

Regular expressions to identify matching values

Under Keywords, Phrases, or Regexes, provide expressions to identify matching values for the entity type.

An entry can be as simple as a single word or phrase, or you can provide a more complex regular expression to identify the values.

Textual maintains an empty row at the bottom of the list. When you type an expression into the last row, Textual adds a new empty row.

To add an entry, begin to type the value in the empty row.

To edit an entry, click the entry field, then edit the value.

To remove an entry, click its delete icon.

Testing an expression

Under Test Entry, you can check whether Textual correctly identifies a value as the entity type based on the provided expression.

To test an expression:

From the dropdown list, select the entry to test.

In the text area, provide the text to test.

As you enter the text, Textual automatically scans the text for matches to the selected expression. The Result field displays the input text and highlights the matching values.

Enabling and disabling the entity type for datasets

Under Activate custom entity, you identify the datasets to make the entity active for. From the dataset details, you can also enable and disable custom entity types for that dataset.

To make the entity active for all current and future datasets, check Automatically activate for all current, and new pipelines and datasets.

To make the entity active for specific datasets, set the toggle for the dataset to the on position.

To filter the list based on the dataset name, in the filter field, begin to type text from the name. Textual updates the list to only include matching datasets.

To update all of the currently displayed datasets, click Bulk action, then click Enable or Disable.

You can also enable and disable custom entity types from within a dataset. For more information, go to .

Create and manage datasets

Viewing the dataset list and details

Viewing the list of datasets

Displaying the Datasets page

To display the Datasets page, in the navigation menu, click Datasets.

The datasets list only displays the datasets that you have access to.

Users who have the global permission View all datasets can see the complete list of datasets.

For each dataset, the Datasets page includes:

The name of the dataset
Any tags assigned to the dataset. For datasets that you can edit, there is also an option to assign tags. For more information, go to Assigning tags to datasets.
The user who most recently updated the dataset
When the dataset was created

Filtering the datasets by name

To filter the datasets by name, in the search field, begin to type text that is in the dataset name.

As you type, the list is filtered to only include datasets with names that contain the filter text.

Filtering the datasets by tag

You can assign tags to each dataset. Tags can help you to organize and provide a quick glance into the dataset configuration.

On the Datasets page, to filter the datasets by their assigned tags:

In the heading for the Tags column, click the filter icon.
On the tag list, check the checkbox for each tag to include.

To find a specific tag, in the search field, type the tag name.

Displaying details for a dataset

Required dataset permission: View dataset settings

To display the details page for a dataset, on the Datasets page, click the dataset name.

The dataset details page includes:

The tags assigned to the dataset, as well as an option to add tags. For more information, go to Assigning tags to datasets.
The list of files in the dataset. For a cloud storage dataset, where the files can be located across multiple folders, Textual navigates to the first folder that contains selected dataset files.
The results of the scan for entity values
The configured handling for each entity type

Creating a dataset

Required global permission: Create datasets

When you create a dataset, you specify:

The type of output to produce
The source location for the files.
If the files are in cloud storage, the connection credentials.

Setting the name, source type, and output type

To create a dataset:

On the Datasets page, click Create a Dataset.

In the Dataset Name field, provide a name for the dataset.
Under Output Format, select the type of output to generate.
Under File Source, select the source type. If the source type is a cloud storage option, then provide the required credentials.
Click Save.
For cloud storage datasets:
1. Textual prompts you to configure the initial file selection. For more information, go to .
2. After you select the files, it prompts you to select an output location. For more information, go to .

Providing credentials for Amazon S3

If the source type is Amazon S3, provide the credentials to use to connect to Amazon S3.

For a self-hosted instance, select the location of the credentials. You can either provide credentials manually, or use credentials that are configured in environment variables. Note that after you save the dataset, you cannot change the selection.
If you are not using environment variables, then in the Access Key field, provide an AWS access key that is associated with an IAM user or role. For an example of a role that has the required permissions for an Amazon S3 dataset, go to .
In the Access Secret field, provide the secret key that is associated with the access key.
From the Region dropdown list, select the AWS Region to send the authentication request to.
In the Session Token field, provide the session token to use for the authentication request.
To test the credentials, click Test AWS Connection.
By default, connections to Amazon S3 use Amazon S3 encryption. To instead use AWS KMS encryption:
1. Click Show Advanced Options.
2. From the Server-Side Encryption Type dropdown list, select AWS KMS.
3. In the Server-side Encryption AWS KMS ID field, provide the KMS key ID. Note that if the KMS key doesn't exist in the same account that issues the command, you must provide the full key ARN instead of the key ID.
Note that after you save the new dataset, you cannot change the encryption type.
Click Save. Textual prompts you to .

Providing Azure credentials

If the source type is Azure, provide the connection information:

In the Account Name field, provide the name of your Azure account.
In the Account Key field, provide the access key for your Azure account.
To test the connection, click Test Azure Connection.
Click Save. Textual prompts you to .

Providing SharePoint credentials

If the source type is SharePoint, provide the credentials for the Entra ID application.

The credentials must have the following application permissions (not delegated permissions):

Files.Read.All - To see the SharePoint files
Files.ReadWrite.All -To write redacted files and metadata back to SharePoint
Sites.ReadWrite.All - To view and modify the SharePoint sites

To provide the credentials:

In the Tenant ID field, provide the SharePoint tenant identifier for the SharePoint site.
In the Client ID field, provide the client identifier for the SharePoint site.
In the Client Secret field, provide the secret to use to connect to the SharePoint site.
To test the connection, click Test SharePoint Connection.
Click Save. Textual prompts you to .

Deleting datasets

Required dataset permission: Delete a dataset

To delete a dataset:

On the dataset details page, click Settings.
On the Dataset Settings page, click Delete Dataset.
Click Confirm Delete.

Assigning tags to datasets

Required dataset permission: Edit dataset settings

Tags can help you to organize your datasets. For example, you can use tags to indicate datasets that belong to different groups, or that deal with specific areas of your data.

You can manage tags from both the Datasets page and the dataset details.

Managing tags from the Datasets page

On the Datasets page, the Tags column displays the currently assigned tags.

To change the tag assignment for a dataset:

Click Tags.
On the dataset tags panel, to add a new tag, type the tag text, then press Enter.
To remove a tag, click its delete icon.
To remove all of the tags, click the delete all icon.

Managing tags from the dataset details page

On the dataset details page, the assigned tags display under the dataset name.

To change the tag assignment:

Click Tags.
On the dataset tags panel, to add a new tag, type the tag text, then press Enter.
To remove a tag, click its delete icon.
To remove all of the tags, click the delete all icon.

Changing the dataset name

Required dataset permission: Edit dataset settings

The dataset name displays in the panel at the top left of the dataset details page.

To change the dataset name:

On the dataset details page, click Settings.
On the Dataset Settings page, in the Dataset Name field, provide the new name for the dataset..

Click Save Dataset.

Changing cloud storage credentials and output location

Required dataset permission: Edit dataset settings

For a cloud storage dataset, you can:

Update the cloud storage credentials. Note that this option is only available if you provided the credentials manually. If you use the credentials set in environment variables, then you cannot change the credentials.
Change the output location for the generated output files

You configure the connection credentials and output location in the Connection Settings section of the Dataset Settings page.

To display the Dataset Settings page, on the dataset details page, click Settings.

After you update the configuration, click Save Dataset.

Changing cloud storage credentials

Amazon S3

To provide updated credentials for Amazon S3:

In the Access Key field, provide an AWS access key that is associated with an IAM user or role. For an example of a role that has the required permissions for an Amazon S3 dataset, go to .
In the Access Secret field, provide the secret key that is associated with the access key.
From the Region dropdown list, select the AWS Region to send the authentication request to.
In the Session Token field, provide the session token to use for the authentication request.
To test the credentials, click Test AWS Connection.

Azure

To provide updated credentials for Azure:

In the Account Name field, provide the name of your Azure account.
In the Account Key field, provide the access key for your Azure account.
To test the connection, click Test Azure Connection.

SharePoint

SharePoint credentials must have the following application permissions (not delegated permissions):

Files.Read.All - To see the SharePoint files
Files.ReadWrite.All -To write redacted files and metadata back to SharePoint
Sites.ReadWrite.All - To view and modify the SharePoint sites

To provide updated credentials for SharePoint:

In the Tenant ID field, provide the SharePoint tenant identifier for the SharePoint site.
In the Client ID field, provide the client identifier for the SharePoint site.
In the Client Secret field, provide the secret to use to connect to the SharePoint site.
To test the connection, click Test SharePoint Connection.

Setting the output location

The output location is where Textual writes the redacted files.

When you create a cloud storage database, after you select the initial set of files and folders, Textual prompts you to select the output location.

For an existing dataset, you set the output location from the Dataset Settings page.

Under Select Output Location, select the cloud storage folder where Textual writes the output files for the dataset.

When you generate output for a cloud storage dataset, Textual creates a folder in the output location. The folder name is the identifier of the job that generated the files.

Within the job folder, Textual recreates the folder structure for the original files.

Textual then writes the output files to the corresponding folders.

Sharing dataset access

Required permissions:

Global permission - View users and groups
Either:
- Global permission - Manage access to datasets
- Dataset permission - Share dataset access

Tonic Textual uses dataset permission sets for role-based access (RBAC) of each dataset.

A dataset permission set is a set of dataset permissions. Each permission provides access to a specific dataset feature or function.

Textual provides built-in dataset permission sets. Organizations can also configure custom permission sets.

To share dataset access, you assign dataset permission sets to users and to SSO groups, if you use SSO to manage Textual users. Before you assign a dataset permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group.

To change the current access to the dataset:

Either:
- On the Datasets page, click the share icon for the dataset to share.
- On the dataset details page, click Share.

The dataset access panel contains the current list of users and groups who have access to the dataset, and displays their assigned dataset permission sets. To add a user or group to the list of users and groups:
1. In the search field, begin to type the user email address or group name.
2. From the list of matching users or groups, select the user or group to add.
For a user or group, to change the assigned dataset permission sets:
1. Click Access. The dropdown list displays the list of custom and built-in dataset permission sets.
2. Under Custom Permission Sets, check the checkbox next to each dataset permission set to assign to the user or group. To remove an assigned dataset permission set, uncheck the checkbox.
3. Under Built-In Permission Sets, click the dataset permission set to assign to the user or group. You can only assign one built-in permission set. By default, for an added user or group, the Viewer permission set is selected. To not grant any built-in permission set, select None.

Manage dataset files

Supported file types

Tonic Textual can process the following types of files:

txt
csv
tsv
docx
xlsx
pdf
png
tif or tiff
jpg or jpeg

Uploading and deleting local files

For a local file dataset, you upload and remove new files directly.

On Tonic Textual Cloud, and by default for self-hosted instances, Textual stores the uploaded files in the application database.

On a self-hosted instance, you can instead configure an S3 bucket where Textual stores the files. In the S3 bucket, the files are stored in a folder that is named for the dataset identifier.

For more information, go to .

For an example of an IAM role with the required permissions, go to .

Adding files to the dataset

Required dataset permission: Upload files to a dataset

From the dataset details page, to add files to the dataset:

In the panel at the top left, click Upload Files.

Search for and select the files.

Textual uploads and then processes the files. For more information about file processing, go to .

Do not leave the page while files are uploading. If you leave the page before the upload is complete, then the upload stops.

You can leave the page while Textual is processing the file.

On a self-hosted instance, when a file fails to upload, you can download the associated logs. To download the logs, click the options menu for the file, then select Download Logs.

Removing files from the dataset

Required dataset permission: Delete files from a dataset

To remove a file from the dataset:

In the file list, click the options menu for the file.
In the options menu, click Delete File.

Selecting cloud storage files

Required dataset permission: Edit dataset settings

For a cloud storage dataset, you manage files from the file selection panel.

When you create a dataset, after you provide the cloud storage credentials and save the dataset, Textual immediately prompts you to select dataset files. After you select the files and click Next, Textual prompts you to set the output location.

For an existing dataset, to display the File Selection panel, on the dataset details page, click Select Files.

The file selection includes:

Whether to restrict the dataset to specific file types
The files or folders to include in the dataset

When you change the file selection, Textual scans the files for entities. For more information, go to .

Filtering files by file extension

When you select files, you can filter the selectable files based on file extension.

To limit the file extensions to include:

Click File Extension Filter. By default, all file extensions are included, and none of the checkboxes are checked.
Check the checkbox for each file extension to include. As you select the file extensions to include, Textual updates the navigation pane so that you can only select files that have one of those file extensions. It hides files that have other file extensions and folders that do not contain files with the selected file extensions.

Selecting files and folders to include

In the file selection area, you navigate to and select the folders and files to add to the dataset.

Navigating through the folders

In the navigation area, to display the contents of a folder, click the Open link for the folder.

Selecting a file or folder

To add a folder or file to the dataset, check its checkbox.

Managing selected folders

In the navigation pane, when you check a folder checkbox, Textual adds it to the Prefix Patterns list.

Adding a folder manually

Instead of navigating to a folder and selecting it, you can add the path to the list manually

To add a folder path:

Click Add Prefix Pattern.
In the field, type the path to the folder, then click the save icon.

Removing folder paths

To remove a folder path from the dataset, either:

In the navigation pane, uncheck its checkbox.
In the Prefix Patterns list, click its delete icon.

For the selected folders, the dataset includes all of the applicable files in the folder that:

Are of a file type that Textual supports
Match the file extension filter

Managing selected files

In the navigation pane, when you select an individual file, Textual adds it to the Selected Files list.

To delete a file, either:

In the navigation pane, uncheck its checkbox.
In the Selected Files list, click its delete icon.

Navigating the file list

For a local files dataset, the file list is a single list of uploaded files.

For cloud storage dataset, the file list initially displays the first folder that contains dataset files. You can then navigate through the folders and files.

For a cloud storage dataset, you can search for folders in the currently displayed bucket or folder.

To search for a folder, in the search field, start to type the folder name.

Tracking and managing file processing

When you add files to a local files dataset, or change the file selection for a cloud storage dataset, Tonic Textual automatically scans the files to identify the entities that they contain.

When you change the dataset configuration, Textual also prompts you to run a new scan. For example, a new scan is required when you:

Configure added and excluded values
Change the available custom entity types

The file list reflects the current scanning status for the file. A file is initially queued for scanning. When the scan starts, the status changes to scanning. When Textual finishes processing a file, it marks the file as scanned.

As Textual processes each file, it updates the results in the dataset details heading and the entity types list.

Pausing the file processing

If needed, you can pause the file processing. To pause the processing, click Pause.

The information in the heading and entity types list only reflect the files that are scanned.

For a cloud storage dataset, when you generate output, Textual only includes files that are scanned.

Starting a scan on a paused file

After you pause the scan, you can start a scan on individual files.

To start a scan on a file:

Click the options menu for the file.
Click Scan.

Configure the redaction

Reviewing the sensitivity detection results

Required dataset permission: View dataset settings

When you first create a dataset, Tonic Textual displays a single list of all of the entity types that it can detect.

As you add and remove files, Textual updates the entity types list to indicate the detected and not detected entity types.

Viewing the number of detected values

At the top of the dataset details view, the Sensitive words tile shows the total number of sensitive values in the dataset that Textual detected.

Viewing the detected entity types

As Textual processes files, it identifies the entity types that are detected and not detected.

The entity type list starts with the detected entity types. For each detected entity type, Textual displays:

The number of detected values that are marked as this type in the output file. Excluded values are not included in the count.
The selected handling option.
Whether there are configured added or excluded values.

Previewing the detected values for an entity type

For each detected entity type, to view a sample of up to 10 of the detected values , click the view icon next to the value count.

Displaying the list of detected values for an entity type

The entities list contains the full list of detected values for an entity type.

To display the entities list, from the value preview, click Open Entities Manager.

Selecting the entity type

When you display the entities list, the entity type that you previewed the values for is selected by default.

To change the selected entity type, from the dropdown at the top left, select the entity type to view values for.

How Textual handles entity values that match multiple types

A detected value might match multiple entity types.

For example, a telephone number might match both the Phone Number and Numeric Value entity types.

Every value is only counted once, for the entity type that it is assigned in the output file.

By default, a detected value is assigned the entity type that it most closely matches. For our example, the telephone number value most closely matches the Phone Number entity type, and so by default is included in the Phone Number count and values list.

If the entity type is turned off, or the value is excluded, then Textual moves the value to the next matching type.

In our example, if you set the handling type for Phone Number to Off, then the telephone number value is added to the count and values list for the Numeric Value entity type.

Information in the entities list

The entities list groups the entities by the file and, if relevant, the page where they were detected.

For each value, the list includes:

The original value.
The original value in the context of its surrounding text.
The redacted or synthesized value in the context of its surrounding text, based on the selected handling option.

Viewing the list of entity types that were not detected

Below the list of detected entity types is the Entity types not found list, which contains the list of entity types that Textual did not detect in the files.

Filtering the entity types

You can filter the entity types list by text in the type name or description. The filter applies to both the detected and undetected entity types.

To filter the types, in the filter field, begin to type text that is in the entity type name or description.

Configuring added and excluded values for built-in entity types

Required dataset permission: Edit dataset settings

In a dataset, for each built-in entity type, you can configure additional values to detect, and values to exclude. You cannot define added and excluded values for custom entity types.

You might add values that Textual does not detect because, for example, they are specific to your organization or industry.

You might exclude a value because:

Textual labeled the value incorrectly.
You do not want to redact a specific value. For example, you might want to preserve known test values.

Note that for a pipeline that redacts files, you cannot add or exclude specific values.

Viewing whether an entity type has added or excluded values

In the entity types list, the add values and exclude values icons indicate whether there are configured added and excluded values for the entity type.

When added or excluded values are configured, the corresponding icon is green.

When there are no configured values, the corresponding icon is black.

Displaying the Configure Entity Detection panel

From the Configure Entity Detection panel, you configure both added and excluded values for entity types.

To display the panel, click the add values or exclude values icon for an entity type.

The panel contains an Add to detection tab for added values, and an Exclude from detection tab for excluded values.

Selecting the entity type to add or exclude values for

The entity type dropdown list at the top of the Configure Entity Detection panel indicates the entity type to configure added and excluded values for.

The initial selected entity type is the entity type for which you clicked the icon. To configure values for a different entity type, select the entity type from the list.

Configuring added values

On the Add to detection tab, you configure the added values for the selected entity type.

Each value can be a specific word or phrase, or a regular expression to identify the values to add. Regular expressions must be C# compatible.

Configuring a new added value

To add an added value:

Click the empty entry.
Type the value into the field.

Editing an added value

To edit an added value:

Click the value.
Update the value text.

Testing an added value

For each added value, you can test whether Textual correctly detects it.

To test a value:

From the Test Entry dropdown list, select the number for the value to test.
In the text field, type or paste content that contains a value or values that Textual should detect.

The Results field displays the text and highlights matching values.

Removing an added value

To remove an added value, click its delete icon.

Configuring excluded values

On the Exclude from detection tab, you configure the excluded values for the selected entity type.

Each value can be either a specific word or phrase to exclude, or a regular expression to identify the values to exclude. The regular expression must be C# compatible.

You can also provide a specific context within which to ignore a value. For example, in the phrase "one moment, please", you probably do not want the word "one" to be detected as a numeric value. If you specify "one moment, please" as an excluded value for the numeric entity type, then "one" is not identified as a number when it is seen in that context.

Adding an excluded value

To add an excluded value:

Click the empty entry.
Type the value into the field.

Editing an excluded value

To edit an excluded value:

Click the value.
Update the value text.

Testing an excluded value

For each excluded value, you can test whether Textual correctly detects it.

To test the value that you are currently editing:

From the Test Entry dropdown list, select the number for the value to test.
In the text field, type or paste content that contains a value or values to exclude.

The Results field displays the text and highlights matching values.

Removing an excluded value

To remove an excluded value, click its delete icon.

Saving the updated added and excluded values

The new added and excluded values are not reflected in the entity types list until Textual runs a new scan.

When you save the changes, you can choose whether to immediately run a new scan on the dataset files.

To save the changes and also start a scan, click Save and Scan Files.

To save the changes, but not run a scan, click Save Without Scanning Files. When you do not run the scan, then on the dataset details page, Textual displays a prompt to run a scan.

Working with custom entity types

From the entity types list, you can set whether each custom entity is active, and edit the custom entity configuration.

You can also create a new custom entity type.

Enabling and disabling custom entity types

Required dataset permission: Edit dataset settings

In the entity types list, custom entity types include a toggle to indicate whether the custom entity type is active for that dataset or pipeline.

To disable a custom entity type, set the toggle to the off position.

When a custom entity type is enabled, then it is listed under either the found or not found entity types, depending on whether the files include entities of that type.

When a custom entity type is not enabled, it is listed under Inactive custom entity types. To enable the custom entity type, set the toggle to the on position.

Editing a custom entity type

Required global permission - either:

Create custom entity types
Edit any custom entity type

To edit a custom entity type, click the settings icon for that type.

Note that any changes to the custom entity type settings affect all of the datasets and pipelines that use the custom entity type.

For information on how to configure a custom entity type, go to .

Creating a custom entity type

Required global permission: Create custom entity types

From the dataset details or pipeline details page, to create a new custom entity type, click Create Custom Entity Type.

For information on how to configure a custom entity type, go to .

Running a new scan to reflect custom entity type changes

When you enable, disable, add, or edit custom entity types, the changes do not take effect until you run a new scan.

For datasets and uploaded file pipelines, to run a new scan, click Scan.

For a cloud storage pipeline, Textual scans the files when you run the pipeline.

Selecting the handling option for entity types

Required dataset permission: Edit dataset settings

For datasets that produce redacted files, for each entity type, you choose how to handle the detected values. This determines how each value displays in the output files.

Datasets that produce JSON output do not use entity type handling options.

Available handling options

The available options are:

Synthesis - Indicates to replace the value with another realistic value. For example, the first name value Michael might be replaced with the value John. The synthesized values are always consistent, meaning that a given entity value always has the same replacement value. For example, if the first name Michael appears multiple times in the text, it is always replaced with John. Textual does not synthesize any excluded values. For custom entity types, Textual scrambles the values.
Redaction - This is the default option, except for the Full Mailing Address entity type, which is Off by default. For text files, Redaction indicates to tokenize the value - to replace it with a token that identifies the entity type followed by a unique identifier. For example, the first name value Michael might be replaced with NAME_GIVEN_12m5s. The identifiers are consistent, which means that for a given original value, the replacement always has the same unique identifier. For example, the first name Michael might always be replaced with NAME_GIVEN_12m5sb, while the first name Helen might always be replaced with NAME_GIVEN_9ha3m2. For PDF files, Redaction indicates to either cover the value with a black box, or, if there is space, display the entity type and identifier. For image files, Redaction indicates to cover the value with a black box. Textual does not redact any excluded values.
Off - Indicates to not make any changes to the values. For example, the first name value Michael remains Michael. This this the default option for the Full Mailing Address entity type.

Selecting the handling option for a specific entity type

To select the handling option for an individual entity type, click the option for that type.

Selecting the handling option for all of the entity types

For a dataset, to select the same handling option for all of the entity types, from the Bulk Edit dropdown above the data type list, select the option.

For a pipeline that generates synthesized files, on the Generator Config tab, use the Bulk Edit options at the top of the entity types list.

Configuring handling of file components

Required dataset permission: Edit dataset settings

The Dataset Settings panel includes options for how Textual handles the following file components:

For .docx files, images and comments
For PDF files, scanned-in signatures

To display the Dataset Settings page, on the dataset details page, click Settings.

These options are not available for pipelines that also redact files.

Configuring how to handle .docx images

For .docx images, including .svg files, you can configure the dataset to either:

Redact the image content. When you select this option, Textual looks for and blocks out sensitive values in the image.
Ignore the image.
Replace the images with black boxes.

On the Dataset Settings page, under Image settings for DOCX files:

To redact the image content, click Redact contents of images using OCR. This is the default selection.
To ignore the images entirely, click Ignore images during scan.
To replace the images with black boxes, click Replace images from the output file with black boxes.

Configuring how to handle .docx tables

For .docx tables, you can configure the dataset to either:

Redact the table content. When you select this option, Textual detects sensitive values and replaces them based on the entity type configuration.
Block out all of the table cells. When you select this option, Textual places a black box over each table cell.

On the Dataset Settings page, under Table settings for DOCX files:

To redact the table content, click Redact content using the entity type configuration. This is the default selection.
To block out the table content, click Block out all table cell content.

Configuring how to handle .docx comments

For comments in a .docx file, you can configure the dataset to either:

Remove the comments from the file.
Ignore the comments and leave them in the file.

On the Dataset Settings page, to remove the comments, toggle Remove comments from the output file to the on position. This is the default configuration.

To ignore the comments, toggle Remove comments from the output file to the off position.

Configuring whether to redact PDF signatures

By default, Textual redacts scanned-in signatures in PDF files. You can configure the dataset to instead ignore the signatures.

On the Dataset Settings page:

To redact PDF signatures, toggle Detect and redact signatures in PDFs to the on position. This is the default configuration.
To ignore PDF signatures, toggle Detect and redact signatures in PDFs to the off position.

Adding manual overrides to PDF files

Required dataset permission: Edit dataset settings

For a PDF file in a dataset, you can add manual overrides to selected areas of a file. Manual overrides can either ignore redactions from Tonic Textual, or add redactions.

Pipelines do not support manual overrides in PDF files.

Editing an individual PDF file

For PDF files, you can add manual overrides to the initial redactions, which are based on the detected data types and handling configuration.

For each manual override, you select an area of the file.

For the selected area, you can either:

Ignore any automatically detected redactions. For example, a scanned form might show an example or boilerplate content that doesn't actually contain sensitive values.
Redact that area. The file might contain sensitive content that Tonic Textual is unable to detect. For example, a scanned form might contain handwritten notes.

You can also apply a template to the file.

Selecting the manual override option for a file

To manage the manual overrides for a PDF file:

In the file list, click the options menu for the file.
In the options menu, click Edit Redactions.

The File Redactions panel displays the file content. The values that Textual detected are highlighted. The page also shows any manual overrides that were added to the file.

Applying a PDF template to a file

If a dataset contains multiple files that have the same format, then you can create a template to apply to those files. For more information, go to .

On the File Redactions panel, to apply a template to the file, select it from the template dropdown list.

When you apply a PDF template to a file, the manual overrides from that template are displayed on the file preview. The manual overrides are not included in the Redactions list.

Adding a manual override

On the File Redactions panel, to add a manual override to a file:

Select the type of override. To indicate to ignore any automatically detected values in the selected area, click Ignore Redactions. To indicate to redact the selected area, click Add Manual Redaction.
Use the mouse to draw a box around the area to select.

Textual adds the override to the Redactions list. The icon indicates the type of override.

In the file content:

Overrides that ignore detected values within the selected area are outlined in red.
Overrides that redact the selected area are outlined in green.

Navigating to a manual override

To select and highlight a manual override in the file content, in the Redactions list, click the navigate icon for the override.

Removing a manual override

To remove a manual override, in the Redactions list, click the delete icon for the override.

Saving the manual overrides

To save the current manual overrides, click Save.

Creating templates to apply to PDF files

A dataset might contain multiple files that have the same structure, such as a set of scanned-in forms.

Instead of adding the same manual overrides for each file, you can use a PDF file in the dataset to create a template that you can apply to other PDF files in the dataset.

When you , you can apply a template.

Creating a PDF template

To add a PDF template to a dataset:

On the dataset details page, click Templates.

On the template creation and selection panel, click Create a New Template.

On the template details page:
1. In the Name field, provide a name for the template.
2. From the file dropdown list, select the dataset file to use to create the template.
3. Add the manual overrides to the file.

When you finish adding the manual overrides, click Save New Template.

Updating an existing PDF template

When you update a PDF template, it affects any files that use the template.

To update a PDF template:

On the dataset details page, click PDF Templates.
Under Edit an Existing Template, select the template, then click Edit Selected Template.
On the template details panel, you can change the template name, and add or remove manual overrides.

To save the changes, click Update Template.

Managing the manual overrides

Adding a manual override

On the template details panel, to add a manual override to a file:

Select the type of override. To indicate to ignore any automatically detected values in the selected area, click Ignore Redactions. To indicate to redact the selected area, click Add Manual Redaction.
Use the mouse to draw a box around the area to select.

Tonic Textual adds the override to the Redactions list. The icon indicates the type of override.

Navigating to a manual override

To select and highlight a manual override in the file content, in the Redactions list, click the navigate icon for the override.

Removing a manual override

To remove a manual override, in the Redactions list, click the delete icon for the override.

Deleting a PDF template

When you delete a PDF template, the template and its manual overrides are removed from any files that the template was assigned to.

To delete a PDF template:

On the dataset details page, click PDF Templates.
Under Edit an Existing Template, select the template, then click Edit Selected Template.
On the template details panel, click Delete.

Preview and obtain output

Previewing file output

Required dataset permission: Preview redacted dataset files

You cannot preview TIF image files. You can preview PNG and JPG files.

Displaying a dataset file preview

From the file list, to display the preview, either:

Click the file name.
Click the options menu, then click Preview.

File preview for a redacted file

For a dataset that generates output files of the same type as the original file:

On the left, the preview displays the original data. The detected entity values are highlighted.
On the right, the preview displays the data with replacement values that are based on the dataset configuration for the detected entity types.

Format of redacted values

Note that in the preview, the redacted values do not include the identifier. They only include the entity type. For example, NAME_GIVEN instead of NAME_GIVEN_1d9w5. The identifiers are included when you download the files.

Preview for PDF and image files

For a PDF or image file, for entity types that use the Redact handling option:

If there is space to display the entity type, then it is displayed.
Otherwise, the value is covered by a black box.

When you hover over a black box, the entity type displays in a tooltip:

To view the entity type labels, you can also zoom into the file.

The preview for a PDF file also reflects any manual overrides.

Selecting entity type handling options from the preview

You can use the preview to select the entity type handling option for each entity type. The options are:

Redact - This is the default value. Textual replaces the value with the name of the entity type followed by a unique identifier. For example, the first name John is replaced with NAME_GIVEN_12345. Note that the identifier is only visible in the downloaded file. It does not display on the preview.
Synthesize - Textual replaces the value with a realistic generated value. For example, the first name John is replaced with the first name Michael. The replacement values are consistent, which means that a given value always has the same replacement. For example, Michael is always the replacement value for John.
Off - Textual ignores the value and copies it as is to the output file.

To select the entity type handling option:

In the results panel, click a detected value.
On the panel, click the entity type handling option. Textual applies the same option to all entity values of that type.

From the preview, you can only select the entity type handling option. For the Synthesize option, you cannot configure synthesis options for an entity type. You must configure those options from the dataset details page. For more information, go to .

Ignoring specific instances in PDF files

From the PDF preview, you can also choose to ignore a specific value.

To configure whether to ignore a specific detected value:

In the results panel, click the value.
On the panel, to ignore the value, toggle Ignore Redaction to the on position.

File preview for a JSON output file

For a dataset that generates JSON output:

On the left is the original content. For files other than .txt files, you can toggle between generated Markdown and the rendered file.
On the right is a set of tabs that summarize the results.

Entities tab - Detected entities in the file

The Entities tab displays the file content with the detected entity values in context.

The actual values are followed by the type labels. For example, the given name John is displayed as John NAME_GIVEN.

JSON tab - Output JSON for the file

The JSON tab contains the content of the actual output file.

For details about the JSON output structure for the different types of files, go to .

Tables tab - Tables in a PDF or image file

For a PDF or image file that contains one or more tables, the Tables tab displays the tables. If the file does not contain any tables, then the Tables tab does not display.

Key-Value Pairs tab - Key-value pairs in a PDF or image file

For a PDF or image file that contains key-value pairs, the Key-Value Pairs tab displays the key-value pairs. If the file does not contain key-value pairs, then the Key-Value Pairs tab does not display.

Downloading local output files

Required dataset permission: Download redacted dataset files

For each file in a dataset, you can download the output file.

Downloading a single output file

From the file list, to download a single output file:

Click the options menu for the file.
In the options menu, click Download File.

Downloading all of the output files

To download all of the output files, click Download All Files.

Generating cloud storage output files

To generate original format output files for a cloud storage dataset, on the dataset details page, click Generate to <cloud storage type>.

Tonic Textual generates the output files to the configured output location. If the output location is not configured, then the generate option is disabled.

For datasets that produce JSON output, Textual generates the output files automatically as soon as the output location is configured.

Textual Python SDK

Installing the Textual SDK

The Tonic Textual SDK is a Python SDK that you can use to redact text and files.

It requires Python 3.9 or higher.

To install the Tonic Textual Python SDK, run:

pip install tonic-textual

Creating and revoking Textual API keys

Required global permission: Create an API key

To be able to use the Textual SDK, you must have an API key.

Alternatively, you can use the Textual API to to use for authentication.

Viewing the list of API keys

You manage keys from the User API Keys page.

To display the User API Keys page, in the Textual navigation menu, click User API Keys.

Creating a Textual API key

To create a Textual API key:

On the User API Keys page, click Create API Key.
In the Name field, type a name to use to identify the key.

Click Create API Key.
Textual displays the key value, and prompts you to copy the key. If you do not copy the key and save it to a file, you will not have access to the key. To copy the key, click the copy icon.

Revoking a Textual API key

To revoke a Textual API key, on the User API Keys page, click the Revoke option for the key to revoke.

Configuring the API key as an environment setting

You cannot instantiate the SDK client without an API key.

Instead of providing the key every time you call the Textual API, you can configure the API key as the value of the TONIC_TEXTUAL_API_KEY.

Obtaining JWT tokens for authentication

Instead of an API key, you can use the Textual API to obtain a JSON Web Token (JWT) to use for authentication.

Configuring the JWT and refresh token lifetimes

JWT lifetime

By default, a JWT is valid for 30 minutes.

On a self-hosted instance, to configure a different lifetime, set the SOLAR_JWT_EXPIRATION_IN_MINUTES.

Refresh token lifetime

You use a refresh token to obtain a new JWT. By default, a refresh token is valid for 10,000 minutes, which is roughly equivalent to 7 days.

On a self-hosted instance, to configure a different lifetime, set the environment variable SOLAR_REFRESH_TOKEN_EXPIRATION_IN_MINUTES.

Obtaining your first JWT and refresh token

To obtain your first JWT and refresh token, you make a login request to the Textual API. Before you can make this call, you must have a Textual account.

To make the call, perform a POST operation against:

The request payload is:

For example:

In the response:

The jwt property contains the JWT.
The refreshToken property contains the refresh token.

Obtaining a new JWT and refresh token

You use the refresh token to obtain both a new JWT and a new refresh token.

To obtain the new JWT and token, perform a POST operation against:

The request payload is:

In the response:

The jwt property contains the new JWT.
The refreshToken property contains the new refresh token.

Instantiating the SDK client

Whenever you call the Textual SDK, you first instantiate the SDK client.

To work with Textual datasets, or to redact individual files, you instantiate TonicTextual.
To work with Textual pipelines and parsing, you instantiate TonicTextualParse.

Instantiating when the API key is already configured

If the API key is configured as the value of the TONIC_TEXTUAL_API_KEY, then you do not need to provide the API key when you instantiate the SDK client.

For Textual datasets, or to use the redact method:

For Textual pipelines:

Instantiating when the API key is not configured

If the API key is not configured as the value of the TONIC_TEXTUAL_API_KEY, then you must include the API key in the request.

For Textual datasets, or to use the redact method:

For Textual pipelines:

Datasets and redaction

You can use the Tonic Textual SDK to manage pipelines and to redact individual strings and files.

Redact individual files

Required global permission: Use the API to parse or redact a text string

You can use the Textual SDK to redact and synthesize values in individual files.

Before you perform these tasks, remember to .

For a self-hosted instance, you can also configure the S3 bucket to use to store the files. This is the same S3 bucket that is used to store files for uploaded file pipelines. For more information, go to . For an example of an IAM role with the required permissions, go to .

Sending a file to Textual

To send an individual file to Textual, you use .

You first open the file so that Textual can read it, then make the call for Textual to read the file.

The response includes:

The file name
The identifier of the job that processed the file. You use this identifier to retrieve a transformed version of the file.

Getting the file with redacted or synthesized values

After you use to send the file to Textual, you use to retrieve a transformed version of the file.

To identify the file, you use the job identifier that you received from textual.start_file_redaction. You can for the detected entity values.

Before you make the call to download the file, you specify the path to download the file content to.

Transcribe and redact an audio file

You can send an audio file to the Tonic Textual SDK. Textual creates a transcription of the audio file, and then redacts the transcription text as a string.

Audio file limitations

The file must be 25MB or smaller, and must be one of the following file types:

m4a
mp3
webm
mp4
mpga
wav

Sending the transcription and redaction request

To transcribe and redact an audio file, you use textual.redact_audio.

redaction_response=textual.redact_audio(<path to the audio file>)
redaaction_response.describe

The request includes the entity type handling configuration.

The redaction response includes the redacted or synthesized content and details about the detected entity values.

Configure entity type handling for redaction

Required dataset permission: Edit dataset settings

By default, when you:

Configure a dataset
Redact a string
Retrieve a redacted file

Textual does the following:

For the string and file redaction, replaces detected values with tokens.
For LLM synthesis, generates realistic synthesized values.

When you make the request, you can:

Override the default behavior.
For individual files and text strings, specify custom entity types to include.

Specifying the handling option for entity types

For each entity type, you can choose to redact, synthesize, or ignore the value.

When you redact a value, Textual replaces the value with a token that consists of the entity type. For example, ORGANIZATION.
When you synthesize a value, Textual replaces the value with a different realistic value.
When you ignore a value, Textual passes through the original value.

To specify the handling option for entity types, you use the generator_config parameter.

Where:

<entity_type> is the identifier of the entity type. For example, ORGANIZATION. For the list of built-in entity types that Textual scans for, go to . For custom entity types, the identifier is the entity type name in all caps. Spaces are replaced with underscores, and the identifier is prefixed with CUSTOM_. For example, for a custom entity type named My New Type, the identifier is CUSTOM_MY_NEW_TYPE. From the Custom Entity Types page, to copy the identifier of a custom entity type, click its copy icon.

<handling_option> is the handling option to use for the specified entity type. The possible values are Redact, Synthesis, and Off.

For example, to synthesize organization values, and ignore languages:

Specifying a default handling option

For string and file redaction, you can specify a default handling option to use for entity types that are not specified in generator_config.

To do this, you use the generator_default parameter.

generator_default can be either Redact, Synthesis, or Off.

Providing added and excluded values for entity types

You can also configure added and excluded values for each entity type.

You add values that Textual does not detect for an entity type, but should. You exclude values that you do not want Textual to identify as that entity type.

To specify the added values, use label_allow_lists.
To specify the excluded values, use label_block_lists.

For each of these parameters, the value is a list of entity types to specify the added or excluded values for. To specify the values, you provide an array of regular expressions.

The following example uses label_allow_lists to add values:

For NAME_GIVEN, adds the values There and Here.
For NAME_FAMILY, adds values that match the regular expression ([a-z]{2}).

Including custom entity types

When you redact a string or download a redacted file, you can provide a comma-separated list of custom entity types to include. Textual then scans for and redacts those entity types based on the configuration in generator_config.

For example:

Textual REST API

About the Textual REST API

The Tonic Textual REST API allows you to more deeply integrate Textual functions into your existing workflows.

You can use the REST API as another tool alongside the Textual application and the Textual Python SDK. The Python SDK supports the same actions as the REST API. We recommend the Python SDK for customers who already use Python.

You can download the Textual OpenAPI specification from:

https://textual.tonic.ai/swagger/v1/swagger.json

REST API authentication

Before you can use the API, you must create a Tonic Textual API key. For information on how to obtain a Textual API key, go to Creating and revoking Textual API keys.

When you call the API, you place your API key in the authorization header of the request, similar to the following curl request, which fetches the list of datasets for the current user.

curl --request GET \
--url "https://textual.tonic.ai/api/dataset" \
--header "Content-Type: application/json" \
--header "Authorization: API_KEY"

Most Textual API requests require authentication. For each request, the reference information indicates whether the request requires an API key.

For requests that require an API key, if you do not provide a valid API key, you receive a 401 Unauthorized response.

Redaction

Use the Tonic Textual REST API to redact text. Redaction means to detect and replace sensitive values.

Redact text strings

You can use the Tonic Textual REST API to redact text strings, including:

Plain text
JSON
XML
HTML

Textual provides a specific endpoint for each format. For JSON, XML, and HTML, Textual only redacts the text values. It preserves the underlying structure.

Datasets

Use the REST API to manage datasets.

Manage datasets

Use the REST API to create and manage datasets.

Manage dataset files

Use the REST API to manage dataset files.

Entity linking

Use the REST API to link entity values.

Access management

Use the API to retrieve information about users and groups, and to manage access to datasets.

Install and administer Textual

Textual architecture

The following diagram shows how data and requests flow within the Tonic Textual application:

Textual application database

The Textual application database is a PostgreSQL database that stores the dataset configuration.

If you do not configure an S3 bucket, then it also stores uploaded files and files that you use the SDK to redact.

Textual datastore in Amazon S3

You can configure an S3 bucket to store uploaded files and individual files that you use the SDK to redact. For more information, go to Setting the S3 bucket for file uploads and redactions.

If you do not configure an S3 bucket, then the files are stored in the Textual application database.

Textual components

Textual web server

Runs the Textual user interface.

Textual worker

A textual instance can have multiple workers.

The worker orchestrates jobs. A job is a longer running task such as the redaction of a single file.

If you redact a large number of files, you might deploy additional workers and machine learning containers to increase the number of files that you can process concurrently.

Textual machine learning

A textual installation can have 1 or more machine learning containers.

The machine learning container hosts the Textual models. It takes text from the worker or web server and returns any entities that it discovers.

Additional machine learning containers can increase the number of words per second that Textual can process.

OCR service

The OCR service converts PDFs and images to text that Textual can then scan for sensitive values.

For more information, go to Enabling PDF and image processing.

LLM service

Textual only uses the LLM service for LLM synthesis.

Setting up and managing a Textual Cloud pay-as-you-go subscription

The Tonic Textual pay-as-you-go plan allows you to automatically bill a credit card for your Textual usage.

The Textual subscription plan charges a flat rate for each 1000 words. You are billed each month based on when you started your subscription. For example, if you start your subscription on the 12th of the month, then you are billed every month on the 12th.

Tonic.ai integrates with a payment processing solution to manage the payments.

Setting up the subscription

To start a new subscription, from a usage pane or upgrade prompt, click Upgrade Plan.

You are sent to the payment processing solution to enter your payment information.

Tracking your usage

The panel on the Home page shows the usage for the current month.

To view additional usage details, click Manage Plan.

The Manage Plan page displays the details for your subscription.

Account summary

The summary at the top left contains an overview of the subscription payment information, as well as the total number of words scanned since you started your account.

From the summary, you can go to the payment processing solution to view and manage payment information.

30-day usage graph

The graph at the top of the page shows the words scanned per day for the previous 30 days.

Current billing period

The Current Billing Period panel summarizes your usage for the current month, and provides information about the next payment.

Next billing date

The Next billing date panel shows when the next billing period begins.

Payment history

The Payment History section shows the list of subscription payments.

For each payment, the list shows the date and amount, and whether the payment was successful.

To download the invoice for a payment, click its Invoice option.

Updating the payment information

You can update the payment information for your subscription. For example, you might need to choose a different credit card or update an expiration date.

To manage the payment information:

On the home page, in the usage panel, click Manage Plan.
On the Manage Plan page, from the account summary, click Manage Payment.

You are sent to the payment processing solution to update your payment information.

Canceling a subscription

To cancel a subscription, from the Manage Plan page:

Click Manage Payment.
In the payment processing solution, select the cancellation option.

The cancellation takes effect at the end of the current subscription month.

Deploying a self-hosted instance

The Tonic Textual images are stored on . During onboarding, Tonic.ai provides you with credentials to access the image repository. If you require new credentials, or you experience issues accessing the repository, contact .

You can deploy Textual using either Kubernetes or Docker.

System requirements

You install a self-hosted instance of Tonic Textual on either:

A VM or server that runs Linux and on which you have superuser access.
A local machine that runs Mac, Windows, or Linux.

Application server or cluster requirements

At minimum, we recommend that the server or cluster that you deploy Textual to has access to the following resources:

Nvidia GPU, 16GB GPU RAM. We recommend at least 6GB GPU RAM for each textual-ml worker.

If you only use a CPU and not a GPU, then we recommend an M5.2xLarge. However, without GPU, performance is significantly slower.

GPU considerations

The number of words per second that Textual processes depends on many factors, including:

The hardware that runs the textual-ml container
The number of workers that are assigned to the textual-ml container
The auxiliary model, if any, that is used in the textual-ml container.

To optimize the throughput of and the cost to use Textual, we recommend that the textual-ml container runs on modern hardware with GPU compute. If you use AWS, we recommend a g5 instance with 1 GPU.

Application database

The Textual application database is a PostgreSQL database that stores the dataset and Textual configuration. If you did not configure an S3 bucket to store files that you upload into a dataset, then the application database also contains those uploaded files.

External database server

The Textual application database is an external database that is hosted on a separate server.

For the PostgreSQL server, we recommend a minimum of an RDS t3.small on AWS, with at least 2GB RAM, 2 vCPU, and 100 GB of storage.

To prevent the loss of Textual metadata, keep regular backups of the PostgreSQL instance.

PostgreSQL version

For the Textual application database, the current minimum supported version is PostgreSQL 14+.

You should keep your PostgreSQL version relatively up-to-date with the current PostgreSQL LTS.

Tonic.ai might periodically conduct a campaign to request updates of self-hosted PostgreSQL instances before a scheduled update in the minimum supported version.

Database user permissions

The user credentials that you provide to Textual for the application database must have permission to create tables, insert, and select.

Setting up Nvidia GPU for Textual

To use GPU resources:

Ensure that the correct Nvidia drivers are installed for your instance.
If you use Kubernetes to deploy Textual, follow the instructions in the NVIDIA GPU operator documentation.
If you use Minikube, then use the instructions in Using NVIDIA GPUs with Minikube.
If you use Docker Compose to deploy Textual, follow these steps to install the nvidia-container-runtime.

Deploying with Docker Compose

The Docker Compose file is available in the GitHub repository .

Fork the repository.

To deploy Textual:

Rename sample.env to .env.
In .env, provide values for the required settings. These are not commented out and have <FILL IN> as a placeholder value:
- SOLAR_VERSION - Provided by Tonic.ai.
- SOLAR_LICENSE - Provided by Tonic.ai.
- ENVIRONMENT_NAME - The name that you want to use for your Textual instance. For example, my-company-name.
- SOLAR_SECRET - The string to use for Textual encryption.
- SOLAR_DB_PASSWORD - The password that you want to use for the Textual application database, which stores the metadata for Textual, including the datasets and pipelines. Textual deploys a PostgreSQL database container for the application database.
To deploy and start Textual, run docker-compose up -d.

Deploying on Kubernetes with Helm

The Tonic Textual Helm chart is available in the GitHub repository https://github.com/TonicAI/textual_helm_charts.

To use the Helm chart, you can either:

Use the OCI-based registry that Tonic hosts on quay.io.
Fork or clone the repository and then maintain it locally.

During the onboarding period, you are provided access credentials to our docker image repository on Quay.io. If you require new credentials, or you experience issues accessing the repository, contact [email protected].

Configure

Before you deploy Textual, you create a values.yaml file with the configuration for your instance.

For details about the required and optional configuration options, go to the repository readme.

Deploy

To deploy and validate access to Textual from the forked repository, follow the instructions in the repository readme.

To use the OCI-based registry, run:

helm install textual oci://quay.io/tonicai/textual -f values.yaml -n textual --create-namespace

The GitHub repository contains a readme with the details on how to populate a values.yaml file and deploy Textual.

Configuring Textual

You can configure your self-hosted instance of Textual to enable Textual features.

How to configure Textual environment variables

On a self-hosted instance of Textual, much of the configuration takes the form of environment variables.

After you configure an environment variable, you must restart Textual.

Docker

For Docker, add the variable to .env in the format:

SETTING_NAME=value

After you update .env, to restart Textual and complete the update, run:

$ docker-compose down

$ docker-compose pull && docker-compose up -d

Kubernetes

For Kubernetes, in values.yaml, add the environment variable to the appropriate env section of the Helm chart.

For example:

After you update the YAML file, to restart the service and complete the update, run:

$ helm upgrade <name_of_release> -n <namespace_name> <path-to-helm-chart>

The above Helm upgrade command is always safe to use when you provide specific version numbers. However, if you use the latest tag, it might result in Textual containers that have different versions.

Configuring the number of textual-ml workers

The TEXTUAL_ML_WORKERS environment variable specifies the number of workers to use within the textual-ml container. The default value is 1.

Having multiple workers allows for parallelization of inferences with NER models. The number of required workers is also affected by the number of jobs that each worker can run simultaneously.

When you deploy Textual with Kubernetes on GPUs, parallelization allows the textual-ml container to fully utilize the GPU.

We recommend 3GB of GPU RAM for each worker.

Configuring processing and parallelism

The following environment variables control job and file processing.

Configuring the number of jobs to run concurrently

By default, each Tonic Textual worker can run eight jobs at the same time. For example, it can process up to eight files simultaneously.

The environment variable SOLAR_MAX_CONCURRENT_WORKER_JOBS controls the number of jobs to run concurrently.

The number of jobs that can run concurrently can affect the number of Textual workers that you need. The more jobs that can run concurrently, the fewer workers that are needed.

Configuring the size of the datetime generator cache

When it generates datetime values, to optimize the processing, Textual stores the redacted datetime values in a cache.

To change the cache size, configure the environment variable SOLAR_DATETIME_GENERATOR_CACHE_CAPACITY.

The default value is 100000, meaning that the cache contains 100,000 values.

Note that while increasing the size of the cache can speed up processing, it also uses more RAM.

Configuring the number of PDF pages to redact simultaneously

When Textual redacts PDF files so that a user can preview or download the output, the following environment variable determines the number of pages that it processes simultaneously:

SOLAR_PDF_PAGE_REDACTION_PARALLELISM

The default value is 4, meaning that Textual processes 4 pages at a time.

Configuring the number of PDF files to plan simultaneously

When Textual plans the redaction of PDF files for a user to preview or download, the following environment variable determines the number of files that it plans simultaneously.

SOLAR_PDF_DOC_PLAN_PARALLELISM

The default value is 3, meaning that Textual plans 3 PDF files at a time.

Configuring how often to purge cached PDF pages

When it redacts PDF files, Textual stores the redacted PDF pages in a cache.

The following environment variable determines how often Textual purges the cache of PDF pages.

PURGE_REDACTED_PAGES_IN_HOURS

The default value is 12, meaning that Textual purges the redacted PDF pages cache every 12 hours.

Configuring the format of Textual logs

Textual writes the worker, machine learning, and API log messages to stdout.

By default, the log messages are in an unstructured format.

To instead use a JSON format for the logs, set the environment variable SOLAR_EMIT_JSON_LOGS_TO_STDOUT to true.

Setting a custom certificate

Tonic Textual provides a certificate for https traffic, but on a self-hosted instance, you can also use a user-provided certificate. The certificate must use the the PFX format and be named solar.pfx.

To use your own certificate, you must:

Add the SOLAR_PFX_PASSWORD environment variable.
Use a volume mount to provide the certificate file. Textual uses volume mounting to give the Textual containers access to the certificate.

You must apply the changes to both the Textual web server and Textual worker containers.

Docker

To use your own certificate, you make the following changes to the docker-compose.yml file.

Environment variable

Add the environment variable SOLAR_PFX_PASSWORD, which contains the certificate password.

Volume mount

Place the certificate on the host machine, then share it to the containers as a volume.

You must map the certificate to /usr/bin/textual/certificates on the containers.

Copy the following:

volumes:
        ...
        - /my-host-path:/usr/bin/textual/certificates

Kubernetes

Environment variable

You must add the environment variable SOLAR_PFX_PASSWORD, which contains the certificate password.

Volume mount

You can use any volume type that is allowed within your environment. It must provide at least ReadOnlyMany access.

You map the certificate to /usr/bin/textual/certificates on the containers. Within your web server and worker deployment YAML files, the entry should be similar to the following:

    volumeMounts:
    - name: <my-volume-name>
      mountPath: /usr/bin/textual/certificates

Configuring endpoint URLs for calls to AWS

For calls to AWS products that are used in Textual, you can configure custom URLs to use. For example, if you use proxy endpoints, then you would configure those endpoints in Textual.

The for custom AWS endpoints include the following:

AWS_S3_FORCE_PATH_STYLE - Whether to always use path-style instead virtual-hosted-style for connections to Amazon S3. The default is false.
This setting is only used if you configured either AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3.
AWS_ENDPOINT_URL - The URL to use for all AWS calls, including calls to Amazon S3, Amazon Textract, and Amazon SES v2. This global endpoint is overridden by service-specific endpoints.
AWS_ENDPOINT_URL_S3 - The URL to use for calls to Amazon S3. This overrides the global URL set in AWS_ENDPOINT_URL.
AWS_ENDPOINT_URL_TEXTRACT - The URL to use for calls to Amazon Textract. This overrides the global URL set in AWS_ENDPOINT_URL.
AWS_ENDPOINT_URL_SESV2 - The URL to use for calls to Amazon SES v2. This overrides the global URL set in AWS_ENDPOINT_URL.

Enabling PDF and image processing

To process PDF and image files, Tonic Textual uses optical character recognition (OCR). Textual supports the following OCR models:

Azure AI Document Intelligence
Amazon Textract
Tesseract

For the best performance, we recommend that you use either Azure AI Document Intelligence or Amazon Textract.

If you cannot use either of those - for example because you run Textual on-premises and cannot access third-party services - then you can use Tesseract.

Azure AI Document Intelligence

To use Azure AI Document Intelligence to process PDF image files, Textual requires the Azure AI Document Intelligence key and endpoint.

Docker

In .env, uncomment and provide values for the following settings:

SOLAR_AZURE_DOC_INTELLIGENCE_KEY=#

SOLAR_AZURE_DOC_INTELLIGENCE_ENDPOINT=#

Kubernetes

In values.yaml, uncomment and provide values for the following settings:

azureDocIntelligenceKey:

azureDocIntelligenceEndpoint:

Amazon Textract

If the Azure-specific environment variables are not configured, then Textual attempts to use Amazon Textract.

To use Amazon Textract, Textual requires access to an IAM role that has sufficient permissions. You must also . The configured S3 bucket is required for uploaded file pipelines, and is also used to store dataset files and individual files that are redacted using the SDK.

We recommend that you use the AmazonTextractFullAccess policy, but you can also choose to use a more restricted policy.

Here is an example policy that provides the minimum required permissions:

After the policy is attached to an IAM user or a role, it must be made accessible to Textual. To do this, either:

Assign an instance profile
Provide the AWS key, secret, and Region in the following environment variables:

Tesseract

If neither Azure AI Document Intelligence nor Amazon Textract is configured, then Textual uses Tesseract, which is automatically available in your Textual installation.

Tesseract does not require any external access.

Setting the S3 bucket for file uploads and redactions

Tonic Textual pipelines can process files from sources such as Amazon S3, Azure Blob Storage, and Databricks Unity Catalog. You can also create pipelines to process files that you upload directly from your browser.

For those uploaded file pipelines, Textual always stores the files in an S3 bucket. On a self-hosted instance, before you add files to an uploaded file pipeline, you must configure the S3 bucket and the associated authentication credentials.

The configured S3 bucket is also used to store dataset files and individual files that you use the Textual SDK to redact. If an S3 bucket is not configured, then:

The dataset and individual redacted files are stored in the Textual application database.
You cannot use Amazon Textract for PDF and image processing. If you configured Textual to use Amazon Textract, Textual instead uses Tesseract.

The authentication credentials for the S3 bucket include:

The AWS Region where the S3 bucket is located.
An AWS access key that is associated with an IAM user or role.
The secret key that is associated with the access key.

To provide the authentication credentials, you can either:

Provide the values directly as environment variable values.
Use the instance profile of the compute instance where Textual runs.

For an example IAM role that has the required permissions, go to Example IAM role for file uploads and redactions.

Docker

In .env, add the following settings:

SOLAR_INTERNAL_BUCKET_NAME= <S3 bucket path>

AWS_DEFAULT_REGION= <AWS Region>

AWS_ACCESS_KEY_ID= <AWS access key>

AWS_SECRET_ACCESS_KEY= <AWS secret key>

If you use the instance profile of the compute instance, then only the bucket name is required.

Kubernetes

In values.yaml, within env: { } under both textual_api_server and textual_worker, add the following settings:

SOLAR_INTERNAL_BUCKET_NAME

AWS_DEFAULT_REGION

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

For example, if no other environment variables are defined:

  env: {
        "SOLAR_INTERNAL_BUCKET_NAME": "<S3 bucket path>",
        "AWS_DEFAULT_REGION": "<AWS Region>",
        "AWS_ACCESS_KEY_ID": "<AWS access key>",
        "AWS_SECRET_ACCESS_KEY": "<AWS secret key>"
       }

If you use the instance profile of the compute instance, then only the bucket name is required.

Required IAM role permissions for Amazon S3

For Amazon S3 datasets and pipelines, you connect to S3 buckets to select and store files.

On self-hosted instances, you also configure an S3 bucket and the credentials to use to store files for:

File upload pipelines. The S3 bucket is required for file upload pipelines.
File upload datasets. If you do not configure an S3 bucket, then the files are stored in the application database.
Individual files that you send to the SDK for redaction. If you do not configure an S3 bucket, then the files are stored in the application database.

Here are examples of IAM roles that have the required permissions to connect to Amazon S3 to select or store files.

Example IAM role for file uploads and redactions

For file upload pipelines, datasets, and individual file redactions, the files are stored in a single S3 bucket. For information on how to configure the S3 bucket and the corresponding access credentials, go to .

The IAM role that is used to connect to the S3 bucket must be able to read files from and write files to it.

Here is an example of an IAM role that has the permissions required to support uploaded file pipelines, datasets, and individual redactions:

Example IAM role for Amazon S3 datasets and pipelines

The access credentials that you configure for an Amazon S3 dataset or pipeline must be able to navigate to and select files and folders from the appropriate S3 buckets. They also need to be able to write output files to the configured output location.

Here is an example of an IAM role that has the permissions required to support Amazon S3 datasets or pipelines:

Configuring model preferences

On a self-hosted instance, you can configure settings to determine whether to the auxiliary model, and model use on GPU.

Configuring whether to use an auxiliary model

To improve overall inference, you can configure whether Textual uses the en_core_web_sm auxiliary NER model.

Entity types that the auxiliary model detects

The auxiliary model detects the following types:

EVENT
LANGUAGE
LAW
NRP
NUMERIC_VALUE
PRODUCT
WORK_OF_ART

Indicating whether to use the auxiliary model

To configure whether to use the auxiliary model, you use the environment variable TEXTUAL_AUX_MODEL.

The available values are:

en_core_web_sm - This is the default value.
none - Indicates to not use the auxiliary model.

Configuring model use for GPU

When you use a textual-ml-gpu container on accelerated hardware, you can configure:

Whether to use the auxiliary model,
Whether to use the date synthesis model

Indicating whether to use the auxiliary model for GPU

To configure whether to use the auxiliary model for GPU, you configure the environment variable TEXTUAL_AUX_MODEL_GPU.

By default, on GPU, Textual does not use the auxiliary model, and TEXTUAL_AUX_MODEL_GPU is false.

To use the auxiliary model for GPU, based on the configuration of TEXTUAL_AUX_MODEL, set TEXTUAL_AUX_MODEL_GPU to true.

When TEXTUAL_AUX_MODEL_GPU is true, and TEXTUAL_MULTI_LINGUAL is true, Textual also loads the multilingual models on GPU.

Indicating whether to use the date synthesis model for GPU

By default, on GPU, Textual loads the date synthesis model on GPU.

Note that this model requires 600MB of GPU RAM for each machine learning worker.

To not load the date synthesis model on GPU, set the environment variable TEXTUAL_DATE_SYNTH_GPU to false.

Viewing model specifications

On a self-hosted instance of Tonic Textual, you can view the current model specifications for the instance.

To view the model specifications:

Click the user icon at the top right.
In the user menu, click System Settings.

On the System Settings page, the Model Specifications section provides details about the models that Textual uses.

Managing user access to Textual

Tonic Textual provides the following options to manage access to Textual and its features.

Textual organizations

In Tonic Textual, each user belongs to an organization. Organizations are used to determine the company or customer that a Textual user belongs to.

A self-hosted instance of Textual contains a single organization. All users belong to that organization.

Textual Cloud hosts multiple organizations. The organizations are kept completely separate. Users from one Textual Cloud organization do not have any access to the users, datasets, or pipelines that belong to a different Textual Cloud organization.

When is a Textual organization created?

A Textual organization is created:

For a standard Textual license, both self-hosted and Textual Cloud, when the first user signs up for a Textual account.
When a user signs up for a free trial or pay-as-you go Textual Cloud license with a unique corporate email domain.
When a user signs up for a free trial or pay-as-you-go Textual Cloud license with a public email domain, such as Gmail or Yahoo. Every user with a public email domain is in a separate organization.

When is a new user added to an existing organization?

Self-hosted instance

A self-hosted instance has a single organization. Every user who signs up for an account on that instance is added to the organization.

Annual Textual Cloud license (not pay-as-you-go)

For companies with an annual Textual Cloud license, the license includes the email domains that are included in the license.

When a user with one of the included email domains signs up for a Textual account, they are automatically added to that organization.

Pay-as-you-go license

For a pay-as-you-go license, when a user with the same corporate email domain signs up for a Textual account, they are automatically added to that organization.

Users with public email domains are always in separate organizations.

Creating a new account in an existing organization

New user on a self-hosted instance

If your company has a self-hosted Textual instance that is installed on-premises, then you navigate to the Textual URL for that instance.

Your self-hosted instance might be configured to use single sign-on for Textual access. If so, then from the Textual login page, to create your Textual user account, click the single sign-on option.

Otherwise, to create your Textual user account, click Sign Up.

Your administrator can provide the URL for your Textual instance and confirm the instructions for creating your user account.

New user for an existing Textual Cloud organization

If your Textual license is on Textual Cloud, then new users that have a matching email domain are automatically added to your Textual Cloud organization.

For a Textual Cloud license other than a pay-as-you-go license, the license agreement specifies the included email domains. When a user with a matching email domain signs up for a Structural account, they are added to that Textual Cloud organization.

For a pay-as-you-go Textual Cloud license, when a user with the same corporate email domain as the subscribed user signs up for a Textual account, they are added to that Textual Cloud organization.

To create your Textual user account, on the Textual Cloud login page, click Sign Up.

Single sign-on (SSO)

Tonic Textual respects the access control policy of your single sign-on (SSO) provider. To access Textual, users must be granted access to the Textual application within your SSO provider.

Self-hosted instances can use any of the available SSO options. Textual Cloud organizations can enable Okta SSO.

To enable SSO, you first complete the required configuration in the SSO provider. You then configure Textual to connect to it. For self-hosted instances, you use Textual environment variables for the configuration. For a Textual Cloud organization, you use the Single Sign-On tab on the Permission Settings page.

After you enable SSO, users can use SSO to create an account in Textual.

For self-hosted instances, to only allow SSO authentication, set the environment variable REQUIRE_SSO_AUTH to true. For Textual Cloud, this is configured in the application. When SSO is required, Textual disables standard email/password authentication. All account creation and login is handled through your SSO provider. If multi-factor authentication (MFA) is set up with your SSO, then all authentication must go through your provider's MFA.

You can view the list of SSO groups whose members have logged into Textual.

Tonic Textual supports the following SSO providers:

Viewing the list of SSO groups in Textual

Required global permission: View users and groups

If you use SSO to manage Tonic Textual groups, then Textual displays the list of groups for which at least one user has logged in to Textual.

To display the SSO group list:

Click the user image at the top right.
In the user menu, click Permission Settings.
On the Permission Settings page, click Groups.

If no users from a group have logged in to Textual, then the group does not display in the list.

The list only displays the group names and indicates the SSO provider. To manage the group permissions:

To assign global permission sets, go to the Global Permission Sets list. For more information, go to Configuring access to global permission sets.
To assign dataset permission sets, go to the Datasets page. For more information, go to Sharing dataset access.
To assign pipeline permission sets, go to the Pipelines page. For more information, go to Sharing pipeline access.

Azure

Use these instructions to set up Azure Active Directory as your SSO provider for Tonic Textual.

Azure configuration

In the portal, navigate to Azure Active Directory -> App registrations, then click New registration.
Register Textual and create a new web redirect URI that points to your Textual instance's address and the path /sso/callback/azure.
Take note of the values for client ID and tenant ID. You will need them later.
Click Add a certificate or secret, and then create a new client secret. Take note of the secret value. You will need this later.
Navigate to the API permissions page. Add the following permissions for the Microsoft Graph API:
- OpenId permissions
- email
- openid
- profile
- GroupMember
- GroupMember.Read.All
- User
- User.Read
Click Grant admin consent for Tonic AI. This allows the application to read the user and group information from your organization. When permissions have been granted, the status should change to Granted for Tonic AI.
Navigate to Enterprise applications and then select Textual. From here, you can assign the users or groups that should have access to Textual.

Textual configuration

After you complete the configuration in Azure, you uncomment and configure the required environment variables in Textual.

For Kubernetes, in values.yaml:

# Azure SSO Config
# -----------------
#azureClientId: <client-id>
#azureTenantId: <tenant-id>
#azureClientSecret: <client-secret>
#azureGroupFilterRegex: <regular expression to identify allowed groups>

For Docker, in .env:

#SOLAR_SSO_AZURE_CLIENT_ID=#<client ID>
#SOLAR_SSO_AZURE_TENANT_ID=#<tenant ID>
#SOLAR_SSO_AZURE_CLIENT_SECRET=#<client secret>
#SOLAR_SSO_AZURE_GROUP_FILTER_REGEX=#"<regular expression to identify allowed groups>

GitHub

Use these instructions to set up GitHub as your SSO provider for Tonic Textual.

Create an OAuth application

In GitHub, navigate to Settings -> Developer Settings -> OAuth Apps, then create a new application.
For Application Name, enter Textual.
For Homepage URL, enter https://textual.tonic.ai.
For Authorization callback URL, enter https://your-textual-url/sso/callback/github.

Replace your-textual-url with the URL of your Textual instance.

Create a client secret

After you create the application, to create a new secret, click Generate a new client secret.

You use the client ID and the client secret in the Textual configuration.

Textual configuration

After you complete the configuration in GitHub, you uncomment and configure the required in Textual.

For Kubernetes, in values.yaml:

For Docker, in .env:

Google

Use these instructions to set up Google as your SSO provider for Tonic Textual.

Create an OAuth 2.0 client ID in Google

Go to
Click Create credentials, located near the top.
Select OAuth client ID.
Select Web application as the application type.
Choose a name.
Under Authorized redirect URIs, add the URL of the Textual server with the endpoint /sso/callback/google. For example, a local Textual server at http://localhost:3000 would need http://localhost:3000/sso/callback/google to be set as the redirect URI. Also note that internal URLs might not work.
On the confirmation page, note the client ID and client secret. You will need to provide them to Textual.

Textual configuration

After you complete the configuration in Google, you uncomment and configure the required in Textual.

The client ID
The client secret

For Kubernetes, in values.yaml:

For Docker, in .env:

Okta (self-hosted and Cloud)

Use these instructions to set up Okta as your SSO provider for Tonic Textual. Okta is supported on both self-hosted instances and on Textual Cloud.

Okta configuration

To enable Okta as your SSO provider for Tonic Textual, you first complete the following configuration steps within Okta:

Create a new application. Choose the OIDC - OpenId Connect method with the Single-Page Application option.

Click Next, then fill out the fields with the values below:
- App integration name: The name to use for the Textual application. For example, Textual, Textual-Prod, Textual-Dev.
- Grant type: Implicit (hybrid)
- Sign-in redirect URIs: For self-hosted instances, <base-url>/sso/callback/okta. For Textual Cloud, on the Permission Settings page, the sign-in redirect URL is displayed on the Single Sign-On tab. Copy the value from there and paste it into the field.
- Base URIs: The URL to your Textual instance
- Controlled access: Configure as needed to limit Textual access to the appropriate users

After saving the above, navigate to the General Settings page for the application and make the following changes:
- Grant type: Check Implicit (Hybrid) and Allow ID Token with implicit grant type.
- Login initiated by: Either Okta or App
- Application visibility: Check Display application icon to users
- Initiate login URI: <base-url>

Make a note of the following values that must be provided to Textual:
- Client ID of the application:
- Your Okta domain (for example, tonic.okta.com)
- If you created a custom authorization server for Textual, the server ID:
- IdP ID (If you use an outside identity provider):

Textual configuration (self-hosted)

On a self-hosted instance, after you , uncomment and configure the relevant in Textual.

Kubernetes

For Kubernetes, the settings are in the Okta SSO Config section of values.yaml:

oktaAuthServerId - If you created a custom authorization server, the server ID. If you do not use a custom authorization server, then you can omit this.
oktaClientId - The client identifier of the application.
oktaDomain - The Okta domain.
oktaIdentityProviderId - If you use a third-party provider, the provider identifier. If you do not use a third-party provider, you can omit this.

Docker

For Docker, the settings are in .env:

SOLAR_SSO_OKTA_CLIENT_ID - The client identifier of the application.
SOLAR_SSO_OKTA_DOMAIN - The Okta domain.
SOLAR_SSO_OKTA_IDENTITY_PROVIDER_ID - If you use a third-party provider, the provider identifier. If you do not use a third-party provider, then you can omit this.

Textual configuration (Textual Cloud)

Required global permission: Manage users and groups

On Textual Cloud, after you , you configure the Textual connection to Okta from the Single Sign-On tab on the Permission Settings page.

To enable Okta SSO, check the Enable Okta SSO checkbox.
In the SSO Client ID field, enter the client identifier of the application.
In the SSO Domain field, enter the Okta domain.
If you use a third-party provider, then in the Identity Provider ID field, provide the provider identifier. If you do not use a third-party provider, then you can skip this field.
If you created a custom authorization server, then in the Authorization Server field, provide the server identifier. If you did not create a custom authorization server, then you can skip this field.
To require your organization users to use Okta SSO to log in to Textual, check the Require SSO for login checkbox.

OpenID Connect (OIDC)

Use these instructions to set up an OpenID Connect SSO provider for Tonic Textual.

SSO setup

When you configure the application/client in your SSO system, you must configure it to use Authorization Code Flow.

You must also make note of the client_id. You must provide the client ID when you complete the configuration for Textual.

Redirect URI

In your SSO provider, configure the following redirect URI:

Sign-in redirect URI: <textual-base-url>/sso/callback/oidc

Textual configuration

Required environment variables

After you set up the SSO provider, you uncomment and configure the required in Textual.

The application client identifier
For HTTP basic authentication (client_secret_basic), the client secret
The base URL of the provider. This is the location of /.well-known/openid-configuration
A regular expression to identify groups that are permitted to use Textual.

For Kubernetes, in values. yaml:

For Docker, in .env:

Optional environment variables

You can optionally uncomment and configure the following optional environment variables:

A space-delimited list of scopes to request from the OIDC SSO provider. Because group information is not part of the standard OIDC specification, for Textual to capture group information, a custom scope must be configured.
The name of the claim that contains the user's first name.
The name of the claim that contains the user's last name.
The name of the claim that contains the user's email address or username.
The name of the claim that contains the user's group membership.

Textual has default values for these settings:

For Kubernetes, in values.yaml:

For Docker, in .env:

Managing permissions

You use permissions and permission sets to manage access to Tonic Textual features and functions.

Learn about permission sets and permissions

View and configure permission sets

Assign global permission sets

About permissions and permission sets

Tonic Textual uses permissions and permission sets to manage role-based access (RBAC) to Textual features and functions.

A permission grants access to a specific feature or function.

A permission set is a collection of permissions that can be assigned to a user or an SSO group.

Built-in and custom permission sets

Textual provides a set of built-in permission sets that you cannot edit or delete.

You can also create custom permission sets.

Global permission sets

Global permission sets control access to features and functions that are outside of the context of a specific dataset or pipeline. For example, global permission control who can manage users and configure custom entity types.

You can also select a default global permission set to assign to all new users.

For the list of global permission sets and available permissions, go to .

For information on how to assign global permission sets, go to .

Dataset and pipeline permission sets

Dataset and pipeline permission sets provide access to specific dataset or pipeline management features and functions.

Dataset and pipeline permission sets are assigned to users and groups within the context of a specific dataset or pipeline. For example, a user might have the Editor permission set in one dataset and the Viewer permission set in another dataset.

For the lists of built-in dataset and pipeline permission sets and available permissions, go to .

For information on how to assign dataset permission sets, go to . For information on how to assign pipeline permission sets, go to .

Viewing the lists of permission sets

Displaying the permission set lists

The Permission Settings page contains the lists of global, dataset, and pipeline permissions.

To display the Permission Settings page:

Click the user icon at the top right.
In the user menu, click Permission Settings.

On the Permission Settings page:

Global Permission Sets contains the list of global permission sets.
Dataset Permission Sets contains the list of dataset permission sets.
Pipeline Permission Sets contains the list of pipeline permission sets.

The lists include:

The permission set name.
Whether the permission set is built-in or custom.
For custom permission sets, when it was most recently modified, and the user who modified it.

Viewing the details for a permission set

To view the details for a permission set, in the permission sets list, click Settings.

The details panel for a permission set includes:

The name of the permission set.
The permission configuration.

Configuring custom permission sets

Required global permission: Manage custom permission sets

You can create custom global, dataset, and pipeline permission sets.

A custom permission set allows you to have more precise control over global, dataset, and pipeline permissions.

For example, you might want a pipeline permission set that allows a user to manage the files but not share or delete the pipeline.

For global permissions, you might want a global permission set that allows a user to manage access to any dataset or pipeline, but not manage Textual users.

Creating a custom permission set

To create a custom permission set:

On the global, dataset, or pipelines permission sets list, click the create permission set button.
On the permission set details panel, in the Permission Set Name field, type the name for the new permission set. Permission set names must be unique for that permission set type (global, pipeline, dataset).
Select the permissions to grant to the permission set. If a permission checkbox is checked, then the permission is granted to the permission set. If a permission checkbox is not checked, then the permission is not granted to the permission set.
To save the new permission set, click Save.
For a global permission set, Textual prompts you to configure access to the new permission set. To display the access management panel for the permission set, click Manage User Access. To not manage access at that time, click Skip.

Editing a custom permission set

You cannot make any changes to a built-in permission set.

For a custom permission set, you can change the permission set name and adjust the assigned permissions.

To edit an existing custom permission set:

On the global, datasets, or pipelines permission sets list, click Settings.
On the permission set details panel, update the permission set configuration.
Click Save.

Deleting a custom permission set

You can delete a custom permission set. You cannot delete a built-in permission set.

You cannot delete a permission set that is assigned to any users or groups. Before you can delete the permission set, you must remove the assignment.

To delete a custom permission set:

On the global, dataset, or pipelines permission sets list, click Settings.
On the permission set details panel, click Delete Permission Set.
On the confirmation panel, click Confirm.

Selecting default permission sets

You can configure:

The global permission set that is assigned to all Tonic Textual users.
The dataset permission set to assign to a user who creates a dataset.
The pipeline permission set to assign to a user who creates a pipeline.

Selecting the global permission set to assign to all Textual users

Required global permission: Manage access to global permission sets

By default, all Textual users are assigned the built-in General User global permission set. You can configure a different global permission set to assign to all Textual users.

The permission set cannot be removed.

When you choose a different permission set to assign to all users, unless they were otherwise assigned the previous permission set, they lose access to it.

To set the default global permission set to assign to all Textual users:

Click the user icon at the top right.
In the user menu, click Permission Settings.
On the Permission Settings page, click Global Permission Sets. The current permission set for all users is marked as Assigned to all users.
To select a different permission set, hover over the permission set row, then click Assign to all users.

The confirmation panel explains the risks of making this change. To confirm the change:
Check I have read and understand the risks.
Click Confirm.

Selecting the permission set to assign to a dataset creator

Required global permission: Manage access to dataset permission sets

By default, when a user creates a dataset, they are assigned the built-in Editor dataset permission set. You can configure a different dataset permission set to assign when a database is created.

Changing the selected permission set does not affect existing datasets. It only applies to datasets that are created after the change.

To select a different permission set for dataset creation:

Click the user icon at the top right.
In the user menu, click Permission Settings.
On the Permission Settings page, click Dataset Permission Sets. The current permission set for database creators is marked Assigned on dataset creation.
To select a different permission set, hover over the permission set row, then click Assign on dataset creation.
The confirmation panel explains the risk of making this change. To confirm the change:
1. Check I have read and understand the risks.
2. Click Confirm.

Selecting the permission set to assign to a pipeline creator

Required global permission: Manage access to pipeline permission sets

By default, when a user creates a pipeline, they are assigned the built-in Editor pipeline permission set. You can configure a different pipeline permission set to assign when a pipeline is created.

Changing the selected permission set does not affect existing pipelines. It only applies to pipelines that are created after the change.

To select a different permission set for pipeline creation:

Click the user icon at the top right.
In the user menu, click Permission Settings.
On the Permission Settings page, click Pipeline Permission Sets. The current permission set for pipeline creators is marked Assigned on pipeline creation.
To select a different permission set, hover over the permission set row, then click Assign on pipeline creation.
The confirmation panel explains the risk of making this change. To confirm the change:
1. Check I have read and understand the risks.
2. Click Confirm.

Configuring access to global permission sets

Required global permissions:

Manage access to global permission sets
View users and groups

From the Global Permission Sets list, you can grant or revoke access to a global permission set. Global permission sets can be assigned to individual users and to SSO groups.

Access to dataset permission sets is managed from the Datasets page. For more information, go to Sharing dataset access.

Access to pipeline permission sets is managed from the Pipelines page. For more information, go to Sharing pipeline access.

You cannot change the assignment of the following global permission sets:

The global permission set that is assigned to all Textual users. Initially, this is the General User permission set, but it can be changed to a different permission set.
The built-in Admin (Environment) global permission set.

Before you assign a global permission set to an SSO group, make sure that you are aware of who is in the group. The permissions that are granted to an SSO group automatically are granted to all of the users in the group.

To manage the permission set assignment:

On the Global Permission Sets list, for the permission set to manage, click Manage Access.
To grant access to a user or group:
1. Begin to type the user or group name.
2. In the list of matching users or groups, click the user or group name.
To remove access from a user or group, click Revoke for that user or group.
To save the changes to the permission set access, click Save.