Tonic Textual comes with built-in models to identify a range of sensitive values, such as:

• Locations and addresses

• Names of people and organizations

• Identifiers and account numbers

The built-in entity types are:

Tonic Textual supports languages in addition to English. Textual automatically detects the language and applies the correct model.

Textual Cloud

On Textual Cloud, we support the following languages:

• Chinese

• German

• Greek

• Italian

• Japanese

• Norwegian (Bokmål)

• Russian

• Spanish

• Ukrainian

Self-hosted instances

On a self-hosted instance, to enable support any languages other than English, you must:

• Configure Textual to support multiple languages

• Provide the language models for the languages to support

Available languages

Self-hosted customers can enable support for the following languages:

• Catalan

• Chinese

• Croatian

• Danish

• Dutch

• Finnish

• French

• German

• Greek

• Italian

• Japanese

• Korean

• Lithuanian

• Macedonian

• Norwegian (Bokmål)

• Polish

• Portuguese

• Romanian

• Russian

• Slovenian

• Spanish

• Swedish

• Ukrainian

Enabling multi-language support

The setting is used by the machine learning container.

Providing the language model assets

For the languages to add, you must provide the language model assets for Textual to use.

By default, Textual looks for model assets in the machine learning container, in /usr/bin/textual/language_models. The default Helm and Docker Compose configurations include the volume mount.

Tonic Textual provides a single tool to allow you to put your file-based data to work for you.

The Textual pipeline option allows you to prepare unstructured text for use in an LLM system. Textual extracts the text from each file and then produces Markdown-formatted output. You can optionally replace sensitive values in the output, to prevent data leakage from your LLM.

You can also use Textual datasets to do simple redaction and synthesis of sensitive values, to produce files in the same format to use for development and training. Each original file becomes an output file in the same format, with the sensitive values replaced.

Need help with Textual? Contact support@tonic.ai.

When you sign up for a Tonic Textual account, you can immediately get started with a new pipeline.

Note that these instructions are for setting up a new account on Textual Cloud. For a self-hosted instance, depending on how it is set up, you might either create an account manually or use single sign-on (SSO).

Signing up for Textual

To get started with a new Textual account:

1. Go to .

2. Click Sign up.

3. Enter your email address.

4. Create and confirm a password for your Textual account.

5. Click Sign Up.

Textual creates your account, and prompts you to provide some additional information about yourself and how you plan to use Textual.

After you fill out the information and click Get Started, Textual displays the Textual Home page.

From the panel, you can create a demo pipeline that uses example data, or you can create a pipeline that uses your own data.

Creating a demo pipeline with example data

Textual provides example pipeline data that consists of files of different types. To create a pipeline that uses the example data, click Try Demo Pipeline.

Creating a Textual pipeline with your own data

From the Getting Started panel, to create a pipeline that uses your own data:

1. Click Create a Pipeline.

1. On the Create A New Pipeline panel, provide the pipeline name.

2. Select the source option for the files:

   • Files uploaded from a local filesystem

   • Amazon S3

   • Databricks

   • Azure Blob Storage

Selecting the file upload option

To upload the files, click File Upload, then click Save.

Selecting the Amazon S3 option

To have the files come from Amazon S3:

1. Click Amazon S3.

1. In the Access Key field, provide an AWS access key that is associated with an IAM user or role.

2. In the Access Secret field, provide the secret key that is associated with the access key.

3. From the Region dropdown list, select the AWS Region to send the authentication request to.

4. To test the credentials, click Test AWS Connection.

5. Click Save.

Selecting the Databricks option

To have the files come from a Databricks volume:

1. Click Databricks.

1. In the Databricks URL field, provide the URL to the databricks data volume.

2. In the Access Token field, provide the access token to use to get access to the volume.

3. To test the connection, click Test Databricks Connection.

4. Click Save.

Selecting the Azure option

To have the files from Azure Blob Storage:

1. Click Azure.

1. In the Account Name field, provide the name of your Azure account.

2. In the Account Key field, provide the access key for your Azure account.

3. To test the connection, click Test Azure Connection.

4. Click Save.

Next steps

After you set up an account on Textual Cloud, you start a Textual free trial, during which Textual scans up to 100,000 words for free. Note that Textual counts actual words, not tokens. For example, "Hello, my name is John Smith." counts as six words.

After the 100,000 words, Textual disables scanning for your account. Until you purchase a pay-as-you-go subscription, you cannot:

• Add files to a dataset or pipeline

• Run a pipeline

• Use the Playground

During your free trial, Textual displays the current usage in the following locations:

• On the Home page

• In the navigation menu

• On the Playground

Whenever you call the Textual SDK, you first instantiate the SDK client.

• To work with Textual datasets, or to redact individual files, you instantiate TonicTextual.

• To work with Textual pipelines, you instantiate TonicTextualParse.

Instantiating when the API key is already configured

If the API key is configured as the value of TONIC_TEXTUAL_API_KEY, then you do not need to provide the API key when you instantiate the SDK client.

For Textual pipelines:

For Textual datasets:

Instantiating when the API key is not configured

If the API key is not configured as the value of TONIC_TEXTUAL_API_KEY, then you must include the API key in the request.

For Textual pipelines:

For Textual datasets:

To install the Tonic Textual Python SDK, run:

pip install tonic-textual

You can use the Textual SDK to parse individual files, either from a local file system or from an S3 bucket.

Textual returns a FileParseResult object for each parsed file. The FileParseResult object is a wrapper around the output JSON for the processed file.

Parse a file from a local file system

To parse a single file from a local file system, use textual.parse_file:

with open('<path to the file>','rb') as f: 
    byte_data = f.read()
    parsed_doc = textual.parse_file(byte_data, '<file name>')

You must use rb access mode to read the file. rb access mode opens the file to be read in binary format.

You can also set a timeout in seconds for the parsing. You can add the timeout as a parameter of parse_file command. To set a timeout to use for all parsing, set the environment variable TONIC_TEXTUAL_PARSE_TIMEOUT_IN_SECONDS.

Parse a file from an S3 bucket

You can also parse files that are stored in Amazon S3. Because this process uses the boto3 library to fetch the file from Amazon S3, you must first set up the correct AWS credentials.

To parse a file from an S3 bucket, use textual.parse_s3_file:

parsed_doc = textual.parse_s3_file('<bucket>','<key>')

Textual uses datasets to produce files with sensitive values replaced.

Before you perform these tasks, remember to instantiate the SDK client.

Create and add files to a dataset

To create a new dataset and then upload a file to it, use textual.create_dataset.

dataset = textual.create_dataset('<dataset name>')

To add a file to the dataset, use dataset.add_file. To identify the file, provide the file path and name.

dataset.add_file('<path to file>','<file name>') 

To provide the file as IO bytes, you provide the file name and the file bytes. You do not provide a path.

dataset.add_file('<file name>',<file bytes>) 

Textual creates the dataset, scans the uploaded file, and redacts the detected values.

Configure a dataset

To change the configuration of a dataset, use dataset.edit.

You can use dataset.edit to change:

• The name of the dataset

• The handling option for each entity type

• Added or excluded values for each entity type

dataset.edit(name='<dataset name>', 
  generator_config={'<entity_type>':'<handling_type>'},
  label_allow_lists={'<entity_type>':LabelCustomList(regexes['<regex>']},
  label_block_lists={'<entity_type>':LabelCustomList(regexes['<regex>']}
)

Get the current status of dataset files

To get the current status of the files in the current dataset, use dataset.describe:

dataset.describe()

The response includes:

• The name and identifier of the dataset

• The number of files in the dataset

• The number of files that are waiting to be processed (scanned and redacted)

• The number of files that had errors during processing

For example:

    Dataset: example [879d4c5d-792a-c009-a9a0-60d69be20206]
    Number of Files: 1
    Files that are waiting for processing: 
    Files that encountered errors while processing: 
    Number of Rows: 0
    Number of rows fetched: 0

Get lists of files by status

To get a list of files that have a specific statuse, use the following:

• dataset.get_failed_files

• dataset.get_running_files

• dataset.get_queued_files

• dataset.get_processed_files

The file list includes:

• File identifier and name

• Number of rows and columns

• Processing status

• For failed files, the error

• When the file was uploaded

Delete a file from a dataset

To delete a file from a dataset, use dataset.delete_file.

dataset.delete_file('<file identifier>')

Get redacted content for a dataset

To get the redacted content in JSON format for a dataset, use dataset.fetch_all_json():

dataset = textual.get_dataset('<dataset name>')
dataset.fetch_all_json()

For example:

dataset = textual.get_dataset('mydataset')
dataset.fetch_all_json()

The response looks something like:

'[["PERSON_Rz8NtJTPONTKgcB95i Portrait by PERSON_blatU6mAWFCQoSa5E, DATE_TIME_Rcl58 ...]'

Textual uses pipelines to transform file text into a format that can be used in an LLM system.

You can use the Textual SDK to create and manage pipelines and to retrieve pipeline run results.

Before you perform these tasks, remember to instantiate the SDK client.

Creating and deleting pipelines

Creating a pipeline

To create a pipeline, use the pipeline creation method for the type of pipeline to create"

• textual.create_local_pipeline - Creates an uploaded file pipeline.

• textual.create_s3_pipeline - Creates an Amazon S3 pipeline.

• textual.create_azure_pipeline - Creates an Azure pipeline.

• textual.create_databricks_pipeline - Creates a Databricks pipeline.

When you create the pipeline, you can also:

• If needed, provide the credentials to use to connect to Amazon S3, Azure, or Databricks.

• Indicate whether to also generate redacted files. By default, pipelines do not generate redacted files. To generate redacted files, set synthesize_files to True.

For example, to create an uploaded file pipeline that also creates redacted files:

pipeline = textual.create_local_pipeline(pipeline_name="pipeline name", synthesize_files=True)

The response contains the pipeline object.

Deleting a pipeline

To delete a pipeline, use textual.delete_pipeline.

textual.delete_pipeline(pipeline_id)

Updating a pipeline configuration

Changing whether to also generate synthesized files

To change whether a pipeline also generates synthesized files, use pipeline.set_synthesize_files.

Adding files to an uploaded file pipeline

To a a file to an uploaded file pipeline, use pipeline.upload_file.

pipeline = textual.create_pipeline(pipeline_name)
with open(file_path, "rb") as file_content:
    file_bytes = file_content.read()
pipeline.upload_file(file_bytes, file_name)

Configuring an Amazon S3 pipeline

For an Amazon S3 pipeline, you can configure the output location for the processed files. You can also identify the files and folders for the pipeline to process:

• To identify the output location for the processed files, use s3_pipeline.set_output_location.

• To identify individual files for the pipeline to process, use s3_pipeline.add_files.

• To identify prefixes - folders for which the pipeline processes all applicable files - use s3_pipeline.add_prefixes.

Configuring an Azure pipeline

For an Azure pipeline, you can configure the output location for the processed files. You can also identify the files and folders for the pipeline to process:

• To identify the output location for the processed files, use azure_pipeline.set_output_location.

• To identify individual files for the pipeline to process, use azure_pipeline.add_files.

• To identify prefixes - folders for which the pipeline processes all applicable files - use azure_pipeline.add_prefixes.

Getting a pipeline or pipelines

Getting the list of pipelines

To get the list of pipelines, use textual.get_pipelines.

pipelines = textual.get_pipelines()

The response contains a list of pipeline objects.

Getting a single pipeline

To use the pipeline identifier to get a single pipeline, use textual.get_pipeline_by_id.

pipeline_id: str # pipeline identifier
pipeline = textual.get_pipeline_by_id(pipeline_id)

The response contains a single pipeline object.

The pipeline identifier is displayed on the pipeline details page. To copy the identifier, click the copy icon.

Running a pipeline

To run a pipeline, use pipeline.run.

The response contains the job identifier.

Getting pipelines runs, files, and results

Getting pipeline runs

To get the list of pipeline runs, use pipeline.get_runs.

The response contains a list of pipeline run objects.

Getting pipeline files

Once you have the pipeline, to get an enumerator of the files in the pipeline from the most recent pipeline run, use pipeline.enumerate_files.

files = pipeline.enumerate_files()

The response is an enumerator of file parse result objects.

Getting the list of entities in a file

To get a list of entities that were detected in a file, use get_all_entities. For example, to get the detected entities for all of the files in a pipeline:

detected_entities = []
for file in pipeline.enumerate_files():
    entities = file.get_all_entities()
    detected_entities.append(entities)

To provide a list entity types and how to process them, use get_entities:

from textual.enums.pii_state import PiiState

generator_config: Dict[str, PiiState]
generator_default: PiiState

entities_list = []
for file in pipeline.enumerate_files():
    entities = file.get_entities(generator_config, generator_default)
    entities_list.append(entities)

generator_config is a dictionary that specifies whether to redact, synthesize, or do neither for each entity type in the dictionary.

For a list of the entity types that Textual detects, go to Entity types that Textual detects.

For each entity type, you provide the handling type:

• Redaction indicates to replace the value with the value type.

• Synthesis indicates to replace the value with a realistic value.

• Off indicates to keep the value as is.

generator_default indicates how to process values for entity types that were not included in the generator_config list.

The response contains the list of entities. For each value, the list includes:

• Entity type

• Where the value starts in the source file

• Where the value ends in the source file

• The original text of the entity

Getting the Markdown output for pipeline files

To get the Markdown output of a pipeline file, use get_markdown. In the request, you can provide generator_config and generator_default to configure how to present the detected entities in the output file.

from textual.enums.pii_state import PiiState

generator_config: Dict[str, PiiState]
generator_default: PiiState

markdown_list = []
for file in pipeline.enumerate_files():
    markdown = file.get_markdown(generator_config, generator_default)
    markdown_list.append(markdown)

The response contains the Markdown files, with the detected entities processed as specified in generator_config and generator_default.

Generating chunks from pipeline files

To split a pipeline file into text chunks that can be imported into an LLM, use get_chunks.

In the request, you set the maximum number of characters in each chunk.

You can also provide generator_config and generator_default to configure how to present the detected entities in the text chunks.

from textual.enums.pii_state import PiiState

generator_config: Dict[str, PiiState]
generator_default: PiiState
max_chars: int

chunks_list = []
for file in pipeline.enumerate_files():
    chunks = file.get_chunks(max_chars=max_chars, generator_config, generator_default)
    chunks_list.append(chunks)

The response contains the list of text chunks, with the detected entities processed as specified in generator_config and generator_default.

You can use the Tonic Textual SDK to manage pipelines and to parse individual files.

The Tonic Textual SDK is a Python SDK that you can use to manage datasets and get access to redacted files.

To be able to use the Textual SDK, you must have an API key.

Viewing the list of API keys

You manage keys from the User API Keys page.

To display the User API Keys page, in the Textual navigation menu, click User API Keys.

Creating a Textual API key

To create a Textual API key:

1. Either:

   • On the Textual Home page, click Create, then click API Key.

   • In the API keys panel on the dataset details page, click Create an API Key.

   • On the pipeline details page, click the API key creation icon.

   • On the User API Keys page, click Create API Key.

2. In the Name field, type a name to use to identify the key.

1. Click Create API Key.

2. Textual displays the key value, and prompts you to copy the key. If you do not copy the key and save it to a file, you will not have access to the key. To copy the key, click the copy icon.

Revoking a Textual API key

To revoke a Textual API key, on the User API Keys page, click the Revoke option for the key to revoke.

Configuring the API key as an environment setting

You cannot instantiate the SDK client without an API key.

Instead of providing the key every time you call the Textual API, you can configure the API key as the value of the environment variable TONIC_TEXTUAL_API_KEY.

For details about all of the available Tonic Textual classes, go to the generated SDK documentation.

You can use the Tonic Textual SDK to manage pipelines and to redact individual strings and files.

In Tonic Textual, a pipeline identifies a set of files that Textual processes into content that can be imported into an LLM system.

Displaying the list of pipelines

The Textual Home page displays both datasets and pipelines.

To display only the list of pipelines, in the Textual navigation menu, click Pipelines.

The Pipelines page displays the list of pipelines.

If there are no pipelines, then the Pipelines page displays a panel to allow you to create a pipeline.

Displaying details for a pipeline

To display the details for a pipeline, on the Textual Home page or Pipelines page, click the pipeline name.

Details for an Amazon S3 pipeline

Here is a pipeline details page for an Amazon S3 pipeline:

For an Amazon S3 pipeline, the details include:

Details for a Databricks pipeline

Here is a pipeline details page for a Databricks pipeline:

For a Databricks pipeline, the details include:

Details for an Azure pipeline

Here is a pipeline details page for an Azure pipeline:

For an Azure pipeline, the details include:

Details for an uploaded file pipeline

And here is a pipeline details page for an uploaded file pipeline:

The pipeline details include:

The Textual LLM preparation workflow transforms source files into content that you can incorporate into an LLM.

You can:

• Upload files directly from a local file system

• Select files from an S3 bucket

• Select files from a Databricks data volume

• Select files from an Azure Blob Storage container

Textual can process plain text files (.txt and .csv), .docx files, and .xslx files. It can also process PDF files. For images, Textual can extract text from .png, .tif/.tiff, and .jpg/.jpeg files.

At a high level, to use Textual to create LLM-ready content:

3. 1. Provide the credentials to use to connect to the storage location.

   2. Identify the location where Textual writes the pipeline output.

   3. Optionally, filter the files by file type. For example, you might only want to process PDF files.

   4. Identify the files to include in the pipeline. You can select individual files or folders. When you select folders, Textual processes all of the files in the folder.
4. For each file, Textual:

   1. Converts the content to raw text. For image files, this means to extract any text that is present.

   2. Uses its built-in models to detect entity values in the text.

   3. Generates a markdown version of the original text.

   4. Produces a JSON file that contains:

      • The Markdown version of the text

      • The detected entities and their locations

From Textual, for each processed file, you can:

For cloud storage pipelines, the JSON files also are available from the configured output location.

Before you perform these tasks, remember to .

You can use the Tonic Textual SDK to redact individual strings, including:

• Plain text strings

• JSON content

• XML content

For a text string, you can also request synthesized values from a large language model (LLM).

The redaction request can include the .

The includes the redacted or synthesized content and details about the detected entity values.

Redact a plain text string

To send a plain text string for redaction, use :

For example:

Redact JSON content

redact_json ensures that only the values are redacted. It ignores the keys.

Basic JSON redaction example

Here is a basic example of a JSON redaction request:

It produces the following JSON output:

Specifying entity types for specific JSON paths

When you redact a JSON string, you can optionally assign specific entity types to selected JSON paths.

To do this, you include the jsonpath_allow_lists parameter. Each entry consists of an entity type and a list of JSON paths for which to always use that entity type. Each JSON path must point to a simple string or numeric value.

The specified entity type overrides both the detected entity type and any added or excluded values.

In the following example, the value of the key1 node is always treated as a telephone number:

It produces the following redacted output:

Redact XML content

redact_xml ensures that only the values are redacted. It ignores the XML markup.

For example:

Produces the following XML output:

Redact HTML content

redact_html ensures that only the values are redacted. It ignores the HTML markup.

For example:

Produces the following HTML output:

Using an LLM to generate synthesized values

You can also request synthesized values from a large language model (LLM).

When you use this process, Textual first identifies the sensitive values in the text. It then sends the value locations and redacted values to the LLM. For example, if Textual identifies a product name, it sends the location and the redacted value PRODUCT to the LLM. Textual does not send the original values to the LLM.

The LLM then generates realistic synthesized values of the appropriate value types.

For example:

Format of the redaction and synthesis response

The response provides the redacted or synthesized version of the string, and the list of detected entity values.

For each redacted item, the response includes:

• The location of the value in the original text (start and end)

• The location of the value in the redacted version of the string (new_start and new_end)

• The entity type (label)

• The original value (text)

• The redacted or synthesized value (new_text). new_text is null in the following cases:

  • The entity type is ignored

  • The response is from llm_synthesis

• A score to indicate confidence in the detection and redaction (score)

• The detected language for the value (language)

• For responses from textual.redact_json, the JSON path to the entity in the original document (json_path)

• For responses from textual.redact_xml, the Xpath to the entity in the original XML document (xml_path)

By default, when you:

• Configure a dataset

• Redact or synthesize a string

• Retrieve a redacted file

Textual does the following:

• For the string and file redaction, redacts the detected sensitive values.

• For LLM synthesis, generates realistic synthesized values.

When you make the request, you can override the default behavior.

Specifying the handling option for entity types

For each entity type, you can choose to redact, synthesize, or ignore the value.

• When you redact a value, Textual replaces the value with <entity type>_<generated identifier>. For example, ORGANIZATION_EPfC7XZUZ.

• When you synthesize a value, Textual replaces the value with a different realistic value.

• When you ignore a value, Textual passes through the original value.

To specify the handling option for entity types, you use the generator_config parameter.

Where:

• <handling_option> is the handling option to use for the specified entity type. The possible values are Redact, Synthesis, and Off.

For example, to synthesize organization values, and ignore languages:

Specifying a default handling option

For string and file redaction, you can specify a default handling option to use for entity types that are not specified in generator_config.

To do this, you use the generator_default parameter.

generator_default can be either Redact, Synthesis, or Off.

Providing added and excluded values for entity types

You can also configure added and excluded values for each entity type.

You add values that Textual does not detect for an entity type, but should. You exclude values that you do not want Textual to identify for that entity type.

• To specify the added values, use label_allow_lists.

• To specify the excluded values, use label_block_lists.

For each of these parameters, the value is a list of entity types to specify the added or excluded values for. To specify the values, you provide an array of regular expressions.

The following example uses label_allow_lists to add values:

• For NAME_GIVEN, adds the values There and Here.

• For NAME_FAMILY, adds values that match the regular expression ([a-z]{2}).

You can use the Textual SDK to redact and synthesize values in individual files.

Before you perform these tasks, remember to .

For a self-hosted instance, you can also configure the S3 bucket to use to store the files. This is the same S3 bucket that is used to store files for uploaded file pipelines. For more information, go to . For an example of an IAM role with the required permissions, go to .

Sending a file to Textual

To send an individual file to Textual, you use .

You first open the file so that Textual can read it, then make then call for Textual to read the file.

The response includes:

• The file name

• The identifier of the job that processed the file. You use this identifier to retrieve a transformed version of the file.

Getting the file with redacted or synthesized values

After you use to send the file to Textual, you use to retrieve a transformed version of the file.

To identify the file, you use the job identifier that you received from textual.start_file_redaction. You can also specify whether to redact, synthesize, or ignore specific entity types. By default, all of the values are redacted.

Before you make the call to download the file, you specify the path to download the file content to.

For each pipeline, you configure the name and the files to process.

Textual pipelines can process the following types of files:

• txt

• csv

• tsv

• docx

• xlsx

• pdf

• png

• tif or tiff

• jpg or jpeg

• eml

• msg

When Textual processes pipeline files, it produces JSON output that provides access to the Markdown content and that identifies entities that were detected in the file.

Common elements in the JSON output

Information about the entire file

All JSON output files contain the following elements that contain information for the entire file:

{
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

For specific file types, the JSON output includes additional objects and properties to reflect the file structure.

Hashed and Markdown content

The JSON output contains hashed and Markdown content for the entire file and for individual file components.

Entities

The JSON output contains entities arrays for the entire file and for individual file components.

Each entity in the entities array has the following properties:

Plain text files

For plain text files, the JSON output only contains the information for the entire file.

{
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown content>",
    "hash": "<hashed content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.csv files

For .csv files, the structure contains a tables array.

The tables array contains a table object that contains header and data arrays..

For each row in the file, the data array contains a row array.

For each value in a row, the row array contains a value object.

The value object contains the entities, hashed content, and Markdown content for the value.

{
  "tables": [
    {
      "tableName": "csv_table",
      "header": [//Columns that contain heading info (col_0, col_1, and so on)
        "<column identifier>"
      ],
      "data": [  //Entry for each row in the file
        [   //Entry for each value in the row
          {    
            "entities": [   //Entry for each entity in the value
              {
                "start": <start location>,,
                "end": <end location>,
                "label": "<value type>",
                "text": "<value text>",
                "score": <confidence score>,
                "language": "<language code>"
              }
            ],
            "hash": "<hashed value content>",
            "text": "<Markdown value content>"
          }
        ]
      ]
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   ///Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.xlsx files

For .xlsx files, the structure contains a tables array that provides details for each worksheet in the file.

For each worksheet, the tables array contains a worksheet object.

For each row in a worksheet, the worksheet object contains a header array and a data array. The data array contains a row array.

For each cell in a row, the row array contains a cell object.

Each cell object contains the entities, hashed content, and Markdown content for the cell.

{
  "tables": [   //Entry for each worksheet
    {
      "tableName": "<Name of the worksheet>",
      "header": [ //Columns that contain heading info (col_0, col_1, and so on)
        "<column identifier>"
      ],
      "data": [   //Entry for each row
        [   //Entry for each cell in the row
          {
            "entities": [   //Entry for each entity in the cell
              {
                "start": <start location>,
                "end": <end location>,
                "label": "<value type>",
                "text": "<value text>",
                "score": <confidence score>,
                "language": "<language code>"
              }
            ],
            "hash": "<hashed cell content>",
            "text": "<Markdown cell content>"
          }
        ]
      ]
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.docx files

For .docx files, the JSON output structure adds:

• A footnotes array for content in footnotes.

• An endnotes array for content in endnotes.

• A header object for content in the page headers. Includes separate objects for the first page header, even page header, and odd page header.

• A footer object for content in the page footers. Includes separate objects for the first page footer, even page footer, and odd page footer.

These arrays and objects contain the entities, hashed content, and Markdown content for the notes, headers, and footers.

{
  "footNotes": [   //Entry for each footnote
    {
      "entities": [   //Entry for each entity in the footnote
        {
          "start": <start location>,
          "end": <end location>,
          "pythonStart": <start location in Python>,
          "pythonEnd": <end location in Python>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
          "exampleRedaction": null
        }
      ],
      "hash": "<hashed footnote content>",
      "text": "<Markdown footnote content>"
    }
  ],
  "endNotes": [   //Entry for each endnote
    {
      "entities": [   //Entry for each entity in the endnote
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed endnote content>",
      "text": "<Markdown endnote content>"
    }
  ],
  "header": {
    "first": {
      "entities": [   //Entry for each entity in the first page header
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed first page header content>",
      "text": "<Markdown first page header content>"
    },
    "even": {
      "entities": [   //Entry for each entity in the even page header
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed even page header content>",
      "text": "<Markdown even page header content>"
    },
    "odd": {
      "entities": [   //Entry for each entity in the odd page header
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed odd page header content>",
      "text": "<Markdown odd page header content>"
    }
  },
  "footer": {
    "first": {
      "entities": [   //Entry for each entity in the first page footer
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed first page footer content>",
      "text": "<Markdown first page footer content>"
    },
    "even": {
      "entities": [   //Entry for each entity in the even page footer
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed even page footer content>",
      "text": "<Markdown even page footer content>"
    },
    "odd": {
      "entities": [   //Entry for each entity in the odd page footer
        {
          "start": <start location>,
          "end": <end location>,
          "label": "<value type>",
          "text": "<value text>",
          "score": <confidence score>,
          "language": "<language code>"
        }
      ],
      "hash": "<hashed odd page footer content>",
      "text": "<Markdown odd page footer content>"
    }
  },
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

PDF and image files

PDF and image files use the same structure. Textual extracts and scans the text from the files.

For PDF and image files, the JSON output structure adds the following content.

pages array

The pages array contains all of the content on the pages. This includes content in tables and key-value pairs, which are also listed separately in the output.

For each page in the file, the pages array contains a page array.

For each component on the page - such as paragraphs, headings, headers, and footers - the page array contains a component object.

Each component object contains the component entities, hashed content, and Markdown content.

tables array

The tables array contains content that is in tables.

For each table in the file, the tables array contains a table array.

For each row in a table, the table array contains a row array.

For each cell in a row, the row array contains a cell object.

Each cell object identifies the type of cell (header or content). It also contains the entities, hashed content, and Markdown content for the cell.

keyValuePairs array

The keyValuePairs array contains key-value pair content. For example, for a PDF of a form with fields, a key-value pair might represent a field label and field value.

For each key-value pair, the keyValuePairs array contains a key-value pair object.

The key-value pair object contains:

• An automatically incremented identifier. For example, id for the first key-value pair is 1, for the second key-value pair is 2, and so on.

• The start and end position of the key-value pair

• The text of the key

• The entities, hashed content, and Markdown content for the value

PDF and image JSON outline

{
  "pages": [   //Entry for each page in the file
    [   //Entry for each component on the page
      {
        "type": "<page component type>",
        "content": {
          "entities": [   //Entry for each entity in the component
            {
              "start": <start location>,
              "end": <end location>,
              "label": "<value type>",
              "text": "<value text>",
              "score": <confidence score>,
              "language": "<language code>"
            }
          ],
          "hash": "<hashed component content>",
          "text": "<Markdown component content>"
        }
      }
    ],
  "tables": [   //Entry for each table in the file
    [   //Entry for each row in the table
      [   //Entry for each cell in the row
        {
          "type": "<content type>",   //ColumnHeader or Content
          "content": {
            "entities": [  //Entry for each entity in the cell
              {
                "start": <start location>,
                "end": <end location>,
                "label": "<value type>",
                "text": "<value text>",
                "score": <confidence score>,
                "language": "<language code>"
              }
            ],
            "hash": "<hashed cell text>",
            "text": "<Markdown cell text>"
          }
        }
      ]
    ]
  ],
  "keyValuePairs": [   //Entry for each key-value pair in the file
    {
      "id": <incremented identifier>,
      "key": "<key text>",
      "value": {
        "entities": [  //Entry for each entity in the value
          {
            "start": <start location>,
            "end": <end location>,
            "label": "<value type>",
            "text": "<value text>",
            "score": <confidence score>,
            "language": "<language code>"
          }
        ],
        "hash": "<hashed value text>",
        "text": "<Markdown value text>
      },
      "start": <start location of the key-value pair>,
      "end": <end location of the key-value pair>
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [   ///Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

.eml and .msg files

For email message files, the JSON output structure adds the following content.

Email message identifiers

The JSON output includes the following email message identifiers:

• The identifier of the current message

• If the message was a reply to another message, the identifier of that message

• An array of related email messages. This includes the email message that the message replied to, as well as any other messages in an email message thread.

Recipients

The JSON output includes the email address and display name of the message recipients. It contains separate lists for the following:

• Recipients in the To line

• Recipients in the CC line

• Recipients in the BCC line

Subject line

The subject object contains the message subject line. It includes:

• Markdown and hashed versions of the message subject line.

• The entities that were detected in the subject line.

Message timestamp

sentDate provides the timestamp when the message was sent.

Message body

The plainTextBodyContent object contains the body of the email message.

It contains:

• Markdown and hashed versions of the message body.

• The entities that were detected in the message body.

Message attachments

The attachments array provides information about any attachments to the email message. For each attached file, it includes:

• The identifier of the message that the file is attached to.

• The identifier of the attachment.

• The JSON output for the file.

• The count of words in the original file.

• The count of words in the redacted version of the file.

Email message JSON outline

{
  "messageId": "<email message identifier>",
  "inReplyToMessageId": <message that this message replied to>,
  "messageIdReferences": [<related email messages>],
  "senderAddress": {
    "address": "<sender email address>",
    "displayName": "<sender display name>"
  },
  "toAddresses": [  //Entry for each recipient in the To list
    {
      "address": "<recipient email address>",
      "displayName": "<recipient display name>"
    }
  ],
  "ccAddresses": [ //Entry for each recipient in the CC list
    {
      "address": "<recipient email address>",
      "displayName": "<recipient display name>"
    }
  ],
  "bccAddresses": [ //Entry for each recipient in the BCC list
    {
      "address": "<recipient email address>",
      "displayName": "<recipient display name>"
    }
  ],
  "sentDate": "<timestamp when the message was sent>",
  "subject": {
    "text": "<Markdown version of the subject line>",
    "hash": "<hashed version of the subject line>",
    "entities": [   //Entry for each entity in the subject line
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "plainTextBodyContent": {
    "text": "<Markdown version of the message body>",
    "hash": "<hashed version of the message body>",
    "entities": [ //Entry for each entity in the message body
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "attachments": [ //Entry for each attached file
    {
      "parentMessageId": "<the message that the file is attached to>",
      "contentId": "<identifier of the attachment>",
      "fileName": "<name of the attachment file>",
      "document": {<pipeline JSON for the attached file>},
      "wordCount": <number of words in the attachment>,
      "redactedWordCount": <number of words in the redacted attachment>
    }
  ],
  "fileType": "<file type>",
  "content": {
    "text": "<Markdown file content>",
    "hash": "<hashed file content>",
    "entities": [ //Entry for each entity in the file
      {
        "start": <start location>,
        "end": <end location>,
        "label": "<value type>",
        "text": "<value text>",
        "score": <confidence score>,
        "language": "<language code>"
      }
    ]
  },
  "schemaVersion": <integer schema version>
}

For an Azure pipeline, the settings include:

• Azure credentials

• Output location

• Whether to also generate redacted versions of the original files

• Selected files and folders

Changing the Azure credentials for a pipeline

When you create a pipeline that uses files from Azure, you are prompted to provide the credentials to use to connect to Azure.

From the Pipeline Settings page, to change the credentials:

1. Click Update Azure Credentials.

1. Provide the new credentials:

   1. In the Account Name field, provide the name of your Azure account.

   2. In the Account Key field, provide the access key for your Azure account.

2. To test the connection, click Test Azure Connection.

3. To save the new credentials, click Update Azure Credentials.

Selecting a location for the output files

On the Pipeline Settings page, under Select Output Location, navigate to and select the folder in Azure where Textual writes the output files.

When you run a pipeline, Textual creates a folder in the output location. The folder name is the pipeline job identifier.

Within the job folder, Textual recreates the folder structure for the original files. It then creates the JSON output for each file. The name of the JSON file is <original filename>_<original extension>_parsed.json.

If the pipeline is also configured to generate redacted versions of the files, then Textual writes the redacted version of each file to the same location.

For example, for the original file Transaction1.txt, the output for a pipeline run contains:

• Transaction1_txt_parsed.json

• Transaction.txt

Indicating whether to also redact the files

By default, when you run an Azure pipeline, Textual only generates the JSON output.

To also generate versions of the original files that redact or synthesize the detected entity values, toggle Synthesize Files to the on position.

For information on how to configure the file generation, go to Configuring file synthesis for a pipeline.

Filtering files in selected folders by file type

One option for selected folders is to filter the processed files based on the file extension. For example, in a selected folder, you might only want to process .txt and .csv files.

Under File Processing Settings, select the file extensions to include. To add a file type, select it from the dropdown list. To remove a file type, click its delete icon.

Note that this filter does not apply to individually selected files. Textual always processes those files regardless of file type.

Selecting files and folders to process

Under Select files and folders to add to run, navigate to and select the folders and individual files to process.

To add a folder or file to the pipeline, check its checkbox.

When you check a folder checkbox, Textual adds it to the Prefix Patterns list. It processes all of the applicable files in the folder, based on whether the file type is a type that Textual supports and whether it is included in the file type filter.

When you click the folder name, it displays the folder contents.

When you select an individual file, Textual adds it to the Selected Files list.

To delete a file or folder, either:

• In the navigation pane, uncheck the checkbox.

• In the Prefix Patterns or Selected Files list, click its delete icon.

For a Databricks pipeline, the settings include:

• Databricks credentials

• Output location

• Whether to also generate redacted versions of the original files

• Selected files and folders

Changing the Databricks credentials for a pipeline

When you create a pipeline that uses files from Databricks, you are prompted to provide the credentials to use to connect to Databricks.

From the Pipeline Settings page, to change the credentials: