For a Databricks pipeline, the settings include:

• Databricks credentials

• Output location

• Whether to also generate redacted versions of the original files

• Selected files and folders

Changing the Databricks credentials for a pipeline

When you create a pipeline that uses files from Databricks, you are prompted to provide the credentials to use to connect to Databricks.

From the Pipeline Settings page, to change the credentials:

1. Click Update Databricks Credentials.

1. Provide the new credentials:

   1. In the Databricks URL field, provide the URL to the Databricks workspace.

   2. In the Access Token field, provide the access token to use to get access to the volume.

2. To test the connection, click Test Databricks Connection.

3. To save the new credentials, click Update Databricks Credentials.

Selecting a location for the output files

On the Pipeline Settings page, under Select Output Location, navigate to and select the folder in Databricks where Textual writes the output files.

When you run a pipeline, Textual creates a folder in the output location. The folder name is the pipeline job identifier.

Within the job folder, Textual recreates the folder structure for the original files. It then creates the JSON output for each file. The name of the JSON file is <original filename>_<original extension>_parsed.json.

If the pipeline is also configured to generate redacted versions of the files, then Textual writes the redacted version of each file to the same location.

For example, for the original file Transaction1.txt, the output for a pipeline run contains:

• Transaction1_txt_parsed.json

• Transaction.txt

Indicating whether to also redact the files

By default, when you run a Databricks pipeline, Textual only generates the JSON output.

To also generate versions of the original files that redact or synthesize the detected entity values, toggle Synthesize Files to the on position.

For information on how to configure the file generation, go to Configuring file synthesis for a pipeline.

Filtering files in selected folders by file type

One option for selected folders is to filter the processed files based on the file extension. For example, in a selected folder, you might only want to process .txt and .csv files.

Under File Processing Settings, select the file extensions to include. To add a file type, select it from the dropdown list. To remove a file type, click its delete icon.

Note that this filter does not apply to individually selected files. Textual always processes those files regardless of file type.

Selecting files and folders to process

Under Select files and folders to add to run, navigate to and select the folders and individual files to process.

To add a folder or file to the pipeline, check its checkbox.

When you check a folder checkbox, Textual adds it to the Prefix Patterns list. It processes all of the applicable files in the folder, based on whether the file type is a type that Textual supports and whether it is included in the file type filter.

When you click the folder name, it displays the folder contents.

When you select an individual file, Textual adds it to the Selected Files list.

To delete a file or folder, either:

• In the navigation pane, uncheck the checkbox.

• In the Prefix Patterns or Selected Files list, click its delete icon.

Creating a pipeline

To create a pipeline, either:

• On the Pipelines page, click Create a New Pipeline.

• On the Home page, click Create, then click Pipeline.

Setting the pipeline name and source type

On the Create A New Pipeline panel:

1. In the Name field, type the name of the pipeline.

2. Under Files Source, select the location of the source files.

   • To upload files from a local file system, click File upload, then click Save.

   • To select files from and write output to Amazon S3, click Amazon S3.

   • To select files from and write output to Databricks, click Databricks.

   • To select files from and write output to Azure Blob Storage, click Azure.

Providing Amazon S3 credentials

If you selected Amazon S3, provide the credentials to use to connect to Amazon S3.

2. In the Access Secret field, provide the secret key that is associated with the access key.

3. From the Region dropdown list, select the AWS Region to send the authentication request to.

4. In the Session Token field, provide the session token to use for the authentication request.

5. To test the credentials, click Test AWS Connection.

6. Click Save.

8. Click Save.

Providing Databricks connection information

If you selected Databricks, provide the connection information:

1. In the Databricks URL field, provide the URL to the Databricks workspace.

2. In the Access Token field, provide the access token to use to get access to the volume.

3. To test the connection, click Test Databricks Connection.

4. Click Save.

6. Click Save.

Providing Azure credentials

If you selected Azure, provide the connection information:

1. In the Account Name field, provide the name of your Azure account.

2. In the Account Key field, provide the access key for your Azure account.

3. To test the connection, click Test Azure Connection.

4. Click Save.

6. Click Save.

Editing a pipeline

To update a pipeline configuration:

1. Either:

   • On the Pipelines page, click the pipeline options menu, then click Settings.

   • On the pipeline details page, click the settings icon. For cloud storage pipelines, the settings icon is next to the Run Pipeline option. For uploaded file pipelines, the settings icon is next to the Upload Files option.

3. Click Save.

Deleting a pipeline

To delete a pipeline, on the Pipeline Settings page, click Delete Pipeline.

Textual pipelines can process the following types of files:

• txt

• csv

• tsv

• docx

• xlsx

• pdf

• png

• tif or tiff

• jpg or jpeg

• eml

• msg

For each pipeline, you configure the name and the files to process.

For an Amazon S3 pipeline, the settings include:

• AWS credentials

• Output location

• Whether to also generate redacted versions of the original files

• Selected files and folders

Changing the AWS credentials for a pipeline

When you create a pipeline that uses files from Amazon S3, you are prompted to provide the credentials to use to connect to Amazon S3.

From the Pipeline Settings page, to change the credentials:

1. Click Update AWS Credentials.

1. Provide the new credentials:

   1. In the Access Key field, provide an AWS access key that is associated with an IAM user or role. For an example IAM role that has the required permissions for an Amazon S3 pipeline, go to #amazon-s3-example-iam-role.

   2. In the Access Secret field, provide the secret key that is associated with the access key.

   3. From the Region dropdown list, select the AWS Region to send the authentication request to.

   4. In the Session Token field, provide the session token to use for the authentication request.

2. To test the connection, click Test AWS Connection.

3. To save the new credentials, click Update AWS Credentials.

Selecting a location for the output files

On the Pipeline Settings page, under Select Output Location, navigate to and select the folder in Amazon S3 where Textual writes the output files.

When you run a pipeline, Textual creates a folder in the output location. The folder name is the pipeline job identifier.

Within the job folder, Textual recreates the folder structure for the original files. It then creates the JSON output for each file. The name of the JSON file is <original filename>_<original extension>_parsed.json.

If the pipeline is also configured to generate redacted versions of the files, then Textual writes the redacted version of each file to the same location.

For example, for the original file Transaction1.txt, the output for a pipeline run contains:

• Transaction1_txt_parsed.json

• Transaction.txt

Indicating whether to also redact the files

By default, when you run an Amazon S3 pipeline, Textual only generates the JSON output.

To also generate versions of the original files that redact or synthesize the detected entity values, toggle Synthesize Files to the on position.

For information on how to configure the file generation, go to Configuring file synthesis for a pipeline.

Filtering files in selected folders by file type

One option for selected folders is to filter the processed files based on the file extension. For example, in a selected folder, you might only want to process .txt and .csv files.

Under File Processing Settings, select the file extensions to include. To add a file type, select it from the dropdown list. To remove a file type, click its delete icon.

Note that this filter does not apply to individually selected files. Textual always processes those files regardless of file type.

Selecting files and folders to process

Under Select files and folders to add to run, navigate to and select the folders and individual files to process.

To add a folder or file to the pipeline, check its checkbox.

When you check a folder checkbox, Textual adds it to the Prefix Patterns list. It processes all of the applicable files in the folder, based on whether the file type is a type that Textual supports and whether it is included in the file type filter.

When you click the folder name, it displays the folder contents.

When you select an individual file, Textual adds it to the Selected Files list.

To delete a file or folder, either:

• In the navigation pane, uncheck the checkbox.

• In the Prefix Patterns or Selected Files list, click its delete icon.

For an Azure pipeline, the settings include:

• Azure credentials

• Output location

• Whether to also generate redacted versions of the original files

• Selected files and folders

Changing the Azure credentials for a pipeline

When you create a pipeline that uses files from Azure, you are prompted to provide the credentials to use to connect to Azure.

From the Pipeline Settings page, to change the credentials:

1. Click Update Azure Credentials.

1. Provide the new credentials:

   1. In the Account Name field, provide the name of your Azure account.

   2. In the Account Key field, provide the access key for your Azure account.

2. To test the connection, click Test Azure Connection.

3. To save the new credentials, click Update Azure Credentials.

Selecting a location for the output files

On the Pipeline Settings page, under Select Output Location, navigate to and select the folder in Azure where Textual writes the output files.

When you run a pipeline, Textual creates a folder in the output location. The folder name is the pipeline job identifier.

Within the job folder, Textual recreates the folder structure for the original files. It then creates the JSON output for each file. The name of the JSON file is <original filename>_<original extension>_parsed.json.

If the pipeline is also configured to generate redacted versions of the files, then Textual writes the redacted version of each file to the same location.

For example, for the original file Transaction1.txt, the output for a pipeline run contains:

• Transaction1_txt_parsed.json

• Transaction.txt

Indicating whether to also redact the files

By default, when you run an Azure pipeline, Textual only generates the JSON output.

To also generate versions of the original files that redact or synthesize the detected entity values, toggle Synthesize Files to the on position.

For information on how to configure the file generation, go to Configuring file synthesis for a pipeline.

Filtering files in selected folders by file type

One option for selected folders is to filter the processed files based on the file extension. For example, in a selected folder, you might only want to process .txt and .csv files.

Under File Processing Settings, select the file extensions to include. To add a file type, select it from the dropdown list. To remove a file type, click its delete icon.

Note that this filter does not apply to individually selected files. Textual always processes those files regardless of file type.

Selecting files and folders to process

Under Select files and folders to add to run, navigate to and select the folders and individual files to process.

To add a folder or file to the pipeline, check its checkbox.

When you check a folder checkbox, Textual adds it to the Prefix Patterns list. It processes all of the applicable files in the folder, based on whether the file type is a type that Textual supports and whether it is included in the file type filter.

When you click the folder name, it displays the folder contents.

When you select an individual file, Textual adds it to the Selected Files list.

To delete a file or folder, either:

• In the navigation pane, uncheck the checkbox.

• In the Prefix Patterns or Selected Files list, click its delete icon.

On a self-hosted instance, before you can upload files to a pipeline, you must configure the S3 bucket where Tonic Textual stores the files. For more information, go to .

For an example IAM role that has the required permissions for file upload pipelines, go to .

Adding files to the pipeline

On the pipeline details page for an uploaded file pipeline, to add files to the pipeline:

1. Click Upload Files.

2. Search for and select the files to upload.

Removing files

To remove a file, on the pipeline details page, click the delete icon for the file.

Indicating whether to also redact the files

By default, Textual only generates the JSON output for the pipeline files.

To also generate versions of the original files that redact or synthesize the detected entity values, on the Pipeline Settings page, toggle Synthesize Files to the on position.

When you choose to also generate synthesized versions of the pipeline files, the pipeline details page includes a Generator Config tab. From the Generator Config tab, you configure how to transform the detected entities in each file.

The Generator Config tab lists all of the available entity types.

For each entity type, you select and configure the handling type. For more information, see Selecting the handling option for the entity types and Configuring synthesis options.

After you change the configuration, click Save Changes. The updated configuration is applied the next time you run the pipeline, and only to new files.