Manage datasets

Use the Tonic Textual REST API to create and manage datasets.

Get all datasets

get

Returns all of the datasets that the user has access to. Each dataset contains a set of minimal info, for more detailed information on a given dataset use the /api/dataset/{datasetId} endpoint.

Path parameters

filterSynthesizedbooleanRequired

When true, excludes synthesized datasets from results

Default: true

Responses

200

Returns all accessible datasets

application/json

Lightweight dataset summary containing only core metadata used in list views and search results.

idstringOptional

namestringOptional

fileSourcestring · enumOptional

The original upload location of source files (Local, S3, Azure, SharePoint, OneLake, or SDK).

Possible values:

tagsstring[]Optional

lastUpdatedobject · InstantOptional

A point in time represented as an ISO 8601 timestamp string.

createdobject · InstantOptional

A point in time represented as an ISO 8601 timestamp string.

fileCountinteger · int32Optional

get/api/Dataset

HTTPcURLJavaScriptPython

GET /api/Dataset HTTP/1.1
Accept: */*

200

Returns all accessible datasets

[
  {
    "id": "text",
    "name": "text",
    "fileSource": "Local",
    "operations": [
      "HasAccess"
    ],
    "tags": [
      "text"
    ],
    "lastUpdated": {},
    "created": {},
    "creatorUser": {
      "id": "text",
      "userName": "text",
      "firstName": "text",
      "lastName": "text"
    },
    "fileCount": 1
  }
]

Gets the dataset by its Id

get

Returns the dataset specified by the datasetId

Path parameters

datasetIdstringRequired

Responses

200

OK

application/json

idstringOptional

namestringOptional

enabledModelsstring[]Optional

lastUpdatedobject · InstantOptional

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

fileSourcestring · enumOptionalPossible values:

customPiiEntityIdsstring[]Optional

404

The dataset cannot be found

get/api/Dataset/{datasetId}

HTTPcURLJavaScriptPython

GET /api/Dataset/{datasetId} HTTP/1.1
Accept: */*

{
  "id": "text",
  "name": "text",
  "datasetGeneratorMetadata": "asdfqwer",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "labelBlockLists": "{\"NAME_FAMILY\": {\"strings\":[],\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{ \"HEALTHCARE_ID\": {\"strings\":[],\"regexes\":[\"[a-z]{2}\\\\d{9}\"]} }",
  "enabledModels": [
    "text"
  ],
  "files": [
    {
      "fileId": "text",
      "fileName": "text",
      "fileType": "text",
      "datasetId": "text",
      "numRows": 1,
      "numColumns": 1,
      "piiTypes": [
        "text"
      ],
      "wordCount": 1,
      "redactedWordCount": 1,
      "uploadedTimestamp": {},
      "fileSource": "Local",
      "processingStatus": "text",
      "processingError": "text",
      "mostRecentCompletedJobId": "text"
    }
  ],
  "lastUpdated": {},
  "docXImagePolicy": "Redact",
  "pdfSignaturePolicy": "Redact",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "fileSource": "Local",
  "customPiiEntityIds": [
    "text"
  ],
  "rescanJobs": [
    {
      "id": "text",
      "status": "text",
      "errorMessages": "text",
      "startTime": {},
      "endTime": {},
      "publishedTime": {},
      "datasetFileId": "text",
      "jobType": "DeidentifyFile"
    }
  ]
}

Creates a new dataset

post

Creates a new dataset with the specified configuration. You must specify a unique, non-empty dataset name

Body

application/json

namestringOptional

Responses

200

OK

application/json

idstringOptional

namestringOptional

enabledModelsstring[]Optional

lastUpdatedobject · InstantOptional

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

fileSourcestring · enumOptionalPossible values:

customPiiEntityIdsstring[]Optional

400

The dataset name must be specified

409

Dataset name is already in use

post/api/Dataset

HTTPcURLJavaScriptPython

POST /api/Dataset HTTP/1.1
Content-Type: application/json
Accept: */*
Content-Length: 15

{
  "name": "text"
}

{
  "id": "text",
  "name": "text",
  "datasetGeneratorMetadata": "asdfqwer",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "labelBlockLists": "{\"NAME_FAMILY\": {\"strings\":[],\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{ \"HEALTHCARE_ID\": {\"strings\":[],\"regexes\":[\"[a-z]{2}\\\\d{9}\"]} }",
  "enabledModels": [
    "text"
  ],
  "files": [
    {
      "fileId": "text",
      "fileName": "text",
      "fileType": "text",
      "datasetId": "text",
      "numRows": 1,
      "numColumns": 1,
      "piiTypes": [
        "text"
      ],
      "wordCount": 1,
      "redactedWordCount": 1,
      "uploadedTimestamp": {},
      "fileSource": "Local",
      "processingStatus": "text",
      "processingError": "text",
      "mostRecentCompletedJobId": "text"
    }
  ],
  "lastUpdated": {},
  "docXImagePolicy": "Redact",
  "pdfSignaturePolicy": "Redact",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "fileSource": "Local",
  "customPiiEntityIds": [
    "text"
  ],
  "rescanJobs": [
    {
      "id": "text",
      "status": "text",
      "errorMessages": "text",
      "startTime": {},
      "endTime": {},
      "publishedTime": {},
      "datasetFileId": "text",
      "jobType": "DeidentifyFile"
    }
  ]
}

Edit a dataset

put

Updates a dataset to use the specified configuration.

Query parameters

shouldRescanbooleanOptional

When true, triggers a rescan of dataset files after the update

Body

application/json

Request to update an existing dataset's configuration, redaction policies, and entity settings.

idstringOptional

The dataset to edit

namestringOptional

The new name of the dataset

generatorSetupobjectOptional

A dictionary mapping entity type names to their transformation type (e.g. Redaction, Synthesis, Off).

Example: {"NAME_GIVEN":"Redaction","NAME_FAMILY":"Redaction"}

generatorMetadataobjectOptional

A dictionary mapping entity type names to their transformation metadata configuration.

labelBlockListsobjectOptional

The new excluded entity value list. Legacy strings inputs are accepted for backward compatibility and normalized into regexes.

Example: {"NAME_FAMILY":{"regexes":[".*\\s(disease|syndrome|disorder)"]}}

labelAllowListsobjectOptional

The new added entity value list. Legacy strings inputs are accepted for backward compatibility and normalized into regexes.

Example: {"HEALTHCARE_ID":{"regexes":["[a-z]{2}\\d{9}"]}}

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

rtfImagePolicystring · enumOptional

Possible values:

• Ignore: Leave images alone
• Remove: Replace each image with an opaque black box of the same size

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

pdfSynthModePolicystring · enumOptional

Possible values:

• V1: Original mode with incorrect font, size and style
• V2: Legacy style-aware mode. Persisted V2 values now use the V5 font detector.
• V5: Style-aware mode backed by the V5 OpenVINO font model

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

llmClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

llmTableClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

outputPathstringOptional

The Location to store dataset files. Not applicable to local datasets

ocrServiceProviderstring · enumOptional

The OCR engine used for text extraction from images, scanned documents, and PDFs

Possible values:

Responses

200

Returns the updated dataset

application/jsontext/plaintext/json

Full dataset details including files, configuration, and permission information.

idstringOptional

namestringOptional

generatorMetadataobjectOptional

A dictionary mapping entity type names to their transformation metadata configuration.

outputFormatstring · enumOptional

The output format for redacted files: Original preserves the source format, Markdown produces a markdown version.

Possible values:

generatorSetupobjectOptional

A dictionary mapping entity type names to their transformation type (e.g. Redaction, Synthesis, Off).

Example: {"NAME_GIVEN":"Redaction", "NAME_FAMILY":"Redaction"}

labelBlockListsobjectOptionalExample: {"NAME_FAMILY":{"regexes":[".*\\s(disease|syndrom|disorder)"]}}

labelAllowListsobjectOptionalExample: {"HEALTHCARE_ID":{"regexes":["[a-z]{2}\\d{9}"]}}

tagsstring[]Optional

lastUpdatedobject · InstantOptional

A point in time represented as an ISO 8601 timestamp string.

createdobject · InstantOptional

A point in time represented as an ISO 8601 timestamp string.

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

rtfImagePolicystring · enumOptional

Possible values:

• Ignore: Leave images alone
• Remove: Replace each image with an opaque black box of the same size

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

pdfSynthModePolicystring · enumOptional

Possible values:

• V1: Original mode with incorrect font, size and style
• V2: Legacy style-aware mode. Persisted V2 values now use the V5 font detector.
• V5: Style-aware mode backed by the V5 OpenVINO font model

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

llmClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

llmTableClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

fileSourcestring · enumOptional

The original upload location of source files (Local, S3, Azure, SharePoint, OneLake, or SDK).

Possible values:

customPiiEntityIdsstring[]Optional

awsCredentialSourcestringOptional

outputPathstringOptional

ocrServiceProviderstring · enumOptional

The OCR engine used for text extraction from images, scanned documents, and PDFs

Possible values:

fileCountinteger · int32Optional

404

The dataset cannot be found

application/jsontext/plaintext/json

409

Dataset name is already in use

application/jsontext/plaintext/json

put/api/Dataset

HTTPcURLJavaScriptPython

PUT /api/Dataset HTTP/1.1
Content-Type: application/json
Accept: */*
Content-Length: 671

{
  "id": "text",
  "name": "text",
  "generatorSetup": {
    "NAME_GIVEN": "Redaction",
    "NAME_FAMILY": "Redaction"
  },
  "generatorMetadata": {},
  "labelBlockLists": {
    "NAME_FAMILY": {
      "regexes": [
        ".*\\s(disease|syndrome|disorder)"
      ]
    }
  },
  "labelAllowLists": {
    "HEALTHCARE_ID": {
      "regexes": [
        "[a-z]{2}\\d{9}"
      ]
    }
  },
  "docXImagePolicy": "Redact",
  "rtfImagePolicy": "Ignore",
  "pdfSignaturePolicy": "Redact",
  "pdfSynthModePolicy": "V1",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "llmClassificationPolicy": "Disabled",
  "llmTableClassificationPolicy": "Disabled",
  "fileSourceExternalCredential": {
    "fileSource": "Local",
    "awsCredentialsSource": "UserProvided",
    "credential": {}
  },
  "outputPath": "text",
  "ocrServiceProvider": "Azure"
}

{
  "id": "text",
  "name": "text",
  "generatorMetadata": {},
  "outputFormat": "Original",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "labelBlockLists": "{\"NAME_FAMILY\":{\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{\"HEALTHCARE_ID\":{\"regexes\":[\"[a-z]{2}\\\\d{9}\"]}}",
  "tags": [
    "text"
  ],
  "files": [
    {
      "fileId": "text",
      "fileName": "text",
      "fileType": "text",
      "datasetId": "text",
      "numRows": 1,
      "numColumns": 1,
      "piiTypes": [
        "text"
      ],
      "wordCount": 1,
      "redactedWordCount": 1,
      "uploadedTimestamp": {},
      "fileSource": "Local",
      "processingStatus": "text",
      "processingError": "text",
      "mostRecentCompletedJobId": "text",
      "fileParseResultId": "text",
      "filePath": "text",
      "generatedFileStatus": "text"
    }
  ],
  "lastUpdated": {},
  "created": {},
  "creatorUser": {
    "id": "text",
    "userName": "text",
    "firstName": "text",
    "lastName": "text"
  },
  "docXImagePolicy": "Redact",
  "rtfImagePolicy": "Ignore",
  "pdfSignaturePolicy": "Redact",
  "pdfSynthModePolicy": "V1",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "llmClassificationPolicy": "Disabled",
  "llmTableClassificationPolicy": "Disabled",
  "fileSource": "Local",
  "customPiiEntityIds": [
    "text"
  ],
  "operations": [
    "HasAccess"
  ],
  "rescanJobs": [
    {
      "id": "text",
      "status": "text",
      "errorMessages": "text",
      "startTime": {},
      "endTime": {},
      "publishedTime": {},
      "datasetFileId": "text",
      "datasetId": "text",
      "jobType": "DeidentifyFile",
      "ocrServiceProvider": "Azure",
      "modelsInfo": {
        "dateSynthesis": {
          "cpuVariant": "text",
          "modelName": "text",
          "runsOnGpu": true
        },
        "fasttext": {
          "libVersion": "text",
          "model": "text",
          "runsOnGpu": true
        },
        "image": {
          "version": "text"
        },
        "spacy": {
          "libVersion": "text",
          "auxModel": {
            "name": "text",
            "language": "text",
            "runsOnGpu": true,
            "version": "text"
          },
          "multilingualModels": [
            {
              "name": "text",
              "language": "text",
              "runsOnGpu": true,
              "version": "text"
            }
          ]
        },
        "torch": {
          "gpuAvailable": true,
          "libVersion": "text"
        },
        "tonicNer": {
          "enModel": "text",
          "xlmModel": "text"
        },
        "tesseract": {
          "model": {
            "id": "text",
            "version": "text"
          }
        }
      },
      "pdfSynthModePolicy": "V1"
    }
  ],
  "mostRecentExternalFileGenerationJob": {
    "id": "text",
    "status": "text",
    "errorMessages": "text",
    "startTime": {},
    "endTime": {},
    "publishedTime": {},
    "datasetFileId": "text",
    "datasetId": "text",
    "jobType": "DeidentifyFile",
    "ocrServiceProvider": "Azure",
    "modelsInfo": {
      "dateSynthesis": {
        "cpuVariant": "text",
        "modelName": "text",
        "runsOnGpu": true
      },
      "fasttext": {
        "libVersion": "text",
        "model": "text",
        "runsOnGpu": true
      },
      "image": {
        "version": "text"
      },
      "spacy": {
        "libVersion": "text",
        "auxModel": {
          "name": "text",
          "language": "text",
          "runsOnGpu": true,
          "version": "text"
        },
        "multilingualModels": [
          {
            "name": "text",
            "language": "text",
            "runsOnGpu": true,
            "version": "text"
          }
        ]
      },
      "torch": {
        "gpuAvailable": true,
        "libVersion": "text"
      },
      "tonicNer": {
        "enModel": "text",
        "xlmModel": "text"
      },
      "tesseract": {
        "model": {
          "id": "text",
          "version": "text"
        }
      }
    },
    "pdfSynthModePolicy": "V1"
  },
  "fileSourceExternalCredential": {
    "fileSource": "Local",
    "awsCredentialsSource": "UserProvided",
    "credential": {}
  },
  "awsCredentialSource": "text",
  "outputPath": "text",
  "externalFilesInfo": {
    "selectedFiles": [
      "text"
    ],
    "pathPrefixes": [
      "text"
    ],
    "selectedFileExtensions": [
      "text"
    ]
  },
  "ocrServiceProvider": "Azure",
  "fileCount": 1
}