Manage datasets

Use the REST API to create and manage datasets.

Get all datasets

get

Returns all of the datasets that the user has access to. Each dataset contains a set of minimal info, for more detailed information on a given dataset use the /api/dataset/{datasetId} endpoint.

Path parameters

filterSynthesizedbooleanRequired

When true, excludes synthesized datasets from results

Default: true

Responses

200

OK

application/json

Lightweight dataset summary containing only core metadata used in list views and search results.

idstringOptional

namestringOptional

fileSourcestring · enumOptional

The original upload location of source files (Local, S3, Azure, SharePoint, OneLake, or SDK).

Possible values:

tagsstring[]Optional

lastUpdatedobjectOptional

A point in time represented as an ISO 8601 timestamp string.

createdobjectOptional

A point in time represented as an ISO 8601 timestamp string.

fileCountinteger · int32Optional

get

/api/Dataset

HTTPcURLJavaScriptPython

GET /api/Dataset HTTP/1.1
Accept: */*

200

OK

[
  {
    "id": "text",
    "name": "text",
    "fileSource": "Local",
    "operations": [
      "HasAccess"
    ],
    "tags": [
      "text"
    ],
    "lastUpdated": {},
    "created": {},
    "creatorUser": {
      "id": "text",
      "userName": "text",
      "firstName": "text",
      "lastName": "text"
    },
    "fileCount": 1
  }
]

Gets the dataset by its Id

get

Returns the dataset specified by the datasetId

Path parameters

datasetIdstringRequired

Responses

200

OK

application/json

idstringOptional

namestringOptional

enabledModelsstring[]Optional

lastUpdatedobjectOptional

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

fileSourcestring · enumOptionalPossible values:

customPiiEntityIdsstring[]Optional

404

The dataset cannot be found

get

/api/Dataset/{datasetId}

HTTPcURLJavaScriptPython

GET /api/Dataset/{datasetId} HTTP/1.1
Accept: */*

{
  "id": "text",
  "name": "text",
  "datasetGeneratorMetadata": "asdfqwer",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "labelBlockLists": "{\"NAME_FAMILY\": {\"strings\":[],\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{ \"HEALTHCARE_ID\": {\"strings\":[],\"regexes\":[\"[a-z]{2}\\\\d{9}\"]} }",
  "enabledModels": [
    "text"
  ],
  "files": [
    {
      "fileId": "text",
      "fileName": "text",
      "fileType": "text",
      "datasetId": "text",
      "numRows": 1,
      "numColumns": 1,
      "piiTypes": [
        "text"
      ],
      "wordCount": 1,
      "redactedWordCount": 1,
      "uploadedTimestamp": {},
      "fileSource": "Local",
      "processingStatus": "text",
      "processingError": "text",
      "mostRecentCompletedJobId": "text"
    }
  ],
  "lastUpdated": {},
  "docXImagePolicy": "Redact",
  "pdfSignaturePolicy": "Redact",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "fileSource": "Local",
  "customPiiEntityIds": [
    "text"
  ],
  "rescanJobs": [
    {
      "id": "text",
      "status": "text",
      "errorMessages": "text",
      "startTime": {},
      "endTime": {},
      "publishedTime": {},
      "datasetFileId": "text",
      "jobType": "DeidentifyFile"
    }
  ]
}

Creates a new dataset

post

Creates a new dataset with the specified configuration. You must specify a unique, non-empty dataset name

Body

application/json

namestringOptional

Responses

200

OK

application/json

idstringOptional

namestringOptional

enabledModelsstring[]Optional

lastUpdatedobjectOptional

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

fileSourcestring · enumOptionalPossible values:

customPiiEntityIdsstring[]Optional

400

The dataset name must be specified

409

Dataset name is already in use

post

/api/Dataset

HTTPcURLJavaScriptPython

POST /api/Dataset HTTP/1.1
Content-Type: application/json
Accept: */*
Content-Length: 15

{
  "name": "text"
}

{
  "id": "text",
  "name": "text",
  "datasetGeneratorMetadata": "asdfqwer",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "labelBlockLists": "{\"NAME_FAMILY\": {\"strings\":[],\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{ \"HEALTHCARE_ID\": {\"strings\":[],\"regexes\":[\"[a-z]{2}\\\\d{9}\"]} }",
  "enabledModels": [
    "text"
  ],
  "files": [
    {
      "fileId": "text",
      "fileName": "text",
      "fileType": "text",
      "datasetId": "text",
      "numRows": 1,
      "numColumns": 1,
      "piiTypes": [
        "text"
      ],
      "wordCount": 1,
      "redactedWordCount": 1,
      "uploadedTimestamp": {},
      "fileSource": "Local",
      "processingStatus": "text",
      "processingError": "text",
      "mostRecentCompletedJobId": "text"
    }
  ],
  "lastUpdated": {},
  "docXImagePolicy": "Redact",
  "pdfSignaturePolicy": "Redact",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "fileSource": "Local",
  "customPiiEntityIds": [
    "text"
  ],
  "rescanJobs": [
    {
      "id": "text",
      "status": "text",
      "errorMessages": "text",
      "startTime": {},
      "endTime": {},
      "publishedTime": {},
      "datasetFileId": "text",
      "jobType": "DeidentifyFile"
    }
  ]
}

Edit a dataset

put

Updates a dataset to use the specified configuration.

Query parameters

shouldRescanbooleanOptional

When true, triggers a rescan of dataset files after the update

Body

application/json

Request to update an existing dataset's configuration, redaction policies, and entity settings.

idstringOptional

namestringOptional

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

pdfSynthModePolicystring · enumOptional

Possible values:

• V1: Original mode with incorrect font, size and style
• V2: New mode

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

llmClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

llmTableClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

awsCredentialSourcestringOptional

outputPathstring · nullableOptional

ocrServiceProviderstring · enumOptional

The OCR engine used for text extraction from images and scanned documents.

Possible values:

Responses

200

OK

application/json

Full dataset details including files, configuration, and permission information.

idstringOptional

namestringOptional

outputFormatstring · enumOptional

The output format for redacted files: Original preserves the source format, Markdown produces a markdown version.

Possible values:

tagsstring[]Optional

lastUpdatedobjectOptional

A point in time represented as an ISO 8601 timestamp string.

createdobjectOptional

A point in time represented as an ISO 8601 timestamp string.

docXImagePolicystring · enumOptional

Possible values:

• Redact: Run images through OCR and redact sensitive text
• Ignore: Leave images alone
• Remove: Cover image with opaque black box

Possible values:

pdfSignaturePolicystring · enumOptional

Possible values:

• Redact: Cover signature with opaque black box
• Ignore: Do not attempt to detect signature

Possible values:

pdfSynthModePolicystring · enumOptional

Possible values:

• V1: Original mode with incorrect font, size and style
• V2: New mode

Possible values:

docXCommentPolicystring · enumOptional

Possible values:

• Remove: Remove all comments for file
• Ignore: Leave comments alone

Possible values:

docXTablePolicystring · enumOptional

Possible values:

• Redact: Treat table content normally, feed into redaction process.
• Remove: Replace all characters and symbols in table with a placeholder.

Possible values:

llmClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

llmTableClassificationPolicystring · enumOptional

Possible values:

• Disabled: Do not use LLM for structured data classification
• Enabled: Use LLM to classify structured data for PII detection

Possible values:

fileSourcestring · enumOptional

The original upload location of source files (Local, S3, Azure, SharePoint, OneLake, or SDK).

Possible values:

customPiiEntityIdsstring[] · nullableOptional

awsCredentialSourcestring · nullableOptional

outputPathstring · nullableOptional

ocrServiceProviderstring · enumOptional

The OCR engine used for text extraction from images and scanned documents.

Possible values:

fileCountinteger · int32Optional

404

The dataset cannot be found

text/plainapplication/jsontext/json

409

Dataset name is already in use

text/plainapplication/jsontext/json

put

/api/Dataset

HTTPcURLJavaScriptPython

PUT /api/Dataset HTTP/1.1
Content-Type: application/json
Accept: */*
Content-Length: 814

{
  "id": "text",
  "name": "text",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "generatorMetadata": {
    "ANY_ADDITIONAL_PROPERTY": {
      "version": "V1",
      "customGenerator": "Scramble",
      "swaps": {
        "ANY_ADDITIONAL_PROPERTY": "text"
      }
    }
  },
  "labelBlockLists": "{\"NAME_FAMILY\": {\"strings\":[],\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{ \"HEALTHCARE_ID\": {\"strings\":[],\"regexes\":[\"[a-z]{2}\\\\d{9}\"]} }",
  "docXImagePolicy": "Redact",
  "pdfSignaturePolicy": "Redact",
  "pdfSynthModePolicy": "V1",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "llmClassificationPolicy": "Disabled",
  "llmTableClassificationPolicy": "Disabled",
  "fileSourceExternalCredential": {
    "fileSource": "Local",
    "credential": {}
  },
  "awsCredentialSource": "text",
  "outputPath": "text",
  "ocrServiceProvider": "Azure"
}

{
  "id": "text",
  "name": "text",
  "generatorMetadata": {
    "ANY_ADDITIONAL_PROPERTY": {
      "version": "V1",
      "customGenerator": "Scramble",
      "swaps": {
        "ANY_ADDITIONAL_PROPERTY": "text"
      }
    }
  },
  "outputFormat": "Original",
  "generatorSetup": "{\"NAME_GIVEN\":\"Redaction\", \"NAME_FAMILY\":\"Redaction\"}",
  "labelBlockLists": "{\"NAME_FAMILY\": {\"strings\":[],\"regexes\":[\".*\\\\s(disease|syndrom|disorder)\"]}}",
  "labelAllowLists": "{ \"HEALTHCARE_ID\": {\"strings\":[],\"regexes\":[\"[a-z]{2}\\\\d{9}\"]} }",
  "tags": [
    "text"
  ],
  "files": [
    {
      "fileId": "text",
      "fileName": "text",
      "fileType": "text",
      "datasetId": "text",
      "numRows": 1,
      "numColumns": 1,
      "piiTypes": [
        "text"
      ],
      "wordCount": 1,
      "redactedWordCount": 1,
      "uploadedTimestamp": {},
      "fileSource": "Local",
      "processingStatus": "text",
      "processingError": "text",
      "mostRecentCompletedJobId": "text",
      "fileParseResultId": "text",
      "filePath": "text",
      "generatedFileStatus": "text"
    }
  ],
  "lastUpdated": {},
  "created": {},
  "creatorUser": {
    "id": "text",
    "userName": "text",
    "firstName": "text",
    "lastName": "text"
  },
  "docXImagePolicy": "Redact",
  "pdfSignaturePolicy": "Redact",
  "pdfSynthModePolicy": "V1",
  "docXCommentPolicy": "Remove",
  "docXTablePolicy": "Redact",
  "llmClassificationPolicy": "Disabled",
  "llmTableClassificationPolicy": "Disabled",
  "fileSource": "Local",
  "customPiiEntityIds": [
    "text"
  ],
  "operations": [
    "HasAccess"
  ],
  "rescanJobs": [
    {
      "id": "text",
      "status": "text",
      "errorMessages": "text",
      "startTime": {},
      "endTime": {},
      "publishedTime": {},
      "datasetFileId": "text",
      "datasetId": "text",
      "jobType": "DeidentifyFile",
      "ocrServiceProvider": "Azure",
      "modelsInfo": {
        "dateSynthesis": {
          "runsOnGpu": true
        },
        "fasttext": {
          "libVersion": "text",
          "model": "text",
          "runsOnGpu": true
        },
        "image": {
          "version": "text"
        },
        "spacy": {
          "libVersion": "text",
          "auxModel": {
            "name": "text",
            "language": "text",
            "runsOnGpu": true,
            "version": "text"
          },
          "multilingualModels": [
            {
              "name": "text",
              "language": "text",
              "runsOnGpu": true,
              "version": "text"
            }
          ]
        },
        "torch": {
          "gpuAvailable": true,
          "libVersion": "text"
        },
        "tonicNer": {
          "enModel": "text",
          "xlmModel": "text"
        },
        "tesseract": {
          "model": {
            "id": "text",
            "version": "text"
          }
        }
      }
    }
  ],
  "mostRecentExternalFileGenerationJob": {
    "id": "text",
    "status": "text",
    "errorMessages": "text",
    "startTime": {},
    "endTime": {},
    "publishedTime": {},
    "datasetFileId": "text",
    "datasetId": "text",
    "jobType": "DeidentifyFile",
    "ocrServiceProvider": "Azure",
    "modelsInfo": {
      "dateSynthesis": {
        "runsOnGpu": true
      },
      "fasttext": {
        "libVersion": "text",
        "model": "text",
        "runsOnGpu": true
      },
      "image": {
        "version": "text"
      },
      "spacy": {
        "libVersion": "text",
        "auxModel": {
          "name": "text",
          "language": "text",
          "runsOnGpu": true,
          "version": "text"
        },
        "multilingualModels": [
          {
            "name": "text",
            "language": "text",
            "runsOnGpu": true,
            "version": "text"
          }
        ]
      },
      "torch": {
        "gpuAvailable": true,
        "libVersion": "text"
      },
      "tonicNer": {
        "enModel": "text",
        "xlmModel": "text"
      },
      "tesseract": {
        "model": {
          "id": "text",
          "version": "text"
        }
      }
    }
  },
  "fileSourceExternalCredential": {
    "fileSource": "Local",
    "credential": {}
  },
  "awsCredentialSource": "text",
  "outputPath": "text",
  "externalFilesInfo": {
    "selectedFiles": [
      "text"
    ],
    "pathPrefixes": [
      "text"
    ],
    "selectedFileExtensions": [
      "text"
    ]
  },
  "ocrServiceProvider": "Azure",
  "fileCount": 1
}