Auditing redaction requests
Required global permission: Use the Request Explorer
The Request Auditor allows you to review the quality of the entity detection in your redaction requests to the Python SDK or REST API.
The Request Auditor uses an LLM to review the redaction results for a sample percentage of the requests. The LLM review specifically looks for entities that were not identified in the original request.
The reviewed requests receive a score based on the LLM findings. You configure a threshold score at or below which a request requires a human review to see what was missed in the original request.
For self-hosted instances, you can configure the LLM to use and parameters for processing the requests.
Displaying the Request Auditor
To display the Request Auditor, in the Textual navigation panel, click the Request Auditor option.
On the Request Auditor:
The Request log page displays the reviewed requests.
The Audit settings page provides configuration options for the audit.
Configuring the audit
On the Audit settings page of the Request Auditor, you configure:
Whether to enable request auditing.
The percentage of requests for the LLM to review.
When to require a user to review the LLM results.
How long to keep the files.
After you change the configuration, to save the changes, click Save.
Enabling or disabling request auditing
To enable request auditing, toggle Enable requests auditing to the on position.
To disable request auditing, toggle Enable requests auditing to the off position. When you disable request auditing, no new requests are sent for auditing. Existing requests remain in the request log until they expire.
Configuring the percentage of requests to audit
For Sampling rate field, set the percentage of requests that the auditor reviews.
You can either use the slider, or type the percentage value directly into the field.
Configuring the threshold for user review
In the Score threshold for user review field, specify the score value to use to determine whether a request requires a user to review it manually.
Requests that receive a score at or below the threshold require the user review.
A perfect score is 1.0.
Configuring the retention period for reviewed files
In the Log retention period field, provide the number of days to retain a reviewed request in the Request log.
The default is 7 days.
Requests that have a score above the threshold are removed after the retention period.
Requests that require a user review and that are not yet reviewed are exempt from the retention period. They remain in the list.
When a request is marked as reviewed, it is then subject to the retention period. For example, the retention period is 7 days. A request requires a user review. After 3 days, a user marks it as reviewed. The request is then removed 7 days after that.
Viewing the request log
The Request log page of the Request Auditor displays the current list of reviewed requests.
Information in the request log
At the top of the page is summary information about the requests:
The average F1 scores from the LLM reviews for the requests in the list.
The number of requests that are waiting for a user review.
The total number of requests that are currently in the list.
For each request, the request list includes:
The identifier of the request.
The recall score assigned during the LLM review.
The current status of the request.
Reviewed by AI - The request was reviewed by the LLM and received a score that is above the threshold for user review.
Pending user review - The request requires and is waiting for a user review.
Reviewed by user - The user review is completed.
The number of days before the request expires.
The source of the request. Currently, the source is always SDK.
The timestamp of the request.
Filtering the request log
You can filter the log by the request status.
To filter the list:
Click Filter.
On the filter panel, select the filters to apply.
Displaying and reviewing a request
To display the details for a request, click the request row.
Information on the request details
The request details include:
Recall score - Measures the model's ability to find all of the entities. Indicates how many of the actual entities it detected. For example, the original request detected 10 correct values. If the LLM review determined that the total number of correct values is 20, then the recall score is lower than if the total number of correct values is 12.
False negatives - List of entities that the LLM detected that were not detected by the original request.
Redacted text. In the text, the detected entities are highlighted to indicate whether they were detected in the original request, or detected during the LLM review.
Entity type settings snapshot - Lists the entity type handling configuration for the request.
Marking a request as reviewed
For requests that require a user review, the details include a Mark as reviewed option. To mark a reviewed request as not reviewed, click Unmark as reviewed.
After it is marked as reviewed, the request is subject to the configured expiration period. For example, if requests expire after 7 days, then the request expires and is removed 7 days after it is reviewed.
Last updated
Was this helpful?