Iterating over the guidelines to use for model training

On the Guidelines refinement page, you prepare the guidelines that define a model.

To test each version of the guidelines, Textual uses the guidelines to detect entity values in the test data. It then generates scores to indicate how closely those detection results match the values that you established during your initial review.

Textual also generates recommendations to improve the guidelines.

Viewing the initial version of the guidelines

To work on the guidelines, click Guidelines refinement. The Guidelines refinement option is enabled when you complete the review on the initial set of test files.

The first time you display the Guidelines refinement page, Textual uses the guidelines that you provided during the entity type creation to populate the Version 1 tab.

Guidelines refinement page for a model-based custom entity type

At the left are the guidelines.

At the right is the list of test data files.

File statuses for the guidelines refinement

For each version of the guidelines, Textual uses the guidelines to detect entity values in the test data.

The file statuses are:

• Queued for annotation - Textual has not yet scanned the file.
• Annotating - Textual is in the process of scanning the file.
• Annotated - The scan is complete.

Reviewing the test scores for the guidelines

When Textual uses guidelines to detect entity values in the test files, it sets the number of detected entities and a set of scores. The scores reflect how well the detections match the entity values that you established in the test data setup. If you change the established values in the test data, Textual updates the scores for the guidelines.

The overall entity count and scores across all files are displayed across the top of the page. The file list displays the entity count and scores for each file.

Overall scores and file scores in the Guidelines Refinement list

The scores are:

• Precision score - Measures the accuracy of positive predictions. Indicates how many of the detected entities were correctly identified. For example, the guidelines detect 10 values. If only 3 of those are correct, then the precision score is lower than if 7 of those are correct.
• Recall score - Measures the model's ability to find all of the entities. Indicates how many of the actual entities it detected. For example, the guidelines detect 10 correct values. If the total number of correct values is 20, then the recall score is lower than if the total number of correct values is 12.
• F1 score - The harmonic mean of precision and recall. The goal is to have a balance between precision and recall. The guidelines should produce annotations that are both accurate and complete. Detecting all of the correct values is not useful if the guidelines also detect a large number of incorrect values. And detecting only correct values is not useful if the guidelines only detect a fraction of the total number of correct values.

Reviewing the guidelines detections

To review the entity values that Textual detected based on the current version of the guidelines, click the file name.

Reviewing the guidelines detections for a test data file

Changing to a different guidelines version

For each file, you can switch between versions, to compare how different guidelines versions performed.

To display the file results for a different version, from the version dropdown list, select the version to change to.

Comparing established values to guidelines detections

By default, the file details use Compare to test data view, which compares the values that you established with the values that the guidelines version detected.

True positive values, highlighted in green, are values that both you established and the guidelines version detected.

False negative values, highlighted in red, are values that you established, but that the guidelines version did not detect.

False positive values, highlighted in yellow, are values that the guidelines version detected, but that you did not include in the established values.

Viewing the guidelines detections only

To change the view to only indicate the values that the guidelines version detected, click Annotations.

In Annotations view, the detected values are all highlighted in yellow.

Editing the guidelines

Based on how accurately Textual detected the entity values, Textual generates suggested changes to the guidelines.

For example, it might suggest additional language to more specifically identify values that the previous version either missed or detected incorrectly.

To start a new version of the guidelines:

1. Click Edit. If there are suggestions, you can also click Review.

Edit and Review options for the annotation guidelines

1. On the Annotation guidelines panel, the current guidelines are displayed in an editable text area on the left. On the right is a summary of the suggested updates to the guidelines. To display the proposed replacement guidelines, toggle Show diff to the on position.

Annotation guidelines panel with AI suggestions to improve the guidelines in the next version

1. To update the guidelines, you can either:

   • Update the guidelines manually.
   • Accept all of the suggestions, and replace the current guidelines. To do this, click Accept changes.
   • Manually copy text from the suggestions and paste it into the guidelines.
2. To save the guidelines version, and start the detection and scoring, click Save new version.

Textual creates a new tab for the new version of the guidelines. The tab label is Version n, where n is incremented for each new version. The most recent version is at the left.

Guidelines version tabs on the Guidelines Refinement page