Iterating over the guidelines to use for model training
On the Guidelines refinement page, you prepare the guidelines that define a model.
To test each version of the guidelines, Textual uses the guidelines to detect entity values in the test data. It then generates scores to indicate how closely those detection results match the values that you established during your initial review.
Textual also generates recommendations to improve the guidelines.
Viewing the initial version of the guidelines
To work on the guidelines, click Guidelines refinement. The Guidelines refinement option is enabled when you complete the review on the initial set of test files.
The first time you display the Guidelines refinement page, Textual uses the guidelines that you provided during the entity type creation to populate the Version 1 tab.
At the left are the guidelines.
At the right is the list of test data files.
File statuses for the guidelines refinement
For each version of the guidelines, Textual uses the guidelines to detect entity values in the test data.
The file statuses are:
Queued for annotation - Textual has not yet scanned the file.
Annotating - Textual is in the process of scanning the file.
Annotated - The scan is complete.
Reviewing the test scores for the guidelines
When Textual uses guidelines to detect entity values in the test files, it sets the number of detected entities and a set of scores. The scores reflect how well the detections match the entity values that you established in the test data setup. If you change the established values in the test data, Textual updates the scores for the guidelines.
The overall entity count and scores across all files are displayed across the top of the page. The file list displays the entity count and scores for each file.
The scores are:
Precision score - Measures the accuracy of positive predictions. Indicates how many of the detected entities were correctly identified. For example, the guidelines detect 10 values. If only 3 of those are correct, then the precision score is lower than if 7 of those are correct.
Recall score - Measures the model's ability to find all of the entities. Indicates how many of the actual entities it detected. For example, the guidelines detect 10 correct values. If the total number of correct values is 20, then the recall score is lower than if the total number of correct values is 12.
F1 score - The harmonic mean of precision and recall. The goal is to have a balance between precision and recall. The guidelines should produce annotations that are both accurate and complete. Detecting all of the correct values is not useful if the guidelines also detect a large number of incorrect values. And detecting only correct values is not useful if the guidelines only detect a fraction of the total number of correct values.
Reviewing the guideline detections
To review the entity values that Textual detected based on the current version of the guidelines, click the file name.
Editing the guidelines
Based on how accurately Textual detected the entity values, Textual generates suggested changes to the guidelines.
For example, it might suggest additional language to more specifically identify values that the previous version either missed or detected incorrectly.
To start a new version of the guidelines:
Click Edit. If there are suggestions, you can also click Review.
On the Annotation guidelines panel, the current guidelines are displayed in an editable text area on the left. On the right is a summary of the suggested updates to the guidelines. To display the proposed replacement guidelines, toggle Show diff to the on position.
To update the guidelines, you can either:
Update the guidelines manually.
Accept all of the suggestions, and replace the current guidelines. To do this, click Accept changes.
Manually copy text from the suggestions and paste it into the guidelines.
To save the guidelines version, and start the detection and scoring, click Save new version.
Textual creates a new tab for the new version of the guidelines. The tab label is Version n, where n is incremented for each new version. The most recent version is at the left.
Last updated
Was this helpful?