Tonic Validate supports the following types of projects:
Development projects
Production monitoring projects
A Validate development project contains the results of Validate runs.
For a set of questions, which can be from a Validate benchmark, the run assesses the quality of the answers from your RAG system. If you also provide expected answers, then the run compares the answers from your RAG system against the provided answers. It also analyzes how the RAG system used additional context to answer the questions.
Each run generates an overall score and metrics.
For information about starting a run, go to Starting a Validate run.
For information about viewing run results, go to Viewing and managing runs.
For development projects, you can also use our Ragas integration, tonic_ragas_logger, which allows you to display Validate visualizations of Ragas results.
A production monitoring project tracks the performance of a production RAG system.
You configure the RAG system to send to the production monitoring project:
The questions the RAG system receives
The answers it provided
The context it used to determine the answer
Validate then generates metrics for each question, and allows you track the RAG system performance over time. Note that production monitoring does not use Ragas.
For information about configuring your RAG system to send questions to a Validate production monitoring project, go to Configuring your RAG system to send questions to Validate.
For information about viewing the results, go to Viewing the metric scores and logged questions.
The Validate home page includes the list of projects.
For each project, the list displays:
The project name
For development projects, when the most recent run occurred
For development projects, a chart that maps the average overall score for each run over time
To display the details for a project, click the project tile.
For a new development project that does not have any runs, the project details page guides you through the required steps to create a run.
If you use tonic_ragas_logger to visualize Ragas results in Validate, then select Ragas as the logging framework.
If you use Validate runs to generate and visualize metrics, then select Validate as the logging framework.
For a development project that has completed runs, the project details page displays the list of runs, and provides an overview of the scores across the runs and questions.
At the left is the list of runs for the project. From there, you can display details for the run results.
The tiles across the top contain the average overall score and average metrics scores for the most recent run.
By default, the graph displays the overall score across all of the runs over time. When you click a metric score tile, the graph updates to show the average metric score across the runs.
Below the graph is the list of questions in the project benchmark. For each question, the list shows the overall score for a month ago and for the most recent run.
To filter the question list, in the filter field, type text from the question.
When you click a question, the graph is updated to show the average overall or metric score across runs for that specific question. To deselect the question, click it again.
For a new production monitoring project that does not have any results, the project details page guides you through the required steps to set up a feed of questions from the RAG system to the project.
For a production monitoring project that has received questions, the project details page shows a set of overall scores based on the most recent questions that the project received.
The overall scores are followed by a timeline that shows changes in the average metric scores over a selected timeframe.
Below the timeline is the list of questions with metric scores. When you click a point in the timeline, the questions are filtered to display questions that were received during that time.
For a new project, you provide a name and select the project type.
To create a project:
On the Validate home page, click Create a Project.
In the Name field, type the name of the project, then click Next.
Click the type of project to create, then click Save.
Validate displays the project details page.
For an existing project, you can change the name.
To edit the project name:
Either:
On the Edit Project panel, in the Project Name field, type the new name for the project.
Click Save.
To delete a project, from the projects list:
Click the options icon for the project.
In the options menu, click Delete.
From the Validate Home page, click the options menu for the project, then click Edit.
From the project details page, click Edit Name.
