> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aide.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals: the tests Aide replays every night

> An eval case is a test made from a conversation Aide got wrong: the customer's message, what a good reply must do, and what it must not say.

An eval case is a test made from a conversation Aide got wrong: the customer's message, what a good reply must do, and what it must not say. Aide replays the cases against your agent's current setup every night, so a mistake that comes back shows up as a failing test. Find them in **AI Agent > Quality > Evals**. Owners and admins see the page.

## What evals are, and what they are not

Evals are a fixed set of test cases, each checked against statements written for it. They are not a replay of your past conversations against the whole agent: you can't pick a week of real conversations and see how today's agent would have answered them. To try the agent on a question yourself, use the preview (see [Test your agent before going live](https://docs.aide.app/aide-ai-agent/test-agent.md)).

## Where eval cases come from

Each case shows its origin:

* **Theme**: drafted from a group of unacceptable replies that share a cause.
* **Draft vs sent**: a draft your team rewrote, compared with the reply they sent.
* **Judge**: a reply Aide's own review judged unacceptable.
* **Teammate review**: a draft a teammate marked **Wrong** (see [Mark Aide's draft as good or wrong](https://docs.aide.app/aide-ai-agent/review-a-draft.md)).
* **Added from a message**: an owner or admin chose **Add to evals** on one of Aide's replies or drafts, in the message's menu in a conversation, with an optional note on what Aide got wrong.

Personal details are removed from every case.

## The nightly run

The run is part of Aide's nightly learning (see [What Aide learns every night](https://docs.aide.app/aide-ai-agent/nightly-learning.md)). Once there is at least one case, Aide replays the newest 30 active cases for each agent and records the result of each:

* **Passing**: the reply met every statement under **A good reply** and said nothing listed under **Must not**.
* **Failing**: the checklist shows which statements were **Met** or **Missed**, with the reason.
* **Needs knowledge**: the reply left a blank exactly where a fact your knowledge doesn't have would go. The case lists the facts to add. These cases are left out of the pass rate.

A case also shows how reliable its result is, such as "Passed 3/3 tries" or "Flaky, 2/3", and a dot for each recent run.

## Read the latest run

The line at the top compares the latest run with the one before: how many cases are passing, how many were fixed, how many broke and how many need knowledge, with one word: **Improved**, **Got worse**, **Mixed** or **No change**. Select a count to see the cases behind it.

When a case that passed starts failing, it shows "Broke after:" with what changed between the two runs. Under **Runs**, each night's run lists the cases it **Fixed** or left **Broken**.

## Run the tests now

Select **Run suite now** to run without waiting for the night. With a topic filter on, the button reads **Run this topic now** and runs that topic's cases. One run at a time per workspace. A run uses Aide's own daily budget for evals, not your AI replies; if that budget is spent, the run stops and you can try again the next day.

## Edit or retire a case

* **Edit** (the pencil): change the statements under **A good reply** (one per line, up to 5) and **Must not** (one per line, up to 4). The case is marked **Edited**.
* **Test is wrong**: use this when the expectation is unfair or out of date, not when Aide got it wrong. Give a reason of up to 200 characters and select **Retire case**. The case leaves the nightly run and the pass rate.
* **Retire**: takes a case out of the nightly run. Switch the status filter from **In the suite** to **Retired** to find it, and select **Put back** to return it.
* **Confirm** or **Reject** on a missing fact: say whether your knowledge really lacks it.

## Evals and what Aide changes on its own

Before Aide changes its own Tone & identity, the new wording must still pass every case that passed that night, three times in a row. A change that later makes the agent fail more of its cases is undone.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.