Skip to main content
An eval case is a test made from a conversation Aide got wrong: the customer’s message, what a good reply must do, and what it must not say. Aide replays the cases against your agent’s current setup every night, so a mistake that comes back shows up as a failing test. Find them in AI Agent > Quality > Evals. Owners and admins see the page.

​
What evals are, and what they are not

Evals are a fixed set of test cases, each checked against statements written for it. They are not a replay of your past conversations against the whole agent: you can’t pick a week of real conversations and see how today’s agent would have answered them. To try the agent on a question yourself, use the preview (see Test your agent before going live).

​
Where eval cases come from

Each case shows its origin:
  • Theme: drafted from a group of unacceptable replies that share a cause.
  • Draft vs sent: a draft your team rewrote, compared with the reply they sent.
  • Judge: a reply Aide’s own review judged unacceptable.
  • Teammate review: a draft a teammate marked Wrong (see Mark Aide’s draft as good or wrong).
  • Added from a message: an owner or admin chose Add to evals on one of Aide’s replies or drafts, in the message’s menu in a conversation, with an optional note on what Aide got wrong.
Personal details are removed from every case.

​
The nightly run

The run is part of Aide’s nightly learning (see What Aide learns every night). Once there is at least one case, Aide replays the newest 30 active cases for each agent and records the result of each:
  • Passing: the reply met every statement under A good reply and said nothing listed under Must not.
  • Failing: the checklist shows which statements were Met or Missed, with the reason.
  • Needs knowledge: the reply left a blank exactly where a fact your knowledge doesn’t have would go. The case lists the facts to add. These cases are left out of the pass rate.
A case also shows how reliable its result is, such as “Passed 3/3 tries” or “Flaky, 2/3”, and a dot for each recent run.

​
Read the latest run

The line at the top compares the latest run with the one before: how many cases are passing, how many were fixed, how many broke and how many need knowledge, with one word: Improved, Got worse, Mixed or No change. Select a count to see the cases behind it. When a case that passed starts failing, it shows “Broke after:” with what changed between the two runs. Under Runs, each night’s run lists the cases it Fixed or left Broken.

​
Run the tests now

Select Run suite now to run without waiting for the night. With a topic filter on, the button reads Run this topic now and runs that topic’s cases. One run at a time per workspace. A run uses Aide’s own daily budget for evals, not your AI replies; if that budget is spent, the run stops and you can try again the next day.

​
Edit or retire a case

  • Edit (the pencil): change the statements under A good reply (one per line, up to 5) and Must not (one per line, up to 4). The case is marked Edited.
  • Test is wrong: use this when the expectation is unfair or out of date, not when Aide got it wrong. Give a reason of up to 200 characters and select Retire case. The case leaves the nightly run and the pass rate.
  • Retire: takes a case out of the nightly run. Switch the status filter from In the suite to Retired to find it, and select Put back to return it.
  • Confirm or Reject on a missing fact: say whether your knowledge really lacks it.

​
Evals and what Aide changes on its own

Before Aide changes its own Tone & identity, the new wording must still pass every case that passed that night, three times in a row. A change that later makes the agent fail more of its cases is undone.