Evals

Changing a system prompt usually feels better. Whether it is better is a different question — one chat message cannot answer it, because your agent is not deterministic and any single answer might have gone either way.

An eval replaces that guesswork. You define the questions that matter, define how a good answer is recognised, and run every version against the same bar.

• DatasetThe questions you want to test, with an optional reference answer for each. Add them by hand or import a CSV of input, expected answer.
• VariantsThe versions being compared — a different system prompt, a different model, a different temperature, or any mix. Leave a field blank and it uses the agent's own setting.
• ScorersHow each answer is judged. Assertions are free and instant, retrieval metrics score your knowledge base, and AI judges grade the things assertions cannot.

1. 1

   Create a dataset

   Go to Evaluation → Datasets, click New dataset, and add ten or so questions real users actually ask. Ten good questions beat a hundred invented ones.
2. 2

   Add a few out-of-scope questions

   Set a couple of cases to Out of scope — things your agent should decline rather than answer. This is how you catch an agent that invents answers.
3. 3

   Build the run

   From Evaluation → Runs click New evaluation. Pick your dataset, then add a second variant with the prompt or model you want to test. The summary rail on the right keeps the answer count and cost estimate in view the whole time, so nothing is a surprise when you press Run.
4. 4

   Read the scorecard

   Each variant gets a column. The strongest variant per metric is marked best, and the difference against your first variant is shown underneath.
5. 5

   Promote the winner

   Promote applies that variant's configuration to the agent and records a new version, so the change is auditable and reversible from the Versions tab.

To sharpen a close call, raise Repeats per case to 2 or 3. Each case runs that many times, and every score is reported as an average with its spread (±). More repeats cost proportionally more, so use them to settle a decision rather than as a default.

• Contains / Does not containThe workhorses. “Every answer about returns must mention 30 days”, or “no answer may ever contain this competitor's name”. Free and exact.
• Valid JSONFor agents that must return structured output. Checks the answer parses and carries the keys you list.
• Recall@k, MRR, nDCG@kRetrieval quality. These need cases labelled with the knowledge-base chunks that should be found, and they answer a question no answer-level score can: did the agent give a bad answer because it reasoned badly, or because the right document never reached it?
• Correct rejectionFor out-of-scope cases. Confirms nothing scored highly enough to be injected, so the agent had no noise to build a confident wrong answer from.
• FaithfulnessAn AI judge checks every claim in the answer against the retrieved context. This is the hallucination check — the one that catches invented phone numbers and addresses.
• Correctness / Custom rubricJudges that compare against your reference answer, or against criteria you write in plain English (tone, length, policy).

They answer different questions and are deliberately kept apart. The Playground is for exploring — you try a prompt, read one answer, get a feel for it. That is judgement on a sample size of one, which is fine for exploration and useless for deciding.

Evals are for deciding. Same questions, every version, scored the same way, with the variance made visible. Use the Playground to come up with a candidate prompt; use an eval to find out whether it is actually better before you publish it.

Eval answers are real API calls: they count against your monthly quota and use your LLM key, exactly like chat traffic. They do not appear in your conversation history, analytics or token-cost charts — eval traffic is tracked separately so it cannot distort your production numbers.

One run is capped at 200 answers. Multiply cases × variants × repeats to stay under it. AI judges add one grading call per answer per judge, which the estimate accounts for.