Evals

Score finished runs against a rubric with an LLM judge, catch regressions, and re-run a golden set when you change something.

A workflow that runs without erroring has not necessarily done its job well. Evals answer the other question: was the output any good, and is it getting worse?

You write a rubric, bind it to a workflow, and every finished run is scored. Two rules decide whether a score is worth interrupting you about.

To measure how well a knowledge base finds its answers, see Evaluate a knowledge base — a separate tool.

The rubric

Field	What it does
Criteria	Named things to judge, each with a `weight` and free-text `guidance`
Scale	The score range. Defaults to 0–10
Floor	An absolute line — below this is a regression
Drop	A fall this far below the recent mean is a regression
Baseline runs	How many recent runs the mean is taken over. Defaults to 10
Judge model	Which model scores — a model picker whose first choice, Workspace judge model, or the default model, follows the workspace's `judge_model`

Guidance is where the work is. "Accurate" judges nothing; "every figure quoted appears in the source documents, and no figure is rounded differently from the source" judges something.

One model call scores one finished run against one rubric — deliberately one call, not one per criterion and not one per step. The model returns an overall score, a score per criterion and a score per step in a single structured answer.

That choice is about three things, in the order they mattered: cost is the whole argument for this product over an autonomous framework, and N calls per run is the easiest way to lose it; a judge that sees the whole run at once can say step 2 is where this went wrong, which a per-step call cannot; and one call is one place for a parse to fail.

What the judge can see

Only what is durable, which is narrower than it looks:

shared_state — every step's output
step_history — the timeline: name, type, status, duration, error
per-step execution memory — the agent, what triggered it, the inputs it saw, every tool call with arguments and a result preview, the turn count, the final output

Explicitly not the run event journal. That is scratch, capped, and truncates exactly the keys a judge would want; on a fleet its counterpart expires within the hour. A judge built on it would work on a laptop and quietly score nothing in production.

Scoring can never fail a run

By the time the judge runs, the run has finished and its result already belongs to you. A judge that raised would turn a successful run into a failed one over an opinion about it, so every entry point is wrapped and a scoring failure is recorded rather than propagated.

Regressions

Two rules, because they catch different failures:

Floor — an absolute line. Answers is this good enough, and is the only rule that works on a brand-new workflow with no history.
Drop — a fall below the mean of recent runs. Answers did this get worse, and catches the slow slide from 9.0 to 7.5 that never crosses any absolute line and is what you actually meet in practice.

The floor is checked first, so a run that breaches both is reported as a floor breach. The drop rule stays silent until there are at least three prior scores — below that, a rolling mean is one opinion with extra steps. The floor rule has no such threshold, which is why a new workflow is not left unguarded.

Either rule can be switched off by leaving it empty. Both off is legitimate: the rubric then scores runs and never flags them.

The baseline is taken before this run's score is written

Otherwise a run is measured partly against itself, which drags the mean toward whatever just happened and makes a sustained decline look like a series of small ones.

Golden sets

Scoring live runs measures quality against whatever people happened to ask that week. A golden set holds the inputs still, so a change in the score is a change in the workflow rather than a change in the questions.

Create a suite, add cases — or have SynapseOrch propose them from the workflow's own definition — and run it when you change a prompt, a model or a step. Suggest cases asks a model for them. A failed call says why — Could not suggest cases., then the wallet, key or timeout that stopped it — and an empty list means the model found nothing to suggest.

A suite run is not a new kind of execution. Each case is an ordinary run of the workflow, started through the same seam every other caller uses and scored by the same hook, because the workflow already carries a rubric.

Cases run sequentially, not in parallel — N at once would be N concurrent runs of the same workflow, which is a self-inflicted load test against whatever that workflow talks to.

Running a set says what it costs first: N full runs, billed as usual, plus N judge calls when a rubric is bound. A set on a workflow with no rubric still runs, says its cases will not be scored, and its history reads not scored: no rubric — bind a rubric under Workflow settings → Guardrails to score them.

A set that stops partway says where: Stopped before case 4 of 10: …, with the reason. On SynapseOrch Cloud each case is billed as the run it is — its credits held when it starts, bounded by the plan's longest run — and an empty model wallet stops the set before its next case; Start is refused (402) when the wallet can't pay at all.

Scoring earlier runs

Binding a rubric to a workflow that already has history does not leave that history unscored. Pick the workflow: while nothing is scored yet, Score earlier runs judges its past runs against the rubric, so a floor or a baseline has something to work with immediately. In SynapseOrch Cloud the button is in the rubric pane.

Feedback

Thumbs up and down on a chat response are recorded alongside judge scores, so a rubric that disagrees with the people using it is visible rather than merely confident.