What we learned by measuring the agent instead of vibing it

ask_clarifying_question is a frontend tool (CopilotKit's human-in-the-loop card) — it has no implementation outside the browser. The first headless runs silently burned their output-retry budget every time the model reached for it. Learning: evaluability isn't free; it required a scripted stub toolset in the harness from day one, and extracting ResearchServicePort so grounded web search could be swapped for a fake like every other data source. All 59 pytest tests stayed green through the refactor.

The very first live run against production gemini-2.5-flash flagged one assertion, itemized_costs_sum_to_subtotals, at 75%. Everything else — budget caps, direct-only max_stops=0, no-red-eye legs, party-size propagation, empty-route fallback, per-island lodging — passed cold on the first try.

Diagnosis of the failing island-hop run showed the model's itemized legs and hotels were correct and complete — but the subtotals it reported were fiction: transportation €600 when its own six legs summed to €1252. This is a data-gathering success and a pure arithmetic failure.

Before — itemized_costs_sum_to_subtotals: 75% (9/12)

Fix: stopped delegating arithmetic to the model. generate_scenarios now re-derives transportation/accommodation subtotals by summing the model's own Leg.cost / Accommodation.cost line items server-side, then the grand total from those subtotals — deterministic code, not a prompt plea. A new unit test (test_generate_scenarios_normalizes_subtotals_from_line_items) locks it in.

After re-run — itemized_costs_sum_to_subtotals: 100%, every category 100%.

Re-running the full suite after the cost fix landed a new, different failure: the heaviest case in the set (a 12-day Greek island hop rendered as one fully-detailed plan) passed in the baseline but errored on the very next run with UnexpectedModelBehavior: Exceeded maximum output retries. Flagged rather than chased immediately — a single data point isn't a diagnosis, it's a reason to run a variance study.

Added human-in-the-loop cases (does the agent ask when info is genuinely missing, and stay quiet when it isn't?), multi-turn refinement cases (shared message_history across turns — "now add a cheaper option", "make it for 4 people instead"), and a small LLM-as-judge (evals/judge.py) for soft criteria no deterministic check can express: is the chat reply a concise recommendation rather than a wall of prose, and do the day-by-day activities actually match the requested vibe ("relaxed foodie", "adventure hiking")? The judge is deliberately decoupled from the agent under test — see below.

The first full variance pass (30 cases × 3 repeats at concurrency 6) came back with almost everything "erroring." Reading the actual error bodies showed 429 RESOURCE_EXHAUSTED quota errors and 504 DEADLINE_EXCEEDED timeouts — the harness was firing more concurrent Gemini requests than the API tier allows, not a model or prompt regression. Practice adopted: re-run flagged cases at low concurrency before treating any failure as a real signal; the runner's per-case aggregation now separates error_runs from genuine assertion failures for exactly this reason.

At safe concurrency, 22 of 30 cases ran clean with 100% of executed assertions passing — budgets, currency propagation, direct-only, no-red-eye, party-size (solo through groups of 6), ferry/rail multi-city chains, HITL ask/don't-ask, multi-turn refinement, and the LLM-judge criteria all held. Re-running the 8 flagged cases individually separated 6 transient infra errors from 2 reproducible failures: greek_island_hop_single_plan and vibe_relaxed_foodie — both single, maximally-detailed 12-day plans — consistently hit UnexpectedModelBehavior: Exceeded maximum output retries. Root cause: on a very large final-synthesis context, Flash intermittently returns an empty completion with no first token.

Raised the agent's output-retry budget from the pydantic-ai default of 1 to 2 (retries={'tools': 2, 'output': 2}) to give a large synthesis a second chance at a non-empty completion. Re-verified on the same two cases: still fails, now as Exceeded maximum output retries (2). This is the honest outcome of the loop, not a hidden one — the fix has to be architectural (lighten the day-by-day detail the prompt asks for in single-plan mode, or split the synthesis into two calls), and that's queued as the next thing the eval suite will verify once it ships.

The LLM judge now defaults to gemini-2.5-flash-lite — cheap and plenty for a boolean rubric — while the agent under test stays on whatever ships in LLM_MODEL, so a judge-cost optimization can never quietly change what's actually being measured. Both are configurable three ways (CLI flag → env var → default) via EVAL_MODEL / EVAL_JUDGE_MODEL in backend/.env, and every JSON report now records exactly which model played which role for full reproducibility.

1) False-negative in the harness itself: infeasible_budget_no_fabrication failed produced_a_plan for correctly refusing to invent a plan for an impossible budget — the evaluator, not the agent, was wrong. Fixed by skipping that assertion when CaseMeta.budget_infeasible is set. 2) Ambiguous case wording: multi_city_rail_europe said Prague→Vienna must be "BY TRAIN" but left the Berlin legs unspecified — the model reasonably flew them, failing ground_legs_present. Reworded to "no flights at all, including the way back" and added explicit forbidden_tools. 3) The real one: group_of_six_propagation — phrasing the travelers as "a group of friends" instead of "6 people" made the agent skip search_flights/search_accommodations entirely and hand-wave numbers straight into generate_scenarios. Casual framing was being read as license to skip the tools. Fixed with a new system-prompt directive: informal phrasing describes the travelers, never a license to fabricate prices.

All three cases: 3/3 repeats clean after their fix; the full 30-case suite re-run confirmed no regressions.

Every case run now flows through the same Langfuse/OpenTelemetry pipeline as production (app/telemetry.py), tagged environment="eval", session_id=<run timestamp> (groups every case + repeat of one evals.run invocation into one Langfuse session), trace_name=<case name>, and tags=["kompass-eval", <category>]. Production traces got the mirror-image environment="production" so the two populations never mix in a shared Langfuse project.

Verified live, not just wired: ran evals.run --case group_of_six_propagation and queried the Langfuse API directly — the trace landed with environment=eval, traceName=group_of_six_propagation, sessionId matching the run's timestamp, and eval_category/eval_repeat metadata on every span, alongside the full token/cost/tool-call detail pydantic-ai already emits.

A full 30-case run flagged empty_route_falls_back_and_flags_estimated failing prices_flagged_estimated. Instead of writing a throwaway debug script, filtered Langfuse to environment = eval, traceName = empty_route_falls_back_and_flags_estimated, and read the generate_scenarios tool call directly: the model's own reasoning_summary said "the total cost is an estimate due to flight prices being sourced from web search" — but the structured estimated argument it actually passed to the tool was false. Right reasoning, wrong structured output — the same failure shape as the itemized-costs bug (§7 in evals.md), just far less frequent.

Quantified before acting: repeated the case 9 more times rather than prompt-patching off one sample. Result: 8/9 passed prices_flagged_estimated cleanly — genuine low-rate model flakiness, not a deterministic bug, so left unfixed rather than overfitting a prompt change to one data point.

Roadmap item 3.5, done. The trace tagging shipped earlier answers "let me look at this run"; this closes the loop with the other question — "how did case X, or category Y, score across every run and prompt/model version?" Each evals.run now mirrors the golden set into a Langfuse dataset (kompass-golden) and publishes the run as a dataset run: one item-run per case linked to its agent trace, one BOOLEAN score per assertion, plus per-category and overall NUMERIC scores on the run itself — so Langfuse's dataset comparison view charts and diffs score history across versions.

Design — mirror, don't re-run: kept pydantic-evals as the source of truth rather than re-scoring on Langfuse's own experiment runner (which would double the model spend). The catch was linking each Langfuse score to the right case: pydantic-evals owns the execution loop, so evals/tasks.py wraps each case in a Langfuse span and stamps its trace id onto the ReportCase via set_eval_attribute("langfuse_trace_id", …) — an exact, per-row link. evals/langfuse_dataset.py then does the sync + publish; failed cases (which lose their attributes) fall back to a per-run trace registry. On by default when Langfuse keys are set, skippable with --no-langfuse-dataset, a no-op otherwise.

Ran the whole suite (uv run python -m evals.run, gemini-2.5-flash agent / flash-lite judge): 29/30 cases clean with 100% of executed assertions passing across all seven categories. The one failure was the known-open greek_island_hop_single_plan output-retry exhaustion (§9), plus a transient 503 from the judge model on one case (infra noise, not a regression). The Langfuse dataset run published correctly and was verified back via the API: 30/30 run items linked to their agent traces — including the failed case, linked via the trace registry fallback and flagged task_error=1 — with per-assertion scores and the run-level overall_pass_rate/category:* aggregates all present.

Follow-up to the dataset-run work above: the runs existed and every score was present via the API, yet the dataset's Experiments tab showed "No data". Root cause: linking a trace to a run with the low-level api.dataset_run_items.create call produces a valid dataset run, but the Experiments tab only surfaces traces that the SDK's own run_experiment() marks as experiment items — environment="sdk-experiment" plus a set of langfuse.experiment.* span attributes. Our traces were tagged environment=eval with none of those attributes, so they never qualified.

Swapped the agent-under-test default from gemini-2.5-flash to gemini-3.1-flash-lite (app/config.py; the judge stayed decoupled on gemini-2.5-flash-lite on purpose) and validated the switch with a same-day A/B rather than trusting the model card. A --concurrency 4 run on the old model came back noisy — several categories (human_in_the_loop, itinerary_quality) produced zero assertions after 429 RESOURCE_EXHAUSTED quota errors ate most of the run, landing 62/63 (98.4%) of what did execute. Re-running at --concurrency 2 on the new model completed cleanly across all seven categories — 152/153 (99.3%) — at roughly 34% lower per-case token cost.

gemini-2.5-flash-lite (the default EVAL_JUDGE_MODEL) began returning 503 UNAVAILABLE ("high demand") on judge-backed assertions (concise_recommendation, vibe_matches_activities) — a transient Google-side capacity ceiling, not a prompt or model problem, but enough to silently drop a check or fail a whole run.

Fix: judge.py now wraps the judge agent in PydanticAI's FallbackModel, retrying the next model in the new EVAL_JUDGE_FALLBACK_MODELS env var (default google:gemini-2.0-flash-lite,google:gemini-2.5-flash — separate Gemini capacity pools) on any ModelAPIError, which covers 503s. Only the judge gets the fallback chain; the agent-under-test's own model is untouched, since decoupling the judge from prod is the whole point (evals.md §6).