Rating methodology
Blind pairwise votes become a statistically principled leaderboard. A standalone Python worker (worker/, omniarena_rating) runs off the request hot path: screen → aggregate → fit → interval → connectivity → write back, appending a history snapshot per refit so ratings can be charted over time. Everything here needs pairwise input — see What the engine cannot rate.
Pipeline
Anomaly screen
anomaly.py · p-value session tests
Aggregate
aggregate.py · SQL GROUP BY → triples
Fit BT
bradley_terry.py · L-BFGS-B MLE
▼
Intervals
confidence.py · Fisher info + bootstrap
Connectivity
connectivity.py · union-find components
Style pass
style.py · joint confounder regression
Write back
writeback.py · upsert ratings + append history
Bradley-Terrylog-param MLEL-BFGS-BRao-Kupper tiesridge priorsum-to-zero anchorFisher-info CIsmultinomial bootstrapwarm-start refitsperiodic full recomputestyle-controlled
Bradley-Terry + Rao-Kupper ties
Votes are pairwise preferences, so each model gets a latent log-strength r_i = log θ_i and outcomes follow the logistic of d = r_a − r_b. Ties (both_good/both_bad) use the Rao-Kupper threshold η ≥ 0 (a fitted parameter), not a y=0.5 hack; skip is dropped.
P(i beats j) = σ(d − η)
P(j beats i) = σ(−d − η) d = r_i − r_j
P(tie) = σ(d + η) − σ(d − η)Working in log-space makes the log-likelihood convex, so SciPy L-BFGS-B with an analytic gradient (unit-tested vs finite differences) finds the unique optimum in a few iterations — no learning rate. Parameter vector: [r_0…r_{n−1}, η].
What the engine cannot rate
Bradley-Terry is a model of comparisons: every term in its likelihood is a contest between two models, and there is no term for a verdict on one answer viewed alone. That bounds what OmniArena can rate.
| Situation | What the engine gets |
|---|---|
| A single round (ARENA_TRIGGER=manual, no opt-in; one model from ARENA_DEFAULT_MODEL) | Nothing. The round persists no matchups row, no responses, no vote token, so it is not filtered out of the fit like a skip vote — it never reaches the database to be filtered. |
| One-sided feedback (a thumbs-up on a lone answer) | No ingestion path. It says nothing about a pair, and BT has nowhere to put it. POST /api/arena/vote accepts only an HMAC-signed matchup token naming two slots; no endpoint, column, or worker path records a per-response rating. |
| A deployment serving mostly single rounds | No ratings. rating, ratingStdError, confidenceInterval, componentId stay null; the leaderboard falls back to counts from whatever matchup rounds occurred, of which there may be none. A model only ever served in single mode never enters the comparison graph. |
Calibrated claim. Rao-Kupper ties instead of a y=0.5 hack, Fisher-information intervals, joint style control, and the anomaly screen are concrete improvements over a hand-rolled Elo loop — given comparisons to fit. None substitutes for having comparisons. For a client that genuinely cannot present two answers (a strict OpenAI-compatible UI with one message channel), OmniArena has a graceful serving story (single, votable: false) and no rating story. Open WebUI, facing the same constraint, shipped its own single-blind thumbs-rated leaderboard on a hand-rolled Elo — see integrations/open-webui/.
Not built: regenerate-as-slot-B
Idea only — not implemented. Turning two sequential single answers into one genuine pair (serve an answer, regenerate the prompt with a second model, persist the two as a matchup, collect a real pairwise vote) would reach these clients. There is no such mode, schema, endpoint, or worker support: the enum admits only matchup, single, and a declared-but-unreachable shadow (server/src/arena/mode.ts). Recorded so it is not mistaken for a feature.
Identifiability: anchoring + ridge
Raw BT strengths are defined only up to an additive constant. Two mechanisms pin them down:
| Mechanism | What it does |
|---|---|
| Anchoring | Re-centre sum-to-zero (default) or pin a reference model, done per connected component (offsets between components are unidentified). |
| Ridge prior ½·ridge·‖r‖² | One term, three jobs: (1) identifiability — strictly convex objective; (2) regularization — sparse models pulled to mean 0 with wide CIs; (3) conditioning — invertible Hessian for Fisher CIs. Never applied to η. RATING_RIDGE=0.01, STYLE_RIDGE=0.05. |
Confidence intervals
| Method | How |
|---|---|
| Primary — Fisher information | Laplace approximation at the optimum ⇒ covariance = inverse Hessian of the penalized NLL (observed Fisher info). Hessian built by central-differencing the analytic gradient (O(n)); covariance projected through the anchoring contrast so the common-mode direction doesn't dominate. One matrix computation, not hundreds of refits. |
| Validation — multinomial bootstrap | Flatten triples → draw each dataset from Multinomial(N, proportions) → re-aggregate → warm-started refit. Column std-devs confirm the analytic SEs, never touching raw rows. A test asserts they agree. |
Reported as rating ± z·stderr, z ≈ 1.96 (95%).
Connectivity → per-component leaderboards
connectivity.py runs union-find over the aggregated pairs; ratings are only comparable within a connected component. Each model gets a componentId so clients can render per-component boards. Isolated models get their own component and wide intervals.
Display scale
display = 1000 + (400 / ln 10) · r SCALE = 400/ln10 ≈ 173.72The classic Elo constant: a 400-point gap ≈ 10× expected win odds, matching P = σ(r_i − r_j). SEs and CI half-widths scale by the same factor; centering is per-component, then baseline 1000 added.
Refit cadence & scaling
| Aspect | Detail |
|---|---|
| One-shot vs loop | python -m omniarena_rating (once) or --loop every REFIT_INTERVAL_SECONDS (300). Docker's worker service runs the loop. |
| Warm start | Most loop refits warm-start from the previous [r…, η] vector (in-memory only) ⇒ few iterations; skipped (cold refit) when the model set changes. |
| Periodic full recompute | Every FULL_REFIT_EVERY refits (12 ⇒ hourly at the default interval; 0 disables) the warm state is discarded and the fit runs from scratch — the ground-truth pass that bounds how long an incremental chain can accumulate drift. The loop's first refit is already cold, so forced passes land on refits 1, 13, 25, … Each refit logs mode=full or mode=incremental. |
| Warm-path validation | The ridge makes the raw optimiser vector the unique minimiser, so the ground-truth pass also re-runs the incremental path over the same aggregates and warns when they disagree by more than half a standard error. SE units, not display points: L-BFGS-B stops on a relative function tolerance, so absolute accuracy loosens as vote volume grows. |
| Aggregate-then-compute | SQL GROUP BY → (model_lo, model_hi, wins_lo, wins_hi, ties) triples; fit input bounded by ~3·C(n,2) rows regardless of vote volume (O(votes) → O(pairs)). Raw rows stay in Postgres. |
Pre-fit anomaly screen
anomaly.py screens anonymous sessions before aggregation (Bonferroni α/3); a rejection excludes the session from both passes. On by default (--no-anomaly-filter off).
| Test | Null | Catches | Runs when |
|---|---|---|---|
| Volume | Poisson upper tail P(X≥n) vs mean votes/session | vote-stuffing | ≥ 20 votes in the session |
| Position bias | two-sided binomial on decisive left/right vs p=0.5 | always-left/right bots | ≥ 15 decisive votes |
| Speed | median inter-vote gap below floor (1.5 s) | automated clicking | ≥ 8 vote timestamps |
α = 1e-3, so each test rejects below α/3 ≈ 3.3e-4. The sample gates are why a small session is never flagged — a handful of votes cannot be distinguished from a fast, one-sided human. Votes with no anonymous_session_id are always kept: they cannot be attributed to a voter, so there is nothing to exclude.
Style-controlled ratings
Voters reward superficial traits. Following LMSYS, style.py folds them into the same BT regression as covariates so strengths and style betas fit jointly:
d = (r_a − r_b) + β · x P(A ≻ B) = σ(d − η)| Feature | Delta of (A − B) |
|---|---|
| position | constant 1.0 → coefficient = systematic left-slot advantage |
| verbosity | output_token_count |
| formatting | markdown_density |
| latency_ttft | ttft_ms |
| latency_duration | stream_duration_ms |
Deltas vary per vote, so this fit can't use pair aggregation — it runs on raw preferences ⋈ matchups ⋈ responses rows, a heavier periodic --style pass. Ridge on all coefficients; SEs via the same inverse-Hessian machinery. Results → model_style_ratings; coefficients → style_control_coefficients; exposed as styleControlledRating.
Leaderboard fields
| Field | Meaning |
|---|---|
| rating | Bradley-Terry rating, Elo-like scale |
| ratingStdError | Standard error (same scale) |
| confidenceInterval | 95% CI { lower, upper } from Fisher information |
| componentId | Connected component; ratings comparable only within it |
| styleControlledRating* | Style-controlled rating, SE, and CI (heavier --style pass) |
rating*/componentId are null until the default pass runs; styleControlled* until the style pass runs. They also stay null for a model with no comparisons to fit (see What the engine cannot rate). Clients fall back to winRate. See API → leaderboard and Data model.
Where the counts come from
The win/loss/tie columns are not the worker's. The server aggregates them directly over models ⋈ matchups ⋈ preferences for every enabled model, so they exist from the first vote and never wait on a refit.
| Field | Counted as |
|---|---|
| wins | votes whose winner_model_id is this model |
| losses | left/right votes on its matchups where the winner is another model |
| ties | both_good + both_bad votes on its matchups |
| skips | skip votes on its matchups |
| totalVotes | every vote on its matchups, skips included |
| winRate | wins / (wins + losses + ties), or 0 when that denominator is zero |
The worker's own games figure (persisted with each rating and exposed by the rating history) is a different number: summed from the aggregated pair triples, so it counts non-skip votes only and credits both models of a pair with the same total. A model whose only comparisons came from a session the anomaly screen excluded therefore shows totalVotes > 0 and games = 0.
Rating history
model_ratings is an upsert keyed by model, so it holds only the latest fit — it says where a model stands, never how it got there. Every refit therefore also appends one snapshot row per model to model_rating_history (migration 005_rating_history.sql), in the same transaction as the upsert, so the two can never disagree about a refit.
| Column | Meaning |
|---|---|
| model_id, computed_at | Primary key. computed_at is NOW() — transaction-stable, so every row a refit writes shares one timestamp and one refit is one point on the x-axis |
| rating, rating_stderr, ci_lower, ci_upper | Exactly the display-scale values written to model_ratings |
| component_id | The component this fit put the model in; it can change between refits as bridging games arrive |
| games | Non-skip comparisons behind this snapshot (see where the counts come from) |
| Before reading a chart | Why |
|---|---|
| The series is per refit, not per vote | One point every REFIT_INTERVAL_SECONDS at the default cadence; a refit that skips (no enabled models, or no comparisons yet) writes nothing at all |
| Both refit modes append | A warm-started incremental refit and a cold ground-truth pass produce indistinguishable rows — which is what makes the warm-drift check a check on the solver rather than something a reader does by eye |
| Ratings move without any model changing | They are anchored sum-to-zero per component, so one model's improvement lowers every other rating in its component, and a component split or merge re-anchors the whole group |
| The style pass has no history | model_style_ratings and style_control_coefficients are upserts with no append-only sibling, so style-controlled ratings have a current value only |
Exposed as GET /api/arena/analytics/rating-history (API → analytics), which is what the rating-over-time chart reads. Empty until the first refit after the migration; rows are never updated or deleted, and a deleted model's snapshots go with it (ON DELETE CASCADE).
Verification
The worker suite (worker/tests/, pure-Python pytest, no database) is what keeps the claims above honest:
analytic gradient vs finite differencesrecovers synthetic ratingstie modeling + anchoringFisher CIs vs bootstrapconnectivity splits a disconnected graphstyle control shrinks a verbosity advantageanomaly screen flags spam/bot sessionsloop alternates cold and warm refitshistory appended in the upsert's transaction
See Setup → Rating worker to run them.