Architecture
Blind multi-turn comparisons streamed through a five-protocol egress-adapter layer (native SSE default), a WebSocket control plane, slot join for one-request-per-model clients, a published headless React SDK, signed votes, multiple providers, smart matchmaking, and an off-path Bradley-Terry rating worker with a style-controlled pass and a pre-fit anomaly screen.
System overview
| Package | Path | Role |
|---|---|---|
| @omni-arena/server | server/ | Fastify API: matchmaking, dual-model streaming through the protocol-adapter layer, WebSocket control plane, slot join, voting, leaderboard, analytics |
| @omni-arena/react | packages/react-sdk/ | Published headless React SDK: hooks over framework-free protocol/session/stream/vote modules; the demo consumes it |
| @omni-arena/web | web/ | Vite + React demo UI; the reference consumer of the SDK |
| omniarena-rating | worker/ | Python Bradley-Terry rating worker (NumPy/SciPy) |
e2e/, examples/*, and integrations/* are consumer code outside the workspace — each with its own package.json and lockfile, installed on demand — so a root install never builds them and an upstream's dependency tree cannot reach the published packages. See Integrations layer.
web (React demo)
imports @omni-arena/react · Vite proxy → :3001
@omni-arena/react (SDK)
useArenaChat · useArenaVote · useArenaLeaderboard · useArena*Analytics · headless hooks
▼ HTTP + native SSE (default) · WebSocket control
Fastify routes
/api/arena/chat (+ /v1/chat/completions) · /models & /v1/models · /control (WS) · /vote · /leaderboard · /analytics/* · /health
ArenaCore
parallel fan-out, one event stream, fault isolation, AbortSignal
Adapter layer
selectProtocol → SSE · AG-UI · A2UI · Vercel AI · OpenAI SSE (egress + ingress)
JoinBroker
pairs two sibling requests into one matchup (opt-in joinKey)
MatchupRegistry
AbortController + steer handler per matchup
SmartMatchmaker
under-sampled / high-variance pairs (RandomMatchmaker fallback)
MatchupTokenService
HMAC token, 15 min TTL, hash stored
▼
PostgreSQL
conversations · turns · matchups · responses · preferences · model_ratings · model_style_ratings · model_rating_history
Model endpoints
Google · OpenAI · Ollama · vLLM · host proxy · mock
▲ writes model_ratings (+ append-only model_rating_history) + model_style_ratings (off the hot path)
worker (Python)
anomaly screen · Bradley-Terry + Rao-Kupper ties · Fisher-info CIs · connectivity · style-controlled pass · periodic refit
Distribution topology (single container)
Self-hosted as a single container per deployment: one Fastify process serves the API and the built web UI on one port (PORT, default 3001), same-origin. Static serving (via @fastify/static) activates only when web/dist exists (production/Docker), with an SPA fallback to index.html for any GET route outside the API prefixes /api, /health, /v1, /models, /chat/completions, /completions, /embeddings (WEB_DIST_DIR overrides the path). That exclusion list matters because the OpenAI-compatible surface lives at top-level paths: otherwise an unmatched GET /v1/models would answer HTML at 200 and an OpenAI client would report a mimetype error instead of a missing route. In npm dev the bundle is absent, so Vite serves :5173 and the API stays on :3001. docker compose up brings up Postgres + rating worker + app (entrypoint: migrate → seed → start). See Setup.
Ports (hexagonal boundaries)
| Port | Implementation | Future replacement |
|---|---|---|
| ModelProviderPort | Google, OpenAI-compatible, Ollama, host proxy, mock | additional providers |
| ProviderResolverPort | ProviderRegistry | unchanged |
| MatchmakingPort | SmartMatchmaker (default) / RandomMatchmaker (fallback) | King-of-the-Hill / bandit variants |
| MatchmakingStatsPort | PostgresRepository (pair counts + CI widths) | unchanged |
| PreferenceRepositoryPort | PostgresRepository | unchanged |
| LeaderboardPort | win-rate SQL + model_ratings + model_style_ratings LEFT JOINs, plus getRatingContext() (component connectivity + fitted style coefficients) | more surfaced rating variants |
| AnalyticsPort | summary · head-to-head · model-metrics · activity · style-control · rating-history aggregations (percentiles/bucketing in TS for pg-mem testability) | prompt-category scoping |
| EventAdapter (egress) | native SSE (default) + AG-UI · A2UI · Vercel AI SDK · OpenAI SSE, via selectProtocol | more wire protocols |
| RequestAdapter (ingress) | AG-UI, OpenAI, and Vercel AI SDK request envelopes; native SSE and A2UI have none and accept only OmniArena's own body | more envelopes |
Core ports are defined in server/src/core/ports.ts; the egress EventAdapter port in server/src/adapters/event-adapter.ts and the ingress RequestAdapter in request-adapter.ts. Tests inject in-memory implementations behind the same interfaces.
Stream orchestration
ArenaCore.stream() merges concurrent provider streams into one async event queue. A single plan omits slot B; a shadow plan runs both, but the chat route only forwards A-facing events while still persisting both slot_done payloads. If one slot errors, the other keeps streaming.
token
slot_error
slot_done
matchup_done
Internal completion events capture TTFT, duration, provider/estimated tokens, Markdown density, and provider-reported model version. Public slot_done strips them; every matchup stores HARNESS_VERSION. The chat route hands each public event to the selected protocol adapter for framing.
Trigger and exposure
| Axis | Env | Values | Default |
|---|---|---|---|
| Trigger (when) | ARENA_TRIGGER | always | manual | sampled | always |
| Exposure (what) | ARENA_EXPOSURE | blind | shadow | blind |
| Plan | Meaning |
|---|---|
| matchup | Blind A/B streamed, votable (blind + engaged) |
| shadow | A streamed, B silent + logged, not votable (shadow + engaged) |
| single | One model, nothing persisted (not engaged) |
Defaults (always + blind) reproduce the historic every-request matchup path. Shadow requires ARENA_DEFAULT_MODEL (incumbent); the challenger is matchmaker-picked and ≠ incumbent. See Setup → Trigger and exposure.
Egress: protocol-adapter layer
One internal event stream (zod publicArenaEventSchema in core/events.ts) is fanned out to five wire protocols through a small egress port, so the chat route knows no framing details.
PublicArenaEvent stream
matchup_started · token · slot_error · slot_done · matchup_done · run_error (schema-validated)
▼ selectProtocol(?protocol → Accept → default)
native SSE
default · byte-for-byte unchanged
AG-UI
typed run/text events over SSE
A2UI
schema-validated NDJSON
Vercel AI SDK
UI message stream v1
OpenAI SSE
chat.completion.chunk
| EventAdapter member | Role |
|---|---|
| headers | Response headers the protocol needs before the first chunk |
| serialize(event) | Wire-ready bytes for one schema-validated event |
| finalize() | Trailing bytes before close (e.g. a [DONE] sentinel) |
| inBandErrors? | Set for a protocol whose clients settle a run on a terminal error event and read a non-2xx as a dead transport (AG-UI): the route then delivers a pre-stream failure as a run_error at 200 rather than an HTTP status |
Unknown ?protocol= values fall back to native SSE, so the demo, the SDK, and existing clients are unaffected. Internal event semantics are identical across protocols; only the framing differs. See API → Protocol selection for aliases, media types, and framing.
Ingress too, for three protocols. A second port, RequestAdapter (request-adapter.ts), is implemented by the protocols with a canonical client request envelope — AG-UI's RunAgentInput, OpenAI's /chat/completions body, the AI SDK's useChat body. Two members: claims(body) (is this the protocol's envelope rather than OmniArena's?) and parse(body), which translates it into the one internal ArenaChatRequest the route runs on. Detection is structural — a messages array and no prompt — so OmniArena's own body is unchanged everywhere, and stock clients of those three need no transport of their own. selectProtocol() resolves both halves from one decision. See Integration → request bodies.
Slot failures on text-only protocols. AG-UI and OpenAI have no per-message error taxonomy, only assistant text, so a slot_error is carried twice there: structurally (AG-UI's CUSTOM slot_error, the OpenAI adapter's omni_arena_error extension) and as text prefixed with the shared [omni-arena:slot-error] marker (adapters/slot-error.ts), so a client that only renders content shows something instead of a permanently blank column — and the marker is what keeps that text distinguishable from the model having said those words.
Roster discovery is part of the surface. GET /models and GET /v1/models (routes/models.ts) return the enabled roster in OpenAI's {object:"list", data} shape — both paths, since a deployment may be configured with or without the /v1 prefix. An OpenAI client's first call is the model list (Open WebUI treats it as the connection check and builds its picker from it), so without it the OpenAI adapter is undiscoverable however well-formed its stream is. The roster is not secret; what stays blind is which two models a matchup drew.
WebSocket control plane
GET /api/arena/control (registered via @fastify/websocket in app.ts) acts on an in-flight matchup out-of-band from the token stream.
MatchupRegistry
AbortController + steer handler per matchup; register / bindSteer / release
ArenaCore.stream(…, signal?, attachSteer?)
stop closes the queue; steer aborts the generation and restarts both slots
| Message | Status |
|---|---|
| stop | Works: aborts the matchup's stream via its AbortController; ok:false for an unknown/finished matchup. |
| steer | Abort-and-restart: cancels the current generation, emits steered, re-runs both slots with the identical system instruction; persists on matchups.steers. Negative ack for unknown/expired or already-completing matchups. |
Slot join: one matchup across two requests
The default shape puts both slots on one connection. Real compare-view chat UIs don't: they fan a multi-model turn out into one request per model sharing a conversation id (Open WebUI v0.10 is the measured case). Each such request has exactly one answer channel, so interleaving both slots on it either garbles the two answers together or yields two unrelated matchups and two half-votes. A client opts in with joinKey; two requests resolving to the same scope inside a window become one matchup, one matchups row, one vote.
request 1 → leader (slot A)
reads conversation · matchmaker · token · inserts the matchup · owns the pump and all persistence
JoinBroker.claim(scope)
synchronous role assignment — no await between deriving the key and recording the claim, so two simultaneous siblings can never both win
request 2 → follower (slot B)
awaits the handshake, then streams slot B only; repeats none of the leader's work
▼ one ArenaCore generation → JoinedRound → one SlotChannel per slot
connection A
matchup_started slots:["A"] → tokens → slot_done → matchup_done
connection B
matchup_started slots:["B"] → tokens → slot_done → matchup_done
| Piece | Role |
|---|---|
| JoinScope | What a join is authorized by: HMAC of {sessionId, conversationId, prompt, joinKey} under a per-process random secret. The joinKey alone is only a correlation id the client already has, so it is deliberately not the capability; the tuple is strictly stronger than the session id that already gates conversation access, and the key never leaves the process. |
| JoinedRound | The single shared generation, demultiplexed per slot. The leader keeps consuming even if its own client disconnects, so slot B still finishes and is recorded. |
| SlotChannel | Single-producer/single-consumer queue drained by one HTTP response. Aborting one connection ends only that connection; each side gets matchup_done as soon as its slot finishes. Backlog capped at ARENA_JOIN_MAX_QUEUED_EVENTS (default 4096), then the slot fails rather than growing without bound. |
| JoinHandshake | What the leader publishes to the follower: matchupId, matchupToken, conversationId, turnIndex. |
| Failure | Response |
|---|---|
| joinKey without a sessionId | 400 join_requires_session |
| Both slots of the scope already claimed | 409 join_slots_exhausted |
| Sibling arrives after the window closed | 409 join_expired |
| More than ARENA_JOIN_MAX_PENDING unpaired scopes | 503 join_unavailable |
| Leader never publishes (window + 30s) | 504 join_leader_timeout |
| Pre-stream failure on the leader | The leader's own status/code, forwarded so both siblings report the same thing (500 join_failed when unclassified) |
A window that closes with no sibling is not an error: the round degrades to exactly the default shape — both slots on the leader's connection, votable, nothing wasted. The leader's control-plane AbortController cascades onto the shared generation, so one stop stops both slots. ARENA_JOIN_WINDOW_MS=0 disables joining and a joinKey is then ignored — see Setup → environment variables.
Linear history
Turn 0
prompt → A/B → decisive vote
Winning response
server derives parent; client history ignored
Turn N
prior winners + next prompt → new A/B
Unique parent_response_id and (conversation_id, turn_index) prevent branching. Session IDs gate ownership. Ties, both-bad, skips, and unvoted turns end the branch.
Provider and key-custody modes
| Mode | Behavior |
|---|---|
| Direct custody | Google, OpenAI-compatible, or Ollama adapters call providers with OmniArena-held configuration; vLLM reuses OpenAI compatibility. |
| Host custody | HostProxyModelProvider calls the host's OpenAI-compatible proxy; upstream keys never enter OmniArena. |
| Mock (demos/e2e) | MockModelProvider streams deterministic tokens with no network; registered only when ARENA_MOCK_PROVIDER=1 so it never shadows a real provider. See Setup. |
| Persistence | Prompts and responses are stored as received; OmniArena does not transform or redact content. |
Blind voting integrity
| Guarantee | Mechanism |
|---|---|
| No identity leak pre-vote | matchup_started carries only slot IDs + token |
| Unforgeable votes | HMAC-SHA256 signed token with matchup + slot claims, 15 min expiry |
| Token never stored | Only its SHA-256 hash lands in the database |
| One vote per matchup | Unique constraint on preferences.matchup_id |
| Holds for every protocol | blindness.test.ts generates its cases from the protocol registry and asserts no identity reaches the wire on a single connection or on either sibling of a joined matchup — so a new adapter is covered without a new test |
Rating worker
The worker/ package (omniarena_rating) computes the style-agnostic default leaderboard rating as a separate Python process — never on the Fastify request path. It follows a strict aggregate-then-compute design.
Its only input is recorded comparisons (preferences ⋈ matchups). A non-votable single round persists neither, so it never reaches the worker and a deployment mostly serving such rounds has nothing to rate. See Rating methodology → what the engine cannot rate.
| Step | Module | What it does |
|---|---|---|
| Anomaly screen | anomaly.py | Runs before aggregation: p-value tests over anonymous sessions flag spam/malicious voters, excluded from the fit. |
| Aggregate | aggregate.py | One SQL GROUP BY → canonical (model_lo, model_hi, wins_lo, wins_hi, ties) triples. skip + flagged sessions excluded. O(votes) → O(model pairs); raw rows stay in Postgres. |
| Fit | bradley_terry.py | Log-parameterized BT MLE via SciPy L-BFGS-B with analytic gradient. Rao-Kupper tie model (ordered logit, threshold ±η). Weak ridge prior, sum-to-zero anchoring, warm-start between loop refits — except every FULL_REFIT_EVERY refits (default 12), where the warm state is discarded for a from-scratch ground-truth fit that also cross-checks the warm path for drift. |
| Intervals | confidence.py | Inverse-Hessian (observed Fisher information) CIs projected through the anchoring contrast, validated by a multinomial bootstrap over the triples. |
| Connectivity | connectivity.py | Union-find components; ratings only comparable within a component, isolated models get wide intervals. |
| Style control | style.py | Heavier periodic pass on raw votes: joint style-controlled BT regression (see below). |
| Write back | writeback.py | Idempotent upsert into model_ratings (default) and model_style_ratings + style_control_coefficients (style pass). The default pass also appends a snapshot per model into model_rating_history in the same transaction — the trail behind /analytics/rating-history. |
Rating methodology: Rao-Kupper gives P(a≻b)=σ(d−η), P(b≻a)=σ(−d−η), P(tie)=σ(d+η)−σ(d−η) with d = r_a − r_b. The ridge-penalized log-likelihood is convex; the ridge (a Gaussian prior) identifies ratings, regularizes sparsely-compared models, and keeps the Hessian invertible for Fisher-information CIs. Reported on an Elo-like scale 1000 + (400/ln10)·r, centered per component. Full story in the rating methodology doc.
L-BFGS-B
Rao-Kupper ties
ridge prior
sum-to-zero anchor
Fisher-info CIs
multinomial bootstrap
warm-start refits
periodic cold refit
Compose runs both passes. The worker service's image CMD is --loop --style, so a stock docker compose up keeps model_style_ratings and style_control_coefficients current alongside model_ratings, and the dashboard's style panels have data from the first refit onward. See Setup → rating worker.
Style-controlled ratings
Following LMSYS, style.py folds voter confounders into the same BT logistic regression as covariates: d = (r_a − r_b) + β·x, P(A≻B) = σ(d − η). Strengths and style coefficients are fit jointly with a ridge on all of them; because the per-vote deltas x vary, it runs on raw preferences ⋈ matchups ⋈ responses rows — a heavier pass (--style) kept off the fast path. Exposed as styleControlledRating.
position (left-slot)
verbosity (tokens)
formatting (markdown)
latency (ttft)
latency (duration)
Anomaly detection
anomaly.py screens anonymous sessions before any fit (Bonferroni-adjusted at α/3); a rejection excludes the session from both passes.
| Test | Null hypothesis | Catches |
|---|---|---|
| Volume | Poisson upper tail P(X≥n) vs mean votes/session | vote-stuffing |
| Position bias | two-sided binomial on left/right vs p=0.5 (slots randomized) | always-left/right bots |
| Speed | median inter-vote gap below a floor (default 1.5s) | automated clicking |
On by default; --no-anomaly-filter disables it.
Smart matchmaking
SmartMatchmaker replaces uniform pair selection behind the same MatchmakingPort. It samples a pair proportional to an information score: coldness = 1/(1+games) (favor under-evaluated pairs) plus normalized rating-interval width (favor high-variance matchups; unrated models count as maximally uncertain). A small floor keeps every pair reachable. Default; MATCHMAKER=random restores the uniform matchmaker.
Frontend
The headless hooks live in the published @omni-arena/react SDK (packages/react-sdk/), and the demo imports them — a single source of truth. The package is deliberately layered: protocol, session, stream, and vote are framework-free modules and each hook is a thin React shell over them, so a non-React client can reuse the same wire logic instead of reimplementing it. Every hook accepts an optional baseUrl (default "" = same-origin/proxied).
| Module | Exports | Responsibility |
|---|---|---|
| protocol.ts | parseArenaMatchup · parseArenaReveal · parseArenaSlotError · isArenaSlot/isArenaVote/isDecisiveVote | Narrow untrusted wire JSON into typed events — the one place the wire shape is known |
| session.ts | getSessionId | Anonymous session id in localStorage, degrading to in-memory when storage is unavailable |
| stream.ts | readArenaStream · createArenaSseDecoder | Incremental SSE decoding over a ReadableStream, independent of any hook |
| vote.ts | submitArenaVote | One POST /api/arena/vote with the matchup token |
| useArenaChat.ts | useArenaChat | POSTs the prompt, drives readArenaStream into per-slot state, votes, and reveals identities only after a successful vote |
| useArenaVote.ts | useArenaVote | Voting on its own, for a UI that renders the stream itself (the integrations do) |
| useArenaLeaderboard.ts | useArenaLeaderboard | Standings plus rating context |
| useArenaAnalytics.ts | useArenaSummary · useArenaHeadToHead · useArenaModelMetrics · useArenaActivity · useArenaStyleControl · useArenaRatingHistory | One hook per analytics endpoint, all with the same { data, refresh, error } shape |
The demo (web/) is a two-page SPA under react-router-dom (the server's SPA fallback makes deep links work in production). / — ArenaPage: prompt box, two anonymous markdown panes, five vote buttons, reveal, and a leaderboard that shows the Elo-like rating ±CI half-width (falling back to win-rate %), plus style <rating> when a style-controlled rating exists. /insights — InsightsPage: a summary stat strip plus four chart tabs (web/src/dashboard/, primitives in web/src/charts/): Rankings (win-rate lollipop with 50% reference line, BT forest plot with CI whiskers, raw-vs-style-controlled dumbbell, rank-shift bump, vote-outcome stacked bars — all from the leaderboard payload), Head-to-head (pair win-rate matrix with games toggle and drill-down, tie-rate bars, connectivity callout by componentId), Style & bias (style-coefficient panel, slot-A vs slot-B position dumbbell, verbosity/latency/markdown vs win-rate scatters, latency p50–p90 spread), and Activity (rating-over-time lines, stacked vote volume, cumulative games). Ranked row charts are hand-rolled CSS grid/SVG; time series and scatters use recharts. Each tab is a nested route (/insights/rankings, /insights/head-to-head, /insights/style, /insights/activity; the index redirects to rankings) so a tab is a shareable deep link rather than component state, and each is lazily imported on its own — the arena never pays for the charting bundle and opening one tab does not load the other three.
Empty dashboards teach. Every chart derives from recorded votes, so a fresh deployment has nothing to plot and a card can only report what is missing. GettingStartedNotice states the remedy once at page level — the demo-data seeder when there are no votes, the worker command when votes exist but no fit does — and disappears once both are present. The page fetches the summary aggregate once and passes it to both the stat strip and the notice.
Reference integrations: two runnable example apps in examples/ — a Next.js + Vercel AI SDK app and an assistant-ui app — drive arena mode through the Vercel AI SDK adapter and are exercised by the deterministic e2e/ suite (npm run e2e, Playwright + mock provider), which also asserts the raw vercel-ai/ag-ui wire streams. See the integration guide.
Integrations layer
integrations/ is the layer above the examples: instead of apps written for the arena, it wires the arena into real upstream chat UIs at pinned revisions — which is what turns "the protocol should work" into evidence that it does.
| Directory | Upstream | Protocol | Shape |
|---|---|---|---|
| vercel-ai-chatbot/ | vercel/ai-chatbot | vercel-ai | Pinned clone + overlay |
| assistant-ui/ | assistant-ui monorepo, examples/with-ag-ui | ag-ui | Pinned clone + overlay |
| open-webui/ | Open WebUI container image | openai | Published image + OpenAI-compatible bridge |
Pinned clone + overlay. upstream.json pins the exact commit, .upstream/ is the gitignored clone, overlay/ holds the arena sources copied in verbatim, and scripts/overlay.mjs applies anchored patches to upstream's own files — each anchor must match exactly once, so an upstream that moved a line fails setup loudly instead of yielding a half-integrated app. The arena-specific code stays reviewable in overlay/ without diffing a vendored tree.
Open WebUI is the case that shaped the server. It sends one request per model for a compare turn — exactly what slot join exists to serve — and reaches the arena through a small OpenAI-compatible bridge (bridge/) rather than an overlay, because the UI ships as a container image. All three run key-free against the mock provider; see Setup.
Intentionally not built yet
Auto-judge for shadow matchups — ARENA_EXPOSURE=shadow persists both responses with matchups.mode='shadow' and rejects human votes; feeding those rows into an LLM-as-judge → recordPreference path is a separate workstream (see Setup → Trigger and exposure). Multimodal input — deferred. OmniArena does not scrub or redact stored prompts/responses — content is persisted as received.