Setup

Self-hosted as a single container per deployment: one Fastify process serves the API and the built web UI on one port. docker compose up brings up Postgres, the rating worker, and the app in one command; an npm path stays available for development.

Quick start · Docker (single-tenant self-host)

cp .env.example .env      # set GOOGLE_API_KEY and MATCHUP_TOKEN_SECRET
docker compose up

Open http://localhost:3001 — web UI and /api/... share the same origin and port (PORT, default 3001). The multi-stage Dockerfile builds all workspaces, ships the migration .sql files next to dist, and prunes dev deps into a slim runtime. The server serves web/dist via @fastify/static only when the bundle exists (production/Docker), with an SPA fallback to index.html for GET routes outside the API prefixes /api, /health, /v1, /models, /chat/completions, /completions, /embeddings — an unmatched path under one of those gets a JSON 404, because HTML at 200 for GET /v1/models makes an OpenAI client report a mimetype error instead of a missing route. Override the bundle location with WEB_DIST_DIR.

Quick start · npm (development)

cp .env.example .env                    # configure the seeded provider
npm install
docker compose up -d postgres worker    # Postgres 16 + rating worker
npm run db:migrate --workspace server
npm run db:seed --workspace server
npm run dev                             # server :3001 · web :5173

Open http://localhost:5173; Vite proxies /api and /health to :3001.

Workspaces

An npm workspace monorepo. The root workspaces array is in build order — server, packages/react-sdk, then web — so the SDK builds before the demo that depends on it.

Environment variables

Host-proxy contract

Trigger and exposure

By default every request to POST /api/arena/chat is a blind A/B matchup. Two orthogonal axes relax that: trigger (ARENA_TRIGGER — when) and exposure (ARENA_EXPOSURE — what the user sees).

# Shadow canary against a fixed incumbent
ARENA_TRIGGER=always
ARENA_EXPOSURE=shadow
ARENA_DEFAULT_MODEL=gemini-3-flash-preview
# Client sees mode:shadow, votable:false, slots:["A"], no token
# Both answers land in Postgres with matchups.mode='shadow'

Database

Demo data

Every chart on the insights dashboard is computed from recorded votes, so a fresh install renders nothing but empty states. server/src/db/seed.demo.ts fabricates a plausible voting record against whichever models are currently enabled.

# npm dev path (host)
npm run db:seed:demo --workspace server
npm run db:seed:demo --workspace server -- --reset --matchups 400 --days 30

# Docker path — run the compiled script directly. `tsx` is a devDependency and
# is pruned from the runtime image, so the npm script is not available there.
docker compose exec app node server/dist/db/seed.demo.js --reset

Each model's personality is derived from its name — display_name plus provider_model_id, lowercased — so the seeded arena matches the intuition a reader already has about the roster. Markers must match a whole name segment, since gemini ends in mini.

Mock provider

docker compose up -d postgres
ARENA_MOCK_PROVIDER=1 npm run db:migrate --workspace server
ARENA_MOCK_PROVIDER=1 npm run db:seed:mock --workspace server
ARENA_MOCK_PROVIDER=1 npm run dev --workspace server   # :3001

Rating worker

The Bradley-Terry rating worker lives in worker/ (Python, NumPy/SciPy). It screens anomalous voting sessions, aggregates votes in-database, fits ratings with Fisher-information CIs, and upserts model_ratings. --style adds the heavier style-controlled pass into model_style_ratings.

cd worker
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# One-shot (needs migrations applied + DATABASE_URL):
DATABASE_URL=postgres://omni_arena:omni_arena@localhost:5432/omni_arena \
  python -m omniarena_rating

# Add the heavier style-controlled pass on raw votes:
python -m omniarena_rating --style

# Periodic loop with warm-started refits (add --style for the style pass):
python -m omniarena_rating --loop --interval 300 --ridge 0.01

# Same loop, but never force a from-scratch refit (default: every 12th):
python -m omniarena_rating --loop --full-refit-every 0

# Skip the pre-fit anomaly screen (keep every session):
python -m omniarena_rating --no-anomaly-filter

python -m pytest          # pure-Python tests, no database

Commands

Reference examples

Third-party UI integrations

Beyond the purpose-built examples, integrations/ wires the arena into real upstream chat UIs at pinned revisions. Each directory is self-contained — its own package.json, lockfile, tests, and README — and outside the root workspace, so a plain root install neither installs nor builds them.

End-to-end tests

Continuous integration