last updated 2026-08-05
Every ranked comparison comes from a blind battle: a user submits a prompt, 2–6 deep-research models run it in parallel, and the responses are shown under anonymous slot labels (A, B, C…). Provider identities, per-provider cost, and token usage are withheld until after the vote is cast — server-side, not just in the UI. Direct (non-blind) comparisons never enter the ranking. Random battles draw at most one model per provider family, and premium-only models are excluded from the random pool. Votes on battles where fewer than two providers produced a completed response are void (a walkover is not a comparison), and a winning vote must point at a response that actually completed.
Voters can pick a single best response, declare a tie, or declare all responses bad. A tie is scored as half a win for each side of every pair. “Both bad” is recorded (it is evidence about absolute quality) but contributes no pairwise preference to the ranking.
A win in a 6-way battle and a win in a 2-way battle carry different information (base rates 1/6 vs 1/2), so multi-way battles are decomposed before scoring: a vote for W in a battle with opponents L1…Lk becomes the pairwise observations W>L1, …, W>Lk. No loser-versus-loser pairs are inferred — the voter only identified a best.
Rankings are fit with a Bradley-Terry model — P(i beats j) = pi/(pi+pj) — over the full pairwise history (the same model family used by chess Elo and LMArena), via the MM algorithm with a weak symmetric prior for numerical stability. Displayed scores are 1000 + 400·log₁₀(strength), so an average model scores 1000 and +400 means 10:1 odds. 95% confidence intervals come from a 200-round bootstrap that resamples whole votes (a multi-way battle’s pairs are correlated, so the vote is the sampling unit) with a fixed seed for reproducibility. The score column is the leaderboard’s sort key — the number shown is the number sorted by.
Rankings are computed by a scheduled job that pages through the complete vote history — never by a request-time scan (which previously truncated silently at 1,000 votes). The leaderboard serves the latest stored snapshot.
Models with fewer than 200 pairwise comparisons are labeled provisional: their scores are shown for transparency but they are not ranked and never receive medals. Small-sample win rates are noise; a 3-for-3 model does not outrank a 120-for-200 one.
Votes aggregate under canonical model keys. Speed variants of the same engine (e.g. Parallel’s “-fast” and 2×/4×/8× compute multipliers of a processor tier) fold into one row so a provider’s sample size isn’t structurally shattered across a dozen keys while single-model providers concentrate in one. Genuinely different capability tiers stay separate rows.
One vote per battle per account (or per IP + browser fingerprint for anonymous voters, with a CAPTCHA verified server-side). Votes are only accepted on completed blind battles. Slot cancellation requires the authenticated battle owner. Per-battle metadata (random vs manual selection, prompt category) and per-response metrics (length, citation count, formatting stats) are persisted with every run so future analysis can regress out verbosity and citation-count bias. Retired models — ones nobody can run anymore — are excluded from the default board; see the model lifecycle policy.