← benchmarks systems External benchmarks public benchmarks for web search and deep research that research.site does not host. we don’t run these — we catalog them: what each measures, who wrote it, how it is graded, and every published score for the systems we track. each score links to the page it was read on (2026-10-21). our own controlled studies are on /benchmarks ; per-provider results from both live on /table .
33 benchmarks · 137 published scores ■ scored by benchmark authors / third party■ vendor-run — published by a vendor in the comparisonvendor-authored benchmark written by a vendor it scores
deep-research reports (8) long-form cited reports, judged against rubrics or reference reports DeepResearch Bench 2025-06-13 # academic PhD-level research tasks answered as long-form reports. RACE scores each report against a reference report on comprehensiveness, insight, instruction-following and readability; FACT checks citation accuracy.
Every tracked system on the official board has 2025-era outputs; Parallel, Exa, Valyu, You.com and Keenable have no official entry. Scores on the GPT-5.5 board are not comparable with the frozen Gemini-judge board or the paper table.
by USTC (Du et al.) · muset-ai tasks 100 (50 Chinese, 50 English) across 22 fields metric RACE 0–100, relative to a reference report grading LLM judge — GPT-5.5 (official since 2026-05-11, replacing Gemini-2.5-Pro) status active leaderboard; only the GPT-5.5 board is maintained after 2026-06-01 links system as published score reported by source Gemini gemini-2.5-pro-deepresearch2025 outputs re-judged by GPT-5.5
49.98 benchmark authors huggingface.co ↗ OpenAI openai-deepresearch2025 ChatGPT deep research; model version not named
47.84 benchmark authors huggingface.co ↗ Perplexity perplexity-ResearchPerplexity app Research mode, 2025 outputs
43.05 benchmark authors huggingface.co ↗ Parallel Parallel Ultra / Ultra8xdifferent metric: share of 50 English tasks beating the reference report; runs dated “Aug 11–29”, no year
82% / 96% win rate Parallel · vendor-run parallel.ai ↗ OpenAI GPT-5same Parallel run; not labeled as o3-deep-research
66% win rate Parallel · vendor-run parallel.ai ↗ Perplexity Perplexitysame Parallel run; tier not specified
6% win rate Parallel · vendor-run parallel.ai ↗ context cellcog-maxtop of the GPT-5.5 board
56.72 benchmark authors huggingface.co ↗
DeepResearch Bench II 2026-01-30 # academic Long-form reports graded against binary rubrics derived from expert-written investigative articles, split into information recall, analysis and presentation.
No official entry for Parallel, Exa, Valyu, You.com or Keenable — research.site's DRB2-20 runs those on a frozen 20-task subset.
by USTC & Metastone Technology tasks 132 across 22 domains (ranking uses 130) metric rubric pass rate 0–100 grading 9,430 binary rubrics judged by GPT-5.5 (default since Aug 2026); Gemini-2.5-Pro board kept separately status active leaderboard; data updated Oct 2026 links here system as published score reported by source OpenAI GPT-o3 Deep ResearchGPT-5.5 judge; 45.40 on the Gemini-2.5-Pro board
43.12 benchmark authors huggingface.co ↗ Gemini Gemini 3 Pro Deep ResearchGemini 2.5 Pro Deep Research 39.89 on the same board
41.52 benchmark authors huggingface.co ↗ Perplexity Perplexity Research38.58 on the Gemini-2.5-Pro board
33.57 benchmark authors huggingface.co ↗ context Beane Harness - Jatin Hans - UAEtop of the GPT-5.5 board
56.70 benchmark authors huggingface.co ↗
DRACO 2026-02-12 # vendor-authored Deep-research answers to de-identified real Perplexity Deep Research requests, graded against task-specific expert rubrics for factual accuracy, depth, presentation and citation quality.
Written by Perplexity from Perplexity traffic, and Perplexity ranks first in its own paper. Valyu's re-run ranks Valyu first and mixes its single-run scores with numbers copied from the paper (Perplexity, Gemini, OpenAI) in one table.
by Perplexity (+ one Harvard author) tasks 100 across 10 domains metric normalized rubric score % grading LLM judge — Gemini-3-Pro primary; GPT-5.2 and Sonnet-4.5 as alternates status static paper (v1); dataset public, re-run by Valyu 2026-05-04 links system as published score reported by source Perplexity Perplexity Deep Research (Opus 4.6)pass rate 72.8%; drops to 50.4 under the GPT-5.2 judge
70.5% Perplexity (paper) · vendor-run arxiv.org ↗ Gemini Gemini Deep Researchversion not specified (pre-Max)
59.0% Perplexity (paper) · vendor-run arxiv.org ↗ OpenAI OpenAI Deep Research (o3) 52.1% Perplexity (paper) · vendor-run arxiv.org ↗ Valyu Valyu DeepResearch (Heavy)$2,500 per 1k queries; single run; Max tier not run
72.7% Valyu · vendor-run github.com ↗ You.com You.com Research (exhaustive)$450 per 1k; frontier tier was sales-gated
53.0% Valyu · vendor-run github.com ↗ Parallel Parallel Ultra8x$2,400 per 1k
51.2% Valyu · vendor-run github.com ↗ Exa Exa Deep ReasoningExa search type=deep-reasoning — not the Agent API; $15 per 1k
25.9% Valyu · vendor-run github.com ↗ context Claude Opus 4.6 (search + code tools) 59.8% Perplexity (paper) · vendor-run arxiv.org ↗
ResearchRubrics 2025-11-10 # independent How well deep-research agents satisfy expert-written, fine-grained rubrics on factual grounding, reasoning soundness and clarity.
Only three systems, all 2025 versions.
by Scale AI (Sharma et al.) tasks 101 prompts · 2,593 rubric criteria metric rubric compliance 0–1 (ternary) grading LLM judges (GPT-5, Claude-Sonnet-4.5, Gemini-2.5-Pro) validated against humans; criterion weights −5 to 5 status static paper (ICLR 2026); no leaderboard links system as published score reported by source Gemini Gemini DRbinary 0.615; top system; 2025 version
0.677 benchmark authors arxiv.org ↗ OpenAI OpenAI DRbinary 0.597
0.664 benchmark authors arxiv.org ↗ Perplexity Perplexity DRbinary 0.487
0.566 benchmark authors arxiv.org ↗
LiveResearchBench 2025-10-16 # academic User-centric research tasks that need live web search, scored on presentation, fact/logic consistency, coverage and citation association.
All listed systems are 2025 versions; the top row is from the benchmark's own lab.
by Salesforce AI Research · UW-Madison · Stanford tasks 100 expert-curated across 7 domains metric average of 4 dimensions, 0–100 grading LLM judge ensemble (Gemini 2.5 Pro + GPT-5) validated against humans status leaderboard page, no update date shown (ICLR 2026) links ResearcherBench 2025-07-22 # academic Frontier AI-science questions; rubric coverage of expert insights, plus whether citations are faithful and claims grounded.
AI-research domain only; 2025 system versions. research.site's RB-30 adds Parallel, Exa, Valyu and You.com on a 30-task subset.
by SJTU · SII · GAIR tasks 65 across 35 AI subjects metric rubric coverage 0–1 grading expert rubrics applied by an LLM; judge not named status README leaderboard; last addition 2026-01-04 (COLM 2026) links here system as published score reported by source OpenAI OpenAI Deep Researchfaithfulness 0.84 · groundedness 0.34; top coverage
0.7032 benchmark authors github.com ↗ Gemini Gemini Deep Researchfaithfulness 0.86 · groundedness 0.59
0.6929 benchmark authors github.com ↗ Perplexity Perplexity Deep Researchfaithfulness 0.85 · groundedness 0.56
0.4800 benchmark authors github.com ↗
DeepResearchGym 2025-05-25 # academic Open sandbox (reproducible search API + evaluation) for research reports: key-point recall against ground-truth documents, citation precision/recall, clarity and insight.
Mainly a sandbox, not a leaderboard; OpenAI and Gemini deep research are not evaluated.
by Carnegie Mellon · INESC-ID · NOVA LINCS tasks 1,000 Researchy Questions metric key-point recall (KPR) 0–100 grading LLM judge gpt-4.1-mini, with a 210-query human study status static paper (v3, 2025-12-15); no leaderboard links system as published score reported by source Perplexity perplexity-sonar-deepsearchcitation recall 99.22, precision 55.65; top KPR
72.50 KPR benchmark authors arxiv.org ↗ OpenAI gpt4-search-previewsearch model, not OpenAI deep research
40.01 KPR benchmark authors arxiv.org ↗
ReportBench 2025-08-25 # academic Reports for prompts reverse-engineered from arXiv survey papers: do cited references overlap the survey's, and are cited and uncited statements correct?
Only OpenAI and Gemini deep research (2025) plus base LLMs; no single headline score.
by ByteDance BandAI tasks 100 survey topics metric citation match rate % · factual accuracy % grading GPT-4o extraction; Gemini 2.5 Pro/Flash web fact-check (6-vote majority) status static paper links system as published score reported by source OpenAI OpenAI Deep Researchreference precision 0.385; non-cited factual accuracy 95.83%
78.87% match benchmark authors arxiv.org ↗ Gemini Gemini Deep Researchreference precision 0.145; non-cited factual accuracy 92.21%
72.94% match benchmark authors arxiv.org ↗
agentic web search (12) hard-to-find answers that need many searches and page reads BrowseComp 2025-04-16 # vendor-authored Hard-to-find, entangled facts that need persistent browsing; answers are short and easy to verify.
Every product number here is vendor-run, on different subsets and harnesses. The same system moves a lot by runner: Parallel Ultra4x is 94% on Parallel's page and 0.56 in Perplexity's search_evals.
by OpenAI tasks 1,266 (vendors usually run 50–200) metric accuracy % grading match against reference answer (LLM grader) status no official leaderboard; the de facto agentic-search metric links here system as published score reported by source OpenAI Deep Researchpaper notes Deep Research was trained on BrowseComp-like data
51.5% OpenAI (paper) · vendor-run arxiv.org ↗ Gemini Gemini Deep ResearchDec 2025 agent (deep-research-pro-preview-12-2025)
59.2% Google · vendor-run blog.google ↗ Perplexity Agent API `high`stated successor to Sonar Deep Research; xhigh 85.33%; undated, no task count
78.00% Perplexity · vendor-run docs.perplexity.ai ↗ Perplexity Sonar Deep Researchsame page; Sonar retires 2026-09-27
29.10% Perplexity · vendor-run docs.perplexity.ai ↗ Parallel Parallel Ultra4x (Task API)100-question subset, 2026-08-26; Lite 88 · Core 91 · Ultra 92
94% Parallel · vendor-run parallel.ai ↗ Perplexity Perplexity highParallel's run of a competitor
86% Parallel · vendor-run parallel.ai ↗ Exa Exa Agent MaxParallel's run; “max” is not a public Exa effort tier
78% Parallel · vendor-run parallel.ai ↗ Exa Exa 3.02026-10-19; read from the chart's alt text; no harness, sample or grader stated
91.8% Exa · vendor-run exa.ai ↗
DeepSearchQA 2026-01-28 # vendor-authored Hand-crafted causal-chain research tasks that require exhaustive answer lists, balancing recall against precision.
Written by Google, which also reports Gemini's score. Vendors mix accuracy and F1, which differ by up to ~25 points for the same run.
by Google DeepMind tasks 900 across 17 fields metric fully-correct accuracy % (some vendors report F1) grading LLM-judge semantic set matching status Kaggle leaderboard run by Google; newest row evaluated 2026-04-13 links system as published score reported by source Gemini Gemini Deep Research Agent±3.2; F1 81.9; evaluated 2025-12-11, row since relabeled deep-research-preview-04-2026 without a re-run
66.1% Google (Kaggle board) · vendor-run kaggle.com ↗ OpenAI o3 Deep Research±3.3; F1 66.5; run by a competing vendor
44.2% Google (Kaggle board) · vendor-run kaggle.com ↗ OpenAI o4 mini Deep Research±3.2; model since shut down
40.4% Google (Kaggle board) · vendor-run kaggle.com ↗ You.com Research API FrontierF1 94.35%; Frontier is $1,200 per 1k, not wired in the arena
86.61% You.com · vendor-run about.you.com ↗ Perplexity Agent API `xhigh` / `high`F1, not accuracy; Sonar Deep Research 48.94% F1
93.00% / 89.60% F1 Perplexity · vendor-run docs.perplexity.ai ↗ Parallel Parallel Ultra4x (Task API)100-question subset
86% Parallel · vendor-run parallel.ai ↗ Perplexity Perplexity highParallel's run — vs 89.60% F1 in Perplexity's own docs (different metric and sample)
68% Parallel · vendor-run parallel.ai ↗ Exa Exa Agent MaxParallel's run
65% Parallel · vendor-run parallel.ai ↗ Exa Exa 3.0chart alt text; no harness stated
88.2% Exa · vendor-run exa.ai ↗ context GPT-5 Pro#2 on the board — a model, not a deep-research product
65.2% Google (Kaggle board) · vendor-run kaggle.com ↗
WideSearch 2025-08-11 # academic Broad information-seeking: collect many independently verifiable facts into a table.
The authors tried the deep-research products but they returned long reports instead of the single required table, so the official rows are web-browsing modes. Vendors run the English half with partial-credit row F1 — not comparable to the paper's success rate.
by ByteDance Seed tasks 200 (100 English, 100 Chinese) metric row-level F1 % (paper: table success rate) grading automated table verification status static leaderboard (paper v2 2025-08-28) links system as published score reported by source Gemini Gemini 2.5 Pro (E2E System)success rate 4.3%; consumer web-browsing mode, not Deep Research
36.6% row F1 benchmark authors widesearch-seed.github.io ↗ OpenAI OpenAI o3 (E2E System)success rate 3.0%
23.9% row F1 benchmark authors widesearch-seed.github.io ↗ Parallel Parallel Search Advancedsearch API behind a GPT-5.6 Sol agent; 100 English tasks
57.6% Parallel · vendor-run parallel.ai ↗ Exa Exa Auto (frontier)same Parallel harness; low-cost tier 53
55.9% Parallel · vendor-run parallel.ai ↗ Perplexity Perplexity (frontier)same Parallel harness; low-cost tier 47
53.5% Parallel · vendor-run parallel.ai ↗ Exa Exa 3.0chart alt text
56.1% Exa · vendor-run exa.ai ↗
xbench-DeepSearch 2025-06-16 # independent Deep search and tool use by consumer AI search and agent products on Chinese-centric, multi-hop questions; the board also lists per-task cost and time.
Consumer products tested in their UIs; no confidence intervals and no API products. The board renders client-side — rows read from its data endpoint.
by xbench (HongShan / UniPat) tasks not stated on the board metric accuracy %, integer grading not stated status 2510 edition; newest evaluation 2026-02-16 links system as published score reported by source OpenAI ChatGPT-5-Protop row; ChatGPT product, not the o3-deep-research API; evaluated 2025-08-28
79 xbench xbench.org ↗ Gemini Gemini Proconsumer app; board doesn't say whether Deep Research mode was used
53 xbench xbench.org ↗ context Kimi K2.5 (Thinking)#3
46 xbench xbench.org ↗
BrowseComp-ZH 2025-04-27 # academic BrowseComp-style hard-to-find questions on the Chinese web; reports accuracy and calibration for LLMs and AI search products.
Early-2025 product versions; Gemini appears only as base models.
by Zhou et al. tasks 289 metric accuracy % grading gpt-4o answer extraction + scripted scoring status static paper (v2 2025-05-01) links system as published score reported by source OpenAI OpenAI DeepResearchbest product row; calibration error 9
42.9% benchmark authors github.com ↗ Perplexity Perplexity (Research)calibration error 53
22.6% benchmark authors github.com ↗ context Doubao (Deep Search)#2 product
26.0% benchmark authors github.com ↗
BrowseComp-Plus 2025-08-08 # academic BrowseComp queries answered over a fixed, human-verified corpus instead of the live web, so the agent model and the retriever can be scored separately.
Bring-your-own-retriever on a fixed corpus — hosted deep-research APIs can't be entered, so there are no product rows. Listed for context.
by University of Waterloo (Tevatron) et al. tasks BrowseComp subset over a fixed corpus metric accuracy % grading LLM judge Qwen3-32B (switched from GPT-4.1; <1% difference reported) status active; 87 rows, newest 2026-10-08 links General-assistant tasks needing browsing, tools and multi-step reasoning, scored by exact-match answers.
None of the 3,994 submissions is from a tracked deep-research product; vendors' GAIA claims appear only on their own blogs. Rows are self-reported.
by Meta · Hugging Face tasks test split (public leaderboard) metric exact-match accuracy % grading automatic exact match status active; 3,994 test submissions, newest 2026-10-21 links system as published score reported by source context Shadow Harness V8.01top row, unverified
94.02% self-submitted huggingface.co ↗ context open Deep Research (HF smolagents, o1)open-source deep-research clone, 2025-02-10
49.83% Hugging Face (self-submitted) huggingface.co ↗
Humanity's Last Exam 2025-01-24 # academic Expert-level closed-ended questions across dozens of subjects; search vendors report it with browsing enabled.
Mostly a reasoning test; with-search scores mix model and retrieval quality.
by Phan et al. tasks 2,500 metric accuracy % grading exact-answer grading status paper revised 2026-10-19 (v12) links system as published score reported by source Gemini Gemini Deep Researchfull HLE set, Dec 2025 agent
46.4% Google · vendor-run blog.google ↗ Perplexity Deep Research on Perplexity2025-02-14
21.1% Perplexity · vendor-run perplexity.ai ↗
search_evals (Perplexity) 2025-09-25 # vendor-authored Open harness that runs five deep-research APIs end-to-end on BrowseComp, DeepSearchQA, HLE and WideSearch. Began in 2025 as a search-API harness.
Run by a vendor that ranks itself first; compare each row with that vendor's own figures above.
by Perplexity tasks browsecomp 1,266 · dsqa 900 · hle 2,158 · widesearch 200 metric accuracy 0–1 (WideSearch: row F1) grading OpenAI-model grader; exact model not stated status active; README results match the 2026-06-01 Search-as-Code post links system as published score reported by source Perplexity perplexity (Agent API, GPT 5.5 high)HLE 0.612 · WideSearch 0.651
BC 0.805 · DSQA 0.871 Perplexity · vendor-run github.com ↗ OpenAI openai (Responses API, GPT 5.5 high)web_search tool — not o3-deep-research; HLE 0.614
BC 0.720 · DSQA 0.733 Perplexity · vendor-run github.com ↗ Parallel parallel (Task API ultra4x)Parallel reports 94% BrowseComp for the same processor
BC 0.56 · DSQA 0.81 Perplexity · vendor-run github.com ↗ Exa exa (Agent API, effort high)HLE 0.387 · WideSearch 0.471
BC 0.38 · DSQA 0.53 Perplexity · vendor-run github.com ↗ context anthropic (Managed Agents) BC 0.598 · DSQA 0.815 Perplexity · vendor-run github.com ↗
WANDR 2026-07-14 # vendor-authored Wide data-collection tasks graded reference-free: each submitted claim is checked against re-fetched cited evidence.
Vendor-authored and vendor-run; Perplexity's own two posts disagree on its score.
by Perplexity tasks 500 (170,495 source-backed records) metric soft F1 0–1 grading evidence-verified automatic grader status active links system as published score reported by source Perplexity Perplexity Search as Codehard F1 0.133; the Search-as-Code post says 0.386
0.363 soft F1 Perplexity · vendor-run perplexity.ai ↗ OpenAI OpenAI 0.130 Perplexity · vendor-run perplexity.ai ↗ Parallel Parallel 0.126 Perplexity · vendor-run perplexity.ai ↗ Exa Exa 0.057 Perplexity · vendor-run perplexity.ai ↗
Mind2Web 2 2025-06-26 # academic Long-horizon agentic search needing real-time browsing and synthesis; answers judged for correctness and source attribution.
Consumer deep-research products evaluated in 2025.
by Ohio State (OSU NLP) & Amazon AGI tasks 130 (leaderboard: 120 private test) metric success rate 0–1 grading Agent-as-a-Judge with tree-structured task rubrics status active leaderboard (data updated 2026-05-17) links Deep Research Bench (FutureSearch) 2025-06-06 # independent Multi-step web research tasks with human-verified answers, run against a frozen offline web snapshot (RetroSearch).
No deep-research product is listed — OpenAI and Gemini rows are base models.
by FutureSearch tasks 89 across 8 categories metric average score 0–1 grading task-specific scoring against verified answers status active (“last updated 2026-06-10”) links system as published score reported by source context Opus 4.6 (high)top entry; only base models in a ReAct harness are listed
0.553 benchmark authors drb.futuresearch.ai ↗
search APIs & retrieval (13) the search layer itself — freshness, factual recall, ranking Artificial Analysis Search Index 2026-08-18 # independent Only the search API changes: a fixed agent (GPT-5.6 Luna, Stirrup harness, ≤25 turns) is scored on DeepSearchQA F1, BrowseComp and AA-Omniscience; the index is their equal-weighted mean.
AA picks each provider's modes. Scores move between snapshots — Exa auto went from 74 at launch to 80 after Exa 3.0.
by Artificial Analysis tasks 1,700 (DSQA 900 + BrowseComp 200 + Omniscience 600 private) metric index 0–100 grading LLM-graded F1 (DSQA), exact answer (BrowseComp), accuracy (Omniscience) status live; data as of 2026-10-19 links system as published score reported by source Exa Exa Search (auto, dynamic highlights)DSQA 86 · BrowseComp 83 · $53.17 search per 1k tasks
80 Artificial Analysis artificialanalysis.ai ↗ Perplexity Perplexity Search (medium)BrowseComp 87, highest on the board · 27.0 s/task
80 Artificial Analysis artificialanalysis.ai ↗ Parallel Parallel Search (advanced)basic 73
75 Artificial Analysis artificialanalysis.ai ↗ OpenAI OpenAI Web Search (medium)web search tool, not o3-deep-research
74 Artificial Analysis artificialanalysis.ai ↗ You.com You.com Search (highlights)19.0 s/task
74 Artificial Analysis artificialanalysis.ai ↗ Keenable Keenable Search (pro / realtime)2026-08-18 launch snapshot — not among the rows on the live board
67 Artificial Analysis artificialanalysis.ai ↗ context Octen Search (highlights)$9.07 search per 1k tasks
77 Artificial Analysis artificialanalysis.ai ↗
Vals Web Search Index 2026-10-02 # independent Whether a fixed model + harness reaches the correct answer on professional legal-research and finance tasks when the only external source is the search tool.
Only native, Exa and Keenable are on the board; Exa, Keenable, Parallel and Tavily are listed as Vals partners.
by Vals AI tasks 658 (legal 208 + finance 450), private metric % correct overall grading legal: GPT-5.4 judge; finance: 3 judges (GPT-5.4, Gemini-3.1-Pro, Claude Sonnet 4.6) status live; updated 2026-10-21 links system as published score reported by source Keenable GPT-5.6 Sol | Keenable#1; same model with native search 43.58%
49.08% Vals AI vals.ai ↗ Exa Claude Fable 5 | Exa#2; GPT-5.6 Sol | Exa 45.24%
48.45% Vals AI vals.ai ↗
NEEDLE 2026-08-27 # vendor-authored Live retrieval benchmark that regenerates queries from fresh sources (news hourly, others daily); each engine is scored as a share of a pooled oracle across news, finance, scholar, rare-agentic and legal benches.
Run by a vendor that ranks itself first; measures retrieval only, not answers. The board renders client-side.
by Keenable tasks rolling (e.g. 20 news queries per run, 100 rare-agentic per day) metric 7-day mean % of oracle grading LLM judge (not named on the blog) status live; last run 2026-10-21 links Multi-turn company search (openbenchmarks) 2026-10-02 # independent Multi-constraint company-discovery questions answered by one agent that may search several times; only the search API behind it changes.
Search endpoints, not deep-research tiers. Unknown operator and funding, undisclosed judge, 45 tasks.
by openbenchmarks.com (operator not disclosed) tasks 45 questions · 12 providers metric F1 % grading not stated; driving model not named status measured 2026-10-02 links AIMultiple Agentic Search 2026-05-13 # independent Relevance and quality of the top-5 results search APIs return for 100 real AI/LLM queries.
Retrieval relevance only, not answers; You.com, Valyu and Keenable absent.
by AIMultiple tasks 100 queries (~4,000 results) metric agent score /20 (relevance × quality) grading LLM judge GPT-5.2, temperature 0 status updated 2026-05-13 links system as published score reported by source Exa Exa#3; 95% CI 13.25–15.50
14.39 AIMultiple aimultiple.com ↗ Parallel Parallel Search (Pro)#4; 13.07–15.47
14.21 AIMultiple aimultiple.com ↗ Perplexity Perplexity#7; 11.77–14.22
12.96 AIMultiple aimultiple.com ↗ context Brave Search#1; the authors call the top four statistically tied
14.89 AIMultiple aimultiple.com ↗
SimpleQA 2024-11-07 # vendor-authored Short fact-seeking questions with one indisputable answer; vendors reuse it to score search-augmented answers.
Each vendor uses its own reader model and subset, so scores do not compare across sources — Exa ranges 71–95% depending on who ran it.
by OpenAI tasks 4,326 metric % correct grading official SimpleQA LLM classifier status saturated for search-augmented systems (vendors report 91–98%) links system as published score reported by source You.com you_searchREADME 2026-10-07; p50 598 ms
96.11% You.com · vendor-run github.com ↗ Exa exa_search_with_textYou.com's run; p50 2.5 s
95.40% You.com · vendor-run github.com ↗ Parallel parallel_search_basicYou.com's run
94.87% You.com · vendor-run github.com ↗ Valyu ValyuParallel 93% · Exa 91% · Google 38% in the same run (2025-10-03)
94% Valyu · vendor-run valyu.ai ↗ Perplexity Perplexity Search API2025-09-25; Exa .781 in the same run
93.0% Perplexity · vendor-run perplexity.ai ↗ Perplexity Deep Research on Perplexity2025-02-14
93.9% Perplexity · vendor-run perplexity.ai ↗ Perplexity Perplexity SearchTavily's run, undated; Exa 71.24%, Tavily 93.3%
85.92% Tavily · vendor-run github.com ↗
SimpleQA Verified 2025-09-09 # vendor-authored A cleaned 1,000-prompt SimpleQA with an improved autorater; built for parametric factuality, reused for search APIs.
On 100 questions a 2–6 point gap is within noise.
by Google DeepMind · Google Research tasks 1,000 (Parallel runs 100) metric % correct grading improved autorater prompt status active (v2 2026-03-10) links system as published score reported by source Parallel Parallel Search Advanced / Basicfrontier agent (GPT-5.6 Sol high), 100 questions, 2026-09-09
97% Parallel · vendor-run parallel.ai ↗ Perplexity Perplexity (frontier)Parallel's run
95% Parallel · vendor-run parallel.ai ↗ Exa Exa Auto (frontier)Parallel's run
91% Parallel · vendor-run parallel.ai ↗
FRAMES 2024-09-19 # vendor-authored Multi-hop retrieval-augmented reasoning questions (fact, fetch and reason).
by Google tasks 824 metric accuracy % grading LLM judge status active (NAACL 2025) links system as published score reported by source You.com you_search 74.03% You.com · vendor-run github.com ↗ Parallel parallel_search_basic 73.79% You.com · vendor-run github.com ↗ Exa exa_search_with_text 73.79% You.com · vendor-run github.com ↗ Perplexity Perplexity Search API2025-09-25, single-step; Exa 39.9% in the same run
45.3% Perplexity · vendor-run perplexity.ai ↗
Time-sensitive questions whose answers change — a freshness test for search-augmented systems.
Results depend on which dated FreshQA version is used; Valyu does not say.
by FreshLLMs authors tasks 600 metric accuracy % grading FreshEval few-shot LLM judge status dated versions; latest 2026-04-21 links system as published score reported by source Valyu Valyu2025-10-03; version not stated
79% Valyu · vendor-run valyu.ai ↗ Parallel ParallelValyu's run; tier not stated
52% Valyu · vendor-run valyu.ai ↗ Exa ExaValyu's run; mode not stated
24% Valyu · vendor-run valyu.ai ↗
Valyu domain search evals 2025-10-03 # vendor-authored Retrieval-only search accuracy on Valyu-built finance and economic-data sets plus MedAgent.
Valyu's own pages disagree on economics (73/52/45 here, 83/48/30 on its DeepResearch launch page).
by Valyu tasks finance 120 · economic 100 · MedAgent 562 metric accuracy % grading 3 LLM graders (Gemini 2.5 Pro, Claude Sonnet 4, GPT 4.1) status single run 2025-10-03 links system as published score reported by source Valyu Valyufinance / economic / MedAgent
73% / 73% / 48% Valyu · vendor-run valyu.ai ↗ Parallel Parallel 67% / 52% / 42% Valyu · vendor-run valyu.ai ↗ Exa Exa 63% / 45% / 44% Valyu · vendor-run valyu.ai ↗
You.com web-search-api-evals 2026-04-21 # vendor-authored Open harness running SimpleQA, FRAMES, BrowseComp, DeepSearchQA and FinSearchComp against search and research endpoints with a fixed synthesis model and grader. Shown here: FinSearchComp T2 (finance).
The README contradicts itself on the synthesis model; sample sizes are not stated.
by You.com tasks not stated metric accuracy % grading GPT 5.4 Mini judge status README results dated 2026-10-07 links system as published score reported by source You.com you_finance_research_deepFinSearchComp T2
87.29% You.com · vendor-run github.com ↗ Parallel parallel_ultrapro 34.45%
73.11% You.com · vendor-run github.com ↗ Perplexity perplexity_sonar_deep_research_high 53.78% You.com · vendor-run github.com ↗ Exa exa_research_proretired endpoint (HTTP 410)
42.02% You.com · vendor-run github.com ↗
WISER-Search 2025-04 # vendor-authored Proprietary blend: 76 same-day news questions (WISER-Fresh) and ~120 hard questions from Parallel customer use cases (WISER-Atomic).
Private dataset; task counts differ between Parallel posts.
by Parallel tasks ~196, not public metric accuracy % grading LLM evaluators against verified ground truth status historical (2025); not on Parallel's current benchmarks page links system as published score reported by source Parallel GPT 4.1 w/ Parallel Search MCP2025-07
74.9 Parallel · vendor-run parallel.ai ↗ Exa GPT 4.1 w/ Exa Search MCPParallel's run
58.67 Parallel · vendor-run parallel.ai ↗ Perplexity Perplexity Deep ResearchParallel Core 77 on the same set
64% (Atomic) Parallel · vendor-run parallel.ai ↗
Search Arena 2025-04-15 # academic Crowdsourced pairwise human preference between search-augmented chat models.
Chat-style search LLMs, not search or deep-research APIs; the only readable figures come via Perplexity's blog. research.site's arena is the deep-research analogue.
by LMArena · UC Berkeley tasks 24k+ multi-turn interactions, ~12k votes (paper) metric Arena score (Bradley-Terry) with CI grading human preference votes status live board returned HTTP 403 — current state unverified links system as published score reported by source Gemini Gemini-2.5-Pro-GroundingGemini grounding, not Deep Research; April 2025 snapshot
1142 Perplexity · vendor-run perplexity.ai ↗ Perplexity Sonar-Reasoning-Pro-Highstatistically tied for #1; April 2025
1136 Perplexity · vendor-run perplexity.ai ↗
