▌
provider specs
what you need to know before integrating — verified against each vendor’s own docs on 2026-08-26; every provider name links to the page it was checked against
| provider | pricing model | est. cost/query | latency | api pattern | citations | special capabilities |
|---|---|---|---|---|---|---|
| OpenAI o3 ↗ o3-deep-research | per-token | $10 / $40 per 1M tok (~$1-10/query) | 10-30+ min | Responses API, async background (poll / webhook) | yes (inline annotations: url, title, start/end index) | MCP serverscode interpreterweb search$10 / 1k web-search calls |
| OpenAI GPT-5.5 Pro ↗ gpt-5.5-pro | per-token | $30 / $180 per 1M tok (no cached rate) | several min (vendor) | Responses API, async background recommended | yes (inline annotations via web search tool) | web searchMCP serversfile searchcode interpreter+10% regional processing |
| OpenAI o4-mini ↗ o4-mini-deep-research deprecated · off the arena roster since 2026-07-23 — jobs fail model_not_found; OpenAI still lists it, no notice | per-token | $2 / $8 per 1M tok | 3-10 min | Responses API, async background | yes (inline annotations) | MCP serverscode interpreterfile searchweb search |
| Perplexity Sonar ↗ sonar-deep-research deprecated · Sonar endpoints retire 2026-09-27 → Agent API preset “high” | per-token + per-search | $2 / $8 per 1M tok + $5 / 1k searches (~$0.30-1.30/query) | 1-3 min | POST /v1/sonar (sync) or /v1/async/sonar (poll) | yes (citations[] URLs + search_results[] w/ snippets) | citation-focusedsearch groundingbilled per token + citation + reasoning + search |
| Gemini Deep Research ↗ deep-research-preview-04-2026 | per-token + per-search | ~$1-3/task (vendor est.) | minutes (60 min max) | Interactions API, async (background=true; poll / stream / webhook) | yes (cited report; schema not documented) | google searchurl contextcode executionMCPfile searchvisualizationmultimodal |
| Gemini Deep Research Max ↗ deep-research-max-preview-04-2026 | per-token + per-search | ~$3-7/task (vendor est.) | up to 60 min | Interactions API, async (background=true; poll / stream / webhook) | yes (cited report; schema not documented) | ~160 searches/taskgoogle searchurl contextcode executionMCPfile searchvisualization |
| Parallel Pro ↗ pro | per-request | $0.10/run | 2-10 min | Task API, async (poll / webhook / SSE) | yes (citations + excerpts + confidence, per element) | structured JSON outputauto-schema1 of 9 tiers (lite→ultra8x)failed runs not billed |
| Parallel Pro-Fast ↗ pro-fast | per-request | $0.10/run | 30s-5 min | Task API, async (poll / webhook / SSE) | yes (citations + excerpts + confidence, per element) | structured JSON outputauto-schemafaster variant (speed over freshness) |
| Parallel Ultra ↗ ultra | per-request | $0.30/run | 5-25 min | Task API, async (poll / webhook / SSE) | yes (citations + excerpts + confidence, per element) | structured JSON outputauto-schemadeep research modeultra2x/4x/8x at $0.60/$1.20/$2.40 |
| Parallel Ultra-Fast ↗ ultra-fast | per-request | $0.30/run | 1-10 min | Task API, async (poll / webhook / SSE) | yes (citations + excerpts + confidence, per element) | structured JSON outputauto-schemafaster variant (speed over freshness) |
| Valyu DeepResearch ↗ deepresearch (fast/standard/heavy/max) | per-request | $0.10 / $0.50 / $2.50 / $15 per task | 5 min - 3 hr (vendor pages disagree) | REST, async (poll / HMAC webhook) | yes (sources[]: title + url) | SEC/academic/clinical/patent sourcesstructured JSON outputpython sandbox $0.10/runscreenshots $0.05/URL |
| You.com Research ↗ research (lite→frontier) | per-request | $0.012 / $0.05 / $0.10 / $0.45 / $1.20 per call | < 10s - 5 min (frontier p50 5 min) | REST, sync (background=true → poll / SSE; frontier background-only) | yes (inline [[n]] + sources[] w/ snippets) | structured JSON output (not lite)source control (domains/freshness)finance research variant |
| Exa Agent ↗ agent (effort minimal→xhigh, auto/max) deprecated · legacy /research API (exa-research, exa-research-pro) retired by Exa — HTTP 410 RESEARCH_RETIRED; arena now runs Agent tiers | per-request | $0.012 - $1.00/run (auto/max metered, $0.10/ACU) | 2-15 min (measured); vendor: not stated | Agent API POST /agent/runs, async (poll 4s / SSE) | yes (output.grounding: field-level citations + confidence) | Exa's own indexstructured output w/ groundingreturns real per-run costDollarsdeep search $12-15/1k |
benchmarks
two kinds of evidence per provider. measured by research.site — our own pre-registered studies: frozen tasks, one harness, published configs and costs (all studies). published elsewhere — academic leaderboards and vendor evals we don’t run, catalogued with methods and authorship on external benchmarks; vendor-run figures were published by a vendor in the comparison, independent ones by the benchmark’s authors or a third party. some studies test the vendor’s deep research API, others its search tool behind an agent — each row says which. for a blind community signal see the arena leaderboard.
at a glance · best configuration per study
| provider | DSE-246 | Astra-26 | LBC-30 | RB-30 | DRB2-20 | external |
|---|---|---|---|---|---|---|
| OpenAI direct compare only | not run | 42.3% #1 of 5 | 46.7% #2 of 8 | not run | not run | 21 scores |
| Perplexity in blind pool | not run | not run | not run | 73.4% #1 of 6 | 34.1% #6 of 9 | 31 scores |
| Gemini in blind pool | not run | not run | not run | not run | not run | 14 scores |
| Parallel in blind pool | not run | 26.9% #4 of 5 | not run | 72.2% #2 of 6 | 29.1% #8 of 9 | 18 scores |
| Exa in blind pool | 60.4% #1 of 2 | 42.3% #1 of 5 | 60.0% #1 of 8 · best of 2 | 66.6% #4 of 6 | 45.0% #1 of 9 · best of 5 | 21 scores |
| Valyu in blind pool | not run | 23.1% #5 of 5 | 26.7% #7 of 8 · best of 2 | 70.2% #3 of 6 | 31.4% #7 of 9 | 4 scores |
| You.com in blind pool | not run | not run | not run | 58.5% #5 of 6 | not run | 8 scores |
| Keenable not yet measured as deep research | not run | 38.5% #3 of 5 | 46.7% #2 of 8 · best of 2 | not run | not run | 3 scores |
each study ranks only the arms it ran — ranks are not comparable across columns. metrics: DSE-246 accuracy · Astra-26 accuracy · LBC-30 accuracy · RB-30 rubric coverage · DRB2-20 rubric pass rate.
OpenAIdirect compare only · 2 research.site results · 21 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| Astra-26 | GPT-6 Astra + Codex built-in web search search tool (agent backend) | 11/26 (42.3%) accuracy | #1 of 5 | Wilson 95% 26%–61% · median 257 s · $3.84 / task · $9.08 / correct |
| LBC-30 | Codex CLI · live web_search search tool (agent backend) | 46.7% accuracy | #2 of 8 | dated 40.0% · undated 53.3% · cost ~$1–6* · 2.4M tokens |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DeepResearch Bench | openai-deepresearch 2025 ChatGPT deep research; model version not named | 47.84 RACE 0–100, relative to a reference report | benchmark authors | huggingface.co ↗ |
| DeepResearch Bench | GPT-5 same Parallel run; not labeled as o3-deep-research | 66% win rate RACE 0–100, relative to a reference report | Parallel vendor-run | parallel.ai ↗ |
| DeepResearch Bench II | GPT-o3 Deep Research GPT-5.5 judge; 45.40 on the Gemini-2.5-Pro board | 43.12 rubric pass rate 0–100 | benchmark authors | huggingface.co ↗ |
| DRACO vendor-authored benchmark | OpenAI Deep Research (o3) | 52.1% normalized rubric score % | Perplexity (paper) vendor-run | arxiv.org ↗ |
| ResearchRubrics | OpenAI DR binary 0.597 | 0.664 rubric compliance 0–1 (ternary) | benchmark authors | arxiv.org ↗ |
| LiveResearchBench | OpenAI o3 Deep Research citation association only 30.9 | 62.9 average of 4 dimensions, 0–100 | benchmark authors | livedeepresearch.github.io ↗ |
| LiveResearchBench | OpenAI o4-mini Deep Research model shut down by OpenAI 2026-07 | 61.8 average of 4 dimensions, 0–100 | benchmark authors | livedeepresearch.github.io ↗ |
| ResearcherBench | OpenAI Deep Research faithfulness 0.84 · groundedness 0.34; top coverage | 0.7032 rubric coverage 0–1 | benchmark authors | github.com ↗ |
| DeepResearchGym | gpt4-search-preview search model, not OpenAI deep research | 40.01 KPR key-point recall (KPR) 0–100 | benchmark authors | arxiv.org ↗ |
| ReportBench | OpenAI Deep Research reference precision 0.385; non-cited factual accuracy 95.83% | 78.87% match citation match rate % · factual accuracy % | benchmark authors | arxiv.org ↗ |
| BrowseComp vendor-authored benchmark | Deep Research paper notes Deep Research was trained on BrowseComp-like data | 51.5% accuracy % | OpenAI (paper) vendor-run | arxiv.org ↗ |
| DeepSearchQA vendor-authored benchmark | o3 Deep Research ±3.3; F1 66.5; run by a competing vendor | 44.2% fully-correct accuracy % (some vendors report F1) | Google (Kaggle board) vendor-run | kaggle.com ↗ |
| DeepSearchQA vendor-authored benchmark | o4 mini Deep Research ±3.2; model since shut down | 40.4% fully-correct accuracy % (some vendors report F1) | Google (Kaggle board) vendor-run | kaggle.com ↗ |
| WideSearch | OpenAI o3 (E2E System) success rate 3.0% | 23.9% row F1 row-level F1 % (paper: table success rate) | benchmark authors | widesearch-seed.github.io ↗ |
| xbench-DeepSearch | ChatGPT-5-Pro top row; ChatGPT product, not the o3-deep-research API; evaluated 2025-08-28 | 79 accuracy %, integer | xbench | xbench.org ↗ |
| BrowseComp-ZH | OpenAI DeepResearch best product row; calibration error 9 | 42.9% accuracy % | benchmark authors | github.com ↗ |
| search_evals (Perplexity) vendor-authored benchmark | openai (Responses API, GPT 5.5 high) web_search tool — not o3-deep-research; HLE 0.614 | BC 0.720 · DSQA 0.733 accuracy 0–1 (WideSearch: row F1) | Perplexity vendor-run | github.com ↗ |
| WANDR vendor-authored benchmark | OpenAI | 0.130 soft F1 0–1 | Perplexity vendor-run | perplexity.ai ↗ |
| Mind2Web 2 | OpenAI Deep Research partial 0.54 · pass@3 0.40 | 0.28 success rate 0–1 | benchmark authors | osu-nlp-group.github.io ↗ |
| Artificial Analysis Search Index | OpenAI Web Search (medium) web search tool, not o3-deep-research | 74 index 0–100 | Artificial Analysis | artificialanalysis.ai ↗ |
| NEEDLE vendor-authored benchmark | chatgpt-search p50 40 s | 48% 7-day mean % of oracle | Keenable vendor-run | keenableai.github.io ↗ |
Perplexityin blind pool · 2 research.site results · 31 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| RB-30 | sonar-deep-research deep research API | 73.4% rubric coverage | #1 of 6 | 95% CI 65%–80% · faithfulness 22.1% · groundedness 31.6% · 9,148 claims checked · token-metered · judges o3-mini / gpt-4.1 |
| DRB2-20 | sonar-deep-research deep research API | 34.1% rubric pass rate | #6 of 9 | info recall 26.1 · analysis 48.4 · presentation 81.3 · 5.6 min mean · token-metered · judge gemini-3.1-pro-preview |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DeepResearch Bench | perplexity-Research Perplexity app Research mode, 2025 outputs | 43.05 RACE 0–100, relative to a reference report | benchmark authors | huggingface.co ↗ |
| DeepResearch Bench | Perplexity same Parallel run; tier not specified | 6% win rate RACE 0–100, relative to a reference report | Parallel vendor-run | parallel.ai ↗ |
| DeepResearch Bench II | Perplexity Research 38.58 on the Gemini-2.5-Pro board | 33.57 rubric pass rate 0–100 | benchmark authors | huggingface.co ↗ |
| DRACO vendor-authored benchmark | Perplexity Deep Research (Opus 4.6) pass rate 72.8%; drops to 50.4 under the GPT-5.2 judge | 70.5% normalized rubric score % | Perplexity (paper) vendor-run | arxiv.org ↗ |
| ResearchRubrics | Perplexity DR binary 0.487 | 0.566 rubric compliance 0–1 (ternary) | benchmark authors | arxiv.org ↗ |
| LiveResearchBench | Perplexity Sonar Deep Research #2 of single-agent deep research | 67.1 average of 4 dimensions, 0–100 | benchmark authors | livedeepresearch.github.io ↗ |
| ResearcherBench | Perplexity Deep Research faithfulness 0.85 · groundedness 0.56 | 0.4800 rubric coverage 0–1 | benchmark authors | github.com ↗ |
| DeepResearchGym | perplexity-sonar-deepsearch citation recall 99.22, precision 55.65; top KPR | 72.50 KPR key-point recall (KPR) 0–100 | benchmark authors | arxiv.org ↗ |
| BrowseComp vendor-authored benchmark | Agent API `high` stated successor to Sonar Deep Research; xhigh 85.33%; undated, no task count | 78.00% accuracy % | Perplexity vendor-run | docs.perplexity.ai ↗ |
| BrowseComp vendor-authored benchmark | Sonar Deep Research same page; Sonar retires 2026-09-27 | 29.10% accuracy % | Perplexity vendor-run | docs.perplexity.ai ↗ |
| BrowseComp vendor-authored benchmark | Perplexity high Parallel's run of a competitor | 86% accuracy % | Parallel vendor-run | parallel.ai ↗ |
| DeepSearchQA vendor-authored benchmark | Agent API `xhigh` / `high` F1, not accuracy; Sonar Deep Research 48.94% F1 | 93.00% / 89.60% F1 fully-correct accuracy % (some vendors report F1) | Perplexity vendor-run | docs.perplexity.ai ↗ |
| DeepSearchQA vendor-authored benchmark | Perplexity high Parallel's run — vs 89.60% F1 in Perplexity's own docs (different metric and sample) | 68% fully-correct accuracy % (some vendors report F1) | Parallel vendor-run | parallel.ai ↗ |
| WideSearch | Perplexity (frontier) same Parallel harness; low-cost tier 47 | 53.5% row-level F1 % (paper: table success rate) | Parallel vendor-run | parallel.ai ↗ |
| BrowseComp-ZH | Perplexity (Research) calibration error 53 | 22.6% accuracy % | benchmark authors | github.com ↗ |
| Humanity's Last Exam | Deep Research on Perplexity 2025-02-14 | 21.1% accuracy % | Perplexity vendor-run | perplexity.ai ↗ |
| search_evals (Perplexity) vendor-authored benchmark | perplexity (Agent API, GPT 5.5 high) HLE 0.612 · WideSearch 0.651 | BC 0.805 · DSQA 0.871 accuracy 0–1 (WideSearch: row F1) | Perplexity vendor-run | github.com ↗ |
| WANDR vendor-authored benchmark | Perplexity Search as Code hard F1 0.133; the Search-as-Code post says 0.386 | 0.363 soft F1 soft F1 0–1 | Perplexity vendor-run | perplexity.ai ↗ |
| Mind2Web 2 | Perplexity Deep Research partial 0.42 | 0.15 success rate 0–1 | benchmark authors | osu-nlp-group.github.io ↗ |
| Artificial Analysis Search Index | Perplexity Search (medium) BrowseComp 87, highest on the board · 27.0 s/task | 80 index 0–100 | Artificial Analysis | artificialanalysis.ai ↗ |
| NEEDLE vendor-authored benchmark | perplexity p50 345 ms | 67% 7-day mean % of oracle | Keenable vendor-run | keenableai.github.io ↗ |
| Multi-turn company search (openbenchmarks) | Perplexity (search_context_size=low) 14.1 s | 37.8% F1 % | openbenchmarks | openbenchmarks.com ↗ |
| AIMultiple Agentic Search | Perplexity #7; 11.77–14.22 | 12.96 agent score /20 (relevance × quality) | AIMultiple | aimultiple.com ↗ |
| SimpleQA vendor-authored benchmark | Perplexity Search API 2025-09-25; Exa .781 in the same run | 93.0% % correct | Perplexity vendor-run | perplexity.ai ↗ |
| SimpleQA vendor-authored benchmark | Deep Research on Perplexity 2025-02-14 | 93.9% % correct | Perplexity vendor-run | perplexity.ai ↗ |
| SimpleQA vendor-authored benchmark | Perplexity Search Tavily's run, undated; Exa 71.24%, Tavily 93.3% | 85.92% % correct | Tavily vendor-run | github.com ↗ |
| SimpleQA Verified vendor-authored benchmark | Perplexity (frontier) Parallel's run | 95% % correct | Parallel vendor-run | parallel.ai ↗ |
| FRAMES vendor-authored benchmark | Perplexity Search API 2025-09-25, single-step; Exa 39.9% in the same run | 45.3% accuracy % | Perplexity vendor-run | perplexity.ai ↗ |
| You.com web-search-api-evals vendor-authored benchmark | perplexity_sonar_deep_research_high | 53.78% accuracy % | You.com vendor-run | github.com ↗ |
| WISER-Search vendor-authored benchmark | Perplexity Deep Research Parallel Core 77 on the same set | 64% (Atomic) accuracy % | Parallel vendor-run | parallel.ai ↗ |
| Search Arena | Sonar-Reasoning-Pro-High statistically tied for #1; April 2025 | 1136 Arena score (Bradley-Terry) with CI | Perplexity vendor-run | perplexity.ai ↗ |
Geminiin blind pool · 0 research.site results · 14 external scores
measured by research.site
not in any research.site study yet
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DeepResearch Bench | gemini-2.5-pro-deepresearch 2025 outputs re-judged by GPT-5.5 | 49.98 RACE 0–100, relative to a reference report | benchmark authors | huggingface.co ↗ |
| DeepResearch Bench II | Gemini 3 Pro Deep Research Gemini 2.5 Pro Deep Research 39.89 on the same board | 41.52 rubric pass rate 0–100 | benchmark authors | huggingface.co ↗ |
| DRACO vendor-authored benchmark | Gemini Deep Research version not specified (pre-Max) | 59.0% normalized rubric score % | Perplexity (paper) vendor-run | arxiv.org ↗ |
| ResearchRubrics | Gemini DR binary 0.615; top system; 2025 version | 0.677 rubric compliance 0–1 (ternary) | benchmark authors | arxiv.org ↗ |
| LiveResearchBench | Gemini Deep Research | 66.4 average of 4 dimensions, 0–100 | benchmark authors | livedeepresearch.github.io ↗ |
| ResearcherBench | Gemini Deep Research faithfulness 0.86 · groundedness 0.59 | 0.6929 rubric coverage 0–1 | benchmark authors | github.com ↗ |
| ReportBench | Gemini Deep Research reference precision 0.145; non-cited factual accuracy 92.21% | 72.94% match citation match rate % · factual accuracy % | benchmark authors | arxiv.org ↗ |
| BrowseComp vendor-authored benchmark | Gemini Deep Research Dec 2025 agent (deep-research-pro-preview-12-2025) | 59.2% accuracy % | Google vendor-run | blog.google ↗ |
| DeepSearchQA vendor-authored benchmark | Gemini Deep Research Agent ±3.2; F1 81.9; evaluated 2025-12-11, row since relabeled deep-research-preview-04-2026 without a re-run | 66.1% fully-correct accuracy % (some vendors report F1) | Google (Kaggle board) vendor-run | kaggle.com ↗ |
| WideSearch | Gemini 2.5 Pro (E2E System) success rate 4.3%; consumer web-browsing mode, not Deep Research | 36.6% row F1 row-level F1 % (paper: table success rate) | benchmark authors | widesearch-seed.github.io ↗ |
| xbench-DeepSearch | Gemini Pro consumer app; board doesn't say whether Deep Research mode was used | 53 accuracy %, integer | xbench | xbench.org ↗ |
| Humanity's Last Exam | Gemini Deep Research full HLE set, Dec 2025 agent | 46.4% accuracy % | Google vendor-run | blog.google ↗ |
| Mind2Web 2 | Gemini Deep Research partial 0.45 | 0.18 success rate 0–1 | benchmark authors | osu-nlp-group.github.io ↗ |
| Search Arena | Gemini-2.5-Pro-Grounding Gemini grounding, not Deep Research; April 2025 snapshot | 1142 Arena score (Bradley-Terry) with CI | Perplexity vendor-run | perplexity.ai ↗ |
Parallelin blind pool · 3 research.site results · 18 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| Astra-26 | GPT-6 Astra + MCP search / fetch search tool (agent backend) | 7/26 (26.9%) accuracy | #4 of 5 | Wilson 95% 14%–46% · median 215 s · $1.37 / task · $5.08 / correct |
| RB-30 | ultra deep research API | 72.2% rubric coverage | #2 of 6 | 95% CI 67%–78% · faithfulness 95.4% · groundedness 48.0% · 5,856 claims checked · $0.30 / request · judges o3-mini / gpt-4.1 |
| DRB2-20 | ultra deep research API | 29.1% rubric pass rate | #8 of 9 | info recall 22.9 · analysis 31.1 · presentation 84.3 · 10.0 min mean · $0.30 / request · judge gemini-3.1-pro-preview |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DeepResearch Bench | Parallel Ultra / Ultra8x different metric: share of 50 English tasks beating the reference report; runs dated “Aug 11–29”, no year | 82% / 96% win rate RACE 0–100, relative to a reference report | Parallel vendor-run | parallel.ai ↗ |
| DRACO vendor-authored benchmark | Parallel Ultra8x $2,400 per 1k | 51.2% normalized rubric score % | Valyu vendor-run | github.com ↗ |
| BrowseComp vendor-authored benchmark | Parallel Ultra4x (Task API) 100-question subset, 2026-08-26; Lite 88 · Core 91 · Ultra 92 | 94% accuracy % | Parallel vendor-run | parallel.ai ↗ |
| DeepSearchQA vendor-authored benchmark | Parallel Ultra4x (Task API) 100-question subset | 86% fully-correct accuracy % (some vendors report F1) | Parallel vendor-run | parallel.ai ↗ |
| WideSearch | Parallel Search Advanced search API behind a GPT-5.6 Sol agent; 100 English tasks | 57.6% row-level F1 % (paper: table success rate) | Parallel vendor-run | parallel.ai ↗ |
| search_evals (Perplexity) vendor-authored benchmark | parallel (Task API ultra4x) Parallel reports 94% BrowseComp for the same processor | BC 0.56 · DSQA 0.81 accuracy 0–1 (WideSearch: row F1) | Perplexity vendor-run | github.com ↗ |
| WANDR vendor-authored benchmark | Parallel | 0.126 soft F1 0–1 | Perplexity vendor-run | perplexity.ai ↗ |
| Artificial Analysis Search Index | Parallel Search (advanced) basic 73 | 75 index 0–100 | Artificial Analysis | artificialanalysis.ai ↗ |
| NEEDLE vendor-authored benchmark | parallel p50 3.4 s | 62% 7-day mean % of oracle | Keenable vendor-run | keenableai.github.io ↗ |
| Multi-turn company search (openbenchmarks) | Parallel basic (/v1/search) top row; $5 per 1k queries | 46.5% F1 % | openbenchmarks | openbenchmarks.com ↗ |
| AIMultiple Agentic Search | Parallel Search (Pro) #4; 13.07–15.47 | 14.21 agent score /20 (relevance × quality) | AIMultiple | aimultiple.com ↗ |
| SimpleQA vendor-authored benchmark | parallel_search_basic You.com's run | 94.87% % correct | You.com vendor-run | github.com ↗ |
| SimpleQA Verified vendor-authored benchmark | Parallel Search Advanced / Basic frontier agent (GPT-5.6 Sol high), 100 questions, 2026-09-09 | 97% % correct | Parallel vendor-run | parallel.ai ↗ |
| FRAMES vendor-authored benchmark | parallel_search_basic | 73.79% accuracy % | You.com vendor-run | github.com ↗ |
| FreshQA | Parallel Valyu's run; tier not stated | 52% accuracy % | Valyu vendor-run | valyu.ai ↗ |
| Valyu domain search evals vendor-authored benchmark | Parallel | 67% / 52% / 42% accuracy % | Valyu vendor-run | valyu.ai ↗ |
| You.com web-search-api-evals vendor-authored benchmark | parallel_ultra pro 34.45% | 73.11% accuracy % | You.com vendor-run | github.com ↗ |
| WISER-Search vendor-authored benchmark | GPT 4.1 w/ Parallel Search MCP 2025-07 | 74.9 accuracy % | Parallel vendor-run | parallel.ai ↗ |
Exain blind pool · 10 research.site results · 21 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| DSE-246 | GPT-6 Astra + Exa Search + Contents search tool (agent backend) | 297/492 (60.4%) accuracy | #1 of 2 | vs DeepSeek Search + Fetch 47.6% · median 87 s · model $462.01 + tools $75.96 total |
| Astra-26 | GPT-6 Astra + MCP search / fetch search tool (agent backend) | 11/26 (42.3%) accuracy | #1 of 5 | Wilson 95% 26%–61% · median 158 s · $1.26 / task · $2.98 / correct |
| LBC-30 | Claude Code · Exa MCP search tool (agent backend) | 60.0% accuracy | #1 of 8 | dated 66.7% · undated 53.3% · cost $50.31 · cheapest Claude arm |
| LBC-30 | Codex CLI · Exa MCP search tool (agent backend) | 40.0% accuracy | #6 of 8 | dated 46.7% · undated 33.3% · cost ~$1–5* · 425 MCP calls |
| RB-30 | high deep research API | 66.6% rubric coverage | #4 of 6 | 95% CI 60%–73% · faithfulness 89.1% · groundedness 36.3% · 2,468 claims checked · $0.50 / request · judges o3-mini / gpt-4.1 |
| DRB2-20 | agent · medium deep research API | 45.0% rubric pass rate | #1 of 9 | info recall 39.3 · analysis 50.2 · presentation 90.5 · 4.2 min mean · $0.10 / request · judge gemini-3.1-pro-preview |
| DRB2-20 | agent · high deep research API | 43.8% rubric pass rate | #2 of 9 | info recall 39.2 · analysis 41.9 · presentation 91.8 · 4.5 min mean · $0.50 / request · judge gemini-3.1-pro-preview |
| DRB2-20 | agent · low deep research API | 43.8% rubric pass rate | #3 of 9 | info recall 38.9 · analysis 44.1 · presentation 87.0 · 3.9 min mean · $0.025 / request · judge gemini-3.1-pro-preview |
| DRB2-20 | agent · minimal deep research API | 40.9% rubric pass rate | #4 of 9 | info recall 34.8 · analysis 46.9 · presentation 86.8 · 2.2 min mean · $0.012 / request · judge gemini-3.1-pro-preview |
| DRB2-20 | agent · xhigh deep research API | 39.5% rubric pass rate | #5 of 9 | info recall 35.3 · analysis 31.6 · presentation 93.4 · 5.0 min mean · $1.00 / request · judge gemini-3.1-pro-preview |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DRACO vendor-authored benchmark | Exa Deep Reasoning Exa search type=deep-reasoning — not the Agent API; $15 per 1k | 25.9% normalized rubric score % | Valyu vendor-run | github.com ↗ |
| BrowseComp vendor-authored benchmark | Exa Agent Max Parallel's run; “max” is not a public Exa effort tier | 78% accuracy % | Parallel vendor-run | parallel.ai ↗ |
| BrowseComp vendor-authored benchmark | Exa 3.0 2026-10-19; read from the chart's alt text; no harness, sample or grader stated | 91.8% accuracy % | Exa vendor-run | exa.ai ↗ |
| DeepSearchQA vendor-authored benchmark | Exa Agent Max Parallel's run | 65% fully-correct accuracy % (some vendors report F1) | Parallel vendor-run | parallel.ai ↗ |
| DeepSearchQA vendor-authored benchmark | Exa 3.0 chart alt text; no harness stated | 88.2% fully-correct accuracy % (some vendors report F1) | Exa vendor-run | exa.ai ↗ |
| WideSearch | Exa Auto (frontier) same Parallel harness; low-cost tier 53 | 55.9% row-level F1 % (paper: table success rate) | Parallel vendor-run | parallel.ai ↗ |
| WideSearch | Exa 3.0 chart alt text | 56.1% row-level F1 % (paper: table success rate) | Exa vendor-run | exa.ai ↗ |
| search_evals (Perplexity) vendor-authored benchmark | exa (Agent API, effort high) HLE 0.387 · WideSearch 0.471 | BC 0.38 · DSQA 0.53 accuracy 0–1 (WideSearch: row F1) | Perplexity vendor-run | github.com ↗ |
| WANDR vendor-authored benchmark | Exa | 0.057 soft F1 0–1 | Perplexity vendor-run | perplexity.ai ↗ |
| Artificial Analysis Search Index | Exa Search (auto, dynamic highlights) DSQA 86 · BrowseComp 83 · $53.17 search per 1k tasks | 80 index 0–100 | Artificial Analysis | artificialanalysis.ai ↗ |
| Vals Web Search Index | Claude Fable 5 | Exa #2; GPT-5.6 Sol | Exa 45.24% | 48.45% % correct overall | Vals AI | vals.ai ↗ |
| NEEDLE vendor-authored benchmark | exa p50 3.0 s; exa-instant 60% | 68% 7-day mean % of oracle | Keenable vendor-run | keenableai.github.io ↗ |
| Multi-turn company search (openbenchmarks) | Exa deep (/search type=deep) 44.7 s mean search time | 45.4% F1 % | openbenchmarks | openbenchmarks.com ↗ |
| AIMultiple Agentic Search | Exa #3; 95% CI 13.25–15.50 | 14.39 agent score /20 (relevance × quality) | AIMultiple | aimultiple.com ↗ |
| SimpleQA vendor-authored benchmark | exa_search_with_text You.com's run; p50 2.5 s | 95.40% % correct | You.com vendor-run | github.com ↗ |
| SimpleQA Verified vendor-authored benchmark | Exa Auto (frontier) Parallel's run | 91% % correct | Parallel vendor-run | parallel.ai ↗ |
| FRAMES vendor-authored benchmark | exa_search_with_text | 73.79% accuracy % | You.com vendor-run | github.com ↗ |
| FreshQA | Exa Valyu's run; mode not stated | 24% accuracy % | Valyu vendor-run | valyu.ai ↗ |
| Valyu domain search evals vendor-authored benchmark | Exa | 63% / 45% / 44% accuracy % | Valyu vendor-run | valyu.ai ↗ |
| You.com web-search-api-evals vendor-authored benchmark | exa_research_pro retired endpoint (HTTP 410) | 42.02% accuracy % | You.com vendor-run | github.com ↗ |
| WISER-Search vendor-authored benchmark | GPT 4.1 w/ Exa Search MCP Parallel's run | 58.67 accuracy % | Parallel vendor-run | parallel.ai ↗ |
Valyuin blind pool · 5 research.site results · 4 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| Astra-26 | GPT-6 Astra + MCP search / fetch search tool (agent backend) | 6/26 (23.1%) accuracy | #5 of 5 | Wilson 95% 11%–42% · median 299 s · $1.56 / task · $6.74 / correct |
| LBC-30 | Claude Code · Valyu MCP search tool (agent backend) | 26.7% accuracy | #7 of 8 | dated 26.7% · undated 26.7% · cost $117.93 |
| LBC-30 | Codex CLI · Valyu MCP search tool (agent backend) | 20.0% accuracy | #8 of 8 | dated 33.3% · undated 6.7% · cost ~$1.5–7* · 858 MCP calls, 2.7M tokens |
| RB-30 | standard deep research API | 70.2% rubric coverage | #3 of 6 | 95% CI 65%–76% · faithfulness 75.9% · groundedness 49.5% · 9,654 claims checked · $0.50 / request · judges o3-mini / gpt-4.1 |
| DRB2-20 | standard deep research API | 31.4% rubric pass rate | #7 of 9 | info recall 25.1 · analysis 37.2 · presentation 78.2 · 15.1 min mean · $0.50 / request · judge gemini-3.1-pro-preview |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DRACO vendor-authored benchmark | Valyu DeepResearch (Heavy) $2,500 per 1k queries; single run; Max tier not run | 72.7% normalized rubric score % | Valyu vendor-run | github.com ↗ |
| SimpleQA vendor-authored benchmark | Valyu Parallel 93% · Exa 91% · Google 38% in the same run (2025-10-03) | 94% % correct | Valyu vendor-run | valyu.ai ↗ |
| FreshQA | Valyu 2025-10-03; version not stated | 79% accuracy % | Valyu vendor-run | valyu.ai ↗ |
| Valyu domain search evals vendor-authored benchmark | Valyu finance / economic / MedAgent | 73% / 73% / 48% accuracy % | Valyu vendor-run | valyu.ai ↗ |
You.comin blind pool · 1 research.site result · 8 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| RB-30 | exhaustive deep research API | 58.5% rubric coverage | #5 of 6 | 95% CI 53%–65% · faithfulness 15.4% · groundedness 32.5% · 1,112 claims checked · $0.45 / request · judges o3-mini / gpt-4.1 |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| DRACO vendor-authored benchmark | You.com Research (exhaustive) $450 per 1k; frontier tier was sales-gated | 53.0% normalized rubric score % | Valyu vendor-run | github.com ↗ |
| DeepSearchQA vendor-authored benchmark | Research API Frontier F1 94.35%; Frontier is $1,200 per 1k, not wired in the arena | 86.61% fully-correct accuracy % (some vendors report F1) | You.com vendor-run | about.you.com ↗ |
| Artificial Analysis Search Index | You.com Search (highlights) 19.0 s/task | 74 index 0–100 | Artificial Analysis | artificialanalysis.ai ↗ |
| NEEDLE vendor-authored benchmark | you p50 513 ms | 58% 7-day mean % of oracle | Keenable vendor-run | keenableai.github.io ↗ |
| Multi-turn company search (openbenchmarks) | You (highlights) 7.7 s | 38.6% F1 % | openbenchmarks | openbenchmarks.com ↗ |
| SimpleQA vendor-authored benchmark | you_search README 2026-10-07; p50 598 ms | 96.11% % correct | You.com vendor-run | github.com ↗ |
| FRAMES vendor-authored benchmark | you_search | 74.03% accuracy % | You.com vendor-run | github.com ↗ |
| You.com web-search-api-evals vendor-authored benchmark | you_finance_research_deep FinSearchComp T2 | 87.29% accuracy % | You.com vendor-run | github.com ↗ |
Keenablenot yet measured as deep research · 3 research.site results · 3 external scores
measured by research.site
| study | tested as | result | rank | detail |
|---|---|---|---|---|
| Astra-26 | GPT-6 Astra + MCP search / fetch search tool (agent backend) | 10/26 (38.5%) accuracy | #3 of 5 | Wilson 95% 22%–57% · median 171 s · $1.30 / task · $3.39 / correct |
| LBC-30 | Codex CLI · Keenable SELECT (WebQL) MCP (via OpenRouter) search tool (agent backend) | 46.7% accuracy | #2 of 8 | dated 33.3% · undated 60.0% · cost ~$15* · 281 select calls; via OpenRouter |
| LBC-30 | Claude Code · Keenable SELECT (WebQL) MCP (via OpenRouter) search tool (agent backend) | 43.3% accuracy | #5 of 8 | dated 46.7% · undated 40.0% · cost $44.39 · 3 failed runs (idx 49, 156, 307: 3×30-min timeouts each); 1,498 select calls; via OpenRouter |
published elsewhere
| benchmark | as published | score | reported by | source |
|---|---|---|---|---|
| Artificial Analysis Search Index | Keenable Search (pro / realtime) 2026-08-18 launch snapshot — not among the rows on the live board | 67 index 0–100 | Artificial Analysis | artificialanalysis.ai ↗ |
| Vals Web Search Index | GPT-5.6 Sol | Keenable #1; same model with native search 43.58% | 49.08% % correct overall | Vals AI | vals.ai ↗ |
| NEEDLE vendor-authored benchmark | Keenable p50 139 ms | 73% 7-day mean % of oracle | Keenable vendor-run | keenableai.github.io ↗ |