Coding-Agent CLIs × Search Backends on LBC-30
When an agent has to find a hard fact on the live web, what decides whether it succeeds: the agent, or the search tool plugged into it? Most benchmarks test an agent with whatever search it ships with, so a score never separates the model from its retrieval. This study isolates the two.
| # | System | vs best | Solved | Gen cost | ||||
|---|---|---|---|---|---|---|---|---|
| 1 | claude-exaclaude code · exa mcp · cheapest Claude arm | 60.0% | reference | 18 / 30 | 66.7 | 53.3 | $50.31 | |
| 2 | claudeclaude code · builtin websearch · 1,632 builtin searches | 46.7% | −13.3 | 14 / 30 | 46.7 | 46.7 | $120.06 | |
| 3 | codexcodex cli · live web_search · 2.4M tokens | 46.7% | −13.3 | 14 / 30 | 40.0 | 53.3 | ~$1–6* | |
| 4 | codex-keenable via openroutercodex cli · keenable select (webql) mcp (via openrouter) · 281 select calls; via OpenRouter | 46.7% | −13.3 | 14 / 30 | 33.3 | 60.0 | ~$15* | |
| 5 | claude-keenable via openrouter 3 failed runsclaude code · keenable select (webql) mcp (via openrouter) · 3 failed runs (idx 49, 156, 307: 3×30-min timeouts each); 1,498 select calls; via OpenRouter | 43.3% | −16.7 | 13 / 30 | 46.7 | 40.0 | $44.39 | |
| 6 | codex-exacodex cli · exa mcp · 425 MCP calls | 40.0% | −20.0 | 12 / 30 | 46.7 | 33.3 | ~$1–5* | |
| 7 | claude-valyuclaude code · valyu mcp | 26.7% | −33.3 | 8 / 30 | 26.7 | 26.7 | $117.93 | |
| 8 | codex-valyucodex cli · valyu mcp · 858 MCP calls, 2.7M tokens | 20.0% | −40.0 | 6 / 30 | 33.3 | 6.7 | ~$1.5–7* |
Results
accuracy and cost per correct answer, one column per setup · keenable select joined 2026-09-04
check on 15 fresh questions with identical plumbing for all three: Keenable SELECT 53% · Exa 40% · built-in 33% (one question = 6.7 pts)
Per-task verdicts are on the ; per-run search behavior from the raw traces is under and per task in the .
caveats
n=30, single run per cell, so one task is worth ~3.3 points; treat gaps under 10 points as noise (claude-exa’s lead and the valyu deficit clear that bar; builtin-vs-codex-exa does not). Codex ran at its CLI-default reasoning effort (“none” in this build). The judge is Gemini, not the paper’s GPT-OSS. Two of the funders’ backends finished first and last; disclosure below. The two Keenable SELECT (WebQL) arms (2026-09-04) measure Keenable’s experimental SQL-over-the-web product, not its search API, and ran with model calls via OpenRouter on a newer Claude Code build than the original six; on a transport-matched held-out 15 the frozen-30 ordering did not reproduce (claude-keenable 53.3%, claude-exa-or 40.0%, claude-builtin-or 33.3%; see the leaderboard and methodology tabs).
Funding
Exa provided $1,000 in API credits for this benchmark series (shared with RB-30 and DRB2-20), and Valyu provided $500 for this benchmark (disclosed post-freeze, pre-main-run; Amendment F). Both are compared search backends here; one funder’s backend won and the other’s lost, under configs frozen mechanically before the first run. Neither had input into task selection, configurations, prompts or judging.
For the 2026-09-04 extension, Keenable provided a $250 OpenRouter credit (of which $174.92 was drawn) that paid for the model calls of its two arms, the held-out slice including the two control arms, and the extension judging pass, plus the API key for the SELECT (WebQL) MCP endpoint. Keenable can observe the benchmark queries after the run; it had no input into task selection, configurations, prompts, the tool-surface rule or judging, all committed before its arms ran (CONFIG.md §9 update, Amendment I).
30 tasks · 15 dated / 15 undated · 6 systems, 180 main runs, 0 failures (2026-08-13) + 2 Keenable SELECT arms, 60 runs, 3 failures (2026-09-04) + held-out 15, 45 runs · judge gemini-3.1-pro-preview, blinded · 13 amendments, none post-score · questions & answers © LiveBrowseComp authors · repository · RESULTS.md ·