research.siteHarness-Bench
One model. Four ways to research.
What does a research harness add when the model stays the same? Claude Fable 5.2 answered 12 ResearchRubrics questions closed-book, with a plain search-and-read loop, inside GPT Researcher and inside LangChain Deep Agents. Two judges graded every report against the dataset’s rubric.
Rubric score
Higher is betterMean of two judges · 12 questions
Range if every judgment the two judges did not settle went against, or for, the report.
What this measures
Each report is graded against its question’s ResearchRubrics criteria. The score is the weight of criteria judged met divided by the total positive weight, so 1.0 means every requirement was met. Error criteria carry negative weight, so a score can fall below zero.
It measures rubric compliance as read from the report text. Nobody checked the reports’ statements against the world, and citation criteria check that sources are cited, not that they support the claim.
Time
Lower is betterMean minutes per run · start to final report
Fastest to slowest of the 12 runs. Time was measured, not capped.
What this measures
Wall-clock time for one run, from the first model call to the finished report, including every model call, search and page read. Runs of the four configurations were interleaved in a seeded random order. Two runs were in flight at a time.
Cost
Lower is betterMean generation cost per run · USD
model retrieval · cheapest to most expensive run. No per-run caps.
What this measures
Model cost is list price applied to every metered token of every call in the run, including thinking, cache writes and reads, and the system’s own retries. Retrieval cost is what Exa reported for the run’s searches and page reads.
Grading, smoke tests and runs that were stopped or dropped are not included here. They are listed under Results.
None of the three registered comparisons met the study’s rule for claiming a direction. On the two-judge mean, Deep Agents scored 0.562, the closed-book answer 0.558, the plain loop 0.518 and GPT Researcher 0.433. The three configurations that search the web cost 6.4× to 18× as much per run as the closed-book answer and took 2.5× to 6.3× as long. Scores are rubric compliance read from the report, not verified accuracy.
The three registered comparisons
Fixed before any main output existed · question is the unitDifference in rubric score, question by question
Mean of the 12 paired differences with a 95% bootstrap interval · positive favours the first configuration
What does ordinary tool-assisted research add?
- Wins / losses / ties
- 4 / 8 / 0
- p, sign-flip
- 0.168
- p, Holm over 3
- 0.292
What does GPT Researcher add beyond a plain loop?
- Wins / losses / ties
- 1 / 11 / 0
- p, sign-flip
- 0.001
- p, Holm over 3
- 0.004
What does Deep Agents add beyond a plain loop?
- Wins / losses / ties
- 9 / 3 / 0
- p, sign-flip
- 0.146
- p, Holm over 3
- 0.292
A direction counts as supported only if the interval excludes zero with the same sign under each judge alone and under the unfavourable end of the worst-to-best range. The judges did not settle 139 of 1,280 judgments; worst and best case resolve all of them one way.
How to read this chart
For each question the first configuration’s score minus the baseline’s is one paired difference. The dot is the mean of those 12 differences and the bar is a 95% percentile bootstrap interval over questions (10,000 draws, fixed seed). The p-value is a paired sign-flip permutation test on the two-judge mean, with a Holm adjustment across the three comparisons.
Twelve questions give wide intervals. A difference that is not distinguished from zero is not evidence that two configurations are equal, and no claim of equivalence is made. Model, effort and research tools were held equal; prompts, loop structure and spending were not, so these are differences between configurations as run here, not pure orchestration effects.
GPT Researcher scored below the plain loop on 11 of 12 questions (−0.085 on the two-judge mean), and that interval excludes zero under each judge alone. It still falls short of the rule: if every unsettled judgment went GPT Researcher’s way, the difference would be +0.048, with an interval that includes zero.
Behind the averages
What did the extra spend buy?
Every run: 48 small marks. Large marks are each configuration’s mean.
The vertical bar through each mean is its range across judgments the two judges did not settle. With one run per question, a single small mark says little on its own. Every run is also listed in the Data Viewer.
How to read this chart
Up is a higher rubric score; right is more money spent on the run. The horizontal axis is logarithmic, so equal distances are equal ratios. Spending was not held equal across configurations: each run spent what its workflow spent, and the study computes no quality-per-dollar score.
All 48 runs used Claude Fable 5.2 at effort high. Small marks of one configuration that sit far apart are different questions, not repeated attempts.
Every question and configuration
Rubric score per run · mean of two judges · bar spans 0 to 1
| Question | A Closed-book | B Plain tools | C GPT Researcher | D Deep Agents |
|---|---|---|---|---|
rr-6053c9Technical Documentation | 0.62 | 0.51 | 0.32 | 0.64 |
rr-6053f5 · datedOther | 0.56 | 0.46 | 0.35 | 0.54 |
rr-60542a · datedTechnical Documentation | 0.71 | 0.48 | 0.47 | 0.69 |
rr-605433 · datedBusiness Planning & Research | 0.85 | 0.87 | 0.71 | 0.70 |
rr-60543e · datedGeneral Consumer Research | 0.04 | 0.01 | −0.03 | 0.01 |
rr-605441 · datedCurrent Events | 0.48 | 0.38 | 0.33 | 0.36 |
rr-605469Hypotheticals & Philosophy | 0.40 | 0.37 | 0.32 | 0.41 |
rr-60548aAI & ML | 0.70 | 0.71 | 0.41 | 0.73 |
rr-9af308STEM | 0.75 | 0.67 | 0.71 | 0.78 |
rr-9af316 · datedCurrent Events | 0.34 | 0.47 | 0.42 | 0.60 |
rr-9af31dHistorical Analysis | 0.60 | 0.59 | 0.53 | 0.64 |
rr-9af332Creative Writing | 0.65 | 0.71 | 0.66 | 0.67 |
| Mean of 12 | 0.56 | 0.52 | 0.43 | 0.56 |
10 ResearchRubrics domains; 6 questions marked “dated” use time-sensitive wording, so a tool-using run may find facts newer than the rubric expects. One run per cell: differences on a single question are not interpreted. Question text is not reproduced here; each id ends with the last six characters of the question’s id in the public dataset.
What stays fixed
- Model
- Claude Fable 5.2 · effort high on every call, enforced at a gateway
- Search
- Exa, 5 results with a 500-character raw snippet each
- Reader
- Exa Contents raw page text, cut at 30,000 characters
- Questions
- 12 from ResearchRubrics, used verbatim · 10 domains
- Runs
- One per question and configuration · no per-run caps
- Judges
- GPT-6 Astra and Jev, every criterion of every report
Read the result with its limits
- 12 questions from one dataset, one run each. It does not rank research harnesses in general.
- The dataset was public before the model’s training cutoff. No question was removed because the model answered it well without tools.
- The two judges left 139 of 1,280 judgments unsettled. The planned manual review has not been done.
- GPT Researcher and Deep Agents are modified configurations with a 20,000-token output limit per call; a call stopped at it in 9 of their 24 runs.
Data & disclosures
The 48 scored runs cost $230.46 to generate at list prices. The protocol, configuration and question selection were frozen at a tagged commit before any main output existed; three amendments followed and are listed with their reasons. The protocol, code, configuration and amendments are in the repository; reports, call logs and judges’ records are not public. 101 questions exist upstream; question, rubric and report text are not published here.