When an agent has to find a hard fact on the live web, what decides whether it succeeds: the agent, or the search tool plugged into it? Most benchmarks test an agent with whatever search it ships with, so a score never separates the model from its retrieval. This study isolates the two.
| # | System | vs best | Solved | Gen cost | ||||
|---|---|---|---|---|---|---|---|---|
| 1 | claude-exaclaude code · exa mcp · cheapest Claude arm | 60.0% | reference | 18 / 30 | 66.7 | 53.3 | $50.31 | |
| 2 | claudeclaude code · builtin websearch · 1,632 builtin searches | 46.7% | −13.3 | 14 / 30 | 46.7 | 46.7 | $120.06 | |
| 3 | codexcodex cli · live web_search · 2.4M tokens | 46.7% | −13.3 | 14 / 30 | 40.0 | 53.3 | ~$1–6* | |
| 4 | codex-exacodex cli · exa mcp · 425 MCP calls | 40.0% | −20.0 | 12 / 30 | 46.7 | 33.3 | ~$1–5* | |
| 5 | claude-valyuclaude code · valyu mcp | 26.7% | −33.3 | 8 / 30 | 26.7 | 26.7 | $117.93 | |
| 6 | codex-valyucodex cli · valyu mcp · 858 MCP calls, 2.7M tokens | 20.0% | −40.0 | 6 / 30 | 33.3 | 6.7 | ~$1.5–7* |
backend moves accuracy 27–33 points · agent at most 6.7
| search ↓ · agent → | Claude Code | Codex CLI | backend spread |
|---|---|---|---|
| builtin | 46.7% | 46.7% | 0.0 pts |
| exa | 60.0% | 40.0% | 20.0 pts |
| valyu | 26.7% | 20.0% | 6.7 pts |
| agent spread | 33.3 pts | 26.7 pts |
Per-task verdicts are on the ; per-run search behavior from the raw traces is under and per task in the .
caveats
n=30, single run per cell, so one task is worth ~3.3 points; treat gaps under 10 points as noise (claude-exa’s lead and the valyu deficit clear that bar; builtin-vs-codex-exa does not). Codex ran at its CLI-default reasoning effort (“none” in this build). The judge is Gemini, not the paper’s GPT-OSS. The two funders’ backends finished first and last; disclosure below.
Exa provided $1,000 in API credits for this benchmark series (shared with RB-30 and DRB2-20), and Valyu provided $500 for this benchmark (disclosed post-freeze, pre-main-run; Amendment F). Both are compared search backends here; one funder’s backend won and the other’s lost, under configs frozen mechanically before the first run. Neither had input into task selection, configurations, prompts or judging.
30 tasks · 15 dated / 15 undated · 6 systems · 180 main runs, 0 failures · judge gemini-3.1-pro-preview, blinded · frozen & run 2026-08-13 · 8 amendments, none post-score · questions & answers © LiveBrowseComp authors · repository · RESULTS.md ·