Applied AI
Open source
DeepResearch Bench

Open Deep Research

A multi-agent research API that turns one question into a long, cited report. I evaluated it on 20 DeepResearch Bench tasks with published graders, and the results traced report quality back to a single retrieval stage.

Live app GitHub

ODR-API — DEEP RESEARCH

Citation accuracy (FACT): 0.847
RACE vs reference, 0.5 = parity: 0.365
Reranker recall vs report quality: r 0.69
Median cost per research task: $0.22
Median time per task: 213 s
Cached UMBRELA relevance labels: 4,552

01Overview

What it does

A client sends a research question over a WebSocket. Four typed agents and a set of shared services build the report while the server streams progress events. The code is organized as agencies built on shared services, so a second agency, a company profile extractor, reuses search, scraping and ranking.

Agencies

Deep Research: Question in, cited Markdown report out.
Company Profile Extractor: Crawls company sites and extracts a profile. Reuses the same services.

Agents

Planner: Sub-queries and a writing plan.
Summarizer: Dense summary per top source.
Writer: Cited draft, may request searches.
Refiner: Integrates new evidence, up to two loops.

Services

Search: Serper web, scholar and news.
Ranking: Cohere cross-encoder rerank.
Scraping: Crawl4AI with Chromium, PyMuPDF.
Chunking: Markdown-aware recursive splits.

Schemas

PlannerOutput: SearchTasks plus a WritingPlan.
WriterOutput: Report with [[CITATION:n]] markers.
SearchTask: Validators repair common LLM slips.
UsageStatistics: Tokens per role, queries, loops.

Every agent is a pydantic-ai agent with a typed output. Claude models use native JSON-schema structured outputs; any other model ID routes through OpenRouter, and each role can run a different model.

Runs are background tasks. If the socket drops, the run continues, state lands in Firestore, and the client polls for the result. A stop endpoint cancels the task.

WS /deep_research/ws/research

[ INITIALIZING ]task_id assigned · Firestore PENDING → PROCESSING
[ PLANNING ]3–7 SearchTasks + a section-by-section WritingPlan
[ SEARCHING ]one Serper batch · /search /scholar /news
[ RANKING ]Cohere rerank-english-v3.0 · keep ≥ 0.25
[ PROCESSING ]scrape · summarize top sources · chunk the rest
[ WRITING ]cited draft · [[CITATION:n]] enforced
[ REFINING ]new evidence only · at most 2 loops
[ COMPLETE ]report + references + per-role token usage

{ step, status, message, details } · ~30 typed events

02System

Architecture

Hover or tap a component to trace its connections. The flows step through a research run, the refine loop and an eval run. The eval harness calls the same orchestration function as the WebSocket route.

Hover or tap any component to trace its connections, or pick a flow to step through a request end to end.

03Pipeline

The research loop

Red numerals mark steps that call a model. A task makes roughly 3 to 14 model calls: one plan, up to ten summaries, one draft and up to two refinements.

[ INITIALIZING ]
Connect
The server validates the query, assigns a task_id, writes PENDING then PROCESSING to Firestore and starts the run as an asyncio task.
[ PLANNING ]LLM
Plan
Sonnet 5.5 returns 3 to 7 SearchTasks, each with a query, an endpoint and a result count, plus a WritingPlan with ordered sections and guidance.
[ SEARCHING ]
Search
One batched Serper call runs every sub-query. Results are flattened and de-duplicated by URL, about 59 per task in the baseline.
[ RANKING ]
Rerank
Cohere scores each result’s title and snippet against the query and drops anything under 0.25. If the rerank call fails, the unranked list is kept.
[ PROCESSING ]
Scrape
Kept URLs are scraped concurrently with one shared Chromium. PDFs are parsed with PyMuPDF and Wikipedia goes through its API.
[ PROCESSING ]LLM
Summarize
Haiku 5.5 writes a dense summary of each top source, up to ten, keeping numbers, methods, findings and limitations.
[ PROCESSING ]
Chunk
The remaining sources are chunked at 2,048 characters, reranked at 0.5, and the top five chunks per source are kept. Every source gets a stable reference number.
[ PROCESSING ]
Budget
tiktoken measures the context. Chunks are trimmed first if summaries and chunks together exceed 600,000 tokens.
[ WRITING ]LLM
Write
Sonnet 5.5 drafts the report with [[CITATION:n]] on every claim. A validator raises ModelRetry if the draft has no citation markers.
[ REFINING ]LLM
Refine
If the writer requested searches, they run through the same retrieval path and the Refiner integrates only the new material. Two loops at most.
[ COMPLETE ]
Finalize
Markers become [n], a numbered reference list is generated from the stable numbers, and the report streams with per-role token usage.

04Evals

How the eval works

Two evals, both built on published benchmarks. Judge prompts are copied verbatim from the upstream repositories; only the output-format lines are adapted to JSON-schema outputs, and every other deviation is listed with its reason.

◈ search_quality

UMBRELA

Are the search results, the reranker’s ordering and the writer’s context relevant?

Method

An open reproduction of the LLM relevance assessor used for the TREC 2024 RAG track. Each passage gets a 0–3 grade, which feeds nDCG@10, precision@10, reranker recall and context precision.

What I changed, and why

Each passage gets two labels: one against the sub-query that retrieved it and one against the full task. UMBRELA expects a single focused query, and a result about one of three requested companies is a perfect hit for its sub-query but only related to the whole task.

castorini/umbrela

◈ report_quality

RACE

How good is the final report?

Method

DeepResearch Bench’s task-specific criteria, 22 to 28 per task with weights. The judge scores our report and the benchmark’s reference report on every criterion, and the result is ours / (ours + reference), so 0.5 means parity.

What I changed, and why

Both orders are scored and averaged to cancel position bias, and the gap is logged (mean 0.015). A per-task JSON schema forces the judge to return exactly that task’s criteria. Citation markers are stripped with a regex instead of an LLM pass.

Ayanami0730/deep_research_bench

◈ report_quality

FACT

Is each cited statement supported by the source it cites?

Method

Every statement–citation pair is checked by the judge. Citation accuracy is supported / (supported + unsupported).

What I changed, and why

Statements come from the structured citation markers, because upstream’s LLM extraction found 23 of 184 pairs in the pilot. Each claim gets its own call, because batching flipped 19 of 48 pilot verdicts. Claims are checked against the evidence the writer was given, which measures grounding in the pipeline’s own context.

DeepResearch Bench · FACT

05Baseline

Results

Baseline configuration: Sonnet 5.5 for planner, writer and refiner, Haiku 5.5 for summaries, Cohere rerank at a 0.25 threshold, two refinement loops. 20 tasks, one repetition, no harness errors. Intervals are 95% normal approximations over tasks.

Report quality, judged by Sonnet 5.5.
Metric	Mean ± 95% CI
RACE overall	0.365 ± 0.054
Comprehensiveness	0.337 ± 0.058
Insight	0.370 ± 0.057
Instruction following	0.360 ± 0.058
Readability	0.403 ± 0.049
FACT citation accuracy	0.847 ± 0.093
Supported citations per report	87.4 ± 21.8

Search quality, UMBRELA labels. Relevant means grade 2 or higher.
Metric	Mean ± 95% CI
Reranker nDCG@10	0.578 ± 0.112
Reranker precision@10	0.470 ± 0.129
Reranker recall	0.426 ± 0.149
SERP precision	0.344 ± 0.071
Context precision	0.376 ± 0.135
Task-level nDCG@10	0.655 ± 0.119

2 / 20

Reports judged better than the reference (0.534, 0.514)

1,748

Supported citations of 2,223 statement–citation pairs

2,805

Median words per report. References run 5k–15k

$25.68

Whole baseline: $5.64 pipeline, $20.04 grading

Task 75 produced no report and scored 0. It stays in the averages, because a run that writes nothing is a product failure. Pipeline cost per task ranged from $0.03 to $1.03 and time from 21 to 1,040 seconds. Grading cost about 3.5 times as much as running the pipeline.

06Finding

What the eval found

Report quality follows reranker recall (r = 0.69). On half the tasks the reranker kept 8 or fewer of about 60 results, and those tasks scored far lower.

Report quality tracks how much evidence survives the reranker

RACE vs reranker recall, one point per task · Pearson r = 0.69 · n = 20

Starved: reranker kept ≤ 8 of ~60 results (10)
Everything else (10)

10 / 20

Tasks where the reranker kept 8 or fewer results

16% · 70%

Relevant results kept: starved tasks · the rest

0.280 · 0.449

RACE: starved tasks · the rest

0 / 57

Results kept on task 75, which wrote no report

Correlation with RACE

Pearson r across the 20 baseline tasks. Retrieval-volume signals highlighted.

Task-level nDCG@100.71

Reranker recall0.69

Results kept by reranker0.68

Report length (words)0.66

Writer-context items0.61

Pipeline cost0.49

SERP precision0.12

Context precision0.05

The cause. DeepResearch Bench prompts are long and multi-part. The reranker scored each result against the whole prompt, so relevant results got low absolute scores and the fixed 0.25 threshold discarded them.

Search was fine. SERP precision barely correlates with quality (r = 0.12), and neither does context precision (r = 0.05). The writer’s context was precise enough. The constraint was how much relevant evidence reached it.

Grounding held. FACT stayed high on starved tasks: 0.967 on task 53 and 1.000 on task 59. Task 53’s report opens by noting that it rests on a single source. The low RACE on those tasks reflects missing coverage.

Refinement did not recover them. Refinement searches go through the same reranker and threshold, so their results were filtered the same way.

07Harness

Eval engineering

The harness is built so results stay comparable while the pipeline changes underneath it.

[ GATE ]

Hash-gated harness

SHA-256 over graders, prompts, runner, sanity checks, data and uv.lock. Every script exits with code 2 until a human re-approves the digest, so no one, including the coding agent doing the hill-climbing, can edit the grader between variants.

[ CACHE ]

Content-addressed labels

UMBRELA labels are keyed by sha256(version | model | query | passage). Variants pay only for unseen passages, and identical passages always get identical labels.

[ FAILURES ]

Typed failure taxonomy

Timeouts, pipeline errors and served-model mismatches are harness errors and are never scored. A run with no context or a failed writer is a product failure and scores 0. Judge errors are typed: refusal, truncated, unparseable, api_error.

[ MODEL ]

Served-model check

The served model is read from every agent result and compared with the requested one, so a server-side fallback cannot silently swap models mid-eval.

[ SANITY ]

Graders tested first

Known inputs run through the real graders before a paid pass: an on-topic passage must score 3, the reference against itself must land in 0.42–0.58, an empty article must score under 0.15.

[ COST ]

Priced per call

Every call is priced from a per-model table, including cache reads and writes. An unknown model raises KeyError, so a silent $0 cannot reach a report.

[ PILOT ]

Pilot before spend

Two tasks ran first, with a written review of graded examples and judge reasoning. The full run started after sign-off.

[ RESUME ]

Resumable stages

Pipeline, search grading and report grading are keyed per (task, rep). A restart skips finished work in all three.

08Iteration

The v1 retrieval change

v1 reranks each result against the sub-query that retrieved it, keeps at least three results per sub-query, and merges the kept lists round-robin so every aspect of the question is represented. One flag restores the baseline.

deep_research/config.py

# False restores the 2026-10-23 eval baseline
rerank_against_subquery: bool = True
rerank_min_results_per_query: int = 3

The change applies to initial results, to chunks, and to results from refinement searches. Its docstring cites the eval directly: the search eval measured 43% reranker recall.

It was piloted first on the two most-starved tasks. The runs are complete but have not been graded yet, so the table shows only what the pipeline logged.

Pilot runs, baseline vs v1. Not yet graded. Cost is computed from per-role token usage at the harness price table.
Run	Results kept	Sources	Words	Time	Cost
Task 53 · baseline	1 / 64	1	1,828	89 s	$0.11
Task 53 · v1	41 / 59	17	3,434	449 s	$0.31
Task 75 · baseline	0 / 57	0	no report	21 s	$0.03
Task 75 · v1	50 / 62	29	5,132	684 s	$0.60

The pilot moved the bottleneck. More evidence now reaches the writer, and summaries run one source at a time, so summarizer input on task 53 rose from 71,683 to 434,942 tokens and wall time grew fivefold. Next: grade v1 on all 20 tasks (cached labels make the search grading cheap), run two repetitions to tighten the intervals, and parallelize summarization.

09Trade-offs

Decisions

ADR 01

Summarize the top sources, chunk the rest

Chose: LLM summaries for up to 10 sources, reranked chunks for the tail
Over: Summarizing every source, or chunking everything

Summaries carry the most useful sources in dense form; chunk reranking covers the long tail cheaply.

Trade-off Summaries run sequentially, which sets most of the latency. The v1 pilot made this visible.

ADR 02

Structured citation markers

Chose: [[CITATION:n]] with reference numbers fixed at ingestion
Over: Free-form citations formatted by the model

References render deterministically, and the FACT eval can extract every statement–citation pair without an LLM.

Trade-off The writer has to comply. A validator enforces it with retries.

ADR 03

Refiner sees only new evidence

Chose: Previous draft plus newly added material, two loops at most
Over: Rewriting from the full context each loop

Tokens per loop stay bounded and the loop count is capped.

Trade-off The refiner cannot revisit evidence it was not shown in that loop.

ADR 04

Typed agents with tolerant validators

Chose: pydantic-ai agents with schema outputs and repairing validators
Over: Parsing free text, or failing the run on any schema slip

Common LLM mistakes, like echoing the endpoint list or an out-of-range count, are coerced into valid values.

Trade-off Validators encode assumptions about which mistakes are safe to repair.

ADR 05

Runs outlive the socket

Chose: Background asyncio task, Firestore state, REST polling
Over: Request-scoped work tied to the connection

Research runs take minutes. A dropped connection should not cost the run.

Trade-off The task registry is in process memory, so cancel only reaches the instance that owns the task.

ADR 06

Published benchmarks for the eval

Chose: UMBRELA, RACE and FACT with upstream prompts
Over: A custom rubric written for this pipeline

The prompts and criteria were not written by me, scores are comparable with other systems, and the method is documented.

Trade-off References are 5k to 15k words against reports of about 2.8k, which caps comprehensiveness.

10History

How it evolved

Apr 2025
Monolithic agent
One 1,201-line DeepResearchAgent class on a LiteLLM wrapper. Together AI reranking, inline XML tags for search requests, context trimmed by character count.
Apr 2025
Refactor to pydantic-ai
Agencies, services and core layers. Typed requested_searches replaced the XML tags. WebSocket callbacks, Firestore and Docker followed in the same week.
Apr–Aug 2025
Second agency
A company profile extractor with its own planner and extractor agents, reusing search, scraping and an email finder.
Apr 2026
Packaging and redesign
uv with a lockfile, Cohere rerank as default, a non-root Docker image with a cached Playwright layer, and the frontend moved to the Nothing design system.
Oct 2026
Eval-driven
Native Claude structured outputs, trace instrumentation, explicit rerank failure semantics, the eval harness and baseline, and the v1 retrieval change.

11Reliability

Failure modes

What breaks, what guards it, and where it shows up.

What breaks	Guard	How I'd know
Rerank API down	The rerank call raises instead of returning an empty list. Results fall back to unranked; a chunk-rerank outage keeps the source’s chunks.	RANKING warning events in the stream.
Nothing survives rerank	Scored as a product failure and kept in the averages. v1 adds a floor of three results per sub-query.	no_context in the eval; task 75 in the baseline.
Draft without citations	Output validator raises ModelRetry; three validator retries inside three orchestrator retries.	Zero citation markers in the FACT extraction.
Silent model fallback	Served model compared with the requested one on every agent result.	model_mismatch in errors.jsonl; never scored.
Client disconnects	The run keeps going as a background task; the client polls REST; stop cancels.	Firestore status and the result endpoint.
Judge flakes	Up to six attempts with jittered backoff, Retry-After honoured, explicit handling of 529s and dropped streams.	Typed JudgeError per case.
Grader drifts during tuning	Hash gate over graders, prompts, data and uv.lock.	Scripts exit with code 2 until re-approved.

12Stack

Stack

API

FastAPI
uvicorn
Pydantic v2
pydantic-settings
Python 3.12
uv

Agents

pydantic-ai
Claude Sonnet 5.5
Claude Haiku 5.5
OpenRouter

Planner, writer and refiner on Sonnet 5.5, summarizer on Haiku 5.5 in the eval baseline. Every role is configurable.

Retrieval

Serper
Cohere rerank-english-v3.0
Crawl4AI
Chromium
PyMuPDF
LangChain splitters
tiktoken

Streaming · state

WebSockets
Firestore

Frontend

Next.js 15
React 19
Tailwind
shadcn/ui
react-use-websocket
react-markdown

Nothing design system: Doto, Space Grotesk, Space Mono, one red accent.

Evals

Anthropic SDK
DeepResearch Bench
UMBRELA
RACE
FACT

Deploy

Docker
python:3.12-slim
non-root user

Engineering portfolio