Real systems are assembled
Useful search requires data inspection, indexing, retrieval, filtering, reranking, and output validation.
Benchmark · Search-system engineering
Search systems built by agents.
Evaluated by execution.
Search-SWE evaluates whether a coding agent can autonomously investigate an environment, construct a search workflow, debug it, and deliver a working system under a fixed budget.
Open-ended methodology · Reproducible execution · Hidden tests
01 · INPUT
04 · OUTPUT
Information task
corpus · sources · constraints
Coding agent
inspect · build · iterate
Executable search
reproducible · scored
→
→
fixed sandbox
same budget
same budget
↯Method agnosticNo prescribed retrieval recipe
▣Execution firstSystems are rebuilt and run
◌Resource awareTime, memory, and network count
✓Result groundedSearch quality meets hidden tests
01 · Motivation · Why
Search quality depends on more than a ranking formula.
Agents face open environments
The best solution is not always known in advance. Agents must discover packages, models, tools, and constraints.
Plans are not deliverables
A convincing explanation is not enough. The final artifact must build, run, and retrieve the right result.
THE QUESTION
Can a coding agent turn an information need into a working, reproducible search system?
02 · Search-SWE · What
A benchmark for search-system engineering agents.
Given a task, data or information sources, a sandbox, public examples, and hidden tests, the agent must deliver an executable search system.
Search-SWE treats every information need as an end-to-end software-engineering problem.
The benchmark does not pre-select BM25, dense retrieval, RAG, reranking, or an agentic workflow. It evaluates the result produced by the agent under the same execution and resource conditions.
CORE PRINCIPLE
Do not score the plan.
Score the executable result.
01ExploreDiscover the environment
→
02ConstructBuild the search system
→
03ValidateTest and iterate
→
04DeliverSubmit a reproducible artifact
03 · Task construction · How
Make the task open-ended for the agent, precise for the grader.
Information need
A realistic query, corpus, source collection, or multi-step information requirement.
Fixed sandbox
The same compute, storage, software, model, and network budget is available to every run.
Public and hidden tests
Examples support iteration while hidden queries measure whether the delivered system generalizes.
The homepage highlights six selected tasks. The complete catalog contains seven defined tasks, including Task 2-3.
04 · Tasks
Six representative Search-SWE tasks.
The homepage gives a quick view of six selected tasks across implementation, optimization, and repair. Select any card to open its detailed task brief.
TASK 1-1 IMPLEMENTATION
Reasoning query rewrite
Recognize when query rewriting is needed and turn noisy natural language into a retrieval-ready query.
query understanding →TASK 1-2 OPTIMIZATION
Memory-constrained retrieval
Search a 1.5M-vector corpus while balancing memory, index size, quality, and query latency.
large-scale dense search →TASK 2-1 OPTIMIZATION
Long-document reranking
Improve Cross-Encoder reranking when the relevant evidence lies beyond the first 512 tokens.
reranking →TASK 2-2 OPTIMIZATION
Embedder training
Improve a fixed-backbone code-retrieval embedder on supplied training data.
representation learning →TASK 3-1 REPAIR
Multilingual BM25 tokenizer
Repair the Pyserini BM25 pipeline so a Korean corpus is indexed and queried correctly.
retrieval debugging →TASK 3-2 REPAIR
Embedding inference repair
Fix the local FlagEmbedding integration so automatic model loading produces valid retrieval embeddings.
integration debugging →05 · Evaluation setting
Every submission has to survive four checks.
Build
Can the system build its index or required assets in a clean sandbox?
Function
Are inputs, outputs, IDs, ordering, and edge cases handled correctly?
Search quality
Does the returned Top-k list contain a verified positive result?
Resources
Does the run stay within the time, memory, disk, GPU, and network budget?
CLEAN-RUN SETTING
Same sandbox profile·Rebuild from submission·Hidden query evaluation
06 · Results
The leaderboard starts with a reproducible run.
Results will be added as public tasks and baseline submissions become available.
RESULTS BOARD● preparing
↗
Benchmark release in preparation
The first release will include task specifications, baseline systems, and reproducible evaluation reports.
Follow developmentSearch-SWE