Search-SWE

04 · Task catalog

The Search-SWE task set.

Seven defined tasks span implementation, optimization, and repair. Each task asks an agent to produce a working change under a concrete search-system scenario.

07 defined tasks·03 task types·Clean-run evaluation

Browse the tasks

Three task types.
One executable standard.

Implementation tasks ask the agent to create a capability. Optimization tasks ask it to improve a working system under a fixed budget. Repair tasks ask it to find and fix a broken integration or pipeline.

IMPLEMENTATIONBuild a missing capability OPTIMIZATIONImprove quality or efficiency REPAIRRestore correctness
TASK 1-1 · IMPLEMENTATION↻

Reasoning query rewrite

Recognize when query rewriting is necessary and design an effective reasoning-query strategy.

Open task brief → TASK 1-2 · OPTIMIZATION◌

Memory-constrained retrieval

Search a 1.5M-vector corpus under strict memory and query-latency constraints.

Open task brief → TASK 2-1 · OPTIMIZATION↗

Long-document reranking

Improve Cross-Encoder reranking when relevant evidence lies beyond the first 512 tokens.

Open task brief → TASK 2-2 · OPTIMIZATION◆

Embedder training

Improve a fixed-backbone code-retrieval embedder using supplied training data.

Open task brief → TASK 2-3 · OPTIMIZATION⌘

Query-side encoder

Optimize a query encoder against fixed document vectors and a runtime gate.

Open task brief → TASK 3-1 · REPAIR⌁

Multilingual BM25 tokenizer

Repair the Pyserini BM25 pipeline for a Korean retrieval corpus.

Open task brief → TASK 3-2 · REPAIR✓

Embedding inference

Fix FlagEmbedding automatic loading and produce correct normalized embeddings.

Open task brief →

Task briefs

What each agent has to solve.

01
TASK 1-1 · IMPLEMENTATION · IN PREPARATION

Reasoning query rewrite for professional search

Build a query-rewriting capability for reasoning-intensive information needs, where the user's natural-language description differs from the terminology used by relevant documents.

ScenarioBRIGHT Biology
Core skillQuery understanding
Expected directionReasoning query
StatusTask under construction
Agent challenge

Identify that adding query rewrite is necessary, extract the real information need from long background and follow-up questions, and design a useful rewrite strategy.

Expected implementation

Use a reasoning-query rewrite approach: remove noise, extract the core need, add professional terminology, and expand useful synonyms before retrieval.

Why this scenario matters: Biology, medicine, law, and research queries often describe phenomena in everyday language while the corpus uses precise professional concepts.
Back to top ↑
02
TASK 1-2 · OPTIMIZATION · DENSE RETRIEVAL

Memory-constrained retrieval over a huge corpus

Build and optimize a dense-vector search system when the complete corpus is larger than the available memory.

Corpus1.5M vectors
Vector shape1024-D float32
Memory budget2 GiB
MetricAccuracy@3
Agent challenge

Choose an index and storage strategy while balancing recall, index size, memory use, disk access, and query latency.

Expected implementation

Use the supplied precomputed vectors with techniques such as ANN, quantization, memory mapping, chunking, candidate retrieval, and exact reranking.

Back to top ↑
03
TASK 2-1 · OPTIMIZATION · CROSS-ENCODER

Improve long-document reranking

Optimize reranking when relevant evidence may appear beyond the first 512 model tokens of a long document.

CorpusNarrativeQA
CandidatesFixed BM25 Top-100
ModelBGE reranker-large
MetricAccuracy@5
Agent challenge

Improve the reranking pipeline without changing the supplied candidate set or replacing, training, or modifying the fixed cross-encoder.

Expected implementation

Make the fixed model see more useful long-document evidence through efficient preprocessing, text views, window selection, batching, caching, and score aggregation.

Back to top ↑
04
TASK 2-2 · OPTIMIZATION · EMBEDDER TRAINING

Improve a code-retrieval embedder

Improve the retrieval quality of a fixed-backbone code embedder using the supplied corpus and training data.

BackboneBGE-base-en-v1.5
DomainCode retrieval
TargetHeld-out Accuracy@1
ConstraintPreserve architecture
Agent challenge

Use the visible training data to improve representations while preserving parameter names, shapes, count, and the supplied backbone architecture.

Expected implementation

Train and validate a self-contained checkpoint that can be loaded locally by the verifier without hidden data, answer mappings, or network access.

Back to top ↑
05
TASK 2-3 · OPTIMIZATION · QUERY ENCODER

Optimize the query-side encoder

Build a query-side retrieval system based on a small backbone while the document vectors remain fixed.

Query modelQwen3-Embedding-0.6B
DocumentsFixed vectors
Corpus8,674 documents
MetricAccuracy@1
Agent challenge

Improve query representations without re-encoding, modifying, or reordering the supplied document vectors, while meeting the runtime gate.

Expected implementation

Train or adapt the query-side system using the supplied backbone and visible data, then submit a deterministic, self-contained executable entry point.

Back to top ↑
06
TASK 3-1 · REPAIR · BM25 / PYSERINI

Repair multilingual BM25 tokenization

Repair a Pyserini BM25 retrieval pipeline so that a Korean corpus is indexed and queried correctly.

RuntimePyserini / Lucene
CorpusKorean retrieval
Core issueTokenizer / indexing
MetricAccuracy@1
Agent challenge

Locate the pipeline or analyzer behavior that causes incorrect multilingual retrieval and repair it without query-specific answer rules.

Expected implementation

Build a fresh index during verification and use the installed Pyserini/Lucene BM25 path for deterministic retrieval.

Back to top ↑
07
TASK 3-2 · REPAIR · EMBEDDING INFERENCE

Repair embedding-model inference integration

Fix a supplied FlagEmbedding inference integration so automatic model loading produces correct retrieval embeddings.

SourceFlagEmbedding
Entry pointAutomatic loading
OutputFinite normalized vectors
MetricHidden retrieval checks
Agent challenge

Find the local integration error without replacing the installed package, bypassing the source, or modifying model and data files.

Expected implementation

Repair the source-level model selection and pooling path so batch consistency, input-length handling, prefix behavior, and retrieval ranking all work.

Back to top ↑

Search-SWE

Every task ends with an executable artifact.

See the evaluation setting