BEIR
BeirBenchmark integrates official preprocessed BEIR retrieval datasets.
BEIR evaluates retrieval quality: recall, ranking, and cross-domain robustness. It does not require PDF parsing, OCR, or answer generation.
Data Layout
corpus.jsonl
document collection with _id, title, text
queries.jsonl
query collection with _id, text
qrels/{split}.tsv
query_id, document_id, relevance
Recommended subsets:
| dataset | Use |
|---|---|
scifact |
Small, stable scientific fact retrieval; good smoke test. |
nfcorpus |
Medical/biomedical retrieval. |
fiqa |
Financial QA retrieval. |
hotpotqa |
Retrieval task derived from multi-hop QA. |
Usage
from heta_framework.evaluation import BenchmarkRunner, BeirBenchmark
benchmark = BeirBenchmark(dataset="scifact")
result = await BenchmarkRunner().run(
benchmark=benchmark,
recipe=recipe,
knowledge_base_name="beir_scifact_vector_v1",
query_modes=("vector_search",),
)
By default, Heta downloads from:
Use local data:
data_root should contain:
Mapping
BEIR labels are document-level, while Heta results are usually chunk-level. The adapter writes each corpus item as one text document:
Evaluators map chunk hits back to benchmark document ids and deduplicate by document before computing metrics.
Default Evaluators
BeirBenchmark uses standard IR metrics:
beir_ndcg@1 / @3 / @5 / @10 / @100
beir_map@1 / @3 / @5 / @10 / @100
beir_recall@1 / @3 / @5 / @10 / @100
beir_precision@1 / @3 / @5 / @10 / @100
beir_mrr@1 / @3 / @5 / @10 / @100
For a lighter run:
from heta_framework.evaluation import BeirRetrievalMetric
evaluators=(
BeirRetrievalMetric(metric="ndcg", k=10),
BeirRetrievalMetric(metric="recall", k=10),
)
Scope
BEIR evaluates retrieval quality only. Use UDA-Benchmark or MultiHop-RAG when you need answer quality or multi-hop evidence evaluation.