Evaluators

Evaluators are the scoring methods used by benchmarks.

Heta Evaluation follows this rule:

Benchmark owns scoring policy.
Common evaluators are reusable building blocks.

Benchmark adapters decide the default scoring policy. Heta provides common evaluators so every benchmark does not need to reimplement recall, exact match, or answer contains.

Protocol

class BenchmarkEvaluatorProtocol(Protocol):
    @property
    def name(self) -> str: ...

    async def evaluate(
        self,
        *,
        case: BenchmarkCase,
        response: QueryResponse,
    ) -> EvaluationScore: ...

Input:

BenchmarkCase
    benchmark query, expected answers, and evidence labels

QueryResponse
    standard output from kb.query(...)

Output:

EvaluationScore(
    name="evidence_recall@5",
    value=0.8,
    passed=None,
    metadata={"matched": 4, "expected": 5},
)

value can be float, bool, or str. metadata carries matched evidence, missing evidence, judge reasons, or other debugging information.

EvidenceRecallAtK

from heta_framework.evaluation import EvidenceRecallAtK

EvidenceRecallAtK(k=5)

Compares BenchmarkCase.expected.evidence against QueryResponse.results[:k].

Matching order:

1. locator match
2. reference_id match
3. text match

Built-in locator matching recognizes fields such as document_id, source_key, object_key, page_index, and chunk_id.

BeirRetrievalMetric

from heta_framework.evaluation import BeirRetrievalMetric

BeirRetrievalMetric(metric="ndcg", k=10)
BeirRetrievalMetric(metric="recall", k=10)

Supported metrics:

ndcg
map
recall
precision
mrr

BEIR qrels are document-level, while Heta results are usually chunk-level. The evaluator maps chunk hits back to benchmark document ids, deduplicates by document, and then computes the metric.

Default BEIR metrics:

from heta_framework.evaluation import beir_default_metrics

evaluators = beir_default_metrics(
    k_values=(1, 3, 5, 10, 100),
)

AnswerContains

from heta_framework.evaluation import AnswerContains

AnswerContains()

Checks whether QueryResponse.answer contains any case.expected.answers. Good for loose QA evaluation.

AnswerExactMatch

from heta_framework.evaluation import AnswerExactMatch

AnswerExactMatch()

Checks whether normalized QueryResponse.answer exactly equals any expected answer. Good for short answers, enums, and classification labels.

Custom Evaluator

class MyEvaluator:
    name = "my_score"

    async def evaluate(self, *, case, response):
        return EvaluationScore(
            name=self.name,
            value=1.0,
        )

Use it in a benchmark:

def evaluators(self):
    return (MyEvaluator(),)

or override a runner call:

BenchmarkRunner().run(
    benchmark=benchmark,
    recipe=recipe,
    knowledge_base_name="kb",
    query_modes=("vector_search",),
    evaluators=(MyEvaluator(),),
)