RAG benchmarks: accuracy, latency, hallucination
A RAG system is only as good as the sources it finds, and only as trustworthy as its discipline not to invent. This page describes how we measure both over an OKF knowledge base – the setup, the metrics, the traps. It deliberately names no numbers: dependable values come only from a defined run against a defined bundle, and we add those here once they exist.
Placeholder. The result tables below are empty until the first measurement run is complete. No value on this page is estimated or illustrative.
What is measured
The Hub's assistant works in two steps: it retrieves relevant concepts from the knowledge base, then lets a model answer from them. Each step has its own failure modes, so we measure them separately.
The test setup
The basis is a fixed bundle and a list of test questions with a known answer that is supported in the bundle. Each question is paired with the concept or concepts that actually support the answer – the reference against which we check retrieval and response.
The real retrieval path being measured has three fixed properties. First the server attempts semantic search with locally computed multilingual embeddings (model multilingual-e5-small, cosine similarity, top 6). If the vector index is not built yet, it falls back to field-weighted keyword search: a title hit counts sixfold, a description hit fourfold, a tag hit threefold, a body hit once. Of the concepts found, the best five go to the model as context, up to 800 characters of body each.
Generation runs at low temperature (0.2) with a 1000-token cap and a system instruction that restricts answers strictly to the supplied sources and requires openly admitting missing evidence.
Metric 1: retrieval hit rate
Does search find the right concept at all? Measured as the share of questions where the reference concept is among the top-k (recall@k, with k equal to the number of delivered sources). This metric is the ceiling for everything else: what retrieval misses, the best generation cannot use.
| Retrieval-Modus | Recall@5 | Median-Rang des Referenzkonzepts |
|---|---|---|
| Semantisch (e5-small) | (nachzutragen) | (nachzutragen) |
| Stichwort-Fallback (TF-IDF) | (nachzutragen) | (nachzutragen) |
The direct comparison of both modes is the interesting part: it shows how much semantic search adds over the fallback – and on which question types the fallback keeps up surprisingly well.
Metric 2: faithfulness
Does every statement in the answer rest on a cited concept? Measured as the share of statements with a correct source number that genuinely supports the statement. The Hub lists only sources the model actually cited by number – that eases the check but does not replace it.
Metric 3: hallucination rate
The counter-test to faithfulness: the share of answers with at least one statement no supplied source supports. One special case counts on its own – the invented URL or brand. The system instruction explicitly forbids any external URL; a violation is a hard hallucination case.
Metric 4: refusal in the right place
A good RAG system says "not covered" when the sources are silent. So we mix in questions whose answer is not in the bundle. We measure both: does the assistant correctly refuse there, and does it answer supportable questions instead of refusing too early?
| Fragetyp | Korrekt beantwortet | Korrekt verweigert |
|---|---|---|
| Im Bundle belegt | (nachzutragen) | – |
| Nicht im Bundle | – | (nachzutragen) |
Metric 5: latency
Time from question to answer, split by two states. The first call after a restart loads the embedding model and is naturally slower; the Hub warms the model at startup for that reason. Afterward the chosen language model's response time dominates. We measure retrieval time and generation time separately, because only the first is in our hands.
| Phase | Kalt (nach Neustart) | Warm |
|---|---|---|
| Retrieval | (nachzutragen) | (nachzutragen) |
| Generierung | (nachzutragen) | (nachzutragen) |
What the numbers won't say
Three limits remain, and we state them. Results depend on the test set – a bundle on technical SEO says little about one on contract law. They depend on the chosen model, which can change at any time. And faithfulness measures whether a statement is supported, not whether the source itself is correct: a cleanly cited but factually wrong source text produces a faithful answer on a false basis. So human review of important answers stays part of the procedure, not its replacement.
Continue
How OKF compares to other formats for AI context is covered in the format comparison. The real retrieval and answer path measured here is described under prompt templates and in the RAG assistant guide.
Testpfad nach dem realen Retrieval in lib/semantic.js und server.js dieses Projekts (e5-small, Top-6, feldgewichtetes TF-IDF, Temperatur 0,2). Messwerte folgen. / Test path per this project's real retrieval in lib/semantic.js and server.js. Measured values to follow.