Your RAG system doesn't need a bigger model.
It needs better retrieval.

A team ships a retrieval-augmented chatbot. It answers well in the demo, then starts making things up in production. The instinct, every time, is to reach for a bigger, smarter model. Here's the part that surprises almost everyone who tries it: that upgrade almost never fixes it. So what actually does?

01  ·  The reflex is wrong

When a RAG answer is bad, one of two things happened: the model reasoned poorly over good context, or it reasoned fine over the wrong context. In practice it's overwhelmingly the second. The model never saw the paragraph that held the answer, so no amount of extra parameters will save it.

Upgrading the model in that situation is paying more to reason more confidently over the same missing information.

02  ·  What “better retrieval” means

Retrieval quality is the share of the time your top-k chunks actually contain what's needed to answer. It has almost nothing to do with the generator and everything to do with chunking, embeddings, and ranking. Get recall@k (the percentage of the time your retrieved chunks actually contain the answer) up and most hallucinations quietly disappear.

$ eval retrieval --set golden.jsonl
recall@5 ............ 0.61   too low
recall@20 ........... 0.88
mrr ................. 0.44
# answers fail where recall@5 misses. fix retrieval, not the model.

03  ·  Measure retrieval before you touch anything

You can't fix what you don't score. Build a small golden set (50 to 200 real questions with the passages that answer them) and measure recall@k and MRR (mean reciprocal rank — how high up the correct answer lands, not just whether it's found) against it. This is a day of work and it reframes every argument that follows.

If recall@20 is high but recall@5 is low, your ranking is the problem, not your index. That distinction alone saves weeks.

04  ·  The four fixes, in order

1. 01Chunk on structure, not length. Split on headings and semantic boundaries so a chunk is a complete thought, not 512 arbitrary tokens.
2. 02Add a reranker. A cross-encoder (a model that re-scores your top candidates for relevance) over the top 50 candidates fixes ordering cheaply, usually the single biggest jump in recall@5.
3. 03Go hybrid. Combine dense vectors (search based on meaning, not exact words) with keyword search. Exact IDs, error codes and names are where pure embeddings quietly fail.
4. 04Then, maybe, the model. Only once retrieval is measured and strong does the generator become the bottleneck worth paying for.

05  ·  When you actually need the bigger model

There is a real case for it: when retrieval is strong (high recall@5, clean context in the window) and the answers still miss because the task needs genuine multi-step reasoning or synthesis across many sources. That's a reasoning ceiling, and a stronger model raises it. But you only know you've hit it once retrieval is off the table as a suspect.

So the order is fixed: measure retrieval, fix retrieval, then, and only then, reach for the model. It's cheaper, it's faster, and nine times in ten it's the whole answer.