"Because it improves search quality."
That's technically true.
But it misses the real problem.
The real issue is the 𝗧𝘄𝗼-𝗧𝗼𝘄𝗲𝗿 𝗕𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸.
In semantic search, the embedding model creates:
• One vector for the query • One vector for the document
They never truly interact.
The system simply compares vectors using cosine similarity or dot product.
You're ranking documents without actually "reading" them together.
Here's the failure mode 👇
Query:
"How do I prevent heart attacks?"
Document:
"Heart attacks kill millions every year."
High semantic similarity?
Yes.
Relevant answer?
No.
One is asking for prevention.
The other is just a statistic.
Semantic similarity does NOT guarantee relevance.
This is where rerankers change everything.
𝗕𝗶-𝗲𝗻𝗰𝗼𝗱𝗲𝗿 (vector search)→ encode(query)
→ encode(doc)
→ compare vectors
Fast.
Scalable.
But shallow.
𝗖𝗿𝗼𝘀𝘀-𝗲𝗻𝗰𝗼𝗱𝗲𝗿 (reranker)→ encode([query, SEP, doc])
Now the model sees:
• Word interactions • Context alignment • Whether the document actually answers the query • Token-level relationships
That [SEP] token is critical.
For the first time, the query and document are processed together instead of independently.
The tradeoff?It's expensive.
Every candidate document requires a full transformer forward pass at query time.
That's why production RAG systems use a 2-stage retrieval pipeline 👇
𝗦𝘁𝗮𝗴𝗲 1:10M docs → Top 100
Fast vector retrieval
𝗦𝘁𝗮𝗴𝗲 2:Top 100 → Top 10
Cross-encoder reranking
Fast recall first.
Deep precision second.
And the difference is massive:
→ Without reranking: ~60% precision → With reranking: ~85% precision
That 25% gap determines whether your RAG system feels intelligent or unreliable.
Rerankers are not optional in production-grade RAG.
They are the precision multiplier.
