Every project we’ve shipped in the last 18 months has had a retrieval layer in it. Some of them were called “RAG” in the proposal. Most of them weren’t. The acronym has become so overloaded that it stopped describing the actual work.
When a customer asks for “a RAG system,” what they usually want is one of three things: a search engine that returns explanations instead of links, a chatbot that grounds its answers in their documents, or an internal tool that lets a junior engineer ask a senior engineer’s question without waiting for a Slack reply. These are different products. They share an architecture, but the failure modes diverge fast.
Vector search alone is a trap
We still see teams ship pure-vector implementations. Embed everything, store in a vector DB, query by cosine similarity, send the top-k chunks to the LLM. It demos beautifully. It also fails on every query that contains a part number, a date, an exact phrase, or a negation.
Vector search is good at semantic similarity. It is not good at exact match. A user typing error code 4017 does not want documents that are “about errors” — they want the page that contains those four characters in that order. A keyword index handles that in milliseconds. Vectors do not.
The fix is hybrid retrieval: run BM25 (or Postgres full-text) and a vector query in parallel, then re-rank the merged set. We use Reciprocal Rank Fusion when we don’t have a budget for a re-ranker, and a cross-encoder when we do. The cost is one extra index and one extra query per request. The win is that the system stops looking stupid on the queries your users actually type.
Re-ranking is where the latency budget goes
If you only have 1.2 seconds end-to-end, you cannot afford to call a 400ms re-ranker on top of a 200ms retrieval call. We’ve learned to stage it: retrieve 50 candidates with a cheap query, re-rank only when the top-1 vector score is below a threshold (the model isn’t confident), and skip re-ranking entirely when the user’s query exactly matches a known intent.
The other latency thief is reading the documents themselves. If your chunks live in S3 and you fetch them on-demand, you’ve added 80ms per chunk. Cache the chunks in Redis or, even better, store them next to the index. The marginal storage cost is nothing compared to the latency tax.
Citations or it didn’t happen
Users will not trust an AI system that confidently states something they have no way to verify. Every answer needs a citation back to the source chunk, and every citation needs to be a real link the user can click. We’ve shipped systems where the citations were faked by the LLM (it knew the format, so it generated plausible URLs). The legal team noticed before the users did.
Build the citation handling outside the model. Pass each retrieved chunk a small numeric ID. Tell the model to use those IDs in its answer. After generation, parse the IDs and render the actual links from your retrieval result. The model never sees a URL, so it can’t hallucinate one.
Evals before launch, evals after
The difference between “works in demo” and “works in production” is an eval suite. Ours has three layers: deterministic retrieval tests (does query X return chunk Y in the top-3?), grounded-answer tests (does the model say what we expected, given the same chunks?), and end-to-end regression (full query, full answer, scored by a stronger model).
We run these in CI on every prompt change, and we re-run the full suite weekly against production traffic samples. The week we shipped Anthropic’s new Claude model, the eval suite caught a regression on negation handling that the demo would never have surfaced. We rolled back, fixed the prompt, shipped two days later.
What we ship with every implementation
After four production builds, the deliverable list has stabilized: a hybrid retriever (BM25 + vectors), a re-ranker invoked conditionally, a chunk cache, a citation parser, an eval harness with at least 50 starter tests, and a small dashboard that surfaces query latency, retrieval recall, and grounded-answer accuracy. The dashboard is what gets us paged when something drifts. Without it, you find out from a customer.
Call it RAG if it helps the procurement form. Internally, call it what it is: search that explains itself, with the receipts to prove it.