RAG That Actually Works: Beyond the Naive Vector Search

Why most retrieval-augmented generation pipelines disappoint, and the hybrid retrieval architecture I reach for instead.

Naive RAG—embed everything, cosine-similarity search, stuff the top-k into a prompt—gets you a demo and not much more. The hard part is retrieval quality. I break down the stack that has consistently worked for me: hybrid search combining BM25 with dense embeddings, a reranker pass to fix the ordering, query rewriting so the user's messy question matches your clean chunks, and metadata filtering to scope the search. I also dig into chunking strategy, why chunk size is a product decision and not a technical one, and how to evaluate retrieval independently from generation so you know which half is failing.