LLMs Are Frozen. Your Data Isn't.
Every large language model has the same fundamental limitation: it knows everything about the world up to its training cutoff date and absolutely nothing about what happened five minutes ago. It doesn't know about your internal wikis, your proprietary codebase, your customer data. If you want a model to work with any of that, you have to solve the context injection problem. How do you get the right data into the model at the right time?
For years there was basically one answer: RAG. Retrieval-augmented generation. You chunk your documents, embed them into vectors, store those vectors in a database, and at query time you run a semantic search to pull the most relevant chunks into the prompt. It works. I've built production RAG systems on AWS Bedrock that handle real traffic, and the patterns hold up.
But the landscape shifted under our feet. In early 2024, context windows were typically 4K to 32K tokens. By mid-2024, Claude hit 200K. By 2025, Gemini pushed past 1 million. A million tokens is roughly 700,000 words. You could fit the entire Lord of the Rings trilogy into the prompt and still have room for The Hobbit.
That kind of jump doesn't just change the numbers. It changes the architecture decision.
The Before: Why RAG Was the Only Option
When context windows were 4K tokens, you couldn't fit a single chapter of a technical manual into the prompt. There was no discussion to be had. You had to build a retrieval layer. The entire RAG ecosystem, embedding models, vector databases, chunking strategies, rerankers, all of it grew up around a hard constraint: the model can only see a small amount of text at a time, so you'd better make sure it's the right text.
I built a production RAG system on Bedrock following exactly this pattern. bedrock-rag-patterns uses hybrid search with Reciprocal Rank Fusion, Claude-based re-ranking, hallucination detection, and Bedrock Guardrails. Five composable classes, one pipeline. The architecture works, and for a lot of use cases it's still the right call. But the reason it exists, the reason any of that machinery exists, is because context windows used to be small.
The entire RAG ecosystem grew up around a hard constraint: models could only see a tiny slice of your data at once. That constraint is dissolving.
The After: Long Context as the No-Stack Stack
Long context is the brute force approach. Skip the database. Skip the embedding model. Take your documents and put them straight into the context window. Let the model's attention mechanism do the heavy lifting of finding the answer.
There are three specific reasons this approach keeps winning in cases where it was never viable before.
1. You collapse the infrastructure
A production RAG system is heavy. You need a chunking strategy (fixed-size? sliding window? recursive?). You need an embedding model to encode the data. A vector database to store it. A reranker to sort the results. A synchronization pipeline to keep vectors in sync with source data. That's a lot of moving parts and a lot of places for things to break.
Long context removes all of it. No database, no embeddings, no retrieval logic. The architecture simplifies down to: get the data, send it to the model. I've started calling this the "no-stack stack" because there's genuinely nothing to operate.
2. You eliminate silent failure
RAG introduces a critical point of failure at the retrieval step itself. When a user asks a question, the system looks at mathematical representations of the data stored as vectors, big arrays of floating-point numbers, and tries to find the closest semantic match. But semantic search is probabilistic. The retrieval might fail to surface the relevant document for all kinds of reasons: the query phrasing doesn't align with the chunk phrasing, the embedding model doesn't capture domain-specific semantics well, the chunk boundaries split the answer across two pieces.
This is called silent failure. The answer existed in your data, but the model never saw it because retrieval didn't return the right results. The user gets a confidently wrong answer and has no idea the system failed at the search step, not the reasoning step.
With long context, there is no retrieval step. The model sees everything.
3. You can reason about what's missing
This is the one that gets underestimated. RAG is fundamentally designed to retrieve what exists. It finds a semantic match between your query and a specific snippet in the database. But what if the answer lies in what's not in the database?
Say you have a product requirements document and a set of release notes, and someone asks: "which security requirements were omitted from the final release?" RAG will search for chunks about security requirements. It'll find snippets from the requirements doc and snippets from the release notes. But it can't retrieve the gap between them. It can only show the model isolated pieces, and the model never sees the full picture needed to spot the missing pieces.
Long context solves this by putting both documents in full into the context window. The model can do the comparison directly. It can find what's present in one document and absent from the other. This kind of cross-document gap analysis is genuinely impossible with chunked retrieval.
RAG Is Not Dead
If long context is so good, why did I still build a production RAG library? Because there are three problems where long context falls apart, and they're common ones.
The rereading tax
Consider a 500-page technical manual. That's roughly 250K tokens. With long context, you're sending those 250K tokens to the model on every single query. Every user question pays the full processing cost for that manual. Every time.
RAG pays that cost once, at indexing time. The embedding model processes the manual when you ingest it, and after that each query only processes the retrieved chunks. Prompt caching can offset some of this for static content, but for data that changes frequently you're stuck paying the full token tax on every request. At scale, the cost difference is enormous.
The needle problem gets worse, not better
There's an intuitive assumption that if data is in the context window, the model will use it. Research says otherwise. As context windows grow past a few hundred thousand tokens, the model's attention mechanism gets diluted. If you ask a specific question about a single paragraph buried in the middle of a 2,000-page document, the model often fails to locate it or hallucinates details from surrounding text.
RAG actually helps here. By retrieving only the top five or ten relevant chunks, you've removed the haystack and presented the model with just the needles. Less noise, more signal. For precise factual retrieval from large corpora, RAG's filtering step is an advantage, not overhead.
Enterprise data doesn't fit
A million tokens sounds like a lot until you measure it against enterprise data. A corporate data lake is measured in terabytes, sometimes petabytes. No context window on earth is going to hold that. If you want a model to search across your entire knowledge base, across years of Jira tickets, Confluence pages, Slack threads, code repositories, customer support logs, you need a retrieval layer to filter information down to something that fits.
The vector database isn't going anywhere for this use case. It's the only viable warehouse for data at that scale.
The Decision Framework
After building systems on both sides of this, here's how I think about the choice:
| Factor | Long Context Wins | RAG Wins |
|---|---|---|
| Data volume | Bounded, fits in context window (under ~500K tokens) | Large corpus, terabytes+ |
| Query type | Summarization, cross-document reasoning, gap analysis | Precise factual retrieval from large data |
| Infrastructure tolerance | You want zero moving parts | You can operate a vector DB and embedding pipeline |
| Query volume | Low to moderate (rereading cost is acceptable) | High volume (indexing cost amortizes over many queries) |
| Data freshness | Static or slow-changing documents | Frequently updated content |
| Failure mode priority | You can't tolerate silent retrieval failures | You can't tolerate attention dilution on large docs |
What Actually Changed
The shift isn't that RAG is dead or that long context replaces it. The shift is that long context went from "not a real option" to "the default choice for bounded datasets." Before 2024, if someone asked me how to give an LLM access to their company's documentation, the answer was always RAG. There was no alternative. Now the first question is: how much data are we talking about?
If the answer is "a few hundred pages of product docs," you probably don't need a vector database. You don't need an embedding model. You don't need a chunking strategy. You just need a prompt.
If the answer is "our entire knowledge base spanning five years of operations," you need RAG. And you need it done well, with hybrid search, re-ranking, and hallucination detection, because the retrieval step is now the single biggest source of quality problems in your system.
Both approaches solve the same problem: getting the right data in front of the model. The difference is where the complexity lives. With RAG, it's in the retrieval infrastructure. With long context, it's in the model's attention mechanism. The question isn't which one is better. It's which one is appropriate for your data volume, your query patterns, and the failure modes you can tolerate.
I've open-sourced the RAG side of this at github.com/ivandir/bedrock-rag-patterns. If your problem actually needs retrieval, those patterns are production-tested. If it doesn't, save yourself the infrastructure and just use the context window. That's the real lesson of 2024: you finally have the option not to build.