Or: the smartest system I ever shipped could not tell you what month it is.

The stories below are composites. Products, customers and companies have been shuffled across many years of shipping software. If one of them sounds like your company, it probably is.

The Reader

The customer had a knowledge base. A few hundred FAQ documents about the same twelve products, written by different people, in different years, with different opinions about what the products do. And they had a request that sounded simple: "Make it answer questions."

So we built the thing everyone was building that year. Cut the documents into pieces. Turn every piece into a vector. When a question comes in, find the pieces that look most like the question, give them to a language model, and ask it nicely to answer. Demo day. It answered. Correctly, even. Applause.

What nobody says in the demo: the system did not read anything. It never will. It has no idea what the documents say. It knows which paragraph is shaped most like your question. That's it. It is a bouncer with a photo, not a reader. We used to call this a search engine. Now it has a demo budget.

The customer, quite reasonably, did not care about the difference. They saw it answer three questions in a row and decided they had hired a very fast employee who had read everything. That was the mental model. That mental model was going to cost us about six months.

Because a fast employee who has read everything knows what day it is.

Bouncer With a Photo

Before the story gets worse, two things about why the bouncer is bad at his job.

The vector is fragile. Same question, one word capitalised, a question mark at the end, and the vector moves just enough. The paragraph that ranked first now ranks third and third doesn't make the cut. So you put keyword search next to the vectors. The industry calls this "hybrid", which is marketing for "we didn't trust either of them". It helps, as long as the question and the answer happen to share words. When they don't, you're back to the photo.

And a paragraph is not a unit of meaning. Paragraph 14 says "this option must never be enabled for these customers." Which option? Which customers? Paragraph 12 knows. Paragraph 14 was embedded alone and it is a vector about nothing in particular. Anthropic wrote this up properly. Their fix: before you embed a chunk, have a model write a short note on top of it, "this is from document X, section Y, it's about Z", and embed that. They report a serious drop in retrieval failures and I have no reason to doubt them. But read what the technique actually is. You are paying a language model, at indexing time, to reconstruct the context you destroyed at chunking time. Every time a document changes, you pay again.

Fine. We paid. The bouncer got a better photo. He still couldn't read.

Black Friday, Forever

The customer ran a promotion. One week, one product, a better rate than usual. They wrote a document about it. Clear, well written, dates in the first line. They uploaded it. And they told us, problem solved: "Now it knows."

For one week, it did. Every question about that product came back with the promotion, because a document about a promotion is the best possible match for a question about the product. Nearest neighbour, working as designed.

Then the week ended. The system kept quoting the promotion. In December. In January. To everyone who asked.

The dates were in the document. The system does not read the document. It does not know what a date is. A paragraph about October 30 is a great match in November, a great match in April, and, this is the part that took a while to explain to the room, a better match than the boring permanent document about the normal rate. The promotion document is more specific. More enthusiastic. More shaped like the question. Retrieval does not care about time. Its only job is to find what looks relevant. Whether it is still true is somebody else's job. We had not hired anybody for that job.

There is now a whole research thread on this. Google published a benchmark of what they call knowledge conflicts, two retrieved sources that disagree, including the boring case where one of them is just old. Meta published work on rerankers showing they will happily promote the more informative but obsolete document and let the model answer confidently and wrong. Both are trying to fix it at query time, teach the ranking step what "current" means. Good luck with that. We fixed it the peasant way. Nothing enters the knowledge base without an expiry date, and there is a job whose only purpose in life is to take expired documents out again.

Nobody budgets for that job. I have been doing this for many years and every project has a pipeline that ingests documents. Somebody wrote it in week one. It has tests. It has a dashboard. I have almost never seen a pipeline that forgets documents. Ingestion gets a birth certificate. Nobody prints the death certificate.

Leonard in Memento runs the same architecture. Every note he writes is true at the moment he writes it. He has no way to find out it stopped being true. So he keeps acting on a Polaroid from three days ago, with total confidence, because the Polaroid is the most relevant thing in his pocket.

What Account Is "It"

The second failure was quieter, and it's the one that finally got the customer to stop calling it "the reader".

User asks: "What is the interest rate on savings?" System finds the rates document. Correct. User asks: "Which account do I need to open to get that?" System finds an account-opening document. For a different product. Because the question it saw contained "account" and "open" and "that", and "that" is not a word the bouncer can do anything with. Third question: "How do I open it?" By now the query is nearly empty. "Open" and "it". You get whatever ranks highest for the verb.

Every question arrives alone, gets answered alone, gets forgotten. The dialog history, if you feed it at all, is used for one thing: rewriting the current question into something that can stand on its own before you search. That works, until the rewrite guesses the wrong "it". At which point you are no longer vaguely searching for nothing. You are confidently searching for the wrong thing. Progress.

And even when the rewrite is right, the user's real question, "what's the best rate and how do I get it", is not in one document. It is a path. Rates, then the product with that rate, then the procedure for that product. Top-k similarity returns the three documents most like the question. That is not the three documents that, together, answer it. Microsoft's GraphRAG is the well-known attempt at this: build an actual graph out of the documents and walk it. It works. It costs a model call per chunk at indexing time, and your graph is now exactly as stale as your documents. Which means as stale as the pipeline that forgets. Which you didn't build. See above.

The Bill

Here is the ladder as I ended up seeing it. Every rung fixes the failure of the rung below by paying more, earlier.

Paste everything into the model. Works for the demo. Then the context fills up, then every question costs as much as your whole knowledge base, then the model starts losing things in the middle anyway. There's a Stanford paper on that, and Google has a paper arguing it's fixed for simple factoid questions. Yours are not simple factoid questions.

So you chunk and embed. Cheaper. Now the paragraphs are orphans. So you pay a model to write context on every chunk. Better. Now every document update is an indexing bill and the trick only sees inside one document. So you build a graph. Now the chain works, the indexing bill went up again, and the graph rots the same way the documents did.

Nowhere on that ladder did anybody fix the actual problem. The corpus is a pile of overlapping documents, written by different people in different years, some of which stopped being true, and nothing in the pipeline knows which ones. Retrieval got smarter at every step. The documents stayed exactly as dumb as the day they were uploaded.

Between Documents

Every failure in this story lived in the same place. Not inside a document. Between them. Paragraph 12 and paragraph 14. The promotion and the day it ended. "Which account", and then "how do I open it", three messages later, with nothing in between that remembers. The tooling is magnificent at looking inside things and nearly blind to the space between them.

The customer got their reader in the end. It is a stateless matcher with a query rewriter in front of it, an expiry date on every document, a job that takes the dead ones out, and a small boring table for the handful of questions that are actually paths. None of that goes on a conference slide. All of it is what made the thing stop lying.

The machine never forgets. That was the bug.