Why retrieval beats fine-tuning

A support engineer asks your app: "does the Pro plan include audit logs?" The model answers, confidently, that it does. It doesn't. Nobody wrote that sentence anywhere. The model produced a plausible continuation of the question, and plausible is all it was ever optimizing for.

The model is not broken. It never saw your pricing page. Everything it knows was frozen when its training finished, and your pricing changed last Tuesday.

There are two ways to close that gap. You can bake your knowledge into the model's weights (fine-tuning), or you can look the answer up at question time and hand it to the model as part of the prompt. The second approach is Retrieval-Augmented Generation, and for the overwhelming majority of production systems it is the right one.

The two ways to give a model your knowledge

Fine-tuning changes the model. You collect thousands of examples, run a training job, and get back a new set of weights that has absorbed your domain. Retrieval changes the prompt. You keep the model as it is, find the three most relevant paragraphs from your own corpus, and paste them above the user's question.

The trade-offs are not close for knowledge-heavy work:

The last two rows are the real dividing line. Fine-tuning teaches behavior; retrieval supplies facts. A model fine-tuned to always answer in your company's terse changelog voice is a good use of fine-tuning. A model fine-tuned to memorize your changelog is a bad one: you will publish another entry tomorrow.

There is also an accountability argument that matters more than either. A retrieved answer can carry its sources. When the model says the Pro plan has audit logs, it can point at the row it read that from, and a human can check. A fine-tuned model that says the same thing offers nothing but its own confidence.

What RAG is

Strip away the acronym and RAG is a search engine wired to the front of a prompt:

flowchart LR
  Q["User question"] --> R["Retriever"]
  R --> D[("Your corpus")]
  D --> C["Top-k chunks"]
  C --> P["Prompt: chunks + question"]
  P --> M["Claude"]
  M --> A["Answer with citations"]

Nothing in that diagram is novel. The retriever is search. The prompt is string concatenation. The model call is one HTTP request. RAG is an architecture rather than a technology, which is why it's worth studying as one. The hard parts are not the individual boxes; they are the decisions connecting them.

Two pipelines, not one

The single most useful thing to internalize early: a RAG system is two pipelines that run at different times and fail in different ways.

flowchart TB
  subgraph Index["Index time: slow, batched, offline"]
    S["Source documents"] --> CH["Chunk"]
    CH --> EM["Embed"]
    EM --> ST[("Vector store")]
  end
  subgraph Query["Query time: fast, per-request, online"]
    U["User question"] --> QE["Embed question"]
    QE --> SE["Search"]
    ST -.-> SE
    SE --> RR["Rerank"]
    RR --> GEN["Generate"]
  end

Index time runs when documents change. It is allowed to be slow: it is a background job. It is measured in throughput and cost, and its failures are silent: a document that fails to index is never found again, and no user sees an error.

Query time runs on every question. It has a latency budget measured in hundreds of milliseconds, and its failures are loud: a slow retriever is a slow product.

Conflating these two is the most common architectural mistake in a first RAG build. Embedding a document inside the request that created it makes uploads slow and couples your write path to a third-party API's uptime. Embedding is a job.

Where this fits in Rails

Every box above maps onto something you already know:

• Chunking and embedding: an ActiveJob, triggered when a source record changes.
• The vector store: a Postgres table with a vector column, the same database, the same transactions, the same backups.
• Retrieval: a query object, the way Enrollment::UserGamificationQuery is a query object in this app.
• Generation: a service object that takes the retrieved context and the question and returns a result.

There is no new infrastructure tier here unless you choose to add one. That is the whole argument for starting with Postgres, and it is the argument this course will keep making.

What this course covers

1. Why retrieval beats fine-tuning: you are here.
2. Indexing: chunking, and what a chunk must carry with it.
3. Embeddings as an architectural choice: dimensions, cost, and lock-in.
4. Storage and retrieval: vectors in the database you already have.
5. Hybrid retrieval and reranking: why pure vector search is not enough.
6. Generation: assembling a prompt that can be held to account.
7. Putting it together: the pipeline as Rails objects.

By the end you will not have memorized an SDK. You will be able to look at a RAG system and say which of its seven decisions were made badly, starting with the one almost everyone gets wrong, which is chunking, and which is where we go next.