Prompting, RAG or fine-tuning? Choosing how to adapt a language model

Your AI assistant is nearly useful: it answers in generalities instead of quoting your policies, drafts replies in the wrong tone, or cannot say where an order is. A common reflex is to ask for a model trained on your own data, often the slowest, costliest option, and one that may not fix the problem. This guide compares three ways to adapt a language model (an AI system that reads and writes text): better prompts, retrieval and fine-tuning. It also covers tools, and how to choose.

Prompting: always the starting point

A prompt is everything you send the model with each request: instructions, background and the user's question. Improving it is the fastest, cheapest change: nothing is trained, and a new version can be tested at once and easily undone.

Weak prompts often lack one of three things:

• Instructions: who the model works for, what it should and should not do, and what to do when unsure, such as handing over to a person.
• Examples: a few samples of good inputs and outputs, which often show what you mean better than a description.
• Output format: a fixed structure, such as named fields, that people can scan and software can read. Many providers can enforce a structure directly (structured outputs), though the content still needs checking.

Prompting has limits. The model knows only what it learned in training plus what is in the prompt, and the most text it can consider at once, including its reply, is its context window. Windows are now large, but answers can become less reliable when the key detail is buried in a long prompt. Long prompts add cost and delay to every request, though prompt caching, which many providers offer, makes resending a fixed block of text cheaper. Long instruction lists may be followed inconsistently.

Retrieval: giving the model current, specific knowledge

If the model gives plausible but wrong answers about your organisation, the gap is knowledge. Retrieval-augmented generation (RAG) closes it: the system searches your documents for the most relevant passages and adds them to the prompt. The model is told to answer from them, and the system can show which sources it used. This reduces made-up answers but does not remove them.

This suits knowledge that is specific to you, changes over time or is too large for every prompt, such as policies, manuals and price lists. Keeping it current means updating documents and the index, not retraining. Sources let people check answers, and the search can return only documents the user may see. That check belongs in the search: telling the model not to reveal certain documents is not a reliable safeguard.

RAG depends heavily on its search: if the right passage is not found, the answer suffers. Even with the right passage, the model can misread it, add details that are not there or cite the wrong source, so answers that matter still need checking. Our article RAG explained covers what decides that quality.

Fine-tuning: changing behaviour with examples

Fine-tuning means further training an existing model on example inputs paired with the outputs you want. It adjusts the model's internal settings (its parameters), so the behaviour is built in rather than requested each time. Many hosted models offer limited or no fine-tuning, so this route can narrow your choice of model.

It suits behaviour: a consistent style or tone, a format that prompting and structured outputs cannot hold reliably, or a narrow, repetitive task such as sorting support requests into your own categories. Because the behaviour is learned, prompts can be shorter, and sometimes a smaller model will do.

Fine-tuning is not a reliable way to add facts: a model may absorb some, but it cannot cite a source, may mix them up and cannot tell when they go out of date. For knowledge, use the prompt or retrieval. Fine-tuning also brings ongoing work:

• Many good, consistent examples, checked by people who know the work for mistakes and bias, which the model learns as readily as good practice.
• Test cases kept out of training, showing that the tuned model is better at the task and no worse elsewhere, such as following instructions or refusing harmful requests.
• Retraining when requirements change or you move to a newer underlying model.
• Care with personal data: use only what the task needs, and only where the law and your provider's terms allow (take legal advice if unsure). Models can sometimes reproduce training text, and data cannot easily be removed from a trained model.

Side by side, the three approaches need different inputs and upkeep.

(opens the full-size image in a new tab)

Tools: when the answer needs an action or exact data

Some requests need live, exact data or an action in another system, such as an order's status or booking an appointment, which neither documents nor training can supply.

Tool use, also called function calling, handles this. You describe functions the model may request, such as looking up an order. The model does not run them: it produces a structured request, your software checks and runs it, and the result goes back to the model to phrase the reply.

Because your systems do the work, figures come from your records and calculations from code, not from the model, which can get arithmetic wrong. The model can still slip when copying a figure into its reply, so show important numbers, such as a balance, exactly as the tool returned them.

Tools need guardrails, not least because text the model reads, such as a customer message or retrieved document, can contain instructions that try to make it misuse a tool (prompt injection). Limit what each tool can do, enforce permissions in your software rather than the prompt, and have a person confirm anything with real consequences. Our article on AI agents covers these safeguards.

How the approaches combine in practice

These are layers, not rivals. Take a hypothetical support assistant for an equipment supplier:

• The prompt sets the tone, scope and when to hand over to a person.
• Retrieval supplies relevant manual pages and the returns policy, with sources.
• A tool checks the status of an order or repair.
• Fine-tuning comes in only if one behaviour stays inconsistent at volume.

A good prompt plus one or two other techniques is often enough. We recommend adding fine-tuning last, not first.

Choosing: is the gap knowledge, behaviour or action?

Look at real cases where the system falls short and ask what is missing. The gap usually falls into one of three groups.

(opens the full-size image in a new tab)

• Knowledge: answers are wrong, out of date or too generic about your organisation. If the material is small and stable, such as one policy, try putting it in the prompt first. For larger or changing collections, or where users may see only some documents, use retrieval.
• Behaviour or format: the facts are right but the tone, structure or judgement is off. Improve the prompt first; consider fine-tuning only if that fails and you have many good examples.
• Action or exact data: the task needs a live figure, a calculation or a change in another system. Use tools.

Weigh running costs, not only the effort to start. Longer prompts add cost to every request, which caching the fixed part can reduce. Retrieval needs current documents and a maintained index. Fine-tuning needs data collection and retraining. Tools need integration work and careful security. The cheapest option to build is not always the cheapest to run.

Where to start

Whatever you choose, measure before and after. Without a fixed test set, changes are judged on a few memorable examples, and an improvement in one area can hide something getting worse in another.

1. Collect real examples, including awkward cases, and note where the current system falls short.
2. Agree what a good answer looks like, and turn the examples into a repeatable test set.
3. Sort each failure into knowledge, behaviour or action.
4. Fix each failure with the simplest matching technique: a better prompt for behaviour, the prompt or retrieval for knowledge, tools for actions, and fine-tuning only as a last resort.
5. Rerun the tests after every change, and keep monitoring once the feature is live.

Our guide to evaluating AI features explains how to build those tests.

If you are weighing these options for your organisation, our AI solutions and automation team can help you name the gap and set up an evaluation before you build anything.