Before choosing between prompt engineering, retrieval-augmented generation (RAG) and fine-tuning, find out why the current answers fail. Missing or outdated knowledge calls for getting the right facts into the model's context at answer time, usually through retrieval. Unclear or incomplete instructions call for a better prompt. A consistent behavior problem that persists despite clear instructions, good examples and the right facts is where fine-tuning starts to make sense. Choose the smallest change that fixes the measured failure, then test again.
Collect failing outputs from testing or a small pilot and label each one with its likely cause. Three questions sort most of them:
- Instruction: did the prompt say clearly what was wanted, including format, language, audience and what to do when unsure?
- Knowledge: did the model have the information the answer required, either in its context or reliably from training?
- Behavior: with clear instructions and the right information, does the model still miss in the same way, again and again?
Check them in that order. Instruction and knowledge fixes are often easier to test first. Fine-tuning can learn behavior from examples, but it does not by itself provide current, permission-filtered evidence. A generated citation still needs checking against an accessible source. Tag the failures in your test set so you can see which category dominates; our guide to evaluating LLM hallucinations describes how to label and slice them.
Figure 1 — Check three likely causes. Instructions, knowledge and behavior.
Typical signs are outputs that change shape from one run to the next, answers in the wrong language, summaries of unpredictable length, or a model that guesses at a format nobody specified. These symptoms may reflect missing instructions, though model capability and conflicting constraints can also be responsible.
The fixes are direct. State the task, audience and purpose. Define the output structure, ideally as a schema your code can validate. Give two or three examples of accepted outputs, including an Arabic one if users write in Arabic. Spell out constraints, and say what the model should do when the information is insufficient. Version prompts like code and re-run the test set on every change. Prompt changes are usually inexpensive to test and easy to roll back; measure their effect before investing in more complex changes.
Typical signs are outdated prices, invented policy details, confident answers about products the model has never seen, or answers that would be right for another company. The model lacks information, and no instruction will create it.
If the knowledge is small and stable, a few pages that rarely change, put the portions the user is authorized to see in the prompt. If it is large, changes often or must respect who is allowed to see what, use retrieval: search approved sources at question time, filter by the user's permissions and cite what was used. Our enterprise RAG guide covers that design. Fine-tuning is a poor way to add facts that change, because keeping facts only in weights requires weight updates and validation when those facts change. Generated citations do not guarantee traceability, and learned knowledge is difficult to restrict by user permission.
For questions about numbers in a database, such as last quarter's sales by region, use an authorized read-only query tool and deterministic calculations where appropriate. Enforce tenant and row permissions, bound the query and return its provenance; text retrieval alone may not support reliable aggregation.
Figure 2 — Supply authorized evidence. Check retrieval before generation.
If you already use RAG and answers are still wrong, check whether the right passages reached the model before changing anything else. If they did not, the problem is retrieval: chunking that separates a clause from its conditions, missing metadata, search settings, or permission filters that are stricter than intended. Changing only the answer-generation prompt does not supply a missing passage. Query rewriting, retriever tuning or reranking may improve retrieval itself. Check permission filters against the actual access policy; do not loosen valid restrictions to improve recall. If the right passages did arrive and the answer still drifted, return to the instruction and behavior questions.
Fine-tuning adjusts the model's weights using examples of the outputs you want. It fits failures that remain once instructions are clear and the facts are present: a business-specific classification scheme the model keeps confusing, a house style it cannot hold across thousands of outputs, or a compact output format needed at high volume, where long prompts full of examples add cost and latency.
Before fine-tuning, compare two alternatives that may avoid a training project. A more capable base model may already handle the task, although its inference cost may be higher. Splitting the task into smaller steps, each with its own prompt and check, often fixes judgment errors that a single prompt cannot. If fine-tuning is still justified, plan for what it requires: a curated set of permitted training examples with an owner and privacy review, a held-out test set the training never sees, a process for retraining when the base model or your requirements change, and a full re-evaluation before each new version goes live.
The approaches are not mutually exclusive. Retrieval for knowledge plus careful instructions for format is a common and sensible pairing, and fine-tuning for a stable behavior plus retrieval for changing facts is also legitimate. Trouble starts when several changes arrive at once without a baseline, because nobody can then tell which one helped or remove one safely.
Anthropic's engineering guidance on building agents recommends finding the simplest solution that works and increasing complexity only when it is needed, noting that more complex systems often trade latency and cost for better task performance; the page marks its December 2024 examples as historical. We apply the same rule here: add one change at a time, measure it on the same test set, and keep it only if the targeted failures decrease and the other quality, security and cost gates still pass. If you are still deciding which workflows deserve this effort, our guide to selecting internal AI copilot use cases helps narrow the list.
Every change discussed here alters answers, so each one is a release: re-run the test set by slice, compare it with the previous version, and record what changed and why. NIST's Generative AI Profile (NIST AI 600-1), published on 26 July 2024 to extend the AI Risk Management Framework, is voluntary guidance that can help structure that documentation and risk review. It does not certify a system or guarantee accurate output.
These recommendations are TecnoStack's practical guidance, not requirements from Anthropic or NIST. If knowledge failures dominate your test results, our RAG systems service can help you design retrieval, permissions and evaluation around your documents.
- Anthropic. Building effective agents: recommends the simplest solution that works and adding complexity only when needed; its December 2024 examples are marked historical.
- National Institute of Standards and Technology. Generative Artificial Intelligence Profile (NIST AI 600-1), published 26 July 2024 as a companion to the AI RMF.
