No test set can remove hallucinations from a language model entirely, so production safety depends on what surrounds each release: a gate a change must pass before more people see it, monitoring of the answers users actually receive, an incident process that contains harm quickly, and a rollback path rehearsed before it is needed. Those four controls let a team keep shipping improvements without betting each one on luck.
This article covers those operational controls. How to build the test set, labels and metrics they rely on is explained in our guide to evaluating LLM hallucinations.
Teams usually gate new features and overlook quieter changes that alter answers just as much: a new model version, an edited system prompt, a different chunking rule or embedding model, a new tool, a large batch of documents added to the index, or a change to permission sync. Each of these deserves a version identifier, a short change note and the same gate.
Two habits make this practical. First, pin model versions where your provider allows it, and record which model, prompt and index version produced each answer, so an investigation can reconstruct the conditions and attempt to reproduce the issue later. Second, keep the release record short: what changed, which test-set version was run, the results per slice, and who approved it and why. That record also gives a governance reviewer useful evidence.
A release gate is a set of conditions agreed in advance. Criteria written after the scores arrive tend to bend toward whatever the new version happened to achieve.
Useful gates separate blocking conditions from advisory ones. For an illustrative HR policy assistant, blocking conditions might be: every permission test passes; no new unsupported claims appear in the pay and leave slice; the system still declines questions that no accessible document covers; and both the Arabic and English slices were reviewed. Advisory conditions might cover answer length or latency. Thresholds depend on the cost of an error in each slice, which is a business decision, so a named business owner should approve them rather than the engineering team alone.
When a blocking gate fails, fix it and re-run, or narrow the release only if the failed capability can be isolated safely. Access-control failures must block release. For advisory criteria, an owner may document a time-limited exception with the remaining risk and a review date. An exception must not waive mandatory security or legal requirements. Silent overrides are the failure mode to design out.
Passing an offline gate does not prove the system will behave on live traffic, because real questions drift away from any test set. Stage the exposure:
- Shadow run. The new version answers real questions in parallel while users still see the current one. Reviewers compare the two on a sample.
- Internal users. Staff who know the domain use it and report problems through a visible channel.
- Small group. A limited set of real users, behind a feature flag that can be switched off at once.
- Wider release. Only after the earlier stages show no new failure pattern.
Throughout, preserve the previous version, prompts, model setting and index snapshot for recovery. Before restoring them, apply current permissions, required deletions and security fixes; an old index must not restore access that has since been revoked.
Figure 1 — Expand access in stages. Keep a tested recovery path.
Monitoring an LLM application relies on sampling and signals, because no single automatic metric establishes correctness across every production question. Combine:
- Sampled human review. Reviewers label a regular sample of live answers with the same guide used for the test set, broken down by slice, including Arabic.
- Automatic checks. Cited passage IDs exist, cited passages were permitted for that user, quoted numbers and units match their sources and outputs follow the required format.
- User reports. A visible way to report a problem that reaches the document owner, not an unread inbox.
- Rate changes. Shifts in decline rates, empty retrievals, tool errors, cost or latency.
A changed signal is a reason to investigate, not proof that quality rose or fell. Watch upstream changes too: edited documents, reorganized permissions and provider model updates can change answers without any release on your side. Keep monitoring logs access-restricted, minimized and subject to a retention limit.
When an answer causes or could cause harm, such as a wrong policy given to many employees, restricted content shown to the wrong person, or a tool action taken on bad information, treat it like any other production incident.
Contain first. Prepare the options before launch: switch the feature off behind its flag, fall back to a safer mode such as search results with links but no generated answer, remove or restrict the document involved, disable a specific tool, or roll back to the previous release.
Then investigate. Preserve the evidence under access control: the question, the retrieved and permitted passage IDs, the answer and the versions involved. Find which stage failed: the document, retrieval, permissions, the prompt or the model. Add the case to the test set so the gate catches it next time, and decide with the owner whether affected users need to be told. When an agent takes actions, approval rules matter even more; our CTO playbook for AI agents covers tool approval.
Figure 2 — Contain harm before investigating. Rollback must preserve current permissions.
Rollback only works if every answer-changing component is versioned together: model setting, prompts, retrieval configuration, tool definitions and the index. Write a short runbook that names who may trigger a rollback and how, then rehearse it in a non-production environment before launch.
Be explicit about what rollback cannot undo. Rolling back application code does not undo emails sent, tickets closed or records changed by a tool. Some changes can be repaired through a separately validated compensating action; sent messages and their consequences may not be recoverable. That is the strongest argument for requiring human approval before an AI system takes actions with external effects.
NIST describes its AI Risk Management Framework as voluntary guidance; it is not a certification or a law, and it is organized into four functions: Govern, Map, Measure and Manage. In our reading, release gates sit between Measure and Manage, incident response and rollback are Manage work, and the ownership behind them belongs to Govern. NIST's Generative AI Profile (NIST AI 600-1), published on 26 July 2024, extends the framework to generative AI. Neither document guarantees safe behavior; both help you organize controls and evidence. For ownership, inventories and legal scope, see our guide to AI governance with NIST and the EU AI Act.
These practices are TecnoStack's guidance, not NIST requirements. If you are preparing an AI feature for launch, our AI systems service can help you set up the gates, monitoring and rollback around it.
- National Institute of Standards and Technology. Generative Artificial Intelligence Profile (NIST AI 600-1), published 26 July 2024 to extend the AI RMF to generative AI.
- National Institute of Standards and Technology. AI Risk Management Framework: voluntary guidance organized around the Govern, Map, Measure and Manage functions.
