Add an LLM to a legacy system in controlled stages rather than one launch. First inventory the touchpoints, the data each one uses and who owns it. Then pilot a read-only feature that suggests rather than acts, run it in shadow mode against real work, and route a growing share of traffic to it behind a switch that can send everyone back to the old path. Features that write to the legacy system come last, once the earlier stages have produced evidence. At every stage, rollback should be a tested routine.
This article is about the order of moves. The target architecture, including adapters, identity, tool boundaries and the separation of reads from writes, is covered in our blueprint for adding an LLM to a legacy application.
Legacy systems carry behavior nobody has written down: batch jobs that run at odd hours, fields with overloaded meanings, integrations that break if a response arrives a second late. An LLM feature adds a dependency that can be slow, wrong or unavailable. Introducing it in stages limits what each surprise can damage.
AWS Prescriptive Guidance describes the strangler fig pattern for modernization: functionality is replaced incrementally behind a routing layer, with the old and new parts operating side by side during the migration. We apply the same idea when adding AI. A routing point decides whether a request takes the existing path or the new one, so exposure can grow gradually and new requests can return to the existing path. In-flight work needs explicit cancellation or draining.
List the candidate places where a model could help: screens, workflows, batch jobs and inbound channels. For each one, record:
- what data it reads, who owns that data and whether it includes personal or confidential information;
- whether it writes anything, and through which interface;
- how it behaves today when a dependency is slow or fails;
- peak load and timing constraints;
- who can approve changes to it, and where an off switch could sit.
Settle data ownership now. During the migration the legacy system remains the system of record. The LLM component may read, summarize and propose, but it must not quietly become a second source of truth for the same data.
Figure 1 — Inventory before building. Keep one authoritative record.
Start in an isolated pilot with permitted test data and designated reviewers. Choose a feature that proposes without automatically changing records: summarizing a customer's history on a CRM screen, drafting a reply an agent edits, or suggesting a category for incoming tickets that a person confirms. If the model is slow or unavailable, the screen should still work exactly as before.
Anthropic's engineering guidance on building agents distinguishes workflows, which follow predefined code paths, from agents, which direct their own tool use, and recommends starting with the simplest solution that works. For a legacy integration that usually means a fixed workflow: the legacy system calls the model at one defined point, and the result is shown to a designated pilot reviewer. Read-only output can still disclose data or mislead a decision, so enforce access and test answer quality.
Before production users see its output, run the new component in parallel with the existing process, using only data authorized for this model and provider. Disable real sends, payments and other side effects in the shadow path; isolate its resource budget so duplicated work cannot degrade the live path. It receives real inputs, but its outputs are recorded rather than shown. Compare them with reviewed expected outcomes and what people or the legacy logic actually did; disagreement alone does not prove the model is wrong or the old process correct. Look at the types of disagreement, not just the rate, and at latency and cost under real load.
Shadow mode surfaces problems that test data misses: unusual record formats, Arabic and English mixed in one field, or timeouts at peak hours. Keep shadow logs under the same access controls and retention limits as production data. Write the criteria for moving to the next stage before the shadow results arrive; our guide to release gates, monitoring and rollback for LLM features covers how to set them.
Figure 2 — Shadow with side effects disabled. Review outcomes; protect data and capacity.
Use the routing point to expose the feature in stages: an internal team first, then a small group of users, then wider groups. Gate each expansion on criteria agreed in advance, and keep the old path fully working and maintained, because it is your rollback. Rollback should mean switching the route back, not redeploying under pressure. Rehearse it before the first real users arrive and again before each expansion. Keep permissions and deletion rules current on the fallback, define how to drain or cancel pending work, and retain compatible data formats. A route switch does not undo disclosed output or completed external effects.
Decide in advance what would allow the old path to be retired: a period without serious incidents, stable quality measures and the owner's sign-off. Until then, old and new coexist, exactly as the strangler pattern intends.
Only after the read-only feature has proven its value should the model's output start changing records. Send writes through the legacy system's existing validated interfaces, such as its APIs, queues or batch imports, rather than writing directly to its database. Require human approval for any write with an external effect, reauthorize and validate the exact proposed action at execution time, bind approval to that action, and make each write idempotent and traceable, and design compensating procedures, because a write that has propagated to other systems cannot simply be rolled back.
This is where data ownership matters most. If the legacy system remains the system of record, every accepted change lands there through its normal rules. Copies, indexes and downstream systems still require reconciliation. Scope idempotency keys to the tenant, operation and payload, store outcomes atomically, and test duplicate and delayed delivery.
If shadow results do not meet the criteria, stop before users see the feature; that avoids widening exposure to a known failure. If the idea turns out to be a new product surface, such as a customer-facing chat or an analyst console, build it alongside the legacy system and read its data through an approved interface instead of grafting it into old screens. For the wider modernization picture, see our guide to the strangler fig pattern for legacy modernization.
This sequence is TecnoStack's guidance, not a requirement of AWS or Anthropic. If you are planning the first AI feature on a legacy platform, our AI systems service can help you inventory the touchpoints and run the stages safely.
- Amazon Web Services. Strangler fig pattern, AWS Prescriptive Guidance: incremental replacement behind a routing layer, with old and new components coexisting during migration.
- Anthropic. Building effective agents: distinguishes predefined workflows from agents and recommends starting with the simplest solution; its December 2024 examples are marked historical.
