AI Agents Are Getting Faster. But Who Is Engineering Their Recovery?
The next challenge in production AI is not simply making agents more capable. It is making their execution recoverable, observable and safe when things go wrong.
An AI agent can investigate a problem, call external APIs, update databases, send messages and coordinate multiple tools without requiring a human at every step.
That is precisely what makes agentic systems valuable.
But it also introduces a question that deserves considerably more attention:
What happens when an agent fails halfway through a task?
Imagine an AI agent handling a customer onboarding workflow:
- It verifies customer information.
- It creates a customer record.
- It sends a confirmation email.
- It provisions an account.
- It updates the billing system.
Now imagine the application crashes after step three.
When the system restarts, should the agent begin again?
If it does, it might create another customer record, send a duplicate email or repeat an operation that has already succeeded.
If it does not, how does it know which operations completed, which failed and which have an uncertain outcome?
This is not fundamentally a prompt-engineering problem. It is a distributed systems and workflow-reliability problem.
And as AI agents move into production, it is becoming increasingly difficult to ignore.
The shift from generating answers to executing operations
For years, the primary concern with generative AI applications was whether the model produced accurate, useful and relevant responses.
Agentic systems change the equation.
An agent can interpret an objective, decide which tools to invoke, process their responses and continue executing until it reaches a result or encounters a failure.
Microsoft highlighted this shift in its June 30, 2026 research on securing AI agents. It described how an indirect prompt injection that might previously have influenced a summarizer could trigger an actual operation when the same kind of vulnerability exists in an agent with write access.
The distinction is significant.
A system that generates an incorrect answer creates an information-quality problem.
A system that executes an incorrect action can create a financial, operational or security incident.
The attack surface also expands beyond the model itself. Tool permissions, MCP servers, credentials, external APIs and connected business systems all become part of the execution environment.
The engineering responsibility therefore moves beyond model quality.
It includes controlling what the agent can do, understanding what it has done and ensuring the application can recover when execution becomes inconsistent.
The evidence is beginning to show the operational consequences
Several developments throughout 2026 illustrate how quickly this problem is evolving.
June: Tool access becomes a security boundary
Microsoft's June 30 research documented an MCP tool-poisoning attack pattern involving an enterprise finance workflow.
The research explained how malicious instructions embedded in tool metadata could influence an agent's behavior.
The important architectural lesson is that an agent's tools are not merely convenient integrations. They are privileged execution interfaces.
If an agent can access financial systems, modify records or send external communications, the security of those operations cannot depend entirely on the model correctly interpreting instructions.
September: Autonomous operations compress attack timelines
On September 8, Google's Threat Intelligence Group published research documenting the evolution of adversarial AI workflows.
One particularly important observation concerned a campaign in Q2 2026.
Threat actors compromised a cloud resource and then planned, built and executed an agent-enabled mass credential-harvesting campaign in under six hours.
This does not mean every stage of every AI-enabled attack now takes six hours. It demonstrates how agent-enabled automation can compress the time required to coordinate complex operations.
The implication for defenders is straightforward: detection and response processes designed around slower, human-driven attack sequences may no longer provide sufficient reaction time.
October: Unauthorized agent actions become an organizational concern
Salt Security's October 13 report, based on a survey of more than 380 security and technology professionals, found that 49.8% of respondents confirmed or suspected an AI agent had taken an unintended, unexpected or unauthorized action during the previous 12 months.
Other findings were equally important:
- 86.4% reported having at least one AI agent in their organization.
- 76.8% reported agents chaining actions across other agents, tools and APIs.
- Only 12.5% could consistently trace a complete agent execution path.
- Only 17.5% could automatically stop an agent behaving unexpectedly in production.
These are survey and vendor-reported findings, not independently established measurements of every enterprise. Nevertheless, they expose an important operational concern: organizations are adopting interconnected agent workflows faster than they are developing the controls to observe and contain them.
October: Agents become part of the offensive security landscape
On October 15, India's CERT-In issued a critical advisory documenting incidents involving AI agents exceeding their assigned authorization and accessing live organizational systems.
The advisory also described financially motivated attacks using open-source agentic offensive-security frameworks.
In those campaigns, agentic tooling compressed the time between initial access and final impact from days to a few hours.
This is particularly important because the problem is no longer limited to accidental behavior inside enterprise applications. Agentic capabilities are also changing the operational speed of adversarial activity.
Together, these developments point toward a broader engineering challenge: autonomous execution requires equally mature controls for authorization, observability, containment and recovery.
The overlooked problem: restarting is not the same as recovering
Consider a workflow that has successfully completed four operations before its fifth operation times out.
A naive recovery strategy might restart the entire agent and ask it to continue the task.
But an agent is not necessarily replaying a deterministic sequence of operations.
It may reconsider its plan, choose different tools, interpret the available context differently or repeat an external operation.
This introduces several failure scenarios:
Duplicate execution: An operation succeeds externally, but the application times out before recording its success. Retrying may execute it twice.
Lost execution state: The process restarts without knowing which operations completed.
Partial completion: One system has been updated while another remains unchanged.
Uncertain external outcomes: An API request times out, leaving the application unable to determine whether the external service processed it.
Changed decisions: The model produces a different plan after recovery, potentially creating a workflow that no longer matches the original intent.
The last issue is especially important.
Traditional workflow systems generally execute explicitly defined transitions. An agent can dynamically decide what the next operation should be.
That means recovery must preserve more than conversational history.
It must preserve the execution state, completed operations, pending operations, relevant decisions and authorization context.
Distributed systems already have answers to many of these problems
Interestingly, engineers do not need to invent an entirely new reliability discipline for AI agents.
Distributed systems engineering has dealt with partial failure, retries, duplicate messages and uncertain external operations for decades.
The challenge is applying those principles to systems whose execution plans may be partially determined by an LLM.
Several established techniques are particularly relevant.
- Durable execution and checkpoints
Persist workflow progress outside the agent's process memory.
After a meaningful operation completes, record its result and the workflow's current state.
If the application crashes, it should resume from the last recoverable point instead of reconstructing the entire task from a conversation.
LangGraph's persistence architecture provides checkpointers for saving graph state and supporting recovery. Its documentation also describes retaining successful task writes when another task in the same super-step fails.
This is a useful implementation pattern, although checkpointing alone does not guarantee that external side effects will not be repeated.
- Idempotency for external operations
Every operation that might be retried should be evaluated for idempotency.
An idempotent operation can be repeated without producing additional unintended effects.
For example, instead of blindly creating a customer record every time an agent retries, the application can use a stable operation identifier and enforce uniqueness at the database or API boundary.
For payment operations, email dispatch and account provisioning, the application should distinguish between a new operation and a retry of an existing one.
Importantly, idempotency does not mean every external API automatically provides exactly-once execution. It requires explicit support and careful handling of uncertain outcomes.
- Retry the failed operation, not the entire workflow
Temporal's retry-policy documentation makes an important distinction between workflow execution and individual activities.
Failure-prone operations, including external API calls and LLM invocations, should be isolated into activities with appropriate retry behavior.
Retrying an entire workflow is generally not the right response to a failure that occurred in one activity.
AWS Step Functions offers another established approach through configurable Retry and Catch behavior, allowing developers to retry specific states or route failures into fallback paths.
These patterns are particularly useful when agent workflows contain a combination of deterministic business operations and nondeterministic model decisions.
- Treat irreversible operations differently
Not every operation should be retried automatically.
Reading a database record is different from transferring money, deleting information or sending an external message.
A production agent should distinguish between:
- Read-only operations.
- Reversible state changes.
- Idempotent writes.
- Irreversible or externally visible operations.
Sensitive operations should have appropriate authorization checks, confirmation requirements, transaction boundaries and compensating actions where possible.
A model deciding that an action is necessary should not automatically mean the action is authorized.
- Preserve execution history and observability
A production agent needs an execution history that answers:
- What was the original objective?
- Which model and tools were involved?
- What operations were attempted?
- Which operations succeeded?
- Which operations failed or have uncertain outcomes?
- What data influenced the decision?
- Who or what authorized the operation?
- Where did execution resume?
This should be structured telemetry, not simply a collection of conversation logs.
Distributed traces, operation IDs, workflow IDs and structured events make it possible to reconstruct what happened without relying on the model to explain its own behavior afterward.
The economics of unreliable autonomy
There is another dimension to this problem that deserves attention: the economics of operating AI agents.
The cost of an agent is often discussed in terms of model pricing, token consumption and inference latency.
But those are only parts of the total operating cost.
Consider the following conceptual model:
Total agent operating cost = successful execution + retries + recovery infrastructure + human intervention + failure impact.
This is not a published industry benchmark. It is a way to think about the full cost structure.
A workflow that normally requires ten model calls might require considerably more if it repeatedly reconstructs context after failures.
A duplicate transaction can create a financial problem. A repeated email can damage customer experience. An incorrect database update can require engineering intervention.
And when organizations cannot trace agent execution, investigating an incident becomes more expensive and time-consuming.
McKinsey's August 25, 2026 global AI survey offers useful context. It found that 20% of respondents reported AI-related operating costs constraining usage. At the same time, 37% reported some enterprise-level EBIT impact from AI, essentially unchanged from the previous year.
These findings do not establish that unreliable agent recovery is causing those costs or limiting financial returns. They do, however, reinforce why measuring AI performance solely through task-completion rates is insufficient.
A faster agent is not necessarily a cheaper agent.
A successful execution is not necessarily an economically successful system.
What engineering teams should measure before scaling agents
Before expanding an agent from a controlled demonstration into a business-critical workflow, teams should evaluate more than answer quality.
A practical production-readiness checklist includes:
Execution reliability
- Can workflows resume after process crashes?
- Are completed operations persisted?
- Are retries limited and configured by failure type?
- Can uncertain external outcomes be reconciled?
Security and authorization
- Does every tool enforce its own permissions?
- Are agent permissions limited to the current task?
- Are sensitive actions subject to independent authorization?
- Can unexpected agent behavior be detected and contained?
Observability
- Can the complete execution path be reconstructed?
- Are model calls and tool operations correlated through trace IDs?
- Can engineers identify the exact point of failure?
- Is there a record of the decisions and permissions involved?
Economic performance
- What is the actual cost per successfully completed workflow?
- How often do workflows require retries?
- How much human intervention is required?
- What is the recovery time after failure?
- How frequently do failures produce duplicate or incorrect operations?
These measurements should be evaluated under realistic failure conditions, not just successful demonstrations.
For example, deliberately terminate an agent process after an external API succeeds but before its result is persisted. Then restart the workflow and verify that the operation is reconciled rather than blindly repeated.
That single test can reveal a class of problems that a hundred successful demonstrations might never expose.
The next stage of agent engineering
The industry has spent considerable effort improving how AI agents reason, plan, use tools and coordinate with other agents.
Those capabilities matter.
But production software is not judged only by what happens when everything works.
It is judged by how it behaves under failure.
The same principle should apply to autonomous AI systems.
An agent that can execute a complicated workflow but cannot reliably resume, explain or contain its actions is not ready for every environment where it might be useful.
The next stage of agent engineering will require more than better models and more sophisticated orchestration.
It will require durable execution, explicit authorization, idempotent operations, recoverable state and observability designed around autonomous behavior.
The real measure of an AI agent is not just whether it can complete a task. It is whether the system can safely handle everything that happens when it cannot.