An AI agent's tool did the work, then failed, then ran again: where to stop the second run on AWS

Your agent's refund tool calls the payment API and the refund goes through. On the next line, before the result is saved, the code throws, or the function times out. Each layer around the tool is doing its job, and its job is to run the tool again.

This is for the Staff or Principal engineer who owns an agent platform and is about to move the first tool that writes (a refund, a ticket, an email, a record update) from demo to production. A CTO asks the same thing in one line: can the agent do it twice?

You get a map of where AWS and MCP re-run a tool. You get the place where the Powertools idempotency decorator stops, and five ways to stop the second action with their trade-offs. Last comes a failure-injection test to run on your own tool.

Everything below comes from AWS documentation, the Powertools documentation and its source code, and the MCP specification. It is a reasoned design, not an incident report. I have not run the test at the end, so its table is a prediction from the docs and is labelled that way.

Where the second run comes from

One approved action travels down a chain: model, orchestrator, queue or workflow, tool function, downstream API. Every link has its own retry rule. Each rule is sensible alone. The trouble is the gap between "the side effect happened" and "the result was saved". No layer closes it.

The diagram shows one attempt that fails inside that gap, and the layers that can run the tool again.

Notice that the write at step 1 has already happened when the failure at step 2 stops the result from being saved. Each layer in step 3 then runs the tool function again. The second run has no way to know the work is done unless something outside the tool recorded it.

This table lists what each layer does, from the AWS pages and the MCP specification.

Switching retries off is not the fix. Duplicates also arrive with no error, as the Lambda and SQS rows show, and with retries off you lose the recovery from a real transient failure. The Lambda best-practices page says "Write idempotent code" and asks for code that can "gracefully handle duplicate events" (Lambda best practices). The question is where that code lives.

The decorator is the first layer, and here is where it stops

Powertools for AWS Lambda (Python) has an Idempotency utility. It stores a record per request in a persistence layer such as DynamoDB. It takes the key from the event with event_key_jmespath, and by default expires records after an hour (Powertools idempotency).

A duplicate that arrives after the first run completed, and before the record expires, gets the saved response back. The page's flow for a record that is "COMPLETE and not expired" ends in "Same response sent to client". A duplicate that arrives while the first is still running raises IdempotencyAlreadyInProgressError (Powertools idempotency).

That covers the common duplicate. Now the case this article is about. The page lists "two failure modes that can cause new invocations to execute your code again despite having the same payload". The first is an unhandled exception: "We catch them to delete the idempotency record to prevent inconsistencies, then propagate them" (Powertools idempotency).

Put that next to the refund. The payment call succeeds and the next line throws. The decorator deletes the record and re-raises, Lambda retries, and the same code runs against the same payment API. The page has no sentence that says a side effect done before the exception will repeat. As I read it, that follows from "execute your code again".

The timeout case needs care. The page says that if the function "times out before completing the first invocation it will only accept the same request when the idempotency record expire". It then says that register_lambda_context puts the remaining invocation time in the record, and that a second invocation after that time, with the record still marked in progress, "we will run the invocation again". It adds, "if an invocation expired during execution, it will be quickly executed again on the next retry".

The same section says you can skip it when you use @idempotent, and does not say why. The Powertools source shows that @idempotent registers the Lambda context itself (checked 2026-11-01). So I read it this way: with the decorator on the handler, a retry that comes after the timeout can run your code again. Lambda starts its first asynchronous retry a minute after the failed attempt. The test below records what really happens.

A related line on the same page says: "If an exception is handled or raised outside your decorated function, then idempotency will be maintained" (Powertools idempotency). It is about exceptions outside the decorated function. A function that catches the late failure itself and returns a result has no unhandled exception, so the decorator has no record to delete, and I would expect it to save that result. That is my reading, not a sentence on the page. The saved result would say "failed after the write", so a duplicate would get that answer back, which may or may not be what you want.

Catching helps with a bug on the next line. It does not help with a timeout, because Lambda stops the function when its time runs out (Lambda timeout), so no code after the external call gets to run.

This default worries me more than a missing feature would, because the demo passes. The failure needs a crash after the write, and a demo rarely has one.

My view: keep the decorator as the first layer. For money, customer messages and anything you cannot undo, also make the write itself check a key.

Make the write check the key

A DynamoDB conditional PutItem that fails its condition returns ConditionalCheckFailedException, and attribute_not_exists on the key is true only when no item with that key exists (DynamoDB items, DynamoDB condition expressions). So the check and the record happen in one call. No second run can slip between them.

If the side effect is a row in a table you own, make the request ID the row's key. The table refund-ledger is keyed on row_id. This function is illustrative and untested.

import boto3
from botocore.exceptions import ClientError

ledger = boto3.resource("dynamodb").Table("refund-ledger")


class KeyReusedError(Exception):
    pass


def create_refund(row_id, request_id, order_id, amount_minor):
    try:
        ledger.put_item(
            Item={
                "row_id": row_id,
                "request_id": request_id,
                "order_id": order_id,
                "amount_minor": amount_minor,
            },
            ConditionExpression="attribute_not_exists(row_id)",
        )
    except ClientError as err:
        if err.response["Error"]["Code"] != "ConditionalCheckFailedException":
            raise
        saved = ledger.get_item(Key={"row_id": row_id}, ConsistentRead=True)["Item"]
        if saved["order_id"] != order_id or saved["amount_minor"] != amount_minor:
            raise KeyReusedError(row_id)
        return "already_done"
    return "done"

In production you pass the same value as row_id and request_id. The harness below passes a random row_id on purpose, to build the "no protection" baseline.

Notice two things. A retry gets already_done and writes nothing, however it reached the function: Lambda retry, SQS redelivery or a client calling again. And a reused key with a different order or amount raises instead of quietly returning the old result. Powertools has the same idea in payload_validation_jmespath, which raises IdempotencyValidationError when the same key arrives with a changed field such as the amount (Powertools idempotency).

If the side effect is an external API call, the same idea works only when the provider accepts a key. Send the same key on every attempt. Provider support differs, and I have not checked any provider's documentation for this article, so read yours.

Without provider support you have three states, not two: not started, done, and unknown. I would write an intent record before the call and mark it done after. A retry that finds an intent record with no result does not know whether the call happened. It should stop and ask the provider by your own reference, not call again. The cost is that some requests wait for a reconcile job or a person. That is the price of not paying twice.

Where the key comes from

The key must be created once, before the first attempt, outside any code that gets retried, and it must be identical on every attempt. A timestamp or a random value created inside the tool fails that test, because each attempt makes a new one.

The harder part is the path from the model to the tool. A tools/call request carries a name and arguments. The Tools page defines no idempotency field. idempotentHint defaults to false, and the spec says clients should not make decisions from annotations received from untrusted servers (MCP schema, MCP Tools). So the key has to travel inside the arguments, and the model writes the arguments. The spec says tool errors are fed back so models can "self-correct and retry with adjusted parameters", and that the model carries values forward between calls (MCP Tools). In my view, a retry the model writes can therefore carry a new ID, and your check never matches.

So I would have the orchestrator mint request_id when the action is approved, or when it accepts the model's proposed call, and overwrite that argument before it calls the tool. The model does not need to see it. Two things I would not rely on:

• The JSON-RPC id. The spec says it must differ between an input-required request and its retry (MCP schema, MCP Tools).
• Gateway context IDs. For an AgentCore Gateway Lambda target, the page describes the event as a map of the inputSchema properties to their values. The context carries IDs such as bedrockAgentCoreMcpMessageId, and the page does not say whether they stay the same across a retry (AgentCore Lambda targets).

So I would declare request_id as a property in the tool's inputSchema. That is the one place the page says the event gets its values from.

If you cannot control the orchestrator, there is a fallback: derive the key from the approved action itself, such as a hash of order ID, action name and amount. It survives retries. Its cost is collision: two legitimate refunds of the same amount on the same order look like one, and the second is dropped. I would take that trade only for actions where a wrongly dropped one is safer than a duplicate.

I have not tested how to inject the argument in any particular agent framework. Check the hook your framework gives you before relying on this.

Five ways to stop the second action

These are the options I would compare, against the same four questions.

Option D has two forms, and both need a closer look.

With return of control in Amazon Bedrock Agents, the application receives the action details with a unique invocationId, runs the action itself and sends the result back with the same invocationId (Bedrock return of control). So the application owns the side effect. The page does not say whether a resent result is deduplicated. Bedrock Agents Classic is also closed to new customers, as an API reference note says as of 2026-11-01 (InvokeAgent).

With Step Functions, the best-practices page puts "all non-idempotent actions" in a Standard workflow, and gives "approving order through human interaction and processing payments" as its examples (Step Functions best practices). Per the docs, in Standard a task is never run more than once unless Retry is specified (Step Functions workflow types). So a payment task with no Retry fails loudly instead of running again. The cost is that the failure ends the execution by default (Step Functions error handling), and someone has to decide what happens next.

If the agent starts that workflow through an AgentCore Gateway tool (AWS What's New, AgentCore Step Functions connector), the count of runs moves up a level. StartExecution is idempotent for Standard only when the name and input match a running execution. If the execution is closed or the input differs, it returns a 400 ExecutionAlreadyExists error. With no name, Step Functions generates a UUID (StartExecution). The connector hides the optional name field unless you expose it (AgentCore Step Functions connector). As I read the docs, a repeated tool call with no name looks like a new execution. So the workflow fixes the order of steps and the count inside one execution. It does not fix the number of executions.

Option E is worth a line. In Bedrock Agents user confirmation, the user chooses CONFIRM or DENY before the function is invoked, and AWS presents it as protection against prompt injection (Bedrock user confirmation). That answers "may this run". It does not answer "did it already run".

A failure-injection test you can run

The goal is one number: side effects per approved action, when the failure comes after the side effect. The refund ledger from earlier is the downstream. The handler below writes a ledger row, then fails on purpose. The Powertools idempotency table needs the default key attribute id. This code is illustrative and untested.

import os
import time
import uuid

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent,
)

FAIL_AFTER_WRITE = os.environ.get("FAIL_AFTER_WRITE", "none")  # none | raise | sleep
CONDITIONAL_WRITE = os.environ.get("CONDITIONAL_WRITE", "off") == "on"
USE_DECORATOR = os.environ.get("USE_DECORATOR", "off") == "on"

persistence = DynamoDBPersistenceLayer(table_name="idempotency")
config = IdempotencyConfig(
    event_key_jmespath="request_id",
    payload_validation_jmespath="amount_minor",
    raise_on_no_idempotency_key=True,
)


def run(event, context):
    row_id = event["request_id"] if CONDITIONAL_WRITE else str(uuid.uuid4())
    create_refund(row_id, event["request_id"], event["order_id"], event["amount_minor"])
    if FAIL_AFTER_WRITE == "raise":
        raise RuntimeError("failed after the write")
    if FAIL_AFTER_WRITE == "sleep":
        time.sleep(context.get_remaining_time_in_millis() / 1000 + 5)
    return {"status": "done"}


@idempotent(persistence_store=persistence, config=config)
def handler_with_decorator(event, context):
    return run(event, context)


def handler_plain(event, context):
    return run(event, context)


handler = handler_with_decorator if USE_DECORATOR else handler_plain

Count the rows for one request after each run. A scan is fine on a small test table.

from boto3.dynamodb.conditions import Attr

rows = ledger.scan(FilterExpression=Attr("request_id").eq("req-001"))["Items"]
assert len(rows) == 1, f"{len(rows)} side effects for one approved action"

Run four injections, each driven by the real retry source:

1. The handler raises after the write, invoked asynchronously with default attempts (Lambda async errors).
2. The handler sleeps past the function timeout, same invoke.
3. The handler returns normally, but the caller gives up first: a Step Functions Task with TimeoutSeconds shorter than the tool and a retrier (Step Functions error handling), or an MCP client with a short timeout (MCP cancellation).
4. An SQS batch of five where record 3 fails, with and without ReportBatchItemFailures (Lambda SQS error handling). Key per record, not per batch. For variant 2 this means moving the decorator to the per-record function.

Run each against four variants:

1. Nothing: USE_DECORATOR=off, CONDITIONAL_WRITE=off.
2. Decorator only: USE_DECORATOR=on.
3. Decorator plus conditional write: both on.
4. Variant 3, then a second call that reuses the key with a different amount_minor.

The pass rule: exactly one row per request_id in every injection, and variant 4 raises on the changed amount. The retry sources are the real services, so this needs an AWS account. Use a test account and delete the stack afterwards.

This is what the docs predict. It is not a measured result.

Variant 4 should raise IdempotencyValidationError (Powertools idempotency). In the SQS case without ReportBatchItemFailures, variant 1 should write the four good records twice (Lambda SQS error handling). With the key per record, variant 3 should hold for all five records. Variant 2 depends on where the decorator sits. In the harness above it wraps the whole handler, so on a batch the key is the batch, and a raise on record 3 should run all five records again. Only a decorator on the function that handles one record, keyed on that record's request_id, could hold the four good ones, and record 3 may still be written twice, as in the raise column. I will not quote counts until the test has run and the numbers are recorded with their date and Region.

If variant 2 writes one row in the raise case, my reading of the Powertools page is wrong for that case, and this article's main claim needs correcting.

If variant 2 writes one row in the sleep case, my reading of how @idempotent handles an in-progress record after a timeout is wrong for that case.

Start with one tool

Open the function behind the riskiest tool your agent has. Read down from the external call and ask whether any code that can throw sits below it. Then invoke the function twice with the same request_id and count what landed downstream. One is a pass. Two means the key is not being checked at the write.

When a different choice is better

• The tool only reads, or its write is idempotent by nature. A PUT of the same state twice leaves the same state. The decorator, or nothing, is enough, and the conditional write adds a write for no gain.
• The provider already deduplicates on a key you control. Pass the key and rely on it. Keep your own record only if you need it for audit.
• A duplicate costs less than the reconcile path. For a low-value, easily reversed action, accepting at-least-once and making duplicates visible can be the right call. I would not make that choice for money.
• You cannot mint a stable key. If you do not control the orchestrator, use option D or a human gate, or the derived key with its collision cost.
• Some things I could not settle from the docs. The pages I read do not say whether Bedrock Agents Classic or AgentCore Gateway retries a Lambda target, or whether a resent invocationId is deduplicated. The Powertools page I used covers Python only. I would test each before relying on it either way.

If you can do only one thing, keep the decorator and make the write for irreversible actions check a key minted before the first attempt.

Sources

Pages were read on 2026-10-26 unless noted, and re-read on 2026-11-01.

AWS Lambda Developer Guide

• How Lambda handles errors and retries with asynchronous invocation: two more attempts, timeouts as errors, 6 hours for throttling, duplicate events.
• Best practices for working with AWS Lambda Event Functions: "Write idempotent code".
• Handling errors for an SQS event source in Lambda: the whole batch visible again, ReportBatchItemFailures.
• Configure Lambda Event Function timeout: the timeout is the maximum time a function can run.

Other AWS documentation

• Amazon SQS visibility timeout, Amazon SQS Developer Guide: 30-second default, redelivery.
• Amazon SQS at-least-once delivery, Amazon SQS Developer Guide, checked 2026-10-16: standard queues can deliver a message again.
• Handling errors in Step Functions workflows, AWS Step Functions Developer Guide: Retry defaults, a retry on timeout invokes the Lambda function again.
• Best practices for Step Functions, AWS Step Functions Developer Guide: non-idempotent actions in Standard workflows.
• Choosing workflow type in Step Functions, AWS Step Functions Developer Guide, checked 2026-10-30: Standard is exactly-once unless Retry is specified.
• StartExecution, AWS Step Functions API Reference, checked 2026-10-30: idempotent for Standard only, name and input, generated UUID.
• Return control to the agent developer by sending elicited information in an InvokeAgent response, Amazon Bedrock User Guide: invocationId.
• Get user confirmation before invoking action group function, Amazon Bedrock User Guide: CONFIRM and DENY.
• InvokeAgent, Amazon Bedrock API Reference, checked 2026-10-20: Bedrock Agents Classic closed to new customers.
• AWS Lambda function targets, Amazon Bedrock AgentCore Developer Guide, checked 2026-10-26: the event is the inputSchema properties; the context IDs.
• AWS Step Functions as Connector Target, Amazon Bedrock AgentCore Developer Guide, checked 2026-10-30: Step Functions workflows as gateway tools, hidden name field.
• AWS Step Functions workflows now available as AI agent tools, AWS What's New, 2026-10-29, checked 2026-10-30.
• Working with items and attributes in DynamoDB and DynamoDB condition expression CLI example, Amazon DynamoDB Developer Guide, checked 2026-10-16: conditional PutItem, attribute_not_exists.
• Idempotency, Powertools for AWS Lambda (Python): records, keys, the two failure modes, timeouts, payload validation.
• Powertools for AWS Lambda (Python) source, idempotency.py, develop branch, checked 2026-11-01: the decorator registers the Lambda context. Source code, not documentation.

Standards and specifications

• Tools and Schema Reference, Model Context Protocol specification, version 2026-07-28: tools/call parameters, annotations as untrusted hints, idempotentHint default.
• Cancellation, Model Context Protocol specification, version 2026-07-28: timeouts, and a cancel arriving after completion.