How to run AI agents in production without burning the budget overnight

The short answer

To run AI agents in production safely, bound the workflow, put human gates where a wrong action is expensive, cap spend per request with a hard kill switch, scope every tool to least privilege, and log every decision so the run is auditable. An agent is not a person you trust with a company card; it is a program that can loop, and anything that can loop needs something that stops the loop by force. The failure mode I refuse to ship is the agent that runs unattended overnight and turns a bug into a bill.

The rest of this page is how I put those controls in place, drawn from a governed Claude Code delivery system I built at ASTM International and from cost and reliability controls on the AI platforms behind it.

Bounded workflows, not open loops

A production agent is a bounded workflow with a defined set of tools and a step ceiling, not an open-ended instruction to figure it out. On the ASTM system the agent follows a fixed loop: read the ticket, design, and application context, confirm intent, write code and tests, run checks with a capped fix loop, then open a pull request. The cap on that fix loop is the whole point. An unbounded retry is exactly how cost and blast radius escape, so the number of attempts, tools, and steps is finite and known before the agent starts.

Human gates at the right points

The best AI workflow still needs human gates, placed where a wrong action is expensive or hard to undo. On the ASTM system, the first gate confirms intent before any code changes, and the second confirms the knowledge-base and session-log updates before the work is treated as complete. Routine steps in between run without a prompt. Gating every single step is its own failure: it trains reviewers to click through without reading. Put the gates on the consequential actions and let the safe ones flow.

Cost ceilings and a kill switch

Every agent request needs a per-request cost ceiling and a hard timeout, enforced in code and not left to the model's judgment. A runaway loop with tool access can spend money fast, so cap every run: meter token and tool cost per request, cut the request the moment it crosses the ceiling, and alarm on it. Keep a kill switch that disables the trigger outright, so a bad loop cannot run up the bill overnight. A semantic cache is a useful partner here: on a RAG platform I built it cut model spend by around 60%, which both lowers the baseline and shrinks the room a bad loop has to run.

Tool-permission boundaries

An agent is only as safe as the tools you hand it, so the boundary is the permission, not the prompt. Give the agent its own least-privilege IAM role, keep read-only tools read-only, and put write and destructive actions behind explicit allow-lists and approval gates. Scoped this way, the worst an agent can do is bounded by what its role permits, regardless of what the model decides or what a malicious input talks it into. I never rely on instruction wording to prevent a dangerous call, because sooner or later something gets past the words.

Observability over every decision

You cannot govern what you cannot see, so log every decision, tool call, input, and cost in a structured, queryable form. When a run goes wrong you need to reconstruct what the agent did and why, the same way you would trace a failed request. On the ASTM system, session logs and pull-request review gates make the agent's work reviewable after the fact; on the RAG platform, returning the source chunks behind every answer turned a black box into something a person can debug. An unexplained agent action deserves the same alarm as an unlogged 500.

Failure modes and security

Once an agent reads untrusted content and holds tools, prompt injection is a real attack surface: treat any retrieved or user-supplied text as hostile input that must never widen the agent's permissions. Keep the tool scopes tight so a successful injection still cannot do much. AI-generated output also needs a review gate before it lands. On Amazon IaC RLHF work I did through Turing, generated infrastructure code was frequently valid but unsafe; with review and feedback in the loop, generation quality rose from 42% to 89% and security violations fell to under 5%. The same discipline applies to any agent: a human or automated check sits between it and anything consequential.

A reference setup on AWS

A pragmatic AWS-native shape: a bounded orchestrator (Step Functions or an equivalent) defines the workflow and the step ceiling, so retries and timeouts are explicit. Each tool is a Lambda with its own scoped IAM role, so permissions are per-tool and least-privilege. A cost meter tallies token and tool spend per request against a hard ceiling, backed by a CloudWatch alarm and a kill switch that disables the trigger. Every decision is written as a structured log to CloudWatch, and human approval steps sit at the expensive or irreversible actions. Nothing here is exotic; it is the same rigor you already apply to production services, pointed at the agent.

Common mistakes

• Running an open-ended agent with no step ceiling or capped retry loop.
• Trusting the prompt to enforce cost limits instead of a hard ceiling in code.
• Giving the agent broad permissions and relying on instructions to stay safe.
• Gating every step, so reviewers click through without reading.
• Shipping with no per-decision logs, so a bad run cannot be reconstructed.
• Letting AI-generated code or actions reach production with no review gate.