In this guide8 sections
An API key is access, not accountability
A shared API key can prove that a piece of software was allowed through a technical boundary. It usually cannot explain which agent acted, which user initiated the task, which policy permitted the action, or who owns the outcome. If several workflows share one powerful credential, their permissions and audit trails collapse into the same identity.
A production agent needs a distinct non human identity with a named business owner. When it acts on behalf of a person, the system should preserve both identities: the user who requested the work and the agent that executed it. This creates a chain of responsibility without pretending that the model is a person.
- Give each production agent or bounded workflow a distinct service identity.
- Record an owner, purpose, environment, and review date for that identity.
- Keep development, testing, and production credentials separate.
- Make revocation possible without disabling unrelated systems.
Make permission follow the task
An agent that qualifies an inquiry may need to read one record, add structured fields, and assign an owner. It does not need permission to export the customer database, change billing settings, or delete records. The useful boundary is the smallest set of data and actions required for the task.
Separate reading from writing. Separate drafting from sending. Separate a proposed database change from the command that commits it. These distinctions let the system remain useful while reducing the impact of a mistaken or manipulated decision. NIST zero trust guidance applies the same principle to resources: trust should not be inherited from network location, and access should be evaluated for the specific identity and resource.
- Use narrow tool scopes and explicit allowlists for actions.
- Validate parameters before a tool or API receives them.
- Prefer short lived credentials where the platform supports them.
- Review unused permissions and remove them from production.
Treat retrieved content as untrusted input
Prompt injection is not limited to a malicious message typed directly by a user. Instructions can arrive through a document, webpage, email, CRM note, tool response, or content written by another agent. A model may interpret that content as a command even when the application intended it only as data.
The control cannot be a sentence telling the model to ignore bad instructions. The system must distinguish policy from content outside the model. Retrieved text should not be able to expand permissions, select an unapproved destination, reveal a secret, or bypass a required approval. The model may recommend an action, but deterministic code should enforce what actions exist and which parameters are acceptable.
- Label and isolate system policy, user instructions, and retrieved content.
- Limit which tools are available in each workflow stage.
- Sanitize or reject tool arguments that violate business rules.
- Test indirect prompt injection through files and external content, not only chat prompts.
Put human approval at the consequence boundary
Human review should not be added to every harmless step. That recreates the delay automation was meant to remove. It belongs where a decision becomes difficult to reverse or materially affects a customer, employee, payment, permission, contract, or public channel.
A strong approval screen shows the proposed action, the source information, the reason, the affected record, and the exact change. A button that says approve without this context transfers responsibility to a human without giving them a useful decision. OWASP recommends human approval for high impact actions as a mitigation for excessive agency.
- Let the system classify, summarize, draft, and route low risk work.
- Require approval before sensitive external communication or irreversible changes.
- Expire stale approvals so an old decision cannot be released later.
- Provide a safe reject, edit, and escalate path.
Design memory and data boundaries deliberately
Agent memory can quietly become a second database. Conversation history, retrieved documents, embeddings, tool output, traces, and backups may all contain sensitive information. A private interface does not make those stores private by itself.
Define what the agent may retain, for how long, and for which user or business unit. Prevent one customer, team, or workspace from retrieving another tenant’s context. Secrets should not be placed in prompts or long term memory. Logs need useful evidence, but they should not become an uncontrolled copy of every sensitive input and output.
- Classify prompts, outputs, documents, embeddings, logs, and backups.
- Apply tenant and role boundaries to retrieval as well as the primary database.
- Redact secrets and unnecessary personal data before storage.
- Set retention and deletion rules that can be verified.
Build evidence into every consequential action
An audit log should answer a practical question: how did this outcome happen? For consequential steps, preserve the initiating user, agent identity, policy decision, model or workflow version, tool call, important inputs, affected object, approval, result, and timestamp. The objective is not to log everything forever. It is to preserve enough evidence to investigate, improve, and demonstrate control.
Operational metrics matter too. Track failure rate, approval rate, exception rate, latency, cost, manual overrides, and the outcomes the workflow was built to improve. CISA recommends protecting logs from unauthorized change and aligning retention with policy and compliance needs.
- Make security and business events searchable by workflow and identity.
- Protect logs from the same agent they are meant to audit.
- Alert on unusual destinations, volumes, failures, and permission use.
- Review traces without retaining more sensitive content than necessary.
Test the system, not only the model
A model evaluation can show whether classification or extraction is accurate. It cannot prove that permissions, integrations, approvals, and recovery paths work together safely. Production readiness requires tests at the workflow level.
Build a test set from normal work, known exceptions, ambiguous inputs, and adversarial attempts. Run it whenever prompts, models, tools, policies, or connected systems change. Measure whether the system reaches the correct business outcome and whether it fails safely. NIST’s AI Risk Management Framework treats measurement and management as continuing lifecycle activities, not a one time launch exercise.
- Test expected tasks, edge cases, abuse cases, and unavailable dependencies.
- Verify that forbidden actions remain impossible even when the model requests them.
- Confirm that retries cannot duplicate payments, messages, or record changes.
- Compare quality and risk metrics before increasing autonomy.
Prepare the stop path before launch
Every agent with production access needs a contained way to stop. That can include disabling one workflow, revoking one identity, removing one tool, switching to read only mode, or routing all actions to human review. A company should not have to shut down its entire automation layer to contain one problem.
The release decision is simple to state: the agent should have an owner, a bounded purpose, the minimum permissions, tested policy enforcement, useful evidence, and a recovery path. If any one of those is missing, the system may still be a prototype, but it is not ready to operate with meaningful authority.
- Document who can pause the agent and how they will know to do it.
- Keep a manual operating path for critical work.
- Rotate or revoke credentials after a suspected compromise.
- Reassess permissions and controls when the purpose changes.