Agents and Governance

Prompt Security and Tool Hardening Checklist

A defense-in-depth checklist for prompt injection, untrusted retrieval, tool permissions, argument validation, sensitive data, confirmations, and incident response.

April 12, 20264 min readAI SecurityPrompt Injection

Prompt injection is not solved by adding a sentence that tells the model to ignore malicious instructions. An agent processes content from users, documents, websites, tools, memory, and system integrations. Any of those channels can contain instructions that conflict with the application's policy. The secure design principle is simple: the model can propose, but trusted software decides what is allowed. The rest of this checklist turns that principle into concrete boundaries.

Map trust boundaries before writing prompts

Classify every input by origin and authority:

system and application policy;
authenticated user input;
retrieved enterprise content;
external web or third-party content;
tool output;
conversation memory and generated summaries.

Retrieved content is evidence, not policy. A document that says “ignore previous instructions” must remain data. Tool responses are also untrusted until their schema, source, and authorization context are validated. Keep policy instructions separate from retrieved evidence in the prompt structure. Add provenance to every context item, limit how much untrusted content can enter the model, and avoid concatenating raw HTML, hidden text, or uncontrolled metadata.

Give tools the minimum authority

Tool security starts outside the model. Each tool should expose the narrowest useful operation, with a typed input contract and an identity that can be authorized independently. For every tool, define:

allowed callers and user context;
required and optional arguments;
accepted values and size limits;
read versus write behavior;
idempotency and retry policy;
timeout and rate limits;
data classification;
audit events;
confirmation requirements.

Avoid generic tools such as unrestricted SQL, arbitrary HTTP requests, or shell execution. If a workflow needs one business operation, expose that operation rather than an open execution surface. Authorization must be checked by the tool or trusted application layer, not inferred from a sentence generated by the model. A valid tool name and valid JSON do not imply that the user is allowed to perform the action.

Validate the complete action path

Before a tool call reaches a side effect:

parse the output against a strict schema;
reject unknown fields and unsafe values;
bind the request to the authenticated identity;
recheck resource-level permission;
apply business invariants;
require confirmation for high-impact actions;
attach an idempotency key where retries are possible;
record the decision and result without leaking secrets.

The confirmation must describe the real action, destination, and important consequences. Asking “continue?” after hiding those details does not create meaningful human control. When a tool returns ambiguous or partial data, the safe behavior is usually to stop, clarify, or escalate. The model should not invent a successful outcome.

Protect sensitive data across the workflow

Sensitive data can leak through prompts, tool arguments, traces, cached entries, model outputs, or human review queues. Apply the same data policy to all of those surfaces. Useful controls include:

data minimization before model calls;
DLP or pattern-based inspection where appropriate;
tenant-aware retrieval and cache keys;
encryption and retention controls;
redaction in logs and evaluation datasets;
provider and regional routing policy;
output validation before display or downstream use.

Memory needs explicit scope and expiry. Do not treat conversation history as an unrestricted database. Store only the state required by the product, associate it with the correct user or tenant, and provide a way to invalidate it. My public DAISI case study describes Model Armor, DLP, GDPR safeguards, identity boundaries, and controlled enterprise integrations as separate layers. Those controls do not make an agent “secure by default”; they demonstrate why security must exist across the model, application, data, and infrastructure layers.

Make unsafe outcomes observable

Security telemetry should answer:

which untrusted source influenced the request;
what policy and tool version were active;
why a tool call was allowed, blocked, or escalated;
whether sensitive data was detected or redacted;
whether confirmation was required and obtained;
what side effect occurred;
which fallback was used.

Do not log raw secrets or sensitive content merely to improve diagnosis. Prefer stable identifiers, classifications, hashes where appropriate, and access-controlled trace views. Alerts should prioritize actual policy violations and suspicious action patterns over generic model uncertainty. Repeated blocked tool calls, attempts to cross tenant boundaries, unusual argument payloads, or missing confirmations are stronger security signals than a low confidence score alone. The enterprise AI governance framework shows how to connect these controls to evidence, ownership, and incident decisions.

The model proposes an action, while trusted software validates identity, arguments, policy, approval, and execution authority.

Test adversarial workflows, not isolated prompts

A red-team set should cover direct and indirect prompt injection, malicious documents, poisoned tool output, encoding tricks, conflicting instructions, excessive payloads, permission changes, retry behavior, and multi-step attacks that appear harmless one action at a time. Test the full path with production-like identities and tool permissions. A model refusing in a chat window does not prove that the backend would reject the same action. Conversely, a model may follow an injected instruction while the trusted application correctly blocks the side effect. Both observations matter. Keep the cases as regression tests and rerun them when prompts, models, parsers, tools, identity rules, or retrieval sources change. Review evaluators carefully so a model judge cannot approve the same unsafe behavior it is supposed to detect.

Prepare a safe operational response

The system needs a documented way to disable a tool, revoke a credential, block a source, reduce capabilities, or route traffic to a read-only fallback. Preserve the evidence required for investigation and identify who owns the application, integration, identity, data, and incident decision. No checklist eliminates all prompt injection risk. A resilient system assumes that untrusted instructions will reach the model and ensures they cannot silently become unauthorized actions.

Sources and references

OWASP Top 10 for LLM ApplicationsThreat categories for LLM applications and agentic systems
NIST AI Risk Management FrameworkGovernance and risk-management principles for AI systems
Microsoft AI red teaming guidanceOperational guidance for adversarial testing of AI applications

From principles to shipped systems

These articles document the methods behind my work. The project case studies show how I apply them across enterprise agents, RAG, and MLOps.

Explore project case studies View my CV

Continue exploring

Related field notes on the architecture, evaluation, and operating decisions behind production AI systems.

April 13, 20264 min readAI EvaluationRegression Testing