Most of the governance conversations we have with engineering teams start in the same place: someone got asked by their CISO or a compliance lead to explain what the company's AI agents are actually doing, and they didn't have a good answer. Not because the agents were doing anything wrong, but because no one had built the infrastructure to answer that question.
The reaction is usually to reach for something heavyweight: a formal AI governance policy document, a committee, a vendor assessment questionnaire. None of that is wrong, but it misses the point. Governance for autonomous AI systems is primarily an operational problem, not a documentation problem. The documentation comes after you have the operational foundation in place.
This is what we've learned building Arrakis and working with the teams that have piloted it: a working governance framework for AI agents has three concrete components. Know what each agent can do. Log what it actually does. Have a defined response when behavior falls outside the boundary. Everything else builds on those three.
Start With the Capability Inventory
You cannot govern what you cannot describe. The first step is building a capability inventory for every autonomous agent your organization runs. This is not a product description or a high-level summary. It is a specific list of the tools, APIs, data stores, and external services each agent has the credentials and permissions to reach.
In practice, this inventory reveals gaps that were not visible at deploy time. An agent provisioned for document review might have been given credentials for a data store that includes materials outside its stated scope. Another agent running IT automation tasks might have inherited a service account with write access to production configurations because the account was convenient, not because anyone decided it should have that access.
The inventory exercise is uncomfortable because it surfaces these decisions, but that discomfort is the point. You want to find them now, not when something unexpected happens.
The capability inventory should answer four questions for each agent:
- What tool calls can this agent make, and against which resource targets?
- What data stores can it read, and can it write to any of them?
- What external services can it call, and does any credential it holds have broader permissions than needed?
- What other agents can it invoke or delegate work to, and do those agents inherit its permissions?
The last question is the one most teams miss. In multi-agent architectures, the effective permission set of an orchestrator agent includes everything its sub-agents can do. If your orchestrator can call an agent that has write access to a financial ledger, then the orchestrator effectively has write access to that ledger, even if it was never explicitly granted.
Define Boundaries Before You Discover Them the Hard Way
Once you have the inventory, the next step is defining what each agent should be permitted to do in normal operation. This is different from what it can do (its credentials and technical permissions). The governance layer sits between those two, expressing intent as policy.
Policies do not need to be complex at the start. A useful early policy structure for a document processing agent might be: reads are permitted on the designated document stores, writes are not permitted to any external destination, API calls to external services are blocked unless the destination is on an approved list. That is three rules and it covers the major categories of overreach.
The goal is not to achieve least privilege on day one. For many teams, that is not realistic. The goal is to make the gap visible: here is what the agent can do, here is what it should do, here is the distance between those two things. Once the gap is visible, you can close it incrementally.
Default-deny configurations are the correct long-term target. Starting with an agent that is permitted only what is on an explicit allow list, rather than blocked from a deny list, produces much cleaner policy over time. Deny lists grow without bound as new resources come online. Allow lists are bounded by the task.
Logging Architecture: What to Capture and Why
The logging question gets answered in one of two ways: either you decide what to capture before an incident, or you figure it out after one and realize you didn't capture the right things. We have talked with enough security teams to know that the second path is genuinely painful.
The minimum viable log for an AI agent session needs to capture the following:
The triggering context: what initiated the session, with what parameters, and from what upstream caller. For a human-triggered workflow, this is straightforward. For an agent that runs on a schedule or is called by another agent, the triggering context is what connects the session to a business process. Without it, you cannot reconstruct why an action happened.
The decision sequence: each tool call the agent made, in order, with the inputs it provided and the outputs it received. Not a summary. The actual inputs, including any data from prior tool calls that the agent incorporated into subsequent ones.
The policy outcome: for each action, whether it was permitted, blocked, or conditionally permitted under a policy rule, and which rule applied. If you only log actions without policy outcomes, you cannot distinguish a correctly executed task from a policy violation that was not caught at runtime.
The session boundary: when the session started and ended, and whether it ended cleanly or with an error condition.
Storing this data for 90 days covers the range of most post-incident investigations. For teams under compliance regimes that specify longer retention (SOC 2 Type II readiness often requires one year for relevant access logs), the log schema should be designed for export from the start.
Response Playbooks: Closing the Loop
A governance framework without a defined response process is incomplete. If your monitoring detects a policy violation, something has to happen with that signal. In practice, three types of response are needed.
Real-time enforcement: the policy engine blocks the action before it completes and records the violation. This is the preferred response for any action that could cause immediate harm, such as a write to a production system or a call to an external service that is not on the approved list. Blocking in real time requires the policy engine to be in the call path, which has latency implications your platform team needs to understand before deployment.
Alert and review: the action is permitted to complete, but the event is flagged and routed to a human reviewer. This is appropriate for actions that are unusual but not clearly harmful, where blocking would disrupt a legitimate business workflow. The flag should include enough context for a reviewer to make a judgment without needing to re-run the session.
Retrospective investigation: a pattern of events triggers a review after the fact. This is not a substitute for real-time enforcement, but it catches classes of behavior that only become visible when you look at multiple sessions together. A single tool call to an unusual API endpoint might not be significant. Ten such calls from the same agent over 48 hours is a pattern worth investigating.
What a Governance Framework Is Not
We want to be direct about the scope here: a governance framework for AI agents is not a guarantee that agents will behave safely in all circumstances, and it is not a substitute for sound agent design. If your agents are poorly designed, have vague task boundaries, or rely on overly broad permissions because that was easier than thinking through what they actually need, governance tooling will expose that rather than fix it.
Governance is also not a compliance checkbox. The framework we've described generates the artifacts that compliance reviews will ask for (inventory documentation, policy records, audit logs) but those artifacts are byproducts of an operational practice, not the goal. Teams that build governance primarily to check a box tend to build it in ways that don't actually answer the question "what did our agents do and why."
Building Incrementally
The teams that have the most success with this don't try to govern everything at once. They pick one or two agents that touch sensitive data or external systems and build the full loop for those: inventory, policy, logging, and response. Once the loop works and the team understands what it takes to maintain it, they extend it to more agents.
That approach also gives you a realistic sense of the operational cost. Reviewing policy violations takes time. Investigating flagged sessions takes time. If you instrument 40 agents at once, you get 40 agents worth of review load before you've built the process to handle it. Start with the agents where the stakes are highest and the learning is sharpest.
The agents in scope at the end of that process will have documented capabilities, policy records, and audit logs. When someone asks what your AI agents are doing, you will have a specific answer. That is what governance looks like in practice.
Audit your AI agents with Arrakis.
Arrakis gives security and platform teams a complete audit trail of every action your AI agents take, with policy enforcement that stops overreach before it reaches your data.
Request a Demo