Agent governance in a regulated industry is the operating system for deciding who an AI agent is, what job it may perform, which data and systems it may use, which actions it may take, when a person must intervene, and what evidence remains after the run.
It doesn't replace the organisation's existing risk, privacy, security, model-governance or compliance programme. It makes those decisions enforceable when an agent starts acting inside a live business process.
This guide answers the questions that security, risk, compliance, technology and business owners most often need resolved before an agent receives production authority.
Why do regulated industries need agent-specific governance?
Agents combine capabilities that enterprises already govern separately. They can authenticate, retrieve sensitive data, interpret changing context, choose tools, update systems and continue without a person approving every step.
That changes the unit of control. A model response is no longer the whole event. The organisation has to govern a run that may cross several systems and produce a business outcome.
The regulatory context varies by sector and jurisdiction, but the operational concerns converge. EIOPA's insurance-sector opinion highlights risk-based governance, record-keeping, fairness, cyber security, explainability and human oversight. Australia's APRA CPS 230 requires regulated entities to manage operational risk, critical operations and service-provider risk. ISO/IEC 42001 establishes an organisation-wide AI management system rather than a product-level compliance badge.
None of these sources says that buying an agent platform makes an organisation compliant. They explain why ownership, proportionate controls, operational resilience and reliable evidence matter.
What should be governed: the model, the agent or the use case?
All three, but they answer different questions.
| Governance layer | Primary question | Example evidence |
|---|---|---|
| Use case | Why is the system being used, who owns it and what outcome is approved? | Intended purpose, owner, risk classification, affected process |
| Model | Which model is used and is it suitable for the task? | Model version, evaluation results, provider terms, limitations |
| Agent | What may this acting workload access and do during this run? | Identity, delegated authority, tool calls, policy results, actions and fallbacks |
| Business process | Did the governed run produce the correct operational outcome? | Claim, case, payment, ticket or record in the system of record |
Model governance doesn't disappear when agents arrive. It simply doesn't answer every question created by an acting workload.
Does every agent need its own identity?
Every production agent should be distinguishable at the level at which the enterprise assigns authority, investigates activity and revokes access.
A shared integration credential may be technically convenient, but it doesn't tell a reviewer which agent, version, deployment or delegated person caused an action. The evidence chain should separate:
- The person or system that requested the job
- The agent that selected and attempted the action
- The business owner accountable for the use case
- The integration identity used to reach the target system
- The customer, claim, account, order or other resource affected
NIST's 2026 software and AI agent identity and authorisation concept paper frames agent identity and authority as a distinct implementation problem. Identity is the anchor for control. It isn't a substitute for control.
How much authority should an agent receive?
Grant the smallest authority that can complete the approved job, then expand it using observed evidence.
"Can access the claims system" isn't a useful permission. A testable authority envelope states:
- Which agent and version may act
- For which business purpose
- In which environment and organisation
- On which customers, claims, records or data classes
- Through which tools and operations
- With which argument, amount, volume or time limits
- Under which person's delegated context
- Which actions require review
- When the authority expires and how it is revoked
For example, an underwriting support agent might read documents attached to a submission, extract named fields and request missing information. That authority doesn't automatically permit it to change coverage terms, bind a policy or browse unrelated accounts.
Is existing IAM enough to govern an agent?
IAM is necessary, but it usually answers only part of the problem.
Existing identity and access management can authenticate a workload, issue credentials and restrict access to applications or APIs. Agent governance must also bind that access to a specific job, evaluate the resolved business action, handle changing context, route exceptions and preserve the complete delegation chain.
Keep the target system's controls. Add an agent-specific boundary rather than replacing the system of record or treating a broad service account as business approval.
Do humans need to approve every agent action?
No. Regulated doesn't mean manual approval of every tool call.
The useful model is bounded autonomy. Routine, reversible and lower-impact actions can be approved in advance. Human review is reserved for consequences, exceptions or ambiguity.
| Action class | Typical treatment | Example |
|---|---|---|
| Read-only, low sensitivity | Pre-approved with scoped access and evidence | Retrieve the status of one bound claim |
| Reversible operational change | Pre-approved inside field and value limits | Add a verified document reference to a case |
| Consequential but rule-bounded | Pre-approved below a threshold | Issue an eligible refund below $100 |
| High impact, ambiguous or difficult to reverse | Human review before execution | Change coverage, approve a large payment or reject a claim |
| Outside purpose or authority | Block or stop | Access an unrelated customer's record |
Approval itself needs governance. The reviewer should see the proposed action, relevant before-and-after state, policy trigger, expected effect and alternatives. Old approvals should expire, and current state should be rechecked before execution.
How should policy be enforced during a run?
The agent may propose an action, but a control outside the model should decide whether that resolved action may proceed.
The decision should include the agent, requesting principal, approved job, target resource, exact arguments, current business state, data classes, policy version and relevant prior steps. It may return:
- Allow: the action is inside the current authority.
- Transform: remove or mask protected information before continuing.
- Narrow: permit only the approved part of the request.
- Review: hold the action for an accountable person.
- Block: prevent the target system from changing.
- Stop: terminate the run because the control path is uncertain.
A prompt isn't the enforcement point. The agent shouldn't be able to grant itself a broader tool, modify the applicable policy or reinterpret a denial as permission.
What if the action is permitted but the purpose has changed?
Authorisation should be evaluated for this action in this context, not treated as a permanent yes or no for a tool.
An agent may be allowed to use a CRM lookup for an active support case. The same lookup shouldn't automatically be allowed to assemble a marketing list. A claims agent may be authorised to prepare a payment for an eligible claim. The same payment operation can require review when the eligibility condition failed or the agent is attempting a discretionary settlement.
This is purpose-bound enforcement. The decision uses grounded context such as the original request, bound case, verified system state, tool arguments and action sequence. It shouldn't trust the agent's own explanation of its intent.
Read AI Agent Enforcement Has an Intent Problem for the complete decision model.
How should sensitive data be controlled?
Map every boundary the data crosses, then apply the configured rule before each destination that shouldn't receive the original value.
An insurance workflow might require a customer identifier for a policy-system lookup while the model only needs the coverage status, incident facts and permitted next steps. The evidence store may need a protected reference rather than a second copy of the customer's personal information.
For each step, record:
- Which data entered the run and from which authoritative source.
- Which destination received it.
- Why that destination required it.
- Which fields were removed, masked, tokenised, generalised or blocked.
- What information was retained as evidence.
Test false negatives and false positives with realistic formats, languages, attachments and structured tool arguments. "PII redaction supported" isn't enough. Buyers should inspect what the model, tool, customer response and evidence record each receive.
Does the agent hold its own credentials?
The stronger question isn't how contained the runtime is. It is whether the agent can reach a system at all without a control deciding first.
An agent that holds no credential of its own can't open a connection, call an API or write to a system directly. Every action it proposes has to pass through a gateway that holds the credential, decides against policy, and acts on the agent's behalf. That gateway should be unreachable from the internet, so nothing outside the platform, including the agent, can call it directly. Network egress policy belongs at the gateway too: it should read the approved destination from configuration at the moment of use, not from the agent's own claim about where it needs to go. A permissive connector still causes harm even inside a well-contained runtime, so the credential boundary matters more than the runtime boundary.
The review should answer:
- Does the agent ever hold a credential it could use to reach a system directly?
- Which gateway decides each proposed action, and can that decision be bypassed?
- Which outbound destinations does the gateway's egress policy allow, and is that policy read from configuration at the moment of use?
- What happens to queued or in-flight work when the agent's authority is revoked?
- What evidence shows that a prohibited action was blocked rather than attempted and silently dropped?
What should an AI agent audit trail contain?
A useful audit trail reconstructs authority, decision and outcome. It isn't simply a transcript or a list of model calls.
The minimum run record should connect:
- Use case, intended purpose and accountable owner
- Requesting principal and agent identity
- Agent, policy, tool and model versions
- Delegated authority and credential reference
- Relevant input context and protected-data handling
- Systems, records and tools reached
- Actions attempted, allowed, changed, blocked and completed
- Human approval, rejection or override
- Fallback and final business outcome
- Timestamps, correlation identifiers and integrity controls
The record should distinguish an attempted action from a completed action. It should also minimise sensitive data and restrict who can inspect or export the evidence.
What happens when the governance service or a dependency fails?
Failure behaviour must be designed before production.
For each dependency, decide whether the run should fail closed, switch to read-only work, retry safely, route to a person or stop. Consequential actions generally need stricter failure behaviour than low-risk retrieval.
Test:
- Policy service unavailable or slow
- Identity or delegation can't be verified
- Target system times out after receiving a request
- Tool returns incomplete or contradictory data
- Human reviewer doesn't respond before expiry
- Agent retries an action that may already have completed
- Revocation occurs while a run is active
Idempotency, timeouts, revalidation and a visible degraded state are part of governance because they determine what the system actually does under pressure.
How should third-party agents and MCP tools be governed?
Govern the integration boundary even when the enterprise did not build every component.
Register the agent runtime and MCP server, verify ownership, approve the exposed tools, restrict discovery by agent and job, validate resolved arguments, use scoped credentials, limit network paths and monitor version changes.
When one agent delegates to another, the child shouldn't silently receive broader authority. Preserve the parent job, delegation, identities and final target-system action across the chain.
Does agent governance make the organisation compliant?
No. Agent governance can provide operational controls and evidence that support a compliance programme. It can't determine legal scope, assign organisational accountability, certify a management system or replace sector-specific obligations.
For regulated teams, start with the rules already attached to the process, data and outcome. Then assess AI-specific overlays such as the EU AI Act and management-system expectations such as ISO/IEC 42001.
The IMDA Model AI Governance Framework for Agentic AI is a useful current reference because it emphasises proportionate controls, meaningful human oversight, technical controls and lifecycle management. It remains guidance, not a universal compliance checklist.
Who should own agent governance?
One named business or operational owner should remain accountable for the agent's purpose, authority and outcome. Control ownership is then shared across established functions.
| Stakeholder | Decision they should own |
|---|---|
| Business owner | Intended outcome, acceptable operating boundary and realised value |
| Technology or platform | Runtime, integrations, reliability and lifecycle operation |
| Security and IAM | Identity, credentials, permissions, the credential boundary and response |
| Privacy and data | Permitted data use, minimisation, retention and protected destinations |
| Risk and compliance | Risk classification, control expectations and evidence needs |
| Human reviewer | Specific escalated decision within recorded authority |
| Internal audit | Independent assessment of control design and operating evidence |
An agent with no active owner shouldn't keep standing production authority.
Where should a regulated organisation start?
Start with one bounded workflow, not an enterprise-wide agent programme.
Choose a job with:
- A named owner
- A measurable business outcome
- Known systems of record
- A small number of consequential actions
- A clear human fallback
- Enough transaction volume to learn
- An impact level the organisation can contain
Then run this sequence:
- Record the intended purpose and prohibited uses.
- Assign agent identity, owner and delegated context from the person the agent acts for.
- Define the minimum useful authority.
- Map data and execution boundaries.
- Configure action-time policy and failure behaviour.
- Test permitted, prohibited, ambiguous and degraded paths.
- Review the evidence with business, security, risk and operations.
- Expand volume or authority only when outcomes and control evidence support it.
For insurance teams, an effective first workflow is usually assistance around an existing claim, submission or service case rather than autonomous underwriting, pricing or claim denial. The agent can create value while the highest-impact decisions remain outside its initial authority.
How should a platform be evaluated?
Do not evaluate agent governance through a generic feature checklist. Run the same consequential workflow through every option.
Require the vendor to demonstrate:
- A distinct agent identity and accountable owner.
- One permitted action and one adjacent prohibited action.
- Sensitive data transformed before a restricted destination.
- A contextual action that receives a different result when the purpose changes.
- A credential boundary: evidence that the agent can't reach a system without the gateway deciding first.
- A human fallback that doesn't expose internal controls to the customer.
- Revocation during an active or queued run.
- One evidence record that reconstructs the full outcome.
- Failure behaviour when policy or a target system is unavailable.
- The exact boundary of what is generally available, preview, roadmap or partner-delivered.
The platform should fit the enterprise's existing systems of record. Replacing the claims, policy, CRM or banking system isn't a prerequisite for governing the agent that acts through it.
Frequently asked questions
What is AI agent governance in a regulated industry?
It is the system of ownership, identity, authority, runtime controls, human oversight and evidence used to keep acting AI workloads within the business and regulatory boundary approved for their job.
Is agent governance the same as model governance?
No. Model governance assesses the model and its lifecycle. Agent governance also controls which workload is acting, its delegated authority, tools, data, business-system actions, fallbacks and evidence across the run.
Do regulated agents need a human in the loop?
Not for every action. Routine lower-impact work can be pre-approved inside explicit limits. Consequential, ambiguous, exceptional or difficult-to-reverse actions should follow a defined human-review path.
Can IAM govern AI agents?
IAM provides essential identity, credentials and access controls. Agent governance adds job-specific authority, action-time context, tool and argument validation, fallbacks and evidence linking the delegated purpose to the business outcome.
What evidence should be retained for an AI agent?
Retain the use case, owner, requesting principal, agent identity, active authority and policy, relevant inputs, data handling, systems reached, attempted and completed actions, human decisions, fallback and final outcome.
How do you prevent an agent from exceeding its authority?
Keep the security boundary outside the model. Scope credentials and tools, evaluate each consequential action against current job context, restrict execution and network access, define review paths and support immediate revocation.
Does an agent governance platform guarantee compliance?
No. It can provide technical controls and evidence that support legal, risk, audit and management-system work. Compliance depends on the organisation, use case, role, jurisdiction and complete control environment.
Turn policy into an operating boundary
The decisive question isn't whether the agent can perform the task. It is whether the organisation can define its authority, enforce that authority when conditions change, recover safely when something fails and record the resulting business outcome.
Explore the Difinity.ai platform or use the AI agent governance platform buyer guide to evaluate one real workflow.
