AI Governance: What It Is and Where It Gets Enforced

AI governance is the set of policies, responsibilities and technical controls that decide what an organization's AI systems can access, say and do, and how that can be checked afterwards. It answers two practical questions: who decides how AI gets used, and how you prove those decisions held.

Most guides to AI governance stop at the first of those questions, with principles, committees and policy documents. That's the right place to start. A policy on its own doesn't stop anything, though. In a running GenAI or agent deployment, a rule only takes effect when the platform applies it to every request, from the documents a user's question can retrieve to the actions an agent is allowed to take. We write from the deployment side, so this guide spends most of its time there.

Key Takeaways

• AI governance is the set of policies, responsibilities and technical controls that decide what AI systems can access, say and do.
• Policy defines the rules. In GenAI and agent deployments, the platform has to enforce them on every request.
• Five areas carry most of the weight in a live system: data and privacy, security, guardrails, accuracy and trust, and agents.
• Eight questions, listed below, show whether a system enforces governance or only describes it.

What Is AI Governance in Simple Terms?

In simple terms, AI governance is how an organization decides what its AI is allowed to do and then makes sure it does only that. It has two layers. The policy layer is what people decide. The enforcement layer is what the system does on every request, whether anyone is watching or not.

The policy layer is familiar territory. It's the AI policy itself, the list of approved use cases, a risk assessment for each one, and the committee or named owner who signs off. Most organizations we work with already have some version of this, often adapted from existing information security and model risk processes.

The enforcement layer is newer, and it's where GenAI changes things.

Take a typical policy line: "Employees may only access information they are authorized to see." For a file share, folder permissions take care of that. For an AI assistant that reads across SharePoint, email archives and a document management system, the same line has to hold for every answer. The assistant has to check, at the moment of each question, what this particular person is allowed to see.

Neither layer works on its own. A policy nobody enforces is a statement of intent, and a control nobody decided on is a setting that nobody owns.

How Is AI Governance Different from Data Governance?

Data governance manages the data itself: its quality, lineage, ownership and who may access it. AI governance manages what an AI system does with that data, including what it reads on a user's behalf, what it answers and what actions it takes. The two meet at access, and that overlap is where most GenAI exposure begins.

Data governance is the older discipline, with its own tools and owners, and AI governance depends on it. If the data underneath is messy or shared too widely, the AI repeats those problems in its answers.

What GenAI adds is a new kind of reader. An assistant that retrieves documents to answer questions reads on behalf of whoever is asking. Suppose the index it searches was built with one shared set of permissions. A junior analyst's question can then surface a board paper the analyst could never have opened directly because permissions simply didn't carry across into the AI.

The FINOS AI Governance Framework, an open-source effort from the financial services community, lists preserving source data access controls in AI systems as a mitigation in its own right. We cover the practical side in our guide to protecting customer data in enterprise GenAI applications.

Why Is AI Governance Important?

AI governance matters because GenAI changes what can go wrong. A model that reads company documents can expose them, and an agent that takes actions can take the wrong one. Governance is how an organization decides which of those risks it accepts, and how it shows a regulator, an auditor or its own board that the controls held.

Particularly for conservative organizations, governance is usually where adoption slows. The demo goes well, then the questions start. Where does the data go? Who can see what? What happens when the answer is wrong? In our experience, more pilots pause over those questions than over model quality.

The caution is well founded. IBM's Cost of a Data Breach Report 2025, which studied 600 breached organizations, found that 63% lacked an AI governance policy or were still working on one. Among the organizations that reported an AI-related breach, 97% said they lacked proper AI access controls.

Governance also pays off over time: once the controls are approved for one use case, the next one doesn't have to win the same arguments from scratch.

What Is an AI Governance Framework?

An AI governance framework is the structure an organization uses to run AI governance: who owns each AI use case, how risk is assessed, which policies apply, and how systems are monitored once they're live. Most build theirs from published reference points.

Whatever the starting point, the frameworks we see tend to share a handful of components:

• An inventory of AI use cases, each with a named owner
• A risk classification for each use case
• Policies covering acceptable use, data handling and human review
• Monitoring once systems are live, with a route for reporting incidents
• Oversight of third-party AI services, including the models themselves

You'll also come across lists of AI governance "pillars," usually some mix of accountability, transparency, privacy, security, safety and fairness. They work as a checklist of themes. There's no single standard list, though, and frameworks group the themes differently, so be wary of any article that presents one list as definitive.

Three published reference points come up in almost every conversation we have with regulated organizations:

Sector supervisors like FINMA and DORA add their own expectations, best read at the source. On where familiar security certifications leave gaps for GenAI, see The Questions SOC 2 Doesn't Answer.

Why Is AI Governance Hard to Implement?

AI governance is hard to implement because policies are usually written about AI models, while most GenAI risk sits in the system around the model, in what it retrieves on a user's behalf and what it's allowed to do. Closing that gap means turning policy statements into controls the platform enforces, which is unfamiliar work for many governance teams.

The pattern we see most often has four parts:

• Policies describe models. Many AI policies grew out of model risk management, which validates a model and watches for drift. A GenAI assistant's riskiest moments tend to come from the documents it reads and the actions it takes.
• Permissions live everywhere. Access rules sit in dozens of systems, and an AI system that reads across them has to respect all of them at once.
• Agents move faster than review. Nobody can approve every step of an agent that runs unattended overnight. Oversight has to be designed in, with clear rules on which actions need a person.
• Evidence is scattered. When an auditor asks why the AI gave a particular answer, the pieces often sit in different tools owned by different teams.

This is also where the term AI model governance causes some confusion. In the classic sense, model governance covers how a model is chosen, validated and monitored. For a GenAI system, that's necessary and only part of the picture. Enterprise-grade LLM governance adds the system around the model: which model handles which task, what it can read, what it can do, and whether you can change the model without rebuilding everything else.

How AI Governance Is Enforced in a Running System

In a live AI system, governance comes down to five areas. Each answers a question a security or risk team will ask before go-live, and each has a deeper guide in this series.

Data and Privacy: What Can the AI See, and Where Does Your Data Go?

Enforcement here means checking permissions at the moment of each question, for that user, against the source systems' own rules. It also means masking personal data before it reaches an external model provider. The deployment model then decides where the model, the search indexes, the logs and the backups physically sit.

In Squirro: Squirro applies project, user and group permissions at both index time and query time. Search and chat results are designed to include only what the signed-in user is allowed to see. For supported connectors, such as SharePoint, the source system's own access lists are synced into the index. An optional Privacy Layer masks personal data before any call to an external model and restores it in the answer. In VPC, on-premises and air-gapped deployments, every component stays inside your own infrastructure, including the model if you run one in-house.

Read more: AI data protection and fine-grained access control · Six questions to ask about sovereign AI infrastructure

Security: What Stops a Document from Giving the AI Instructions?

The security risk most specific to GenAI is prompt injection: text the AI reads, in a question or inside a document, that tries to override its instructions. The harder version hides the instruction in content the AI retrieves on someone's behalf. Researchers have already shown this working against a widely used enterprise AI assistant, where a crafted email could expose data without the recipient ever opening it.

Enforcement starts with treating everything the AI retrieves as untrusted input, and with limiting what the AI can do if an instruction gets through. An assistant that can only read and cite is a much smaller target than an agent that can send email.

In Squirro: Agents and tools are scoped by configuration, so an agent can only use the models and tools its project allows, and the guardrails described below add input filtering on top. On the organizational side, Squirro is ISO 27001 certified and holds SOC 2 Type I, with Type II in progress. Current status is on our Security and Trust Center.

Read more: AI agent security: why the threat isn't at the perimeter

Guardrails: Are the Rules Enforced on Every Request?

Guardrails are the rules on what an AI system will and won't answer or do, applied automatically to every request. They typically cover what comes in (input filtering), what's in scope (topic restrictions), what goes out (output validation) and the handling of personal data. The test is whether the platform enforces them on every request, or whether they exist only in a policy document.

In Squirro: Squirro's guardrails management, available as an add-on, supports input filtering, topic restriction, output validation and PII checks. Rules can be set at deployment, project or agent level. A customer-facing assistant and an internal research tool can therefore run under different rules on the same platform.

Read more: A CISO's guide to AI safety guardrails

Accuracy and Trust: Can Every Answer Be Traced Back to Its Source?

Traceability means every answer points to the passages it relied on, a reviewer can open them, and the system declines when no approved source supports one. Behind each answer sits a record of who asked what, and which model responded.

Hallucination is usually discussed as a question of model quality. It's also a traceability question, because a wrong answer with a citation can be caught in seconds by the person reading it.

In Squirro: Squirro retrieves at paragraph level, so answers are designed to cite the specific passages they used, and users can click through to the source. Agents are configured to answer only from retrieved content and to decline when nothing approved supports an answer. If your organization maintains a taxonomy or knowledge graph, Squirro Graphite can use it to sharpen classification and retrieval. It helps where it exists, and you don't need one to start.

Activity is logged and can flow into your own security monitoring. And because Squirro works with a wide range of models, you can change the model without rebuilding everything around it.

Read more: AI grounding: the hidden infrastructure behind trustworthy enterprise AI · AI audit trails: what happens when the regulator calls

Agents: Can You Limit What an Agent Does and Review What It Did?

Agents shift governance from what the AI can say to what it can do. An assistant that answers a question can be wrong. An agent that sends an email or updates a record can be wrong in a way that's harder to undo. Governing agents means limiting what each one can do, deciding which actions need a person's approval, and keeping a record of what happened.

In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027. Inadequate risk controls are one of the three reasons it gives, alongside escalating costs and unclear business value.

In Squirro: Each agent runs under its project's limits on which models and tools it can use. An agent's own configuration can narrow those limits and can never widen them. In Squirro, approval rules are set per tool. Each tool can be allowed to run, refused, or paused until a person approves it in the conversation. Only the central project policy can auto-approve an action, and an individual task can only tighten the rules. In unattended runs, an action that needs approval and has no rule granting it is refused. In practice, a search can run freely while an outgoing email waits for a person.

Read more: AI agent governance: why auditability is the foundation of agentic AI · Enterprise AI architecture: how your first AI agent creates most value

AI Governance at a Glance: Eight Questions Your AI System Should Be Able to Answer

If you take one thing from this guide into your next platform review, make it this list. Each question shows whether a system enforces governance or only describes it. Ask them of any AI platform you're evaluating, ours included.

1. Does the AI respect the same document permissions as the person asking? If it doesn't, the AI becomes a way around your existing access controls.
2. Do you know what data leaves your environment when someone queries the AI, and where it goes? Ask about the model, the indexes, the logs and the backups separately.
3. Can every answer be traced back to the source documents it used? A citation is what lets a reviewer check an answer quickly.
4. Are the rules on what the AI will and won't answer or do enforced on every request? A rule in a policy document only counts once the platform applies it.
5. What stops text inside a document from giving the AI instructions? This is prompt injection, the security risk most specific to GenAI.
6. If an AI agent can take actions, can you limit what it does and review what it did? Look for per-tool limits and a clear point where a person approves.
7. Is there a log of who asked what, what was retrieved and what was returned? Ask how long it's kept and who can change it.
8. Can you show which model produced an answer, and change it without rebuilding everything around it? Models change often, and your controls shouldn't have to be rebuilt each time.

How Do You Implement AI Governance?

Start small and concrete. Implementing AI governance usually begins with knowing which AI use cases you have and who owns them. From there, the work is turning each policy into something the platform enforces and you can test.

1. List your AI use cases and give each one a named owner.
2. Classify each use case by risk, so the controls match the stakes.
3. Write policies as testable statements. "Staff see only what they're cleared to see" can be tested. "AI should be used responsibly" can't.
4. Check that the platform enforces each policy, using the eight questions above.
5. Ask for evidence during the pilot, such as citations and logs, before you approve production.
6. Put approval gates on agent actions that are hard to undo.
7. Review regularly, because use cases, models and rules all change.

On ownership, the split we see working most often is simple. Policy owners in risk, compliance and legal decide the rules. Platform owners, usually the CISO, CTO or Head of AI, make sure the system enforces them. One global industrial manufacturer we work with co-governs its AI program with us through quarterly steering committees that bring both sides to the same table. For finding where existing policies fall short, see our piece on compliance gap analysis.

Before your next AI review, pick one use case and run it against the eight questions. The answers you can't get yet tell you where to start.

See the Eight Questions Answered on a Working System

The quickest way to judge any AI platform is to watch it answer these questions for itself. In a demo, we'll walk you through a governed Squirro deployment. Two people ask the same question and get different results, because each one only sees what they're cleared to see. Every answer cites passages you can open at the source. An out-of-scope request gets declined, and an agent pauses for a person's approval before it acts.

Bring the governance question your security team asks most often. We're happy to work through it with you.

Book a Demo