Governing Employee Use of Public LLM APIs
Employees are already running company data through public LLM APIs, and IT usually finds out after the fact, if at all. Legal teams paste contract language into ChatGPT, while engineers drop stack traces and source code into Claude. Finance analysts run projections through Gemini, hunting for a faster way to build a forecast. None of this happens because people are careless; it happens because the tools are fast, free at the point of use, and nobody stopped them at the door. Governing this behavior takes a real control layer, not a memo, and that's the case this piece is going to make.
Why acceptable-use policies fail as the primary control
Most companies responded to the AI wave the way companies respond to most things: they wrote a policy. Somewhere in a shared drive sits a PDF telling employees not to paste confidential data into public AI tools. Almost none of these companies paired that document with any way to check whether anyone actually follows it.
That's the flaw, structurally speaking. A policy assumes the employee already knows what counts as sensitive in an AI context, without being told outright, and most don't. A customer's first name feels harmless until you remember it's sitting next to a purchase history and a support ticket, and together those add up to something a privacy officer would flag as regulated data. Policies also can't see anything, since there's no audit trail behind a Google Doc that says "be careful," and no way to catch an employee routing around the approved tool with a personal account on their phone. Telling someone not to share confidential data with a public LLM is a sentence, and a sentence has no hands, no logs, no idea what just happened.
Outright bans do worse. One fintech I spoke with banned AI tools outright and watched employees route around the ban with personal accounts, mobile apps, and AI features quietly baked into SaaS products they already used. Ban the tool and the behavior doesn't stop; it just moves somewhere IT can't see it. A healthcare SaaS company I know gave employees a sanctioned, working alternative instead of a ban and saw unauthorized use drop within weeks. People don't actually want the shadow tool. They want the fastest tool in reach, and if the sanctioned one is fast enough, they'll take it without complaint.
Then there's the part that breaks the policy model completely: agents. An autonomous agent chaining API calls across five systems doesn't pause to read the acceptable-use policy before its next tool call, and it doesn't take a lunch break during which someone reminds it about data classification. It just acts, at machine speed, and every system it touches becomes part of the compliance perimeter. A minor policy gap that would earn a human employee a talking-to can cascade through an entire agent chain before anyone notices. Every "AI governance in place" claim I've heard from a security lead this year turned out, on closer inspection, to describe controls built for humans clicking buttons, not software making its own decisions.
What a structured governance layer for public LLM API use actually covers
A governance layer isn't a document, and it isn't a procurement checklist either, no matter how many vendor questions it contains. It's an enforced system with parts that talk to each other: policy, risk assessment, compliance mapping, technical controls, and ongoing monitoring.
For public LLM API use, four pieces do most of the work. Identity-bound access decides who can use which tool, under what conditions, checked against a live directory rather than a spreadsheet someone updates twice a year. Data egress policy decides what categories of information can leave the building through an AI interface, enforced actively rather than described politely in a wiki page. Real-time threat detection catches prompt injection, exposed secrets, and PII sitting in a prompt before it ever reaches the model. Audit logging keeps a searchable record of who used what, why, and at what cost, and shadow AI detection finds the tools nobody approved, running on personal accounts or bolted invisibly onto software the company already pays for.
These aren't separate projects sitting on separate roadmaps. Identity feeds the access decision, access control decides what egress rules apply, logging records what actually got through, and shadow AI detection tells you what your policy missed entirely. Pull one piece out and the rest gets weaker; that's the whole point of calling it a layer instead of a list.
This sits between the employee and the external LLM API, as infrastructure, the same way a load balancer or a firewall is infrastructure rather than something an admin configures once and revisits at quarterly reviews. It's the same logic that produced API gateways for web services years back, and CASB tools for SaaS after that: a company can't inspect every vendor by hand, so it builds a chokepoint that inspects traffic on the way through. AI needs the same chokepoint, and the industry just took a minute to build one.
Tying LLM access to existing identity providers instead of managing it separately
Nothing else here works without this piece. You can't enforce a data egress rule, and you can't write a useful audit log entry, if you don't know who sent the request in the first place.
The setup isn't exotic. Single sign-on through whatever identity provider a company already runs, Okta, Entra ID, whatever handles SAML or OIDC today, authenticates the employee before any call to an LLM API goes through, with no separate AI password to remember and no separate credential sitting in a spreadsheet waiting to leak. Role-based access control then maps permissions to job function. A finance analyst gets a code-assist model but not a general-purpose API with no filter on what comes back out. A developer gets broader model access, but code-specific rules apply to what leaves through that channel. A contractor gets scoped, time-limited access instead of the same permissions as a ten-year employee, because a contract ends and access should end with it, on the same day, not whenever someone remembers to revoke it.
Sorting tools into Fully Approved, Limited-Use, and Prohibited gives employees something they can reason about day to day. But the tiering only works if the identity layer applies the right bucket automatically, based on who's logged in, instead of trusting the employee to remember which bucket their tool sits in. The enterprise tiers of the big platforms, ChatGPT Enterprise, Claude for Enterprise, Gemini for Workspace, Microsoft Copilot inside M365, already ship with SSO, admin console controls, and data-protection terms that consumer accounts don't get. Those features sit dormant, though, until someone wires them into a central identity system that applies them consistently, not just to the employees who bothered to sign up correctly.
Agents need this too. An agent or an MCP server calling an external LLM API on a hardcoded key is a blank check with no name on it. Give the agent a scoped, verifiable credential instead, and its actions become attributable and revocable, the same way a person's actions are. Skip this step, and every audit log you produce later is a log of API keys, not people, which is forensically useless the day something actually goes wrong.
Enforcing data egress policy at the point where prompts leave the organization
The exposure point is the prompt itself. The moment an employee pastes a contract clause or a customer record into a public LLM, that data has left the company's control boundary, and it doesn't matter what happens to the response after that.
One payments company I talked to found that a large share of what employees uploaded to public generative AI tools contained PII or payment data, and most of that traffic started on personal accounts sitting completely outside IT's field of view. Classification has to come first: an employee can't make a good call about what's safe to paste if nobody told them, in concrete terms, which categories are off-limits. And that classification only means something once it's wired into enforcement, not filed away in a policy appendix nobody opens.
At the technical level, egress enforcement means machine-learning-based detection of PII and PHI in outbound prompts, since regex alone misses too much and flags too much. It means scanning for secrets, API keys, credentials, connection strings, before they leave inside a prompt or a pasted code snippet. It means checking content against whatever categories the company has already defined as restricted, and responding in proportion: block, redact, or alert depending on severity, rather than one blunt block-everything rule that trains employees to find a workaround within a week.
Source code, legal documents, and financial projections made up most of the sensitive data one SaaS security vendor found exposed in enterprise AI prompts during a 2025 audit, and each needs its own rule set, not a generic reminder to use good judgment. Hardcoded secrets showing up in AI-assisted code commits have climbed sharply as coding assistants spread through engineering teams, so secrets leakage is now something the governance layer owns outright, not a habit developers are supposed to maintain on their own between sprints. The 2025 OWASP LLM Top 10 places Sensitive Information Disclosure and Supply Chain Vulnerabilities right behind Prompt Injection, and egress controls built at the infrastructure layer handle all three at once.
Detecting prompt injection and other runtime threats that egress rules alone don't catch
Prompt injection doesn't look like the threats egress controls were built to catch. The attacker's payload is a sentence, not a virus, and it arrives hidden inside a document, a webpage, an email, or a code comment that an agent was designed to read and act on.
One injected instruction buried in a retrieved document can redirect an agent's behavior entirely, push data to an outside endpoint, or trigger a system action nobody authorized, all without a stolen credential or a byte of malware anywhere in the chain. The EchoLeak vulnerability, tracked as CVE-2025-32711, showed this in the wild: zero-click data exfiltration through a single email an LLM processed on its own. The attack surface isn't what the user types; it's everything the agent reads.
Even the best foundation models don't hold up as a backstop here. Independent testing shows that even well-defended models carry a meaningful attack success rate at a modest number of attempts, and that rate climbs fast the more times an attacker tries. For agentic systems, the 2026 OWASP Top 10 for Agentic Applications pushes the injection risk further: indirect injection through trusted data sources, and propagation across multiple agents, where one compromised agent quietly poisons every agent downstream of it.
Runtime guardrails need to cover four categories at once. Content safety catches harmful or policy-violating output. Security guardrails catch injection attempts, jailbreaks, and attempts to extract the system prompt. Data protection guardrails catch PII, PHI, and secrets on the way in and the way out, while compliance guardrails enforce jurisdiction-specific rules and business restrictions on what the model can say. None of this is the same thing as model alignment, which is what the model provider builds into the foundation model before anyone ever calls it. Guardrails are what the platform team enforces on every request, regardless of which vendor or model is answering, and that job belongs to the governance layer, not the vendor.
System prompts deserve their own warning label. Any secret, API key, or authorization rule tucked into a system prompt is exposed the moment someone extracts that prompt, and prompt extraction isn't rare or exotic as attacks go. Authorization needs to live in a deterministic control outside the model. A model is a language generator, not a vault, and treating it like one is how companies end up explaining themselves to a regulator.
Building the audit log and observability layer that compliance and incident response both need
An audit log of AI interactions isn't a nice-to-have for anyone under GDPR, HIPAA, or the EU AI Act. It's the thing you hand the regulator when they ask you to prove the controls you described on paper actually ran.
EU AI Act obligations for general-purpose AI models, which took effect in August 2025, require technical documentation and traceability. A company that can't produce a record of how its AI was actually used has nothing to show when asked. ISO/IEC 42001 certification, which now shows up as a hard requirement in enterprise procurement conversations, is out of reach without a live inventory of AI systems and a working usage record behind it, not a document written once and forgotten in a compliance folder.
A useful log captures the identity of the user or agent making the request, tied to the identity provider rather than a bare API key, along with which tool or model got called and which version. It captures the timestamp, the session context, whether an egress rule or a threat detection rule fired, and the token cost of the request, rolled up by user, team, and department.
Cost visibility gets overlooked constantly in governance conversations, and it shouldn't. Public LLM APIs bill by the token, and agentic workflows can call a tool over and over with no ceiling built in anywhere. Without usage telemetry, the first sign of a cost problem is the invoice, weeks after the spend already happened. MCP-based agent workloads don't come with native token tracking or rate limiting out of the box, so the governance layer has to bolt that on, or financial control over AI spend simply doesn't exist.
This data serves three audiences that rarely talk to each other otherwise. Security teams use it to spot anomalous usage, shadow tools, or a credential behaving strangely. Compliance teams use it to produce evidence when an audit or a regulatory inquiry lands on their desk. Platform and finance teams use it to attribute spend, pick better models for the job, and enforce a budget before it's blown through. Skip this layer and incident response after a breach turns into guesswork. One logistics company that got breached through an AI channel and had no access controls in place also had no forensic record of what the model was asked, or what it handed back.
Detecting shadow AI use that operates outside every control described above
Shadow AI is shadow IT in a new coat. Employees reach for personal accounts, browser extensions, mobile apps, and AI features that quietly appeared inside SaaS tools the company already approved for something else entirely, and all of it slides right past a governance layer built to watch the sanctioned tools.
A majority of employees at one mid-sized software company admitted to using AI tools nobody approved, and among the technical staff there that number climbed higher still. In most companies today, the governed path is the minority path, not the default one, which is a strange thing to admit out loud but true almost everywhere you look.
Detection has to cover several sources at once: direct browser use of consumer LLM interfaces on personal logins, browser extensions quietly forwarding page content or clipboard data to an outside model, AI features switched on inside an approved SaaS product without anyone re-reviewing it, developer tools calling external APIs outside the sanctioned stack, and MCP servers or agent integrations employees stood up on their own because filing a ticket took too long. Watching for this means monitoring more than one layer: endpoint behavior, including processes, extensions, and local tool activity; network egress, including DNS queries and outbound traffic headed to known LLM API endpoints; and SaaS activity, including OAuth grants and new integrations connected to tools the company already sanctioned.
None of this is about catching someone and writing them up. The real output of shadow AI detection is a map, an honest picture of what employees are actually using, and that map tells the company which tools deserve a seat inside the governed path, not which employees deserve a warning. The pattern held at every company I looked at: give people a governed alternative that actually works, and unauthorized use falls off fast. Employees were never trying to break the rules. They were trying to get their work done, and they took the fastest door they could find. Build a faster, sanctioned door, and most of them will walk through it without being asked twice.
