PII Detection and Blocking in AI Agent Outputs

In 2023, roughly 11% of enterprise ChatGPT inputs contained sensitive data. By Q4 2025, that figure had climbed to 34.8%. This is not careless behavior by outlier employees. It is how enterprise AI is actually being used, by competent people doing their jobs, at scale.

Harmonic's analysis of tens of millions of enterprise AI prompts in 2025 found source code and proprietary algorithms appearing in hundreds of thousands of instances. Credentials showed up far less often, in far fewer instances, but a notable share of coding tool exposures contained API keys or tokens. A single leaked credential can enable broad infrastructure access. The low count is not reassuring; it is misleading.

The Samsung incidents from 2023 are still the most instructive data point. Three separate incidents within twenty days: engineers pasting proprietary source code, confidential meeting notes, and chip-yield test sequences into consumer ChatGPT, under terms that gave Samsung no deletion path. Nobody involved was being reckless. They were using the best tool available to get their work done. The data was gone before the controls existed to stop it. That gap, between what people naturally reach for and what security teams have actually locked down, is where most of this happens.

IBM's 2025 Cost of a Data Breach Report quantifies what that gap costs: shadow AI breaches averaged $4.63 million per incident, $670,000 above breaches without AI involvement, with a mean detection time of 247 days. Nearly eight months of PII flowing through systems, into logs, into third-party APIs, before anyone with accountability for it knew it had happened.

Why LLM-Only Detection Fails and What a Hybrid Approach Actually Does

Using an LLM to detect PII in LLM outputs has a foundational flaw: the model has no native concept of data sensitivity. To the model, a Social Security number and a city name are both tokens. No flag, no weight, no inherent classification. PII items are also frequently split across multiple tokens during inference, which means an LLM-as-detector produces false negatives and partial matches at the exact moments when accuracy is most consequential.

The reliable baseline for structured PII detection has existed for twenty-five years: pattern matching via regex and lookup tables. Emails, SSNs, phone numbers, card numbers: these can be caught with high precision. Pattern-based detection for credit card numbers approaches 99% accuracy. Named entity recognition handles context-dependent entities like names and addresses, where pattern matching would fail entirely, because "the patient" followed three sentences later by a name is not a format any regex will reliably catch. NER accuracy ranges from 88 to 95% in vendor studies; combined NER and pattern-based systems achieve F1 scores in the low-to-mid 90s on standard entity types.

The hybrid model uses each tool for what it actually does well. Compiled regex and pattern dictionaries handle structured formats with surgical precision. An NER classifier picks up the unstructured, contextual PII that would otherwise slip through. An AI gateway integrates both into the workflow so text is scanned before it reaches the model, not after the model has already routed it somewhere inconvenient.

One honest caveat: those accuracy benchmarks come from vendor studies on standard entity types. Real-world performance against proprietary PII, internal employee IDs, coded project names, or identifiers that carry confidential meaning only inside a single organization, will vary. Any deployment should be tested against actual data flows before vendor numbers are trusted operationally.

The Four Enforcement Actions That Turn Detection Into Policy

Detection identifies PII. Policy decides what happens next. Conflating the two is where most enterprise AI governance programs quietly collapse.

Four modes cover most enforcement scenarios.

Redaction replaces detected PII before data reaches its destination. "John Smith, DOB 15/03/1985" becomes "[PERSONNAME], DOB [DATEOF_BIRTH]" in the agent's output. The agent's task can usually complete without the raw PII appearing downstream. This is the safest default.

Blocking stops the entire action when PII appears somewhere it has no business being. If an agent attempts to send customer PII to an external API that was never authorized to receive it, the call does not go through.

Allow with audit flag handles the legitimate case where the agent actually needs PII to complete its task. Processing continues, but the event is logged for review. This is a governance decision, not a detection failure: an intentional, recorded choice to process PII, which is exactly what compliance frameworks require as evidence.

Escalation routes ambiguous cases to a human reviewer when neither redaction nor blocking is clearly the right call. The system's job is to ensure a person makes that judgment rather than having it resolved silently by default behavior.

A fifth option is maturing quickly: real-time tokenization, which replaces SSNs and credentials before a prompt reaches a third-party model, then rehydrates the original values in the response. The employee receives complete, usable output; the sensitive data never left enterprise control.

The policy choice between these modes must be made at the governed infrastructure layer, not inside individual applications. Application teams under deadline pressure default to the path of least resistance.

How Multi-Agent Architectures Multiply the Enforcement Problem

A single-agent system has one input boundary and one output boundary. A multi-agent pipeline has as many enforcement points as it has agent transitions, and most organizations have controls at essentially none of them. The surface area scales with the architecture. The governance rarely does.

Each agent in a chain should carry PII policies appropriate to its specific role. A retrieval agent is legitimately authorized to access raw PII. A summarization agent downstream should receive only redacted data. Without per-agent policy enforcement, PII authorized for one stage flows unfiltered through every subsequent stage. Scoped authorization becomes effectively global, which defeats the purpose entirely.

The log gap is where this gets genuinely embarrassing in production. A voice agent was found to have logged complete credit card numbers for three weeks: not in the transcript display, which was correctly redacted, but in OpenTelemetry debug spans the engineering team was using for latency analysis. The team had done the obvious right thing in the obvious right place. They had simply not thought about the non-obvious place. Every telemetry pipeline, every distributed trace, every structured log is a potential PII leak that sits outside the application's redaction logic. Observability infrastructure is typically excluded from data loss prevention reviews, and that gap is now a live production vulnerability in nearly every agentic deployment.

Prompt injection compounds all of it. A poisoned document or tool response at one stage can instruct a downstream agent to exfiltrate data it legitimately holds. The agent is not compromised in the traditional sense; it is doing exactly what it was told, by something that was not who it appeared to be. Detection must cover payloads flowing between agents, not just user-facing outputs. OWASP's Top 10 for LLM Applications placed sensitive information disclosure second in 2025, up from sixth the prior year. That ranking reflects exactly this expansion of the attack surface in production agentic systems.

Where MCP Servers Enter the Picture and Why They Widen the Leakage Surface Further

The Model Context Protocol, released by Anthropic in November 2024, has become the de facto standard for connecting AI agents to external tools and data sources. By December 2025, the SDK was seeing 97 million monthly downloads, with backing from every major AI lab. Adoption grew more than 400% in 2025, at a pace security reviews rarely match.

The majority of those deployments occurred outside any formal security review. More than 10,000 active public MCP servers exist, and installing one requires no procurement process. A single overpermissioned MCP server can give an agent direct SQL access to a production database, write access to internal file systems, and the ability to call external APIs, with no audit trail on any of it.

GitGuardian's research found tens of thousands of unique secrets exposed in MCP configuration files in 2025 alone. Google API keys and PostgreSQL connection strings were among the most common. The secrets are not in the AI output. They are in the configuration layer that most security teams are not examining.

The structural problem is this: when an agent calls an MCP server, the payload flowing through that tool call contains PII the agent retrieved from an upstream source. Most MCP deployments have no inspection of what passes through those payloads. PII policy enforcement at the application level does not reach MCP server interactions unless it was explicitly engineered to do so, and in practice, it almost never is.

OAuth 2.1 was added to the MCP specification in June 2025, but implementation remains uneven. Some deployments still run under shared service accounts with no per-user identity propagation, meaning the agent has broader access than any individual user would be granted directly.

The ForcedLeak exploit against Salesforce Agentforce, surfacing in September 2025, made the tool-call leakage vector concrete. A poisoned CRM lead record sat dormant until a routine employee query triggered the agent to hand contact details, deal values, and pipeline data to an attacker-controlled server. No user reviewed the payload before it left; by the time the response was visible, the data was already gone. This is not an edge case. It is a demonstration of how the vector works in production, at scale, against a major enterprise platform.

What a Governed Interception Layer Does That Application-Level Controls Cannot

Application-level controls are built by individual teams for individual use cases. They do not cover the full data path. They do not enforce consistent policy across agents. They are invisible to security teams outside the team that wrote them. This is a structural consequence of building controls inside applications rather than beneath them, and no amount of engineering diligence at the application layer resolves it.

A governed interception layer sits in front of the AI stack: between agents, tools, and external systems, enforcing PII policy regardless of how individual applications were built. The logic is identical to what moved TLS termination and authentication out of individual services and into the gateway. Security belongs in infrastructure, not distributed across application code written under different pressures by different teams at different times.

Core capabilities at this layer include real-time scanning of prompts and outputs before they reach the model or downstream systems; consistent enforcement of redaction, blocking, allow-with-audit, and escalation policies across every agent in the organization; per-agent, role-based PII policies tied to existing identity providers; and audit logs covering every PII event, including tool-call payloads and inter-agent handoffs, not just user-facing outputs.

Identity propagation closes a vulnerability application-level controls cannot address. If a user cannot access a record directly, the agent acting on behalf of that user should not be able to access it either. Enforcing this requires identity context to travel with every request through the full agent chain and into every MCP server call, not to be assumed in a system prompt that any downstream agent can be instructed to ignore.

SpeakeasyAPI's control plane operates as this kind of governed layer, providing real-time PII and prompt injection detection, role-based access control tied to Okta and Entra ID, and telemetry across agent interactions and MCP server calls.

Prompt injection detection must operate at the tool-call boundary, not at the chat interface. The ForcedLeak attack did not enter through the chat interface; it entered through a CRM record. Detection at the wrong boundary produces confidence without protection, which is a more dangerous state than no detection at all.

The Compliance Backdrop That Is Turning PII Governance from Best Practice into Obligation

GDPR and CCPA have always required organizations to know where personal data flows and to demonstrate control over it. AI agents create new flows that most organizations cannot currently map or document. That puts most organizations in some degree of non-compliance they cannot yet fully articulate.

The EU AI Act's risk classification framework, phasing into enforcement across 2025 and 2026, requires organizations deploying high-risk AI systems to maintain technical documentation, logging, and human oversight mechanisms. The audit trail produced by a governed interception layer is precisely what those requirements demand.

Shadow AI compounds the compliance problem in a specific way. When an organization cannot detect or inventory its AI tools, it cannot demonstrate that its data handling complies with applicable law, because it does not know what data is being handled. That is not a partial answer to an auditor's question. It is no answer.

The "allow with audit flag" enforcement mode is worth isolating here. It produces the documented, intentional processing record that regulators require when PII is legitimately processed by an automated system. GDPR's lawful basis requirements, CCPA's processing disclosures, the AI Act's transparency obligations: all require evidence that PII processing was a deliberate, recorded decision. An allow-with-audit log is that evidence. Without it, processing happened but cannot be demonstrated to have been intentional.

IBM's 247-day shadow AI breach detection window is not just an operational problem. It is a 247-day compliance exposure window: undocumented PII flows and accumulating regulatory liability in systems nobody is watching. The governed interception layer that enforces PII policy in real time is also the system of record that answers the auditor's core question: show me every time your AI agents touched personal data, and show me what happened to it. Without that layer, the honest answer is that no one knows.