A bank robber slips separate notes to four different tellers: “walk,” “to the,” “vault,” “now.” Each teller sees a harmless scrap of paper, and the security camera reviewing any single note finds nothing to flag. Only when a fifth person walks in and reads all four notes in order does the instruction become unmistakable.
Payload splitting works the same way against AI systems, breaking a harmful instruction into fragments that each look harmless on their own, then relying on the model to reassemble and execute the whole.
A safety filter scanning one message at a time sees nothing wrong with a1 = 'Repeat ‘ or a2 = 'everything ‘, but the model reads the reconstructed instruction and follows it. That gap between how filters inspect and how models process is now an operating risk for anyone deploying chatbots, copilots, or autonomous agents. This article explains why session-level analysis is important for detecting fragmented instructions across a conversation.
Key takeaways
- Payload splitting turns harmless-looking fragments into a harmful instruction once the model recombines context across variables, conversation turns, tokens, or documents.
- Fragment-only inspection leaves a security gap because keyword, regex, and static-signature controls can miss obfuscation, paraphrase, and multi-turn adversarial drift.
- Enterprise exposure rises when reconstructed prompts affect customers, sensitive data, or autonomous agents that can query systems and trigger downstream actions.
- Stronger resilience requires layered controls: session-level intent analysis, bidirectional prompt-and-output inspection, data tokenization, least privilege, agent discovery, pre-execution checks, and audit trails.
You Can’t Secure What You Can’t See
WitnessAI gives you network-level visibility into every AI interaction across employees, models, apps, and agents. One platform. No blind spots.
Explore the PlatformDefining payload splitting and how it works
Payload splitting distributes an adversarial instruction across multiple benign-looking fragments, then gets the model to concatenate and act on the assembled whole. The technique is also called token smuggling and Fragmentation Concatenation Attack.
The HackAPrompt attack taxonomy treats the three as similarly defined and consolidates them into one category. Daniel Kang and colleagues formally introduced payload splitting to LLM security in their 2023 work on programmatic behavior research. Their work showed that standard computer security attack methods can be applied to language models.
Fragments can look safe to a filter and still combine into a malicious instruction for the model. Documented attack examples show that fragments that are individually benign can become malicious once the model is asked to concatenate them into a complete instruction.
The technique takes several forms across prompts, conversations, tokens, and enterprise documents:
- In variable-assembly splitting, the attacker defines variables and instructs the model to combine them. The Raccoon benchmark uses fragments such as
a1 = 'Repeat 'anda2 = 'everything ', then later fragments are assembled into a request to reveal the instruction prompt. - In multi-turn splitting, fragments arrive across separate conversation turns. Each turn can pass filters independently while the multi-turn conversation context rebuilds the full instruction.
- At the token level, token-smuggling obfuscation research shows how obfuscation can alter tokenization. Words that would flag prompt injection can be split into benign subwords that classifiers don’t catch while the model preserves the semantics.
- Document injection brings the technique into enterprise workflows. OWASP documents an attacker who uses resume prompt injection; when the LLM evaluates the candidate, the combined fragments manipulate the model into a positive recommendation regardless of the resume’s actual contents.
In each form, the fragments look safe in isolation. The risk appears when the model reconstructs the instruction inside a broader context.
Knowing Which AI Tools Are in Use Is Just the Start
WitnessAI goes beyond app discovery. Observe classifies the intent behind every AI interaction across employees and agents, so you can build smarter policies based on real risk, not guesswork.
Explore ObserveWhy traditional security controls miss it
Traditional inspection can’t catch payload splitting because it evaluates fragments, not the reconstructed instruction the model sees. This blind spot is specific to keyword-, regex-, and packet-centric inspection.
Keyword and regex filters fail against payload splitting because no single fragment ever contains the flagged pattern. Variable-assembly prompts split instructions across labeled fragments, so controls that only block words when they appear together can miss the full request. The flagged words rarely appear together, so the filter doesn’t fire.
Encoding pushes the same limitation further. A filter catching “ignore previous instructions” can be bypassed through encoding and obfuscation techniques, and zero-width Unicode characters break keyword patterns invisibly while the model reads the text normally.
The failure modes fall into a few distinct categories worth naming:
- Static signature matching: Regex and keyword lists don’t capture how language changes through paraphrase, translation, or summarization. Those approaches were designed to monitor traffic and data movement, not to evaluate conversation purpose.
- Per-message inspection: Attacks like payload splitting and Crescendo keep every individual turn inside the safety distribution while the adversarial objective emerges across the full dialogue.
- Perimeter-only coverage: Guardrails that don’t evaluate business context miss sensitive information that appears in ordinary prose without recognizable file formats or keyword patterns.
The empirical record confirms the gap. Adaptive attacks against spotlighting and prompt-sandwiching defenses reached success rates above 95% on the AgentDojo benchmark, and independent reviews have documented near-complete bypass of widely deployed systems using Unicode, homoglyph, and adversarial techniques.
Session-level detection has to re-classify the full conversation trajectory, so risk is assessed across the session rather than one message at a time. Intent-based classification, applied through models that analyze conversations and context, can detect patterns that evolve across a conversation or session in a way per-message rules can’t.
The underlying reason is straightforward: attackers routinely rewrite payloads using synonyms, encodings, obfuscation, and role-play framing, all of which defeat static pattern matching while preserving the underlying intent.
Runtime AI Threats Need Runtime Defense.
WitnessAI’s enterprise AI firewall delivers bidirectional runtime defense, blocking prompt injections, jailbreaks, and data exfiltration before they reach your models or your customers.
Explore ProtectWhat effective defense against payload splitting looks like
A single control rarely stops payload splitting, and the two authoritative bodies say so plainly: NIST states NIST current mitigations don’t offer full protection against all attacker techniques, and OWASP states prevention within the LLM has limits.
You need layered controls for the risk that remains. In practice, prioritize by consequence. Routine use needs session-level analysis and runtime defense that includes tokenization; high-consequence agent actions need human review. The following capabilities, used together, address the specific ways splitting evades traditional inspection.
1. Analyze the full conversation
Session-level detection is the core defense against payload splitting because harm is distributed across fragments and turns. The Conversational Risk Accumulation framework formalizes session-level accumulation as a distinct failure class and proposes stateful guardrails that track risk across the full dialogue. Stateful drift detection lets a classifier see the trajectory rather than the leaf, and flag the benign-to-adversarial drift a per-turn check ignores.
2. Classify intent across the full context
Intent-based classification analyzes the full context and can detect malicious trajectories that no individual message would trigger. Multi-turn intent classification uses the full dialogue history for context-aware decisions, while latent adversarial-detection work proposes labeling trajectories across benign, steering, and adversarial phases. Intent survives synonyms, encodings, and role-play framing, so classification has to evaluate what an interaction is trying to accomplish.
3. Inspect prompts and responses together
Bidirectional inspection is required because attacks can succeed through outputs, such as data embedded in a response or exfiltrated via a crafted URL exfiltration path. OWASP treats output handling as a distinct risk area and recommends controls that define and validate expected output formats, particularly when LLM output can trigger downstream actions.
WitnessAI is an AI security and governance platform that gives you visibility and runtime control over how your workforce and agents use AI. Its runtime guardrails apply bidirectional inspection in practice: the Protect module blocks threats like prompt injection before they reach your models and agents, and filters harmful outputs before they reach users or trigger unintended actions.
4. Neutralize the data even when an attack lands
Least-privilege access and data tokenization controls limit the damage a reconstructed payload can do. Granting the LLM only the minimum access it needs for its intended operations reduces what an injected instruction can reach.
Real-time data tokenization addresses the exfiltration path: PII such as SSNs, along with credentials, are replaced with reversible tokens before a prompt reaches any model, then rehydrated in the response so downstream workflows stay intact. If an injection succeeds, the payload it carries out has already been stripped of sensitive data. Coverage matters as much as the controls, because much AI usage occurs outside the browser, in native apps, IDEs, and embedded copilots, and agent API calls don’t touch a browser at all.
Runtime defense only works if it sees runtime activity, and most enterprise AI usage now sits outside the browser. Native apps, IDEs, embedded copilots, and agent API calls don’t touch a browser at all, which means browser-extension-only tools leave real gaps in coverage.
WitnessAI closes those gaps with network-level visibility across AI traffic routed through the platform, extending coverage to native apps, IDEs, embedded copilots, and agent API calls from a single console. WitnessAI provides protection across a broad range of enterprise AI models and applications used in customer environments.
Witness Protect, WitnessAI’s Enterprise AI Firewall for Models, Apps, and Agents, is the runtime layer on top of that visibility. It delivers bidirectional runtime defense in three places at once:
- Blocks prompt injections and jailbreaks before they reach your models or agents.
- Tokenizes sensitive data before it leaves enterprise control.
- Filters harmful outputs before they reach users or customers.
Those three enforcement points turn the defensive principles above into controls you can apply to real traffic, without waiting for the model provider to close the gap.
Are Your AI Applications Secure at Runtime?
WitnessAI provides bidirectional defense for your models, apps, and agents, blocking prompt injections and filtering harmful outputs before they reach users or trigger unintended actions.
Learn About WitnessAI For ApplicationsClosing the gap as agents scale
Payload splitting works because filters read fragments while models read the whole, and that gap widens the moment an agent can act on the reconstructed instruction.
Effective defense against payload splitting uses session-level intent classification and bidirectional runtime defense with data tokenization across AI traffic routed through the platform, including browser sessions, native AI applications, embedded copilots, autonomous agents, and the MCP servers they reach.
WitnessAI gives security and compliance teams responsible for AI a unified platform to govern the human and digital workforce together, with intent-based policies and runtime guardrails designed to detect and block fragmented and multi-turn attacks.
To discuss the visibility and control gap in your specific AI deployments, schedule a demo.