Prompt Injection Defense Kit
Detection controls for prompt injection, tool misuse, and data-exfiltration attempts — a rule + classifier hybrid with measured efficacy at the input, output, and tool-argument boundaries.
Problem statement
Most agents rely on a single prompt-level filter, which misses output-side leaks and tool-argument attacks entirely.
Why it matters
Prompt injection against a tool-calling agent is consequential because the agent has privileges. Defense in depth — separate detection per boundary — is the only approach that holds up under red-team testing.
Threat model
- Instruction override embedded in untrusted content.
- Exfiltration encoded into tool arguments.
- Secret echo in agent output.
- Unauthorized tool calls triggered by injected instructions.
Architecture
Three independent detection layers; any one can block or escalate. No single filter is load-bearing.
Key features
- Rule + classifier hybrid per boundary.
- Category-level efficacy metrics.
- Re-runnable eval suite for CI.
- Configurable block-or-review thresholds.
- Reproductions for every detected attack.
Guardrails & controls
Test scenarios
- Direct and indirect instruction override.
- Exfil via URL, query string, and webhook arguments.
- Secret echo and PII leak in outputs.
- Multi-turn injection that builds across messages.
Example workflow
Classify input
Untrusted content is scored before it reaches the agent.
Validate arguments
Tool calls are checked against schemas and exfil heuristics.
Scan output
Agent output is screened for secret echo and leakage.
Escalate
Above threshold, the call is blocked or routed to review and logged.
Tech stack
- Language
- Python 3.12
- Models
- OpenAI · local classifiers
- Eval
- Pytest-style suite
- CI
- GitHub Actions
- Output
- Category metrics + reproductions
Screenshots
Visual placeholders. Replace with real screenshots and a demo video.
Lessons learned
- Tool-argument detection catches attacks the prompt filter never sees.
- Efficacy must be measured per category, not as a single number.
- An eval suite that the team owns matters more than a one-time test.
Future roadmap
- Pluggable classifier backends.
- Cross-tenant isolation checks.
- Benchmark pack against published injection corpora.
Want this pattern in your stack?
This kit is a re-runnable eval suite for agent prompt-injection defense and is open work — get in touch if you're building something similar or want to compare notes.