Agentic SOC Lab

Problem statement

There is no safe place to evaluate whether an AI agent triages alerts well — production is too risky, and ad-hoc tests are not repeatable or measurable.

Why it matters

Teams adopting AI in the SOC need evidence, not vibes. A lab gives repeatable evals, accuracy metrics per agent revision, and a controlled place to exercise red-team scenarios before anything touches production.

Threat model

• Agents that look accurate on a demo but drift between revisions.
• Unbounded tool access during experimentation.
• Test data leaking real customer telemetry into prompts.
• No baseline for false-positive containment risk.

Key features

• Synthetic and sanitized telemetry generators per alert class.
• Scoped tool surface with argument schemas.
• Replay harness that re-runs sessions deterministically.
• Accuracy, precision, and false-positive metrics per revision.
• Red-team scenario library for triage manipulation.

Guardrails & controls

Test scenarios

• Triage accuracy across phishing, malware, and suspicious-login alerts.
• False-positive containment rate under noisy input.
• Prompt-injection attempts inside synthetic alert bodies.
• Tool-scope escalation attempts during enrichment.

Tech stack

Language

Python 3.12

Agent

LangChain · Claude

SIEM

Splunk (sandbox)

Runtime

Docker

Eval

Replay harness + labeled sets

Lessons learned

• Labeled synthetic data is the bottleneck — invest there first.
• Replay determinism matters more than raw accuracy for tuning.
• Scope tools even in the lab; habits formed there carry to production.

Future roadmap

• More alert-class generators (cloud, identity, OT).
• Built-in red-team prompt library with scoring.
• Exportable eval reports for stakeholder readouts.