Rethinking how we evaluate security agents for real-world use

Rethinking how we evaluate security agents for real-world use by Mudita Khurana Security agents are gaining momentum across industry, but the way we evaluate them remains rooted in narrow, outcome-only benchmarks. These evaluations tell us whether an agent produced a correct answer, but not “how” it arrived there or whether that behavior will remain stable once deployed. In practice, enterprise security is not a sequence of isolated tasks. It is a connected, end-to-end workflow that follows a find → confirm exploit → patch → validate loop. Agents that perform well on task-specific benchmarks often fail in these multi-stage settings due to contextual loss and brittle transitions across steps. This talk introduces a practical framework for evaluating security agents by mapping agentic capabilities (planning, reasoning, memory, perception, tool use) to security functions (reconnaissance, exploit confirmation, root-cause analysis, patching, validation) across the full lifecycle. We also share insights from our large-scale survey of existing agentic systems, highlighting which capabilities consistently drive success at each stage of the security lifecycle. Finally, we present a lightweight, unified end-to-end scoring perspective that teams can use to assess an agent’s readiness for real operational environments.