The Detection Content Lifecycle: CI/CD for SOC Rules

A detection rule is code that decides whether your firm sees an attack. It is strange, then, that most detections are still written by hand in a SIEM console, pushed straight to production, and never tested against anything.

Every other artefact that governs how a regulated firm runs — application code, infrastructure definitions, database migrations — has moved to version control, review and automated testing. Detection content, the rules that tell a security operations centre something is wrong, is often the last thing still edited live in a web console by whoever is on shift. The rule fires, or it does not, and nobody finds out which until an incident or an auditor forces the question. Detection-as-code closes that gap. It treats a detection like any other unit of code, with a lifecycle of authoring, testing, staging, deployment and measurement.

The console is where detections rot

Rules written directly in a SIEM share the failure modes of any code with no source of truth. There is no history of who changed a threshold and why. There is no review before a change reaches production. There is no test proving the rule still fires on the attack it was written for, or that a platform upgrade has not silently renamed the field it depends on. And there is no record of how noisy the rule is — so the analyst who turns it down at 2am to stop the pager screaming is making an undocumented change to a control the firm relies on. Over a couple of years a busy SOC accumulates hundreds of these rules, and no one can say with confidence which of them still work.

The lifecycle, stage by stage

The pattern borrows directly from software delivery. Five stages, each with an artefact you can point a supervisor at:

1. Author as code. The detection is a file — a Sigma rule, or the SIEM’s native query wrapped in YAML — in a repository, with metadata: the technique it covers, the data source it needs, its owner and its intended false-positive budget.
2. Test against replay data. On every change a pipeline runs the candidate detection over two labelled datasets: telemetry from the attack it targets, and benign telemetry from normal operations. It must fire on the first and stay quiet on the second.
3. Stage. Merged rules deploy in an audit or shadow mode, generating alerts that are reviewed but page no one, so real false-positive behaviour surfaces before the rule goes live.
4. Deploy. A pipeline pushes the rule to production through the SIEM’s API, not by hand, so the content running in the platform always matches the repository.
5. Measure. Each rule reports its true- and false-positive behaviour continuously, and rules that stop earning their place are retired on evidence rather than on a hunch.

None of this is exotic. It is the same pipeline shape a firm already runs for application code — authored, reviewed, tested, staged, deployed — applied to a class of artefact that has escaped it for too long. A firm that has already instrumented its delivery pipeline for lead time and change-failure rate has most of the machinery; detection content is one more thing to route through it.

Replaying attack telemetry against candidate rules

The heart of the pipeline is the test stage, and its logic is simple to state: take events you have labelled malicious or benign, run the candidate detection over them, and count what it caught and what it flagged by mistake. A rule that misses the attack has no value. A rule that alerts on ordinary activity has negative value, because it trains analysts to ignore it. The script below is a minimal version — in production the predicate is compiled from Sigma or the SIEM’s query language, and the events come from replayed datasets rather than a single file — but the scoring is exactly what a detection pipeline computes on every change.

"""Replay labelled telemetry against candidate detections and report efficacy.
Usage: python replay.py events.jsonl
Each line in events.jsonl is one event carrying a "label" of "malicious" or "benign".
Exit code is non-zero if any detection breaches its false-positive budget."""
import json
import sys
from dataclasses import dataclass
from typing import Callable

@dataclass
class Detection:
    name: str
    predicate: Callable[[dict], bool]   # True when the event should raise an alert
    max_false_positive_rate: float      # the build fails above this

# Candidate detections under test. In practice these are compiled from Sigma
# or the SIEM's own query language; here they are plain Python predicates.
DETECTIONS = [
    Detection(
        name="suspicious_lsass_access",
        predicate=lambda e: e.get("event_id") == 10
            and e.get("target_image", "").endswith("lsass.exe")
            and "0x1010" in e.get("granted_access", ""),
        max_false_positive_rate=0.01,
    ),
]

def score(det, events):
    tp = fp = fn = tn = 0
    for e in events:
        fired = det.predicate(e)
        malicious = e.get("label") == "malicious"
        if fired and malicious:
            tp += 1
        elif fired and not malicious:
            fp += 1
        elif not fired and malicious:
            fn += 1
        else:
            tn += 1
    benign = fp + tn
    fpr = fp / benign if benign else 0.0
    precision = tp / (tp + fp) if (tp + fp) else 0.0
    recall = tp / (tp + fn) if (tp + fn) else 0.0
    return fpr, precision, recall

def main(path):
    with open(path, encoding="utf-8") as fh:
        events = [json.loads(line) for line in fh]
    failed = False
    for det in DETECTIONS:
        fpr, precision, recall = score(det, events)
        breached = fpr > det.max_false_positive_rate
        failed = failed or breached
        print(f"{det.name}: recall={recall:.2f} precision={precision:.2f} "
              f"fpr={fpr:.4f} budget={det.max_false_positive_rate} "
              f"{'FAIL' if breached else 'ok'}")
    sys.exit(1 if failed else 0)

if __name__ == "__main__":
    main(sys.argv[1])

Wire that into CI so a rule cannot merge if it fails to fire on its own attack data or breaches its false-positive budget, and “the rule is broken” becomes a build failure instead of a post-incident discovery. It is the same discipline as a pre-commit baseline that actually runs: the gate belongs in the pipeline, not in a reviewer’s memory.

Test data is the hard part

The pipeline is easy; the data is not. Detection-as-code lives or dies on having labelled telemetry to test against, and that comes in two halves. The malicious half comes from executing the attack in a controlled range and capturing the logs — frameworks such as Atomic Red Team give you small, repeatable tests mapped to MITRE ATT&CK techniques, and public collections such as Splunk’s attack-data repository publish curated datasets you can replay without running the attack yourself. The benign half is harder and matters more: a representative sample of ordinary production telemetry, so a rule’s false-positive rate is measured against how your firm actually behaves, not against a clean lab. Curating and refreshing that benign corpus, and keeping it free of anything sensitive, is the real work of a detection programme — and the reason detection-as-code is a governance commitment rather than a weekend project.

The metrics that retire rules

A detection that fired twice last year and generated four hundred false alarms is not a control; it is noise with a name. The lifecycle only pays off if measurement feeds back into it. The metrics worth tracking per rule are few: precision — of the alerts it raised, how many were real — the raw alert volume it contributes, and its coverage against the techniques you care about. A rule whose precision has collapsed goes back to authoring for tuning, or is retired outright, with the decision recorded in the same repository as the rule itself. Coverage mapped to ATT&CK tells you where you are blind, which matters more than the count of rules: a hundred detections clustered on the same three techniques is not defence in depth.

What a supervisor is really asking

The point of all this is not tooling for its own sake. It is that a regulated firm should be able to answer one question — “how do you know your detections work?” — with evidence rather than faith. Detection content edited by hand in a console cannot answer it. Detection content that is authored as code, tested against replayed attacks, staged, deployed through a pipeline and measured in production can. The rules that guard the firm deserve at least the engineering rigour the firm already applies to the code that bills its customers.