Turn attacker behavior into a decision you can trust.
Reverse-engineer a real incident, inspect the rule that caught it, build and test a behavioral detection, tune its operating point, ship it as code, and decide how it should be measured in production.
What is detection engineering?
Detection engineering is the discipline of designing, testing, deploying, measuring, and maintaining analytics that turn security telemetry into reliable decisions. It treats every detection as a production system, with a threat hypothesis, data contract, tests, owner, response, and feedback loop, not merely a saved query.
01Study the incident
Trace linked identity, endpoint, and network evidence.
02Engineer the rule
Move from telemetry contract to a tested decision boundary.
03Prove its quality
Backtest, emulate, tune, correlate, and peer review.
04Operate the result
Measure trust, data health, analyst cost, and drift.
The brief will collapse into a pill in the top right so you can reopen it at any time.
01Threat model and operating pointStrategyThe response defines how much error the detection can safely tolerate
Detection strategy · first principle
The response defines the quality bar.
Before choosing fields or syntax, decide what the alert will cause. The cost of a wrong decision determines the operating point you can safely deploy.
Low response costHunting lead
A human reviews the result. Broader recall and some noise can be acceptable.
Optimize for discovery
Medium response costAnalyst page
Interrupts the queue. Precision must protect analyst attention and trust.
Balance signal and coverage
High response costAutomatic isolation
A wrong match disrupts a healthy host. Precision and fail-safes dominate.
Optimize for trustworthy action
Detection engineering applies the software lifecycle to security analytics: define the threat, understand the telemetry, express a hypothesis, test both sides of the decision boundary, deploy through review, then observe and maintain the result.
Recall asks, “Of the attacks that happened, how many did we catch?” Precision asks, “Of the alerts we produced, how many were truly malicious?” An automatic host-isolation rule needs a very high precision bar; a low-cost hunting lead can tolerate more noise in exchange for broader recall.
TP ÷ (TP + FN)Recall
How much of the target behavior the detection catches. Missed attacks are false negatives.
TP ÷ (TP + FP)Precision
How trustworthy the alerts are. Incorrect alerts are false positives.
Decision rule
State the threat behavior, protected asset, response action, acceptable error cost, and owner before writing the query. Those constraints tell you what “good” means.
Facilitation note
Open with consequences, not syntax. Ask what happens when a false positive triggers a page, an account disablement, or host isolation. The acceptable operating point changes with the response cost.
02Incident evidenceINV-2024-0087 · six alertsFollow one evidence-backed attack chain across email, identity, endpoint, CASB, and network
At 06:47 UTC, Marcus Chen opened an invoice-themed adversary-in-the-middle URL on FINANCE-WS-01. Thirty-seven minutes later, ransomware impact began on RESEARCH-STATION-01.
The six records below are the complete alert membership of INV-2024-0087. Select each one to inspect what its native tools contributed to the chain.
Have learners distinguish alert chronology from causal confidence. ALT-7272 through ALT-7276 establish initial access, replay, and persistence; ALT-7278 establishes credential access and lateral movement; ALT-7287 and ALT-7291 establish C2/data loss and impact. All six are required to support the investigation narrative.
03The telemetry contractData qualityA perfect query cannot recover a field the pipeline never collected
Before DET-1002 can recognize suspicious PowerShell, five things have to work: the host must emit the event, collection must preserve it, normalization must map it consistently, the rule must evaluate it, and the alert must carry enough context for triage.
Select each stage to inspect its contract. These are separate failure domains: a rule can be logically correct and still have zero effective recall because command lines are truncated upstream.
Failure mode
If CommandLine logging is disabled, DET-1002 cannot observe -EncodedCommand in the execution evidence that contributes to ALT-7287. “No alert” then means “no decision was possible,” not “the behavior did not occur.” Coverage reporting should separate rule coverage from telemetry coverage.
Ask learners to name an owner and an observable health check for each stage. Detection engineering includes data contracts: schema, latency, completeness, timestamp fidelity, and retention are part of the product.
04Anatomy of DET-1002T1059.001Build around behavior, then preserve the context an analyst needs
DET-1002 - Suspicious Encoded PowerShell Execution detects PowerShell with an encoded argument, then decodes the payload and looks for behaviors such as Invoke-Expression or DownloadString.
This is stronger than alerting on every encoded command, and more durable than matching only the staging domain recorded in ALT-7287. Select a line to see what it contributes to the decision.
Contrast the rule’s minimum detection logic with enrichment. Parent ancestry is powerful context, but requiring Outlook as the parent would miss delivery through another process. Keep confidence-raising context separate from unnecessarily narrow gating.
05The test benchPrecision + recallPositive, negative, and variant fixtures probe different failure modes
A rule is a claim about future data. Test that claim against a known malicious behavior, a benign lookalike, and an adversarial variant before you trust the metrics.
The fixtures below are labeled validation inputs, not alerts. Run each one to see which clauses evaluate true. Then build the decision boundary in the scored task.
Incident replay · expected alert
Ready. Select “Run fixture” to evaluate DET-1002.
-true positive
-false positive
-precision
-recall
Why variants matter
A test that replays only the DET-0039 command represented in ALT-7287 proves reproduction, not generalization. The -enc fixture checks whether the rule catches a semantically equivalent flag without binding to the original string.
Facilitation note
Separate unit tests from backtests. Unit fixtures prove known cases. A production backtest estimates noise on historical data. Neither proves future performance alone, so good releases use both.
06Tune deliberatelyDET-1001Thresholds move the decision boundary; they do not remove the tradeoff
DET-1001 - Ransomware File Rename Pattern - fires when an endpoint renames more than 100 files in five minutes. ALT-7291 crossed that boundary with 847 files in 238 seconds.
Move the threshold to see how a broad operating point catches smaller attacks but also captures benign bulk jobs. The sample set is a tuning simulation; ALT-7291 is the only alert record represented.
50 · broad500 · strict
Guardrail
Never tune only until the alerts stop. Explain which benign behavior you are excluding and which attacks the new boundary might miss.
67%precision
100%recall
3events fire
The ransomware event fires, but two benign bulk jobs also cross this boundary. Add contextual features before raising the threshold past attacks you still need to catch.
The simulator intentionally makes a threshold-only solution uncomfortable. The mature response is feature engineering: process trust, path, extension entropy, ransom-note creation, and user context can improve separability without blindly sacrificing recall.
07Validation gatesRelease engineeringA green syntax check is necessary, but it is nowhere near sufficient
A production detection should fail closed in CI when its schema, tests, mappings, or operational contract break. The gate should be repeatable and reviewable, not a screenshot of one analyst’s successful query.
Run the release suite. One gate is deliberately red: the rule has no documented telemetry-health monitor. That is a real deployment risk because silent collection loss looks exactly like “zero attacks.”
Schema + syntaxwaiting
Required fields exist; operators and value types compile for the target backend.
Positive fixtureswaiting
Incident replay behavior and the short-flag variant both produce the expected decision.
Negative fixtureswaiting
Approved encoded maintenance stays quiet; exclusions are narrow and documented.
Historical backtestwaiting
Thirty days of representative data fit the alert-volume budget and reveal top talkers.
Attack emulationwaiting
A controlled T1059.001 execution proves the full sensor-to-alert path, not just the query.
Telemetry health SLOwaiting
Collection freshness and command-line completeness have an owner, threshold, and alert.
Six gates ready. Run the suite to prepare the peer-review decision.
Release rule
A failed health gate blocks production promotion until the telemetry SLO is defined. Shipping an unobservable detector creates confidence without evidence.
Facilitation note
Ask which checks belong at pull-request time and which require a scheduled job. Syntax and fixtures are fast PR gates; longer backtests, drift checks, and emulations may run nightly or before promotion.
08Correlation and coverageINV-2024-0087Correlation combines independent evidence; it should not erase the source records
A single alert says a behavior crossed a boundary. The investigation says multiple behaviors, entities, and timestamps belong to one threat story.
INV-2024-0087 spans six alerts across MailGuard, Identity, EDR, CASB, and NGFW. This creates higher confidence and a richer response decision, while preserving each alert’s native provenance for audit and retuning.
Credential AccessALT-7272 · stolen session through an AiTM proxy.
Coverage maps are not scorecards where more is always better. Prioritize techniques relevant to the threat model and assets, then distinguish no rule, no telemetry, untested rule, and proven end-to-end coverage.
09Detection-as-codeReview + CIThe rule, fixtures, metadata, and runbook ship as one reviewable unit
The query is only one artifact. A production change also needs ownership, severity rationale, ATT&CK mapping, expected false positives, test fixtures, response guidance, and a retirement path.
The repository specimen below packages DET-1002’s behavioral intent without hard-coding the exact ALT-7287 staging domain. That preserves the incident alert as evidence while keeping the rule general.
title:Suspicious Encoded PowerShell Executionid:DET-1002status:enabledlogsource:category:process_creationproduct:windowsdetection:image:powershell.exeflags:[-EncodedCommand, -enc]decoded_behavior:[IEX, Invoke-Expression, DownloadString, WebClient, Start-BitsTransfer]condition:image and flags and decoded_behaviortags:[attack.t1059.001]evidence:[DET-0039, ALT-7287]owner:detection-engineeringreview_interval:90d
Peer review lens
Review intent before syntax: What threat behavior should this catch? Which observable carries that meaning? Which benign behaviors overlap? What evidence proves the release is safe?
A portable schema does not guarantee portable semantics. Field names, process ancestry, decode functions, and aggregation windows vary by backend. Require compiled-query review and backend-specific tests before promotion.
10Production healthOperateMeasure trust, coverage, data health, and analyst cost, not just alert count
Deployment starts the operational phase. Watch the rule’s true-positive rate, alert volume, entity concentration, telemetry freshness, field completeness, and time-to-triage. Drift can come from attackers, administrators, software, or the collection pipeline.
Move the observed true-positive rate to see how a fixed alert volume changes analyst cost. A low rate is a prompt to investigate top talkers and feature overlap, not an automatic reason to delete the detector.
30-day rolling window
5% · noisy95% · precise
62alerts / month
26true positives / month
36false positives / month
4.5hmonthly triage cost
Investigate the 36 false positives by entity and parent process. The rule still produces meaningful signal, but its overlap budget is too high.
Keep healthy
Data SLO: endpoint process events arrive within five minutes and retain complete command lines.
Quality SLO: alert volume stays inside the triage budget and no single benign entity dominates.
Coverage SLO: positive fixtures and scheduled emulation continue to reach the alert layer.
Tune or retire
Tune when false positives have a defensible shared cause. Rework when evasion variants expose a brittle observable. Retire when the threat is irrelevant, telemetry is gone by design, or another detector provides better coverage with lower operational cost.
Facilitation note
Close by asking learners to write a one-sentence rule charter: threat behavior, protected asset, response action, acceptable alert budget, and owner. If those cannot be named, the detector is not ready to operate.