> For the complete documentation index, see [llms.txt](https://0xzoro.gitbook.io/0xzoro/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://0xzoro.gitbook.io/0xzoro/soc-operations/detection-engineering/01-intro-and-lab-setup.md). # 01 intro and lab setup ## 1. Introduction, Theory & Lab Setup > **Course:** TCM Security — Detection Engineering **Notes standard:** see [Note-Taking Standard](https://claude.ai/cowork/00-notes-skill.md) · **Course map & architecture:** see [Course Reference](https://claude.ai/cowork/00-course-reference.md) **Lab stack:** VirtualBox (Win11 / ParrotOS / Ubuntu), Elastic + Elastic Agent, Sysmon, Zeek, Atomic Red Team, Python, TOML, GitHub Actions. > > **Theory module coverage:** Security Operations · Role Variety · SIEM · The Detection Engineering Workflow · What Makes a Good Detection · Technology Stack · MITRE ATT\&CK · Navigating the Matrix. Section 1.1 below covers the Workflow; remaining theory lessons slot in as 1.2+ as you watch them. *** ### 1.1 The Detection Engineering Lifecycle (I/O Model + Workflow) *Process/methodology topic — the technical sections below are reframed around the process itself rather than a single technical control.* *** #### Definition Detection engineering is often mistaken for "writing SIEM rules." It isn't. It's an **engineering discipline** — a repeatable process that takes raw signal from multiple parts of a security organization (hunting, threat intel, incident response, direct requests, business context), turns it into tested and documented detection logic, ships it into production, and then feeds the results of that logic back into the next cycle. The word "engineering" is doing real work in that phrase. An engineer doesn't ship something because it compiled once. They gather requirements, understand constraints, test against reality, document the design, get it reviewed, and monitor it in production. Detection engineering applies that same discipline to security alerting. The output of the discipline isn't a rule — it's a **detection capability** with a known scope, known blind spots, known false-positive behavior, and a paper trail that lets someone other than its author maintain it. > **Think of it as** the difference between *writing a query* and *building software*. Anyone can write a query that returns rows. A software engineer ships a component that's specified, tested, reviewed, documented, and monitored — and can be handed to someone else. Detection engineering is that second thing, applied to alerts. Two complementary mental models describe this discipline: 1. **The I/O Model** — a systems view: what goes into the detection engineering function, what happens inside it, and what comes out. 2. **The 5-Stage Workflow** — a process view: the literal sequence of steps an engineer walks through to take one requirement from idea to deployed, documented, reviewed detection. Together they answer two different questions. The I/O model answers *"where does detection work come from, and what does the function produce?"* The workflow answers *"what do I actually do, step by step, to build one detection correctly?"* *** #### Why It Exists Before detection engineering existed as a discipline, most security teams treated detection as an ad-hoc activity: someone had an idea, opened the SIEM, wrote a query, and turned it on. This works for a handful of rules. It fails at scale, for predictable reasons: * **No traceability.** Nobody remembers why a rule exists, what it was supposed to catch, or whether it still works after a platform migration. * **No validation.** Rules are written from assumptions about what telemetry *should* look like, not from what it *actually* looks like in the environment. They silently fail — the SIEM shows green, but the rule never fires because a field was never populated the way the author assumed. * **No feedback loop.** Incidents and hunts produce enormous amounts of knowledge about attacker behavior, but if there's no process to route that knowledge into new detections, the organization relearns the same lessons in every incident. * **No prioritization.** Without a structured intake, engineers work on whatever is loudest or most recent, not what's actually highest risk. * **No handover.** A rule that lives only in its author's head becomes unmaintainable the moment that person is unavailable, and unusable to the analyst who receives its alert at 2 a.m. Detection engineering, as a discipline, exists to close these gaps. It borrows the same instinct that produced software engineering out of ad-hoc coding: formalize intake, testing, review, and documentation so the result is durable, explainable, and improvable — rather than a pile of undocumented queries that nobody trusts or fully understands. *** #### How It Works **The I/O Model, end to end** The I/O model treats the detection engineering function as a system with three phases: **Input → Processing → Output**. Each phase has multiple channels feeding it, and — critically — the output of the system loops back around to become input again. It is not a one-way pipe. ``` INPUT │ ┌──────────────┼──────────────┐ │ │ │ Hunting Intel Response │ │ │ ├──────── Detection ──────────┤ │ │ │ └──────── Business Units ─────┘ │ ▼ PROCESSING │ ┌──────────────┼──────────────┐ │ │ │ Testing Alert Creation Alert Tuning │ │ │ ├──── Documentation ──────────┤ │ │ │ └────── Knowledge Transfer ───┘ │ ▼ OUTPUT │ ┌──────────────┼──────────────┐ │ │ │ Detection Docs Incident Alerts Gap Identification │ ▼ Metrics ``` **Why five different inputs, and not just "requests"?** Because a mature detection program treats detection requirements as something it actively *harvests* from across the security org, not something it passively waits to receive. A direct request ("build a detection for X") is only one channel among five, and arguably not even the richest one — hunts and incidents tend to produce detections grounded in things that actually happened in *this* environment, which is a stronger foundation than a hypothetical "what if." **Why does processing include tuning and knowledge transfer, not just "writing the rule"?** Because a rule that fires accurately in a lab but drowns the SOC in false positives, or fires correctly but nobody knows how to investigate it, has not actually produced a working detection capability. Processing is the whole set of activities required to turn "a rule that matches a pattern" into "an operationally trustworthy alert that someone downstream can act on." **Why does the output include gap identification and metrics, and not just alerts?** Because the value of a detection program isn't fully captured by "does this individual rule fire." A program also needs to be able to answer *"what can't we see at all,"* and *"is the program, as a whole, getting better over time."*\* Those two questions can only be answered by looking across the whole rule set, not any single rule — which is why they're modeled as first-class outputs, not afterthoughts. **The 5-Stage Workflow, end to end** Where the I/O model describes the function as a system, the workflow describes the literal steps for turning **one** requirement into **one** deployed detection. ``` Information Gathering ↓ Unit Testing ↓ Alert Development ↓ Documentation Creation ↓ Peer Review / Handover ``` **Why does the process start with "gathering information" instead of "writing the rule"?** Because most bad detections are bad requirements, not bad logic. "Detect suspicious PowerShell" is not an engineering requirement — it's a topic. An engineer has to convert it into something testable: a specific behavior, on a specific platform, with specific expected telemetry, before there's anything to build. **Why is "unit testing" a separate stage from "alert development," instead of testing the rule after it's written?** This is the single most important structural decision in the whole workflow, and it inverts the instinct most beginners have. The instinct is: write the rule, then see if it fires. The engineering approach is: *prove the behavior produces the telemetry you think it does, before you invest any effort writing detection logic against it.* If you skip straight to writing the rule, and the rule doesn't fire, you don't know whether your logic is wrong or your assumption about the telemetry was wrong — you've conflated two separate failure modes into one debugging session. Separating unit testing from alert development forces you to validate the *data* first, then reason about *logic* against data you've already confirmed is real. > **Mindset:** unit testing is *not* "testing my rule." It's testing my **assumption about the relationship between behaviour and telemetry**. The rule comes after you understand that relationship. Beginner path: *rule idea → write rule → hope it works.* Engineer path: *behaviour → hypothesis → generate behaviour → observe telemetry → understand reality → build detection.* **Why does documentation come after development but before review?** Because a reviewer cannot meaningfully evaluate a detection without the context documentation provides — the intended scope, the expected false positives, the MITRE mapping. Undocumented logic can only be reviewed for syntax, not for whether it's the *right* detection. **Why is review the last gate before deployment, rather than deployment itself being the last step?** Because the workflow's implicit claim is that a second set of eyes catches classes of error the original author is structurally unlikely to catch alone — false-positive blindness, missing edge cases, incorrect severity — the same reason software engineering treats code review as non-optional before merge. **Where the two models intersect** The 5-stage workflow is what happens *inside* the "Processing" box of the I/O model, for a single detection. Zooming out, one full turn of the workflow (from a single input) produces the outputs of the I/O model (a documented detection, alerts once deployed, and — if telemetry turns out to be missing during unit testing — a gap identified instead of a shipped rule). *** #### Workflow **Unit Testing sub-workflow** (the mechanism inside stage 2 of the 5-stage process — this is the part beginners skip and is worth its own diagram): ``` Detection Requirement ↓ What behavior am I trying to catch? ↓ What telemetry should that behavior produce? ↓ Review Log Source ── "What can this source actually see?" ↓ Perform Test ── deliberately generate the behavior in a lab ↓ Review Actual Data ── compare reality to expectation │ ┌────┴────┐ ▼ ▼ Matches Doesn't match │ │ ▼ ▼ Build Update the model — detection it's either a gap logic or a wrong assumption ``` **The feedback loop across the whole lifecycle** (why the workflow is a loop, not a line — see *Real-World Notes* for the worked example): ``` ┌──────────────────┐ │ ▼ Requirement → Development → Deployment → Alerts ▲ │ │ ▼ └──── Tuning ← Investigation ``` *** #### Components **Input components (I/O model)** **Hunting.** A threat hunter goes looking for behavior without a pre-existing alert — e.g. hunting for PowerShell used to download payloads. If they find something real, that hunt converts into a standing detection: `Hunt → Interesting behavior discovered → Create detection → Test → Production rule`. This is one of the most common origin stories for a durable detection, because the behavior was proven to exist in the environment before anyone wrote a line of logic. **Intel.** Threat intelligence describes what attackers are doing *right now*, elsewhere, and asks whether that maps to something detectable here: `Threat report → APT uses specific technique → What telemetry would expose it? → Can we detect it?`. Intel can arrive as IOCs (a hash, an IP, a domain), or as TTPs (a technique, a procedure, a campaign pattern). This distinction matters enormously: an **IOC-based** detection catches *this specific* artifact and expires the moment the attacker changes infrastructure. A **behavioral** detection catches the underlying tradecraft regardless of which IOC is used today, and survives infrastructure churn. Mature programs prefer behavioral detections and treat IOC matching as a short-lived supplement, not the strategy. > **Think of it as** the difference between memorising a criminal's current license plate versus recognising *how they break in*. Swap the car (the IOC) and the plate-watcher is blind; the person who knows the break-in method (the behaviour) still catches them. **Response.** Incident response investigations routinely surface attacker behavior nobody had a detection for — e.g. discovering during an incident that the attacker used a scheduled task for persistence, with no prior alert on that technique: `Detection (missing) → Incident → Investigation → New attacker behavior discovered → Detection improvement`. This is why detection engineering cannot sit organizationally isolated from IR — every incident is a source of unpaid detection requirements, and if there's no pipe connecting IR findings back into detection engineering, those lessons evaporate. **Detection.** The direct, explicit ask — "we need a detection for suspicious LSASS access," or "detect Office spawning PowerShell." This is the most obvious input and the one beginners assume is the *only* input, which is exactly the misconception the model is designed to correct. > **Mindset:** the whole point of the five-input model is that *"detection engineering" is not "waiting for someone to request a rule."* Four of the five inputs (hunting, intel, response, business context) are ones you go and *harvest*. If your only input is direct requests, you're a ticket-taker, not an engineer. **Business Units.** Non-technical context that reshapes priority without changing the underlying technical detection at all: which applications are business-critical, which servers hold sensitive data, which accounts are highly privileged. The same raw behavior means something different depending on where it happens: ``` Same behavior ├── Normal workstation → medium priority └── Domain Controller → potentially critical ``` Business context changes what gets detected first, how it's prioritized, its severity, how it alerts, how it's responded to, and how much risk it represents — all without changing the detection logic itself. **Processing components (I/O model)** **Testing.** Before a detection is trusted, prove it actually behaves as intended, using two symmetric tests: * **Positive testing** — perform the malicious/suspicious behavior (e.g. via Atomic Red Team) → expect an alert. * **Negative testing** — perform benign, look-alike behavior → expect *no* alert. Both are required. A detection that only passes positive testing might also fire constantly on legitimate activity; you don't know until you've tried to make it *not* fire. **Alert Creation.** The gap between "a query that finds something interesting" and "a production detection" is precision. A hunting query like *find all PowerShell executions* is far too broad to alert on directly — it needs refinement into something like *PowerShell + encoded command + suspicious parent process + unusual user* before it earns a place in production alerting. Hunting queries and production alerts are different artifacts with different bars for precision, even when they start from the same idea. **Alert Tuning.** A newly deployed detection is rarely calibrated correctly on day one. Tuning means investigating volume and precision after the fact — e.g. `1000 alerts → 850 legitimate / 140 suspicious / 10 confirmed malicious` — and adjusting conditions, thresholds, time windows, exclusions, allow-lists, correlation, or severity in response. The target of tuning is **not** zero false positives; a detection tuned to zero false positives is usually also tuned to miss real attacks, since attackers can approximate benign behavior. The actual target is *a useful alert with an acceptable noise level* — precision high enough that an analyst trusts and acts on it, without being so narrow it stops catching variations of the real behavior. **Documentation.** Covered in depth in its own workflow stage below — but as an I/O-model *processing* activity, its role is converting a working rule into an artifact someone else can maintain. **Knowledge Transfer.** A detection's value is capped by how well the people who *receive* its alerts understand it. If a complex detection fires and the analyst on shift has no way to know why, what to investigate, what evidence to collect, or when to escalate, the detection has produced noise, not signal — regardless of how well-engineered the underlying logic is. Knowledge transfer serves SOC analysts, incident responders, threat hunters, other security engineers, and management, each of whom needs a different slice of the same underlying understanding. **Output components (I/O model)** **Detection Documentation.** A structured artifact per detection — see the `Documentation Creation` workflow stage below for the full field list. **Alerts for Incident Response.** The pipeline a detection ultimately feeds: `Detection → Alert → SOC analyst → Investigation → Incident Response`. In the personal lab context this maps directly onto `GOAD attack → Wazuh → Detection → Alert → TheHive → Investigation`. **Gap Identification.** Arguably the most mature output of the three, because it requires the program to look at what it's *missing*, not just what it has: `ATT&CK Technique → Do we have telemetry? → Can we detect it? → NO → Detection Gap`. A program that can only describe its rule count ("we have 500 rules") is less mature than one that can describe its coverage against a framework: ``` T1059.001 PowerShell → GOOD T1053.005 Scheduled Task → GOOD T1547.001 Registry Run Keys → PARTIAL T1055 Process Injection → POOR ``` That coverage table *is* the roadmap for what to build next — gap identification converts "what we don't have" into a prioritized backlog, rather than an unknown unknown. **Metrics.** How the program proves it's improving over time, across five dimensions: coverage (MITRE techniques covered), detection quality (true/false-positive rates), alert volume (alerts/day), operational performance (MTTD, MTTR), and detection health (rules tested, rules failing, rules stale, rules needing tuning). These are the numbers a dashboard would eventually be built around. **The 5-stage workflow, per stage** **1. Information Gathering** — three sub-activities: * *Initial Requirements.* The first filter question is whether a request contains enough information to actually build against. "Create a detection for suspicious PowerShell" fails this test; it doesn't specify what "suspicious" means, which environment, which systems, or what telemetry is expected. A workable requirement looks like: ``` Detect PowerShell executing encoded commands on domain-joined Windows endpoints. Reason: Observed during recent incident. Required telemetry: Windows process creation. Expected fields: Image, CommandLine, ParentImage, User, Host, Timestamp ``` Linked tickets and prior documentation matter here more than they first appear to — an old incident ticket attached to a vague request can hand you real attacker behavior, commands executed, affected hosts, relevant Event IDs, and known false positives, turning a one-sentence ask into an evidence-backed requirement: `Incident Response → Information → Detection Engineering → New Detection`. * *New Log Source.* Before writing anything against a new telemetry source (e.g. adopting Microsoft Defender for Endpoint), establish whether the data actually reaches the platform, whether it's searchable, what fields exist, how they're normalized, and what the vendor documentation says. This matters because detection logic can be perfect and still be inert if the required telemetry simply isn't collected — that's a **visibility gap**, a categorically different problem from a logic bug: ``` Want to detect: Process Injection Required telemetry: Process + memory-related visibility Reality: endpoint only sends basic Windows Security logs Result: detection may not be possible with current telemetry ``` * *Prioritization.* With a large backlog (e.g. 20 critical / 100 high / 300 medium / 380 low), two questions decide what gets worked next: how does this compare to the rest of the backlog, and does it need to happen immediately? A request tied to an *active* incident jumps the queue even if the same technique, in the abstract, would normally sit at medium priority. **2. Unit Testing** — see the dedicated *How It Works* and *Workflow* sections above for the full mechanism. As a workflow stage, its purpose is narrower than it sounds: it validates the **behavior ↔ telemetry relationship**, not the rule. That validation happens in three steps — review the log source's actual capabilities, deliberately perform the test behavior in a lab, and review what data actually resulted, comparing it against the hypothesis rather than assuming the hypothesis was right. **3. Alert Development** — only now is detection logic actually written (in this course, an Elastic KQL/EQL query stored in a TOML file), because by this point the requirement is specific and the telemetry has been proven to exist and look a particular way. Writing the rule earlier than this point is precisely the beginner mistake the whole workflow exists to prevent. **4. Documentation Creation** — in this course, the "document" *is* the detection: a TOML file whose schema is enforced by CI. The real minimum the course requires (from `validation.py`) is `description`, `name`, `rule_id`, `risk_score`, `severity`, `type`, `query`, plus `metadata.creation_date` and the MITRE `threat` mapping — see the [Course Reference](https://claude.ai/cowork/00-course-reference.md#the-real-detection-document-schema) for the exact schema and a real example. > **Enterprise extension (beyond this course).** A mature program often documents more than the course enforces — investigation guidance, known false positives, and tuning history. That richer template is worth aspiring to, but it is *not* the course bar. Illustrative example of the fuller form: > > ``` > Name: Suspicious PowerShell Child Process Logic: ParentImage = cmd.exe AND Image = powershell.exe > MITRE: T1059.001 Severity: Medium Known False Positives: Administrative scripts > Testing: Atomic test #001 Investigation: Review command line, user, host, parent > Tuning: Exclude approved management software > ``` > > Treat the extra fields as good habits to layer on once the core TOML-in-git loop is second nature. In this course's Detection-as-Code repo, each detection is a single TOML file in `detections/`; validation, MITRE checking, Elastic sync, and metrics are all driven off that file automatically. Part 4 and Part 5 build this out. **5. Peer Review / Handover** — a second engineer or analyst reviews the detection *and* its documentation before deployment, checking for bad logic, excessive false positives, missing telemetry, incorrect MITRE mapping, poor severity, missing investigation guidance, bad exclusions, performance problems, and blind spots. The anti-pattern this stage exists to prevent: `Engineer writes rule → Engineer enables rule → Nobody else knows what it does`. *** #### Example Walking one requirement through the entire lifecycle, start to finish: 1. **Input (I/O model):** During an incident, IR observes an attacker running `powershell.exe -EncodedCommand ...` spawned from Microsoft Word, on a domain-joined workstation. This is a **Response**-origin input, backed by a real incident ticket. 2. **Information Gathering:** The ticket already supplies the specific behavior, the host type, and the relevant fields — `Image`, `CommandLine`, `ParentImage`, `User`, `Host`. Business Units input flags that this workstation belongs to a finance user with access to sensitive data, raising priority. 3. **Unit Testing:** Review the log source (Windows Process Creation via Sysmon/EDR) — confirm `CommandLine` is actually populated in this environment (not assumed). Perform the test — actually launch an encoded PowerShell command from Word in a lab VM. Review the data — confirm the expected fields appear, and note that the platform represents the flag as `-enc` rather than `-EncodedCommand` in some tools, so the logic needs to account for both. 4. **Alert Development:** Write the Sigma/KQL/FortiSIEM logic: `ParentImage contains WINWORD.EXE AND Image contains powershell.exe AND CommandLine contains ('-enc' OR '-EncodedCommand')`. 5. **Documentation Creation:** Fill in `DET-WIN-023` per the template above, including the MITRE mapping (T1059.001) and known false positives (approved macro-based automation). 6. **Peer Review:** A second engineer checks whether legitimate mail-merge or reporting macros in this org would trigger this — and adds an exclusion for a known signed internal tool before approval. 7. **Output (I/O model):** The detection deploys, generates real alerts routed to the SOC → TheHive for investigation. Over the following weeks, tuning data shows a manageable false-positive rate. This detection also closes part of a previously identified gap in T1059.001 coverage, and the whole cycle updates the program's coverage metrics. This single walk-through touches every box in both diagrams — which is the point: the two models aren't separate topics, they're two views of the same one lifecycle. *** #### Common Failure Modes *What actually goes wrong when this lifecycle is skipped or done badly — and why each failure is invisible from the inside.* * **IOC-only detection.** Skipping the intel → behavioural-detection distinction produces detections that are trivially defeated by infrastructure churn. A new hash, a new domain, and the rule goes dark while the technique keeps working. Nothing alerts you that it stopped being useful. * **Untested rules.** A rule that looks correct on paper but never fires — because of a field-name mismatch, a missing telemetry source, or a parsing issue — is worse than no rule, because it manufactures false confidence. The platform shows a healthy, enabled detection; the technique goes unseen; nobody learns otherwise until an incident. * **Writing logic before confirming telemetry.** The single most common process failure. It conflates two separate failure modes — wrong logic and wrong assumptions about the data — into one undiagnosable debugging session. * **No gap identification.** An organisation that can't enumerate its blind spots is in the same practical position as one whose blind spots an adversary has already mapped — except it will never find out which. * **Documentation as an afterthought.** Recreates the exact "only the author understands this" problem the discipline exists to eliminate, and slows response even when an alert fires correctly. * **Tuning toward zero false positives.** Usually tunes out true positives at the same time, since real attacker behaviour varies around the edges of any narrow rule. * **Measuring the program by rule count.** "We have 500 rules" says nothing about coverage or quality, and rewards volume over usefulness. The inverse of this list is the payoff: a detection program built through this lifecycle survives platform migrations, survives staff turnover, can prove its own coverage, and improves measurably each cycle — rather than accumulating undocumented, half-trusted rules indefinitely. *** #### Process Health & Metrics *How you know the lifecycle itself — not any single rule — is working.* There is no event ID for a healthy detection program, but the model supplies equivalent signals at two levels: **Per detection.** For any rule in production, can you point to which input produced it, which unit test proved the telemetry existed, who reviewed it, and where its documentation lives? If any of those four is missing, the process broke down for that detection — even if the rule technically works. **Per program.** The *Metrics* output of the I/O model: | Dimension | Signal | | ----------------------- | ------------------------------------------------------ | | Coverage | ATT\&CK techniques covered, and at what depth | | Detection quality | True-positive / false-positive trends per rule | | Alert volume | Alerts per day, and whether analysts still triage them | | Operational performance | MTTD, MTTR | | Detection health | Rules tested vs. stale vs. actively failing | The health question these answer together is not *"do we have detections?"* but *"do we know which of our detections still work, and what we still can't see?"* *** #### Limitations & Assumptions Where this model quietly assumes conditions that may not hold: * **It assumes organisational maturity that may not exist.** A one-person detection function can't run formal peer review on every rule. The principle — a second perspective before production — still applies, but the mechanism must scale down (a self-review checklist, or review batched weekly). * **It assumes a lab exists for unit testing.** Without one, "generate the behaviour and observe the telemetry" is difficult to do safely, particularly for destructive techniques or anything requiring elevated privileges. * **Prioritisation is vulnerable to urgency bias.** Anything framed as tied to an active incident jumps the queue, which can indefinitely starve lower-drama but higher-value work — like closing a known coverage gap. * **Metrics can be optimised against their own intent.** Chasing ATT\&CK coverage percentage encourages many shallow, noisy detections across many techniques instead of fewer strong ones against the techniques that actually matter to this environment's threat model. *** #### MITRE ATT\&CK The lifecycle itself isn't a technique, but it's the mechanism by which ATT\&CK coverage gets built and tracked. The worked example above maps to: * **Tactic:** Execution * **Technique:** T1059 — Command and Scripting Interpreter * **Sub-technique:** T1059.001 — PowerShell It maps here because the detection specifically targets PowerShell as the execution mechanism — as opposed to T1053.005 Scheduled Task, the other worked example used earlier for IR-driven persistence detection. Gap Identification, as an output, is this same mapping exercise run across the entire matrix rather than one detection at a time — which is what converts a single rule's ATT\&CK tag into a program-wide coverage map. *** #### Real-World Notes **The workflow looks linear on the page — in practice it's a loop, and going backward mid-process is a sign of good engineering, not failure.** The clearest example: partway through Alert Development, you may discover the telemetry doesn't actually support the detection you were asked to build (a New Log Source problem, discovered too late). The correct move is not to force a weaker rule that technically compiles — it's to go back: ``` Alert Requirement → Information Gathering → Telemetry unavailable → Detection Gap → Request new data source ``` This is one of the most important professional instincts the source material calls out explicitly: a junior engineer under pressure will often "make the rule somehow" rather than admit the requirement can't be met with current visibility. A mature engineer reports the gap, because a documented gap is more valuable to the organization than an undocumented, unreliable rule pretending to cover it. **Incident response and detection engineering should not be organizationally siloed.** Every real incident in the source discussion produces a new detection requirement almost by definition — an attacker did something that wasn't caught, which is itself evidence about what to detect next. Programs that don't route IR findings back into detection engineering relearn the same lessons repeatedly. **"Zero false positives" is a beginner's goal, not an expert's goal.** Every discussion of tuning in the source material converges on the same point: the target is a usable signal-to-noise ratio, not silence. A rule tuned to never fire falsely will, in practice, also frequently fail to fire on real (but slightly varied) attacker behavior — attackers who know a detection exists tend to test around its exact boundary conditions. *** #### Key Takeaways (Quick Reference) * Detection engineering ≠ writing SIEM rules. It's a repeatable process: gather → test → develop → document → review → deploy → learn. * Five inputs feed the function: Hunting, Intel, Response, (direct) Detection requests, and Business Units. Direct requests are only one of five, not the default. * IOC-based detection is brittle; behavioral detection is durable. Prefer behavioral whenever intel allows it. * Unit testing validates **behavior ↔ telemetry**, not the rule's logic — and it happens *before* the rule is written, not after. * "Detection gap" and "detection logic problem" are different diagnoses — don't force a rule when the real issue is missing visibility. * Testing requires both positive (should alert) and negative (should not alert) cases. * Alert tuning's goal is useful signal at acceptable noise, not zero false positives. * Documentation is a first-class deliverable, not an afterthought — it's what makes a detection maintainable by someone other than its author. * Peer review is a required gate, not an optional courtesy — it catches classes of error the original author structurally can't catch alone. * Program-level outputs (Gap Identification, Metrics) matter as much as individual alerts — "how many rules do we have" is a weaker health signal than "what's our coverage, and is quality trending up." * The whole lifecycle is a loop: alerts, false positives, missed detections, and new intel all become new input for the next cycle. Going backward mid-workflow (e.g. discovering a telemetry gap) is correct engineering behavior, not failure. *** #### External Resources * **MITRE ATT\&CK** (attack.mitre.org) — the framework used throughout for technique mapping and gap identification; essential for turning "what can't we detect" into a structured coverage map. * **SigmaHQ** — the open-source Sigma rule repository/format; directly relevant to the Alert Development stage and to the personal Detection-as-Code project. * **Atomic Red Team** (Red Canary) — the standard tool referenced for repeatable positive/negative testing during Unit Testing and Purple Team validation. * **Red Canary / SpecterOps / Huntress blogs** — real-world detection engineering write-ups that model the Real-World Notes style of thinking (lessons learned, false-positive patterns, novel attacker behavior). * **SANS reading-room papers & blog** — deep, vendor-neutral write-ups on log sources, Windows internals, and detection topics that go beyond this course; useful for the technology-deep-dive topics that follow. *** #### Lab Ideas * **Course lab (Elastic)** — the environment built across Parts 2–5: Elastic Agent shipping telemetry into Elasticsearch, detections authored as TOML, Atomic Red Team generating the behaviour, and the Elastic API + GitHub closing the loop on metrics. This lab is a complete, hands-on instance of the entire I/O model. * **Unit testing dry-run** — before building any detection, practise the 3-step unit test cycle in isolation: pick a technique, review what the log source actually captures for it, execute the technique via Atomic Red Team, and compare the resulting document against expectation *before* writing any rule. * **Detection-as-Code repo scaffold** — one folder per detection containing the rule, a test case, and documentation per the template above. This operationalises the Documentation and Peer Review stages as a real Git workflow, with pull requests as the review gate. Part 4 covers this directly. * **Gap-mapping exercise** — take a narrow slice of the ATT\&CK matrix (Execution and Persistence only) and build a coverage table like the one under *Components → Gap Identification* against your current detections. * **Scaling beyond atomic tests** — once single-technique testing is comfortable, CALDERA or PurpleSharp chain individual techniques into multi-step scenarios, which is where detections that look fine in isolation start revealing sequencing and timing gaps. * **Personal project thread (GOAD + Wazuh + TheHive)** — separate from the course stack, but the same lifecycle applies: attack GOAD, let Wazuh detect, route to TheHive for investigation. Useful for practising the `Detection → Alert → Analyst → Investigation → IR` output pipeline against a different toolset. *** #### Related Topics ``` Detection Engineering Lifecycle → Unit Testing (deep dive, own topic) Detection Engineering Lifecycle → Sigma Rule Authoring Detection Engineering Lifecycle → Detection-as-Code / Git workflow Detection Engineering Lifecycle → Threat Hunting methodology Detection Engineering Lifecycle → Incident Response ↔ Detection feedback loop Detection Engineering Lifecycle → MITRE ATT&CK coverage mapping Detection Engineering Lifecycle → Purple Team exercises (Atomic Red Team, CALDERA) ``` ### 1.2 What Makes a Good Detection *Theory module, episode 6. A process/methodology topic — the criteria a detection must satisfy to be worth deploying, and the working habits that produce them.* *** #### Definition A "good" detection is not simply one that fires when the bad thing happens. That's the beginner's definition, and it's the trap. A rule that fires on the attack but *also* fires two hundred times a day on benign activity is not good — it's a liability, because it trains the SOC to ignore it. A rule that fires accurately but three hours too late to matter is not good either. "Good," in detection engineering, is **multi-dimensional**. This lesson names six dimensions to judge a detection against, and the important shift is understanding that a detection can pass on one axis and fail on another — accurate but too slow, specific but untestable, timely but noisy. A genuinely good detection is one that holds up across *all six* at once. The six criteria are the checklist you run a candidate detection through before it earns a place in production. > **Mindset:** "does it fire?" is the *beginner's* question, and it hides every way a detection can be quietly useless. The six criteria exist to replace that one question with six better ones — *will anyone act on it, in time, without drowning in noise, on a rule I can prove works, that's worth building at all?* *** #### Why It Exists Without explicit criteria, "is this detection good?" collapses into "does it fire?" — and that single question hides every way a detection can be quietly useless. Naming the dimensions forces the questions that "does it fire?" skips: *Will anyone act on it? Will they act in time? Is it drowning in noise? Can I even prove it works? Is it worth building at all?* These six criteria exist so that quality is **assessed deliberately rather than assumed**. They turn a vague gut feeling ("this rule seems fine") into a repeatable evaluation that any engineer on the team can apply to any detection and reach the same verdict. That repeatability is what lets a *team* — not just the rule's author — agree on whether something is ready to ship. *** #### The Six Criteria **1. Response Feedback — *****"In practice, is the alert missing information or logic?"*** The ultimate judge of a detection is the analyst who has to act on it. Response Feedback is the loop where the responders who work your alerts tell you what's wrong with them: an alert that fires correctly but doesn't include the command line, the user, or the host forces the analyst to go hunting for context the detection should have handed them. "Missing information" and "missing logic" are the two failure shapes — either the alert lacks the data needed to triage it, or its logic is subtly wrong in a way only becomes visible once real analysts run real investigations against it. This criterion is why detection engineering must stay connected to incident response: the people downstream are your most honest source of quality signal. **2. False Positive Ratio / Fidelity — *****"If out of 100 triggers, 2 are true positive. Bad alert."*** Fidelity is the ratio of true positives to total triggers. The slide's example is deliberately brutal: 2 real hits out of 100 alerts means an analyst wastes effort on 98 false alarms to catch 2 real ones — and long before they get through 100, they've started reflexively dismissing the alert, at which point it catches *nothing*, including the 2 that mattered. High-noise detections don't just waste time; they actively erode trust and get muted, which is worse than not having the detection at all. > **Reconciling this with §1.1's "the goal isn't zero false positives."** These are not in tension. Zero false positives is usually a sign of an over-narrow rule that also misses real variants — so it's not the target. But *fidelity still has to be high enough that analysts trust the alert.* The goal is a workable ratio: low enough noise that the alert stays credible, without tuning so hard that you carve away real detections. "Not zero FPs" and "2/100 is a bad alert" both point at the same middle ground. **3. Timeliness — *****"Will trigger in enough time for Response to analyze and contain?"*** A perfectly accurate detection that fires after the attacker has already exfiltrated the data and left is a post-mortem, not a detection. Timeliness asks whether the alert reaches responders while there's still an incident to contain. This has real design consequences: a detection that depends on a batch job running overnight, or on correlating a week of logs, may be accurate but structurally too slow. In the course's terms, the `from = "now-6m"` window on every TOML rule is a timeliness decision — the rule looks at the last six minutes so it fires close to when the behaviour happens, not hours later. **4. Specificity — *****"Is the alert catching unrelated activity?"*** Specificity is the flip side of fidelity, viewed from the logic rather than the numbers. A specific detection targets *the behaviour you mean* and nothing adjacent to it. A rule that alerts on "any PowerShell execution" is unspecific — it sweeps up mountains of legitimate administrative activity. Adding conditions (encoded command *and* suspicious parent *and* unusual user) narrows it toward the actual malicious pattern. Specificity is the lever you pull to fix a bad fidelity ratio — but pulled too hard, it becomes the over-narrow rule that fails on the next attacker variation. The craft is finding the specificity that isolates the behaviour without pinning to one brittle artifact. **5. Testability — *****"Can unit tests be performed to confirm?"*** If you cannot reproduce the behaviour and watch the detection fire, you cannot claim the detection works — you can only hope it does. Testability asks whether there's a repeatable way to generate the behaviour (an Atomic Red Team test, a script, a manual procedure) and confirm the rule triggers on it. This is why testability is a *quality* criterion and not just a nice-to-have: an untestable detection is an unverifiable claim, and unverifiable detections are how programs accumulate rules that silently stopped working. A good detection ships with the test that proves it. **6. Compensating Controls — *****"Is this detection alerting on an identified gap in security tooling?"*** This one is about *value*, not correctness — it asks whether the detection is worth building in the first place. A compensating control is a detection that covers a gap where you have no *preventive* control. If your EDR already blocks a technique outright, a detection for that technique is lower value — the bad thing can't happen anyway. But where there's a gap — something you can't prevent, only observe — a detection *compensates* for the missing control and is disproportionately valuable. This criterion is really a prioritisation lens: given limited time, build the detections that cover what nothing else is covering, rather than duplicating protection you already have. *** #### Example Run the course's `data_exfil_over_ftp` detection (`.zip` files sent via FTP `STOR`, caught in Zeek data) through the six criteria: * **Response Feedback** — does the alert give the responder the source host, destination, and filename? Zeek's FTP fields do, so an analyst can immediately see what left and where it went. Passes. * **Fidelity** — how often do legitimate users FTP `.zip` files out? In many modern environments, rarely — FTP itself is unusual — so the noise is likely low. Reasonable. * **Timeliness** — with a `now-6m` window, it fires within minutes of the transfer, while there may still be a session to kill or an account to disable. Passes. * **Specificity** — it targets `.zip` over FTP `STOR` specifically, not all FTP traffic. Reasonably specific, though an attacker using a different extension or protocol slips past (a known limitation, not a flaw). * **Testability** — trivially reproducible: stand up an FTP server in the lab, `STOR` a zip, confirm the alert. Passes. * **Compensating Controls** — if the environment has no DLP and no egress filtering blocking FTP, this detection is *compensating* for that missing prevention. High value. The point of the walk-through: no single criterion decides it. The detection is "good" because it holds up across all six — and where it's weak (specificity against a determined attacker), you now know exactly where, which is itself a useful output. *** #### Common Failure Modes Each criterion names a way detections go bad — reading them as failures is the fastest way to internalise them: * **Fires but useless to the analyst** — passes "does it fire," fails Response Feedback (no context to act on). * **The boy who cried wolf** — low fidelity; so noisy it gets muted, then catches nothing. * **The post-mortem alert** — accurate but too slow to enable containment; fails Timeliness. * **The dragnet** — unspecific; buries the real hit under unrelated activity. * **The unverifiable claim** — untestable; nobody can prove it works, so nobody knows when it breaks. * **The redundant rule** — technically correct but duplicates a preventive control that already blocks the behaviour; spends effort for little marginal coverage. *** #### Key Points (from the lesson) The lesson closes on five habits that produce detections satisfying the criteria above — these are the working discipline, and they map straight back to the lifecycle in §1.1: * **Detection engineers follow a methodological process** — quality is the output of a repeatable workflow, not of individual cleverness. (This is the §1.1 workflow.) * **Unit testing is always preferred** — the Testability criterion, made into a default habit: prove behaviour ↔ telemetry before trusting a rule. * **Alerts should be as specific and timely as possible** — the Specificity and Timeliness criteria restated as a design goal for every rule. * **Documentation should follow a standard** — not free-form notes, but a consistent schema. In this course that standard is *code-enforced*: `validation.py` rejects any TOML detection missing required fields, so "follow a standard" is not a suggestion, it's a build gate. * **All final alert work should be peer reviewed** within the detection engineering team *and* accepted by incident response leadership. Two audiences: engineers check the logic and telemetry; IR leadership confirms the alert is actually actionable for the people who'll work it. This is the Peer Review / Handover stage of the lifecycle, with the explicit reminder that IR — the Response Feedback source — has sign-off. *** #### Key Takeaways (Quick Reference) * A good detection is judged on **six axes at once**, not just "does it fire": Response Feedback, Fidelity, Timeliness, Specificity, Testability, Compensating Controls. * **Fidelity** = true positives ÷ total triggers. 2/100 is a bad alert — low fidelity gets the rule muted, and a muted rule catches nothing. * **Specificity** is the lever for fixing fidelity — but over-tightening creates brittle rules that miss variants. Find the middle. * **Timeliness** is a design property: `from = "now-6m"` is a timeliness decision. Accurate-but-late = post-mortem, not detection. * **Testability** turns a detection from a hope into a verifiable claim; an untestable rule is one that can silently break. * **Compensating Controls** is a *value/prioritisation* lens: build detections that cover gaps nothing else covers, over ones that duplicate existing prevention. * The habits that produce all six: methodological process, unit testing by default, specific + timely design, a code-enforced documentation standard, and peer review by both DE and IR leadership. * "Not zero false positives" (§1.1) and "2/100 is bad" (this lesson) agree: aim for a fidelity high enough to stay trusted, without tuning away real detections. *** #### Related Topics ``` What Makes a Good Detection → The Detection Engineering Lifecycle (§1.1) [the process that produces "good"] Fidelity / Specificity → Alert Tuning (§1.1 Components) [the levers] Testability → Unit Testing (§1.1) · Atomic Red Team (Part 3) Documentation standard → TOML schema & validation.py (Part 4 · Course Reference) Peer Review → Peer Review / Handover (§1.1 workflow stage) ``` ### 1.3 Technology Stack for Detection Engineering *Theory module, episode 7. What tooling a detection engineer actually needs — organised not as a shopping list, but by which workflow stage each tool serves. This is the lecture where the course quietly justifies its entire detection-as-code architecture.* *** #### Definition The "technology stack" is the set of tools that make the detection engineering workflow *possible to execute*. Every stage of the §1.1 workflow — gathering information, unit testing, developing the alert, documenting it, handing it off — needs somewhere to actually happen. The stack is the collection of platforms that provide those somewheres. The important framing, and the one to hold through the whole lecture: **you don't pick tools and then find work for them. You look at each workflow stage, ask what capability it requires, and choose a tool that provides it.** The workflow drives the stack, not the other way around. *** #### The organising idea: every workflow stage needs a tool The lecture opens by walking the five workflow stages from §1.1 and asking, for each: *what do I need to be able to do this?* ``` Workflow stage → Capability it demands ───────────────────────────────────────────────────────────── Information Gathering → somewhere to query logs (a SIEM) Unit Testing → a way to perform AND validate tests (testing tools) Alert Development → query logs + review the security tools (SIEM + stack knowledge) Documentation Creation → a place to store alert documentation (a doc repository) Peer Review / Handover → all of the above feed this step ``` Two of these deserve a second look: **Alert Development needs two things, not one.** You need somewhere to query logs (obvious), *and* you need to review the security tools already in the environment. Why the second? Because a detection is shaped by what other controls already exist — if an EDR already blocks a technique, your detection's value changes (this is the *Compensating Controls* criterion from §1.2). You can't develop a well-targeted alert without knowing what's already covering the ground. **Peer Review / Handover has no tool of its own — "all prior solutions impact this step."** The reviewer is judging the whole package: the query (from the SIEM), the test (from the testing tool), the documentation (from the repository). If any of those tools produced something weak, it surfaces here. Handover is where the quality of every earlier tool choice gets paid back. > **Mindset:** the stack is not a pile of products. It's one tool per capability the workflow demands. When you evaluate any security tool, the right first question is *"which stage of my workflow does this serve?"* — if the answer is "none," you don't need it. *** #### The security tooling landscape The lecture then shows the broader landscape a detection engineer works alongside — seven categories of security product: | Category | Examples (illustrative — not a list to memorise) | Why a detection engineer cares | | --------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | **Firewalls / Proxy** | Palo Alto, Cisco, Checkpoint, pfSense | Network telemetry source; also a preventive control that shapes what you still need to detect | | **Endpoint (EDR/AV)** | CrowdStrike, Tanium, Cylance, Windows Defender | Richest host telemetry (process, file, registry); often the primary data source | | **Network Analysis** | Suricata, Corelight/**Zeek**, ExtraHop, Darktrace | Sees what endpoints can't — traffic, protocols, exfil over the wire | | **Web-Based Tools** | DomainTools, APIVoid, Menlo/urlscan.io, CyberChef | Enrichment and analysis — decode, reputation-check, investigate indicators | | **Automation** | XSOAR, Phantom/Splunk SOAR, TheHive, **GitHub/Python** | Glue and orchestration — where detection-as-code lives | | **SIEMs** | Splunk, **Elastic**, Sentinel, QRadar | Where logs land and detections run — the workflow's centre of gravity | | **Email Security** | Proofpoint, Microsoft 365, Mimecast, FortiMail | Email is the top initial-access vector; a major telemetry source and detection surface | > **The point is not the vendor names.** In six months you will not be quizzed on whether Cylance is endpoint or network. The point is the *categories* — because each category is a **telemetry source and/or a control** that shapes your detections. When you land in a new environment, you inventory which category is filled by which product, because that inventory tells you what you can see and what's already being blocked. Understand the architecture; the vendors are interchangeable. Notice the **bold** entries — they're the ones this course actually uses: **Zeek** (network analysis), **Elastic** (SIEM), **GitHub/Python** (automation). The course isn't teaching you every tool; it's teaching you one representative from each capability the workflow needs. *** #### Testing capabilities (unit tests) For the Unit Testing stage specifically, the lecture calls out the tools that can *generate and validate* attacker behaviour: ``` Splunk Attack Range AttackIQ \ / Atomic Red Team ── BloodHound / \ Caldera Custom VMs ``` * **Atomic Red Team** — the course's choice. Small, per-ATT\&CK-technique tests; lightweight and repeatable. (Covered hands-on in Part 3.) * **Caldera** — chains techniques into full automated adversary emulation; heavier than atomics. * **Splunk Attack Range** — a purpose-built, disposable lab that generates attack telemetry. * **AttackIQ** — a commercial breach-and-attack-simulation platform (continuous validation). * **BloodHound** — maps Active Directory attack paths; a way to identify *what* to test in a domain. * **Custom VMs** — roll your own when nothing off-the-shelf reproduces the behaviour you need. They're not interchangeable: atomics test *one technique cleanly*, Caldera/AttackIQ test *chains and continuous coverage*, BloodHound tells you *where the paths are*. The course picks Atomic Red Team because for learning, the cleanest signal is one technique fired deliberately. *** #### The documentation repository — four requirements Before naming a tool, the lecture defines what a place to store detections must actually provide. This is the setup for the whole detection-as-code argument: ``` Searchable ↓ Change Control ↓ Programmatic Access ↓ Restrictive Roles for Access ``` * **Searchable** — you must be able to find a detection later ("do we already have something for FTP exfil?"). A pile of files nobody can search is write-only. * **Change Control** — every change tracked: who changed what, when, and the ability to see the history. This is version control, in one phrase. * **Programmatic Access** — other tools (validators, sync scripts, metrics) must be able to read and act on the store automatically. A format a machine can parse — which is exactly why the course stores detections as **TOML**. * **Restrictive Roles for Access** — not everyone should be able to push a detection to production; access is gated by role. Hold these four next to the next slide — they're the criteria the CI/CD answer is about to satisfy. *** #### Why CI/CD instead of a wiki? The obvious place to store detection docs is a wiki (Confluence, a Notes app). The lecture argues against it, and the argument is really "which option satisfies the four requirements above?" A wiki is *searchable* — and stops there. What it can't do: * **Data validation** — a wiki will happily let you save a malformed, broken detection. CI/CD tooling can *reject* it before it ships. * **Automation (e.g. GitHub Actions)** — a wiki can't run your tests, sync your rules, or regenerate metrics on its own. * **Revert releases and changes** — a wiki's history is for reading; git's history is for *rolling back* a bad detection instantly. * **Integration with other security tools** — CI/CD sits naturally in a pipeline that pushes to your SIEM; a wiki is a dead end. > **This is the hinge of the whole course.** "Documentation should follow a standard" (§1.2) plus "the store needs validation, automation, revert, and integration" leads to exactly one answer: store detections as code, in git, behind a CI/CD pipeline. Everything in Parts 4 and 5 is the build-out of this one decision. *** #### What *is* a CI/CD pipeline? CI/CD (Continuous Integration / Continuous Delivery/Deployment) comes from software engineering. Its three properties, per the lecture: * **Typically used for code development** — it's a developer tool the course is borrowing. * **Allows staging, pushing, and reverting changes** — the change-control and revert requirements, met. * **Can enforce requirements prior to pushing to production** — the validation requirement, met: the pipeline is a *gate*. The standard software pipeline, and the detection-engineering parallel the lecture draws directly beneath it: ``` Continuous Integration Continuous Delivery Continuous Deployment ──────────────────────────────────── ────────────────── ───────────────────── Software: Develop → Test → Merge → Updated Repo → Updated Production Detection: Develop → Validate → Merge → New Detection Ready → Push to SIEM ``` Read the bottom row — it *is* the detection engineering workflow, mechanised: * **Develop** — author the detection (Alert Development stage) as a TOML file. * **Validate** — CI runs your checks (`validation.py`, `mitre.py`) and *fails the build* if the detection is malformed or mis-mapped. This is Unit Testing + the documentation standard, enforced by a robot. * **Merge** — Peer Review / Handover: the pull request is where a human approves before it can merge. * **New Detection Ready** — the repo now holds a validated, reviewed detection (Continuous Delivery). * **Push to SIEM** — the sync script deploys it to Elastic automatically (Continuous Deployment). The distinction between the two CDs matters: **Delivery** gets a validated change into the repo, ready to go; **Deployment** takes that final automated step of pushing it live to the SIEM. The course automates all the way to deployment — you merge a pull request, and the detection appears in Elastic with no manual step. *** #### Example — the course's own stack, mapped Everything above resolves into the specific stack this course builds, one tool per capability: | Workflow need | Course's choice | Category | | ----------------------------------------------------------------------- | ------------------------------ | ------------------ | | Query logs (Information Gathering, Alert Dev) | **Elastic** | SIEM | | Host telemetry | **Sysmon** (via Elastic Agent) | Endpoint | | Network telemetry | **Zeek** | Network Analysis | | Perform + validate unit tests | **Atomic Red Team** | Testing | | Store detections (searchable, change control, programmatic, role-gated) | **GitHub** (TOML in git) | Documentation repo | | Validate + deploy (the pipeline) | **GitHub Actions + Python** | Automation / CI-CD | This is why the [Course Reference](https://claude.ai/cowork/00-course-reference.md) stack table looks the way it does: each row is one capability this lecture said the workflow demands. The tech-stack lecture is the *why*; the Course Reference is the *what*. *** #### Key Takeaways (Quick Reference) * The stack is chosen **workflow-first**: each stage demands a capability, and you pick one tool per capability — not the reverse. * The seven tool categories (firewall, endpoint, network, web tools, automation, SIEM, email) matter as **architecture, not brand names**. Each is a telemetry source and/or a control. Don't memorise vendors; know what fills each slot in an environment. * **Alert Development needs stack awareness**, not just log access — what's already prevented changes what's worth detecting (ties to *Compensating Controls*, §1.2). * Unit-test tools aren't interchangeable: **atomics = one clean technique** (course choice), Caldera/AttackIQ = chains/continuous, BloodHound = find the paths. * A detection store must be **Searchable, Change-Controlled, Programmatically Accessible, and Role-Restricted.** A wiki gives you only the first. * **CI/CD beats a wiki** because it adds validation, automation, revert, and integration — the exact gaps a wiki leaves. * The detection pipeline **Develop → Validate → Merge → Ready → Push to SIEM** is the §1.1 workflow mechanised: validate = testing + standard enforced by CI, merge = peer review, push = automated deployment. * This lecture is the *justification* for the whole detection-as-code architecture; Parts 4–5 are the *implementation*. *** #### Related Topics ``` Technology Stack → The Detection Engineering Lifecycle (§1.1) [the workflow the stack serves] Alert Development / stack awareness → Compensating Controls (§1.2) Documentation repository → TOML schema & validation.py (Part 4) CI/CD pipeline → GitHub Actions & Elastic sync (Part 5 · Course Reference) Testing capabilities → Atomic Red Team (Part 3) The whole architecture → Course Reference (Detection-as-Code) ``` ### 1.4 MITRE ATT\&CK & the Cyber Kill Chain *Theory module, episodes 8–9. The framework the entire course maps its detections to — what ATT\&CK is, how it's structured, how to read the matrix and the Navigator, and how it compares to the older Cyber Kill Chain. This is the shared language of detection engineering.* *** #### Definition **MITRE ATT\&CK** (Adversarial Tactics, Techniques, and Common Knowledge) is a free, continuously updated **knowledge base of real-world adversary behaviour**, organised into a matrix. It's not a tool you run and not a standard you comply with — it's a catalogue. It takes everything defenders and threat intel have observed attackers actually *doing* and gives each behaviour a name, an ID, a description, and notes on how to detect and mitigate it. The reason it dominates detection engineering is that it turns a chaotic, infinite space ("all the things an attacker might do") into a **finite, addressable list** you can point at, map to, and measure against. Every detection in this course carries an ATT\&CK mapping for exactly this reason — it's how a single rule connects to the bigger picture of "what are we covered against." > **Think of it as** a periodic table for attacker behaviour. Chemists didn't invent the elements; they organised known ones into a grid so everyone could refer to the same thing and spot the gaps. ATT\&CK does that for adversary techniques — and, like the periodic table, new entries get added as they're discovered. *** #### Why It Exists — "Why MITRE?" The lecture gives three reasons, and each maps to a real problem detection engineers hit: * **A common language.** Before ATT\&CK, one vendor's report said "the malware achieved persistence via a registry key," another said "startup autorun," a third said "run-key implant" — all the same thing, three names. ATT\&CK gives it *one* ID (T1547.001). Now security products, vendors, threat intel, and internal teams can all point at the same identifier and mean the same thing. This is the single biggest reason it caught on. * **Showing detection coverage.** Because the matrix is a finite grid, you can colour in the cells you can detect and leave the rest blank — instantly turning "how good is our detection program?" from a hand-wave into a *picture*. (This is the Navigator, below.) * **Prioritising detection engineering effort.** Once you can see coverage, you can decide *what to build next* rationally — target the techniques your threat intel says are relevant and that you currently can't see, instead of building whatever's top of mind. > **Mindset:** ATT\&CK's value to a detection engineer isn't "a list of attacks to memorise." It's that it makes coverage *visible* and *measurable*. You can't manage what you can't see — ATT\&CK is what lets you see the shape of your own blind spots. *** #### Two key concepts: Tactics vs Techniques The entire matrix is built from two layers, and the cleanest way to hold them is the **Why vs How** split from the slide: | | **Tactics** | **Techniques / Sub-techniques** | | -------------- | -------------------------------- | ----------------------------------------------- | | Answers | **"Why"** — the adversary's goal | **"How"** — the method to achieve it | | Count | **14 categories** (Enterprise) | 190+ techniques, 400+ sub-techniques\* | | ID format | **TA\*\*\*\*ID** (e.g. `TA0002`) | **T\*\*\*\*ID** (e.g. `T1059`, sub `T1059.001`) | | Role in matrix | The **columns** — the headings | The **cells** under each column | *\*The technique/sub-technique counts grow with every release (the slide is from ATT\&CK v13; later versions have more). The 14 tactics have been stable. Don't memorise the numbers — know the structure.* **Tactics = the adversary's objective at a moment.** "I need to get *in*" (Initial Access), "I need to *run code*" (Execution), "I need to *stay*" (Persistence), "I need to *steal the data*" (Exfiltration). There are 14 of these, and they roughly read left-to-right as an attack progresses — Reconnaissance → Resource Development → Initial Access → Execution → Persistence → Privilege Escalation → Defense Evasion → Credential Access → Discovery → Lateral Movement → Collection → Command and Control → Exfiltration → Impact. **Techniques = the specific way an adversary achieves that objective.** Under Execution (TA0002) sits Command and Scripting Interpreter (T1059); under *that* sits the sub-technique PowerShell (T1059.001). A **sub-technique** is just a more specific flavour of its parent. > **Think of it as** a filing system: the **tactic** is the goal on the folder tab, the **technique** is a document inside, the **sub-technique** is a specific paragraph in that document. `T1059.001` = "Execution → Command and Scripting Interpreter → PowerShell." The relationship, drawn out: ``` TACTIC (why) TECHNIQUE (how) SUB-TECHNIQUE (specific how) ───────────────────────────────────────────────────────────────────────────── Execution (TA0002) ──▶ Command & Scripting ──▶ PowerShell (T1059.001) Interpreter (T1059) Windows Cmd (T1059.003) Python (T1059.006) ``` *** #### The Ransomware example — mapping an incident to MITRE The lecture's clearest demonstration: take a real incident and translate each step into ATT\&CK. This *is* the core skill. ``` What actually happened (Incident) Its ATT&CK mapping ───────────────────────────────── ────────────────────────────────────── Phish TA0001 — Initial Access user downloads phishing attachment → T1566.001 Spearphishing Attachment ↓ ↓ PowerShell TA0002 — Execution attachment runs PowerShell, pulls .exe → T1059.001 Command & Scripting: PowerShell ↓ ↓ Ransomware TA0040 — Impact the .exe encrypts the hard drive → T1486 Data Encrypted for Impact ``` What this exercise teaches: * **An incident is a *****chain***** of tactics, not one label.** A single ransomware case touches Initial Access, Execution, and Impact. Real intrusions walk across the matrix — this is why coverage matters across columns, not just in one. * **The mapping is how a detection gets its `threat` block.** In the course's TOML, the `msfvenom_powershell_payload` detection carries exactly this: `TA0002 Execution → T1059 → T1059.001`. The ransomware example is the manual version of what you'll encode in every rule. * **Reading an incident in ATT\&CK terms is the daily language of a SOC.** "We caught them at Execution but missed Initial Access" is a precise, shared sentence only because ATT\&CK exists. *** #### Navigating the matrix (lecture 9) The [**Enterprise Matrix**](https://attack.mitre.org/matrices/enterprise/) on attack.mitre.org is the primary interface. Reading it: * **Columns are the 14 tactics**, left-to-right roughly following attack progression. Each column header shows the technique count under it. * **Cells are techniques**; a small number badge (e.g. "PowerShell (4)") means it has sub-techniques you can expand. The `show/hide sub-techniques` toggle controls this. * **Platform filters** (Windows, macOS, Linux, Cloud, Network, Containers, etc.) narrow the matrix to what's relevant to your environment — a Windows shop hides the macOS-only techniques. * **Click any technique** to reach its page: description, procedure examples (real groups that used it), **data sources** needed to see it, **detection** guidance, and **mitigations**. For a detection engineer, the *data sources* and *detection* sections are gold — they tell you what telemetry a detection for this technique requires. > **Mindset:** don't try to "learn the matrix." It's a *reference*, not a syllabus. The skill is knowing how to *navigate* to the technique you care about and read what telemetry detects it — the same way you don't memorise a dictionary, you know how to look a word up. *** #### The ATT\&CK Navigator The [**ATT\&CK Navigator**](https://mitre-attack.github.io/attack-navigator/) is a separate free web tool for **annotating the matrix** — colouring, scoring, and commenting on cells to build a visual picture. Its killer use for detection engineering is **coverage mapping**: * You create a **layer** — a copy of the matrix you can colour. * Colour green the techniques you can detect, red the ones you can't, yellow partial. Now your entire detection posture is one heat-map. * Layers can be **combined** (e.g. overlay "techniques APT29 uses" onto "techniques we detect" to see exactly where you're exposed to that specific actor). * Layers export/import as **JSON**, which means they can be *generated by code*. This last point connects straight to the course: the repo's `toml_to_navigator.py` script reads every detection's ATT\&CK mapping and **auto-generates a Navigator layer** (`metrics/navigator.json`). So your coverage map isn't hand-maintained — every time you add a detection, the pipeline redraws the heat-map for you. That's Gap Identification (from §1.1) made automatic. > **Think of it as** a "fog of war" map for your defenses. The Navigator lights up the parts of the attack landscape you can see; everything still dark is where an attacker moves unobserved. The job is to light up more of the map — starting with the areas your threat intel says matter most. *** #### The Cyber Kill Chain (the predecessor) Before ATT\&CK, [**Lockheed Martin's Cyber Kill Chain**](https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html) (2011) was the dominant model. It describes an intrusion as **7 sequential stages**: ``` 1. Reconnaissance — research/select the target 2. Weaponization — build the payload (e.g. malware + exploit in a doc) 3. Delivery — send it (email, USB, web) 4. Exploitation — the payload's code executes on the victim 5. Installation — malware installs, establishes a foothold 6. Command & Control — the foothold beacons out for remote control 7. Actions on Objectives — the attacker does the actual damage (steal, encrypt) ``` **How it compares to ATT\&CK — and why ATT\&CK largely superseded it:** | | **Cyber Kill Chain** | **MITRE ATT\&CK** | | -------- | ----------------------------------------------- | ---------------------------------------------------- | | Shape | 7 **linear, sequential** stages | 14 tactics, **non-linear** — attackers loop and skip | | Depth | High-level phases only | Hundreds of **specific techniques** under each | | Detail | "Delivery happened" | "T1566.001 Spearphishing Attachment happened" | | Best for | Explaining an intrusion's **arc** to leadership | **Engineering and measuring** detections | > **Mindset:** the Kill Chain and ATT\&CK aren't rivals; they're different zoom levels. The Kill Chain is the *story* of an attack (great for a briefing slide); ATT\&CK is the *engineering detail* (what you actually build detections against). ATT\&CK "won" for detection work because you can't write a rule against "Delivery" — but you can write one against T1566.001. *** #### Relation to detection engineering ATT\&CK is not a side-topic in this course — it's woven through the whole pipeline: * Every TOML detection's **`[[rule.threat]]` block** is an ATT\&CK mapping (tactic + technique + sub-technique). * `mitre.py` in CI **validates those IDs against MITRE's live** [**ATT\&CK data**](https://attack.mitre.org/resources/attack-data-and-tools/) (the STIX/JSON feed) — so a typo'd or deprecated technique ID fails the build. Your mappings can't silently rot. * `toml_to_navigator.py` turns those mappings into a **coverage layer** automatically. * Which means "what can't we detect yet?" (Gap Identification, §1.1) is answered by a picture your pipeline regenerates on every commit. *** #### Key Takeaways (Quick Reference) * **ATT\&CK is a knowledge base, not a tool** — a finite, addressable catalogue of real adversary behaviour. It's the shared language of detection engineering. * **Tactics = Why (the goal), Techniques = How (the method).** 14 tactics (stable), hundreds of techniques (grows each version — don't memorise counts). IDs: **TA** for tactics, **T** for techniques, `T####.###` for sub-techniques. * **An incident is a chain across tactics**, not one label — the ransomware example spans Initial Access → Execution → Impact. * The three reasons ATT\&CK matters: **common language, coverage visibility, prioritisation.** * The **matrix** is a reference to navigate, not a syllabus to memorise — the payoff is each technique page's *data sources* and *detection* guidance. * The **Navigator** turns coverage into a heat-map (layers as JSON) — and the course auto-generates it from your detections via `toml_to_navigator.py`. * **Cyber Kill Chain vs ATT\&CK:** 7 linear stages (the story, for leadership) vs 14 non-linear tactics with deep technique detail (the engineering, for detections). Different zoom levels, not rivals. * In this course, ATT\&CK is **validated in CI** (`mitre.py`) and **visualised automatically** — mappings can't rot, and coverage is always current. *** #### External Resources * [**MITRE ATT\&CK Enterprise Matrix**](https://attack.mitre.org/matrices/enterprise/) — the primary reference; live matrix, technique pages with detection guidance. * [**ATT\&CK Navigator**](https://mitre-attack.github.io/attack-navigator/) — the coverage-mapping tool; build and export layers. * [**ATT\&CK Data & Tools**](https://attack.mitre.org/resources/attack-data-and-tools/) — the machine-readable STIX/JSON feed of the whole framework; this is the source `mitre.py` validates against. * [**Lockheed Martin Cyber Kill Chain**](https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html) — the 7-stage predecessor model. * [**IBM — What is the MITRE ATT\&CK framework?**](https://www.ibm.com/think/topics/mitre-attack) — a clean written primer if you want the concepts in prose. * [**YouTube: MITRE ATT\&CK overview**](https://www.youtube.com/watch?v=LCec9K0aAkM) — video walkthrough of the framework and matrix. * **MITRE MAD (MITRE ATT\&CK Defender)** — MITRE's own training/certification subscription; deeper, structured ATT\&CK skilling. * **Center for Threat-Informed Defense** — MITRE's R\&D arm publishing open ATT\&CK-based projects (mappings, tools, research). *(Sources for the framework facts in this section:* [*MITRE ATT\&CK*](https://attack.mitre.org/matrices/enterprise/)*,* [*CrowdStrike ATT\&CK primer*](https://www.crowdstrike.com/en-us/cybersecurity-101/cyberattacks/mitre-attack-framework/)*,* [*Lockheed Martin*](https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html)*.)* *** #### Related Topics ``` MITRE ATT&CK → Technology Stack: "coverage" & "prioritisation" (§1.3) Tactics/Techniques mapping → TOML [[rule.threat]] block & mitre.py (Part 4 · Course Reference) Navigator coverage layer → Gap Identification (§1.1) · toml_to_navigator.py (Part 5) Ransomware example → the 10 course detections mapped to ATT&CK (Course Reference) Cyber Kill Chain → MITRE ATT&CK (zoom levels of the same intrusion) ``` *** ### 1.5 The Lab *Course lectures 010–017 (the VirtualBox install click-through — importing ParrotOS / Windows 11 / Ubuntu, VM snapshots, disabling Defender, installing Zeek) are **not written up here**. A custom purple lab was built instead of following the course's setup step for step.* 📎 **Lab writeup:** [GOAD Replication Fix — Purple Lab (LinkedIn)](https://www.linkedin.com/posts/ali-khaled-57b606236_goadreplicationfixpurplelab-ugcPost-7496978344727650304-Zyzc/) #### Lab structure The course ships a small single-box lab. It's swapped out for a **GOAD (Game of Active Directory)** build — a multi-domain forest with a real trust and genuine attack paths — because detections should be tested against something that looks like an environment worth defending, not one machine. ``` ┌─────────────────────────────────────┐ │ GOAD FOREST (red side) │ │ │ │ Domain A ──── forest trust ──── Domain B │ (DCs, member servers, real AD attack paths) └───────────────────┬──────────────────┘ │ telemetry ┌───────────────────▼──────────────────┐ │ BLUE / PURPLE TOOLING │ │ ELK / Elastic · Wazuh · (more) │ ← being layered in └───────────────────┬──────────────────┘ │ ┌───────────────────▼──────────────────┐ │ Detection engineering + purple loops │ │ rule creation · testing · tuning │ └───────────────────────────────────────┘ ``` | Layer | What's in it | Role | | -------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------- | | **Attack surface** | GOAD — multiple domains, forest trust, member hosts | A realistic AD environment to attack and detect in (vs. the course's single box) | | **Telemetry / SIEM** | ELK / Elastic, Wazuh *(being layered in)* | Collect host + AD logs; where detections run | | **Workflow** | detection rules, purple-team loops | The detection-engineering practice this whole notebook feeds | > **Why GOAD over the course lab:** more domains and trusts = more attack *scenarios* = more behaviours to detect. A single box can only teach you so many techniques; a forest gives you lateral movement, trust abuse, and cross-domain paths — the things real detections are written for. The build is transitioning from **red-team-only** to a full **purple-team / detection-engineering range** — the ELK + Wazuh layer is what turns an attack playground into somewhere detections get built, tested, and tuned. #### Field note — fix the ground truth before you build detections A war story from bringing this lab back online, worth keeping because the *lesson* is pure detection engineering: The lab sat powered off for months. On restart the domain controllers wouldn't replicate. It **looked** like networking (adapters had been changed earlier) — but that was a red herring. **Real root cause:** the lab had been off longer than the **AD tombstone lifetime (180 days)**. Replication had dropped into its protective lockout — `repadmin` throwing **error 8614**: *"the time since the last replication has exceeded the tombstone lifetime."* No amount of DNS or Kerberos poking clears that; it's a deliberate safety stop. **The fix — non-destructive, no rebuild:** 1. Confirm it's genuinely tombstone, not time skew or DNS (clocks were fine, DNS resolved cleanly). 2. Temporarily set **"Allow Replication With Divergent and Corrupt Partner"** on both DCs. 3. `repadmin /syncall`, verify **0 failures both directions**, run a lingering-object advisory check. 4. Turn the override back **off**. Result: a clean, replicating forest — users, GPOs, and trusts all intact. > **Mindset — why this matters for a detection person:** a broken forest doesn't fail *quietly*. It **floods the logs with replication error events**, and that noise poisons the exact baseline you're trying to learn. If you build detections on top of a sick environment, you're modelling the sickness, not the threat. **Fix the ground truth first, then build detections on top of it.** "Is my telemetry healthy?" is a prerequisite question, not an afterthought — it's the same instinct as *Unit Testing* (§1.1): validate the environment produces clean, trustworthy data before you trust anything built on it. #### Why it's built this way — the concepts that carry over The mechanical steps don't need notes; the reasoning does. Each choice maps to a detection-engineering idea covered earlier: * **Snapshots** → let an attack re-run from a clean state, which is what makes a unit test *repeatable* (§1.1, §1.2 *Testability*). * **Defender disabled in the lab** → so attacker behaviour actually executes and produces telemetry to detect, instead of being blocked before it logs anything. In production you keep the preventive control; in a *detection lab* you deliberately remove it to study the behaviour it would otherwise stop — the *Compensating Controls* idea (§1.2), inverted for a lab. * **Zeek** → network telemetry the endpoint can't see (§1.3 *network analysis*; feeds the FTP-exfil and web-scanner detections in Part 3). * **Component "why" in general** → [Course Reference → technology stack](https://claude.ai/cowork/00-course-reference.md#the-technology-stack--and-why-each-piece). > **Mindset:** skipping the click-through is fine; skipping the *why* is not. You don't need notes on which button imports an OVA — you need to know what each component contributes to the `attack → telemetry → detection` pipeline.