17.5 How to Triage Alerts and Avoid Panic
Triage Is a Checklist, Not a Feeling
An alert is an unverified claim, and panic comes from treating it as a verdict. Work a fixed order. First, validity: does the event exist in the raw source, and is the alert describing what the rule claims? Second, scope: one process on one host, one account on many hosts, or a whole subnet -- scope sets severity far more reliably than the word "critical" in a rule title. Third, actor: who or what performed the action, and was that identity expected to be doing it at that hour. Fourth, timeline: pull events from before and after the alert, because the alerting event is almost never the first interesting one. Fifth, impact: what data or privilege was actually reachable. Write every answer down as you go; a triage note is what makes the next similar alert take four minutes instead of forty.
The Hash Check and the Allow-List
When a file is involved, compute its SHA-256 before touching it -- never open the sample to "see what it does" on your own workstation. Look the hash up against your internal allow-list of shipped software, then vendor and public multi-engine reputation feeds, then a detonation sandbox if it is unknown. A known-good hash whose path and publisher match is usually an allow-list gap, not an incident; a hash unseen anywhere, unsigned, or written into a temp directory minutes ago is the opposite. Reputation is only a population signal, so brand-new legitimate binaries look "unknown" and uniquely packed malware looks "clean" -- corroborate with path, signature validity, parent process, and delivery method.
Where False Positives Come From
Most noise is legitimate tools behaving like attackers. Vulnerability scanners fan out across ports and look like horizontal scans; backup agents read every file at odd hours; deployment and patch systems spawn shells and install services; RMM agents reach into other processes; admins run encoded PowerShell for inventory. Exercise calendars belong in triage context, because a sanctioned penetration test produces real-looking events with no adversary behind them. The fix is rarely deleting a rule: tag known actors, scope rules to asset groups, and keep an allow-list with owners and review dates so stale entries get questioned instead of masking an intrusion.
Escalate, Contain, or Close
Three outcomes, each with a bar. Escalate when scope or privilege is real -- domain account misuse, egress to an unknown host, persistence mechanisms -- preserve evidence first (process lists, memory, the file, the timeline), then contain the narrowest footprint that stops the harm. Closing as false positive requires a stated benign explanation and a fix: tune, allow-list, or ticket the asset owner. Reimaging a host before scoping destroys the answer to "how many machines?" and turns one incident into a repeat.
Architecture Diagram
Key Takeaways
- Answer validity, scope, actor, timeline, then impact -- in that order, written down, every time.
- Severity comes from scope and asset criticality, not from the alert rule's own label.
- Hash-check first with SHA-256, then allow-list, then reputation, then sandbox; corroborate with path, signature, and parent.
- Most false positives are scanners, backups, deployers, RMM agents, and admin shells -- tag the actor instead of deleting the rule.
- Escalate with evidence preserved and contain narrowly; closing as FP needs a stated benign explanation plus a fix.