Security Alert Tuning

An alert nobody reads is worse than no alert: it trains people to ignore the one that matters. You make every alert worth a human's attention: fewer, more accurate, each with a clear reason to exist and a first step for whoever receives it. The rule for every change is the same: noise goes down and the real case still fires, and you have proof of both.

When to use

• "We get 300 security alerts a day and ignore them all."
• "This alert fires every night at 2am; can we fix it?"
• "Did we miss something because the alert was muted?"
• "Set up sensible alerting for our new monitoring tool."
• A monthly or quarterly detection review.

What you need

• Access to the alerting tool the owner already uses (a SIEM, a log platform, the cloud provider's security alerts, an endpoint tool, uptime or error monitoring with security rules): the console in your browser signed in by the owner, a connected app, or exports. Read access for analysis; change access only after the owner approves the tuning plan.
• 30 days of alert history (90 if volume is low), with dispositions if analysts recorded them.
• The rule definitions (queries, thresholds, suppressions).
• Who receives alerts and how (pager, chat channel, email, ticket).

Method

1. Measure. Export the alerts and build a table per rule with execute_code: total count, alerts per day, distinct entities (users, hosts, IPs), and dispositions. Sort by volume. A few rules usually produce most of the noise; start there.
2. Label a sample when dispositions are missing: up to 20 alerts per noisy rule (all if fewer), each labelled:
   • TP: true positive, malicious or policy-breaking
   • BTP: benign true positive, the rule worked but the activity was expected (a scheduled scan, an admin doing admin work)
   • FP: false positive, the logic matched something it should not Confirm labels you are unsure of with the owner or the system owner (ask_teammate).
3. Compute precision per rule: TP / (TP + BTP + FP). Note acknowledgement time if available. A rule with high volume and near-zero precision is the priority.
4. Find the cause of each FP and BTP pattern: wrong field, missing condition, threshold too low for this environment, one noisy entity, duplicate alerts for one event, a test environment included, a known scheduled job.
5. Choose the fix, in this order of preference:
   1. Fix the logic (correct field, add the missing condition).
   2. Narrow the scope precisely: exclude one known entity doing one known thing (this service account, from this host, running this job), never a whole category such as "all admins" or "all internal IPs".
   3. Adjust the threshold using the environment's own baseline (for example above the 99th percentile of that entity's normal), not a guess.
   4. Aggregate: one alert per entity per time window instead of one per event.
   5. Re-route: lower severity or send to a daily digest or ticket queue instead of a page.
   6. Disable: only when the rule cannot be made useful, the owner signs off, and another detection covers the same threat (or the gap is recorded).
6. Every exclusion and allowlist entry gets: what it excludes, why, who asked for it, and an expiry or review date (90 days is a sensible default). Expired entries come back for review, not silent renewal.
7. Prove it still works. Before and after the change:
   • replay the past 30 days through the tuned logic and compare counts
   • confirm known true positives from history would still fire
   • where safe and approved, generate a harmless test event that should trigger it (for example a few failed logins on a test account you own), and confirm the alert arrives at the right place Record the evidence.
8. Route by urgency. Page a human only for alerts that are urgent, actionable and high-confidence. Everything else goes to a ticket queue or digest. Each alert must say what it means and what to check first.
9. Document each detection using the strategy template in references/detection-strategy-template.md: goal, ATT&CK mapping, how it works, blind spots, known false positives, how to validate, priority and response steps. Keep the documents next to the rules, ideally in version control.
10. Check coverage. List the threats that matter for this company (from its threat model if one exists, otherwise account takeover, cloud key misuse, admin changes, data exports, malware on endpoints, web attacks) and mark which detections cover each. Gaps go to the owner as a list.
11. Schedule a monthly review with cronjob: top rules by volume, precision trend, expiring exclusions, rules that never fired (test them; silence can mean broken).

Worked example

• Rule: "Multiple failed logins" fires when an account has 5 or more failures in 10 minutes.
• Measured: 410 alerts in 30 days, about 14 a day. Sample of 20: 0 TP, 17 BTP, 3 FP.
• Causes: 15 of the 17 BTPs come from one integration account whose stored password expired monthly; the 3 FPs came from a test tenant.
• Changes: fixed the integration's credential handling with its owner (logic fix at the source); excluded the test tenant by its exact tenant ID with a 90-day expiry; raised the threshold for human accounts to 10 failures in 10 minutes based on the 99th percentile of normal; aggregated to one alert per account per hour.
• Validation: replayed 30 days: 410 alerts became 9. Two historic true positives from last year's incident both still fire. A harmless test with 12 failed logins on a test account produced one alert in the right channel.
• Routing: stays a ticket, pages only when followed by a successful login from a new country.

What every alert must carry

• what fired and why, in one plain sentence
• the entity (user, host, IP) and a link to the raw events
• the first two or three checks to make, from the detection document
• who owns the rule

Output

• Tuning table per rule: before volume, precision, cause, change made, after volume, validation evidence, owner, review date.
• Updated detection documents.
• Coverage list with gaps.
• A show_card with total alerts per day before and after, precision before and after, and rules changed.

Checks before you finish

• Every change was approved by the owner before it was applied.
• Every tuned rule has evidence that it still fires on a true positive.
• Every exclusion has a reason, a requester and an expiry date.
• No rule was disabled without a recorded compensating detection or accepted gap.
• Detection documents are complete for every rule touched.

Pitfalls

• Broad allowlists. Excluding "all traffic from the office" also excludes the attacker on a compromised office laptop.
• Tuning by gut feel. Use the counts and labels; a rule that feels noisy may be the only one catching real activity.
• Disabling instead of fixing. The threat does not go away when the alert does.
• Never testing quiet rules. A rule that has not fired in six months may be broken by a log format change.
• Paging for everything. Pages are for urgent and actionable; the rest belongs in a queue.
• Sign-off. The owner approves all rule changes, disables and exclusions. For regulated environments, a qualified security lead should review changes to critical detections.

Template: references/detection-strategy-template.md. Credits: references/CREDITS.md.