Log Threat Hunting

You look for an attacker who has not set off any alarm yet. Each hunt starts from a written hypothesis or a baseline of normal, runs against the logs that would show it, and ends in one of three honest answers: found, not found, or could not tell because the data is missing. Good hunts also leave something behind: a new detection, a logging fix, or a cleaner picture of normal.

When to use

• "Can you check the logs and see if anyone has got in?"
• "Do a monthly log review."
• "There's news about attacks on ; are we affected?"
• "We saw one weird login; look for anything else like it."
• After hardening or an incident, to confirm nothing was missed.

What you need

• Read-only access to the log sources, chosen by what the owner has:
  • Linux servers: SSH to a read-only or sudo account for journalctl, /var/log/auth.log or /var/log/secure, web server logs
  • cloud audit logs: console in your browser signed in by the owner with a read-only role, or exports
  • email and identity (Google Workspace, Microsoft 365, the identity provider): admin audit and sign-in logs through the console or a connected app
  • code host audit log (GitHub, GitLab organisation audit log)
  • firewall, VPN, DNS or proxy logs if they exist
• The retention window of each source. If logs only go back seven days, every conclusion is limited to seven days; say so.
• What is normal: admin names, office and home countries, working hours, known service IPs. Ask the owner with clarify.

Method

1. Inventory the data. For each source, record what it covers, from when to when, time zone, and gaps. Missing sources are findings in their own right: "no record of SSH logins before 3 days ago because journald is not persistent".
2. Pick the hunt type:
   • Hypothesis-driven: a specific idea, for example "an attacker reused a leaked password against our email". Write it as: "If happened, we would see in ."
   • Baseline: describe normal for a field (countries, user agents, processes, destinations), then examine the rare values. Choose from references/hunt-catalogue.md, which lists starter hunts with their ATT&CK technique IDs, sources and queries. Prioritise by what the company exposes: public SSH, email accounts, cloud keys, a web app.
3. Plan before querying. Note the hypothesis, data sources, time range, what a positive looks like, and what benign activity looks similar. Use todo for multi-hunt sessions.
4. Query with free tools on your own computer: grep, awk, jq, journalctl, and Python with pandas in execute_code for larger sets. Work on copies of exported logs, never edit the originals. Normalise time zones to UTC first.
5. Use these analysis patterns:
   • stacking (least frequency): count values of a field and look hardest at the rarest ones (a user agent seen once, a country seen once)
   • sequence: many failures followed by a success from the same source or for the same account
   • first seen: new admin, new key, new country, new process, new outbound destination compared with the baseline period
   • volume and time: spikes in downloads or API calls, activity outside the account owner's normal hours
6. Triage every hit. Look for the benign explanation first: travel, a new office, a VPN, a deploy, a scheduled job, a new employee. Confirm with the account owner or the owner through ask_teammate or clarify; do not message staff outside the company. Record the verdict and how you reached it.
7. Escalate, do not act. If a hit looks malicious and the benign explanations fail, stop hunting that thread, tell the owner immediately, and move to the security-incident-triage process. Do not disable accounts, block IPs or delete anything yourself.
8. Leave something behind. For each hunt, decide whether it should become a scheduled detection (hand the logic to whoever runs alerting), whether a logging gap needs fixing, and whether the baseline should be saved for next time.
9. Schedule recurring hunts with cronjob: a weekly check of authentication anomalies and new admins or keys; a monthly rotation through the rest of the catalogue.

Choosing what to hunt first

If the company has	Start with
Google Workspace or Microsoft 365	Sign-ins from new places, forwarding rules, third-party app grants (H2, H4, H5)
Servers with SSH open to the internet	Logins after failures, new keys, new cron and services (H1, H6, H7)
A cloud account with access keys	New keys, unusual API use, logging changes, new regions (H10 to H12)
A public web app	Probing patterns and error spikes in access logs (H9)
Sensitive data in storage or SaaS	Unusual downloads and exports (H13)

A worked hunt, start to finish

• Hypothesis: "If someone guessed or reused a password for SSH on web-1, we would see a run of failed logins followed by an accepted login from the same IP in the SSH journal during the last 30 days."
• Data check: journald is persistent on web-1 and holds 41 days; logs are UTC. Good enough.
• Query: count failures by source IP, list accepted logins with source IP (hunt H1 in the catalogue), then join the two lists in execute_code.
• Result: 18,400 failures from 1,312 IPs (normal internet noise for an open SSH port); 96 accepted logins, all by key, from 3 IPs.
• Triage: the 3 IPs are the office, one admin's home connection and the CI runner, each confirmed with the owner. None of the 3 appear in the failure list.
• Conclusion: not found, for web-1, for the last 41 days.
• Follow-ups: SSH is open to the whole internet; recommend restricting it (hardening). Turn the join into a weekly scheduled check.

Output

A hunt record per hypothesis, saved to the workspace, plus a session summary on a show_card.

Hunt H-<n>: <title>   ATT&CK: <technique ID(s)>   Date: <date>   Analyst: <you>
Hypothesis: If <threat>, we would see <evidence> in <source>.
Data: <sources, time range UTC, gaps>
Method: <queries or code, saved at path>
Results: <counts, notable values>
Triage: <each hit, benign explanation checked, who confirmed>
Conclusion: found | not found | inconclusive (why)
Follow-ups: <new detection, logging fix, escalation>

Checks before you finish

• Every conclusion states the time range and sources it rests on.
• Data gaps are listed as findings with a recommended fix.
• Each hit has a recorded verdict and how it was confirmed.
• Anything suspicious was escalated to the owner; no containment action was taken by you.
• Queries are saved so the hunt can be repeated.

Pitfalls

• "Nothing found" without saying what was searched. Always give the sources and window; absence of evidence in seven days of logs is weak.
• Ignoring time zones. Mixed local and UTC timestamps create false sequences. Normalise first.
• Assuming a country means an attack. VPNs and travel are common. Check before raising the alarm, but check quickly.
• Hunting without a hypothesis or baseline. Scrolling raw logs finds nothing. Start with a question.
• Touching the evidence. Work on copies; never rotate, truncate or edit logs during a hunt.
• Sign-off. You report; the owner decides on escalation and response. For suspected serious compromise, recommend a qualified incident response provider.

Starter hunts: references/hunt-catalogue.md. Credits: references/CREDITS.md.