Logging & Alerting

# ===== What to log (security-relevant events) =====
# Authentication
#   - Login success / failure (with user_id, ip_hash, ua, mfa_used)
#   - MFA challenge issued / failed / rate-limited
#   - Password change, reset, email change
#   - Session created / destroyed
# Authorisation
#   - Access denied (resource, action, user, decision reason)
#   - Privilege escalation (admin role granted)
# Inputs + boundaries
#   - Validation rejections (schema mismatch, suspicious shapes)
#   - WAF blocks, rate-limit hits
# Data lifecycle
#   - Sensitive data viewed / exported
#   - DDL changes; bulk deletes
# Infrastructure
#   - Deploys (who, what, when, version)
#   - Config changes; cron triggers; webhook receipts

# ===== What NEVER to log =====
# - Passwords or password hashes
# - Full session tokens / API keys / JWTs
# - Credit card numbers / PAN / CVV
# - PII unless explicitly approved + retained per policy
# - Personal health data
# - Full request bodies containing the above

# Mask: keep the last 4 chars for support; mask the rest.

# ===== A good log line shape =====
{
  "ts": "2024-04-10T03:14:00Z",
  "level": "info",
  "event": "auth.login",
  "outcome": "success",
  "user_id": "u-42",
  "ip_hash": "sha256(salt + ip)",
  "ua": "Mozilla/...",
  "mfa_used": true,
  "route": "/login",
  "deploy_id": "v1.42",
  "request_id": "req-..."
}

# ===== Retention =====
# - Auth events: 12 months (often required by compliance)
# - Access denied: 6 months
# - Application errors: 30-90 days
# - Sensitive data access: 12 months
# - Debug logs: 7-14 days
# - PII in logs: should not exist; if any, shortest legal retention

# ===== Alerts (what turns logs into detection) =====
# - Burst of failed logins for one user
# - Distributed failed logins (credential stuffing)
# - MFA fatigue patterns (many MFA challenges, few approvals)
# - Impossible travel (login from AU then US within 10 minutes)
# - Privilege escalation followed by data export
# - 5xx errors clustered by route + payload shape (probes)
# - WAF block + 500 within seconds = bypass attempt
# - Suspicious user-agent ratios on auth endpoints

# ===== Storage + transport =====
# - Append-only; HMAC chains or signed batches for integrity
# - Encrypted at rest, in transit
# - Separate environment per tenant
# - Time-synced clocks (NTP) on every host

# ===== Common patterns =====
# Structured logs (JSON) -> shipper (Fluent Bit, Vector) -> central store (Loki, Elastic, Splunk)
# -> SIEM rules / dashboards
# -> Pager + ticket integrations

# ===== Practical first steps =====
# 1. Tag every log with user_id, request_id, deploy_id
# 2. Standardise event names (auth.login, auth.password_change, ...)
# 3. Move PII out of logs immediately; refactor producers
# 4. Add 3-5 alerts on the highest-leverage events (failed login, denied admin, MFA fatigue)
# 5. Tabletop a breach: 'we got paged at 02:00; what do we have to investigate?'

# ===== When good logging wins =====
# - Compliance (SOC 2, ISO 27001, PCI DSS, HIPAA)
# - Incident response speed
# - Customer support (what happened on the account)
# - Engineering debug

# ===== Pitfalls =====
# - Logging the entire request body
# - PII drifting in via 'helpful' debug logs
# - Retention policies that store logs forever
# - Alert fatigue: too many noisy rules; tune or delete
# - No log review until something happens

// Log auth failures, role changes, data exports.
// Forward to a SIEM (Datadog SIEM, Elastic, Splunk).
// PII-scrub logs before sending; rotate creds on alert.