Logging & Monitoring

# ===== The model =====
# - Containers write to stdout / stderr
# - kubelet collects per-pod log files on the node
# - A log SHIPPER (DaemonSet) reads files + sends them off-node
# - An aggregation backend stores + indexes

# Never write log files inside containers (lost on restart, no rotation).

# ===== View logs =====
kubectl logs my-pod
kubectl logs my-pod -c api          # specific container in multi-container pod
kubectl logs -f my-pod              # follow
kubectl logs -l app=shop --tail=100 # by label
kubectl logs my-pod --previous       # previous container instance

# ===== Stern (multi-pod tail) =====
brew install stern
stern -n default '.*'                # tail all
stern -l app=shop                    # by label

# ===== Structured logs =====
# JSON is king: each log line is a JSON object with level, timestamp, message, fields.
# Shippers parse JSON; backends index fields for search.

# Example (Node + pino):
import pino from 'pino';
const logger = pino();
logger.info({ user_id: 'u-1', route: '/api/users' }, 'request handled');
# -> {"level":"info","ts":..., "user_id":"u-1", "route":"/api/users", "msg":"request handled"}

# ===== Fluent Bit (the standard shipper) =====
# Deploy as a DaemonSet; mounts /var/log/containers/.
# Helm:
helm install fluent-bit fluent/fluent-bit --set backend.loki.host=loki

# Reads container logs, enriches with Kubernetes metadata (pod, namespace, labels),
# ships to backend.

# ===== Loki (Grafana stack) =====
# Stores logs as a stream + label index (like Prometheus for logs).
# Cheap, fast, integrates with Grafana for dashboards.

helm install loki grafana/loki-stack

# Query:
# {app="shop"} |= "error"
# {app="shop"} | json | level="error"

# ===== Elastic / OpenSearch =====
# Full-text indexing; rich queries; heavier.
# Use when search + analytics on log content matter more than cost.

# ===== CloudWatch Logs =====
# AWS EKS native option. Fluent Bit -> CloudWatch.
# Insights query language for searching.

# ===== Datadog / New Relic / Splunk =====
# Commercial; usually best UX, highest cost.

# ===== Retention =====
# Live searchable: 7-30 days typical
# Archived (S3): 90 days - 7 years (per compliance)
# Tier down to cheaper storage automatically

# ===== Patterns =====
# - JSON logs from every service
# - Pod / namespace / app labels for filtering
# - request_id propagated through every log line
# - Error logs trigger alerts; info logs are searched ad-hoc

# ===== Pitfalls =====
# - Logging secrets / PII (review producers!)
# - Unstructured logs -> regex parsing fragility
# - Logging too much (cost) or too little (no debug ability)
# - No log retention policy -> infinite storage cost

# Fluent Bit -> Loki / Elasticsearch / CloudWatch.
# Prometheus + Grafana for metrics.