iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Observability Hooks

By · Reviewed & last updated · Part of CI/CD Tutorial

CI/CD observability turns deploys from "we shipped, lets hope" into "we shipped, the metrics tell us its working". Three signals to wire up: build success rate + duration in CI, deployment frequency + MTTR in the pipeline, and post-deploy SLO health in production. Combined, they make every release boring.

Build telemetry, deploy markers, SLO gates

EXAMPLE
# ===== 1) Build telemetry =====
# Per-job: status, duration, queue time, runner type
# Aggregated: success rate, p50/p95 duration, weekly trend
# Where to send it: Datadog, Honeycomb, Grafana Cloud, BuildBuddy, or your own
# Most CI providers emit OpenTelemetry — point it at your collector.

# GitHub Actions example: send job status to Datadog
# - name: Send build status to DD
#   if: always()
#   run: |
#     curl -X POST 'https://api.datadoghq.com/api/v1/events' \
#       -H "DD-API-KEY: $DD_API_KEY" -H 'Content-Type: application/json' \
#       -d "{ \"title\": \"CI ${{ github.workflow }}\", \"text\": \"${{ job.status }}\", \"tags\": [\"repo:${{ github.repository }}\",\"branch:${{ github.ref_name }}\"] }"

# Auto-instrument via BuildBuddy / DataDog CI Visibility for build + test timing
# at no code cost.

# ===== 2) Deployment markers =====
# Send a custom event to your APM every time you deploy, tagged with:
# - commit sha
# - version
# - environment
# - changed services
# Use it to overlay 'deploy' lines on every chart, so the cause of a regression
# is one glance away.

# Datadog example
# datadog-ci deploy --version 1.4.0 --env production --service api
# Honeycomb
# honeycomb-ci-markers add --type=deploy --message="v1.4.0" --dataset=api
# Sentry
# sentry-cli releases new $VERSION
# sentry-cli releases set-commits $VERSION --auto
# sentry-cli releases finalize $VERSION
# sentry-cli releases deploys $VERSION new --env production

# ===== 3) Post-deploy SLO gates =====
# A 'soak' job inspects production telemetry for N minutes after deploy and
# fails the pipeline if SLOs breach.

# soak.yml
# - name: SLO soak
#   run: |
#     for i in 1 2 3 4 5; do
#       sleep 60
#       err=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_requests_total{status=~\"5..\"}[1m])" | jq -r '.data.result[0].value[1]')
#       if [ "$(echo "$err > 0.01" | bc)" -eq 1 ]; then
#         echo 'error rate exceeded' >&2; exit 1
#       fi
#     done
# - name: Auto rollback
#   if: failure()
#   run: ./scripts/rollback.sh

# ===== 4) The four DORA metrics =====
# - Deployment Frequency: how often you ship to production
# - Lead Time for Changes: PR opened -> shipped
# - Change Failure Rate: % of deploys that cause incidents / rollbacks
# - Time to Restore: from incident start to resolution
# Track them in a dashboard; do not obsess over absolute values — track the trend.

# Tools that compute DORA for you: LinearB, Sleuth, Athenian, Faros.

# ===== 5) Trace IDs across the boundary =====
# Have CI inject a trace id (commit sha + run id) and propagate it to your
# observability stack so a failed build can be linked to the test that broke,
# the PR that introduced it, and the spans of the test run.

# ===== 6) Synthetic checks =====
# Lambda / k6 / Datadog Synthetics / Checkly hit critical endpoints from
# multiple regions every minute. Page on-call when they fail. These catch
# the 'works in the lab, broken in the wild' regressions a green CI cannot.

# ===== 7) Feature-flag observability =====
# Every flag rollout is a deploy. Treat flag toggles as deploy markers in
# observability. LaunchDarkly + Datadog integrate this out of the box.

# ===== 8) Build performance =====
# Slow CI hurts shipping cadence more than most things. Watch:
# - p95 pipeline time over a week
# - Cache hit rates (dependency, build, test caches)
# - Test pass rate per file (top-N flaky tests)
# Invest a half-day every quarter in shrinking the slowest path.

# ===== 9) Sample dashboards =====
# Build status grid (red / green / amber per workflow)
# DORA quarterly trend
# Deploy frequency by service
# Post-deploy 5xx rate, p95 latency, queue depth — overlaid with deploy marker
# Top-N flaky tests (sorted by failure-rate * frequency)

# ===== 10) Pitfalls =====
# - Treating green CI as 'deploy is safe' without a soak gate
# - Reporting absolute DORA numbers without context (compare to your own past)
# - Slack-flooding the team with every CI run; alert only on regressions
# - No deploy markers; production graphs are a mystery during incidents
# - Ignoring p95 build time; the long-tail kills focus more than the average

Why it matters

Wire deploy markers to your observability stack on day one. Every production graph becomes "what changed and when?" instead of "what is going on?" — and the time from incident-start to "found the cause" drops from hours to minutes because the offending deploy is a vertical line on the same chart as the error rate.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
# Tell the world about the deploy.
- run: curl -X POST $SENTRY_HOOK -d 'version=${{ github.sha }}'
- run: dd-agent event 'deploy ${{ github.sha }}'
Try it Yourself »

Discussion

Loading…