Rollback

# A rollback playbook (paste into runbooks/)

## Decision: do we roll back or roll forward?
- Roll BACK when the last deploy is the obvious cause AND data shape is
  compatible with the prior version (additive migrations only).
- Roll FORWARD when the prior version cannot read the current data shape,
  OR the issue is data corruption that a redeploy will not fix.
- The first 60 seconds are about deciding which path; the next 5 are about executing.

## Layers to roll back, in order
1. **Code**: redeploy the prior immutable tag/image.
2. **Routing / feature flags**: flip off the offending flag if applicable.
3. **Config**: revert config maps / Parameter Store / env file change.
4. **Data**: STOP if migrations are not additive. Engage IC.
5. **Background work**: pause queue consumers running the new code; drain.

## What 'additive migration' means
Old code MUST still work with the new schema:
- Adding a column with a default: ok
- Adding an index: ok
- Adding a table: ok
- DROP COLUMN / RENAME COLUMN / type changes: NOT additive — design as two-step.

## Two-phase migration recipe (so rollback is always possible)
- Release N:    add new column, dual-write, prefer old on read
- Release N+1:  switch reads to new column, keep dual-write
- Release N+2:  stop writing the old column
- Release N+3:  drop the old column
- At every step, the prior release can run unchanged. Rollback never breaks reads.

## Communication
- One incident channel, one IC, one timekeeper.
- Customer-facing status page updated within 5 minutes for >1% impact.
- Internal Slack post every 15 minutes until resolved, even with 'no change'.

---

# GitHub Actions one-command revert workflow
# .github/workflows/rollback.yml
name: rollback
on:
  workflow_dispatch:
    inputs:
      tag:
        description: 'Previous good tag (e.g. v1.3.9)'
        required: true

permissions:
  id-token: write
  contents: read

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.DEPLOY_ROLE_ARN }}
          aws-region: ap-southeast-2

      # 1) Re-deploy the prior image. ECR images are immutable -> safe.
      - name: Deploy previous image
        run: |
          aws ecs update-service --cluster shop-prod --service api \
            --task-definition shop-app:${{ github.event.inputs.tag }} --force-new-deployment
          aws ecs wait services-stable --cluster shop-prod --services api

      # 2) Pause new consumers if needed (the new code MAY have started workers)
      - name: Drain risky queues
        run: scripts/pause-queue.sh checkout-events

      # 3) Smoke + health
      - name: Verify rollback
        run: scripts/check-health.sh --window 3m --error-rate-max 1 --p95-ms-max 800

      # 4) Notify
      - name: Notify
        if: always()
        run: scripts/notify.sh slack --channel deploys \
                                     --status ${{ job.status }} \
                                     --message 'Rolled back to ${{ github.event.inputs.tag }}'

# Companion scripts
# scripts/pause-queue.sh   <name>     # set max-in-flight to 0
# scripts/check-health.sh  ...        # query SLO dashboards
# scripts/notify.sh        ...        # post to Slack/PagerDuty

# Tag every artifact (commit SHA / semver).
# Re-deploy the previous tag, or use:
kubectl rollout undo deployment/api