jenkins-migration: from webhook receiver to declarative pipeline

Where this starts

deploy-receiver turned a GitHub push into a production deploy with a webhook, a health check, and an automatic rollback. It worked. It also hit the ceiling every script hits: no build history, no parallelism, no PR awareness, and every new guarantee meant more hand-rolled code to maintain.

The next move was Jenkins. Same guarantees, plus the CI the script could not give.

What it is

One declarative Jenkinsfile, versioned next to the app it deploys. It runs on a self-hosted, dockerized Jenkins controller on the same host as the compose stacks.

GitHub (push)
    │
    │  signed webhook → Jenkins
    ▼
┌──────────────────────────────┐
│  Jenkins (dockerized)        │
│                              │
│  declarative · agent any     │
│  timeout 30m · failFast      │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Checks (parallel)           │
│                              │
│  lint ──── pnpm lint         │
│  unit ──── tests + migrate   │
│  e2e ───── playwright        │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Build                       │
│  tags: app:latest, app:<sha> │
└──────────────┬───────────────┘
               │
       ┌───────┴────────┐
       ▼                ▼
  non-prod branch   prod branch
       │                │
       ▼                ▼
┌──────────────┐  ┌─────────────────────┐
│  dev stack   │  │  prod stack         │
│  compose up  │  │  health loop 20×5s  │
│  --build     │  │  fail? → rollback   │
└──────────────┘  └─────────────────────┘
               │
               ▼
Outputs:
  GitHub commit statuses  — ci/lint · ci/test · ci/e2e · ci/deploy
  Discord embeds          — branch · commit · author · build # · duration

How it works

Parallel checks, throwaway databases

Lint, unit tests, and Playwright E2E run in parallel with failFast. Each test stage gets its own MariaDB 11 sidecar on a shared CI Docker network: wait for the container's healthcheck.sh to report ready, run migrations, run the suite, tear the container down in post { always }. The E2E branch seeds dedicated CI users with a CI-only password, so real credentials never enter the pipeline.

One red check kills the whole stage. The shared network outlives the parallel branches and is removed only after every branch has cleaned up its container.

Build once, tag by commit

The image builds with BuildKit and BUILDKIT_INLINE_CACHE, cached from app:latest, and gets two tags: app:latest and app:<short-sha>. The SHA tag is what makes rollback cheap later.

Branch-gated deploys

Any non-prod branch lands on the dev stack with docker compose up --build --force-recreate. The prod branch takes the guarded path:

capture PREV_IMAGE from the running container
recreate the app container, restart nginx
health loop: poll docker inspect, 20 × 5s
  ├── healthy → done
  └── failing → re-tag PREV_IMAGE as latest → redeploy → fail the build

disableConcurrentBuilds() replaces deploy-receiver's filesystem lock. The engine serializes builds, so two deploys cannot race.

Visibility where the work happens

Every stage reports a GitHub commit status (ci/lint, ci/test, ci/e2e, ci/deploy), so the commit page answers "which check failed" without opening Jenkins. Discord embeds carry branch, commit, author, build number, and duration, with distinct messages for pass, fail, and rollback.

Why these choices

Decision	Rationale
Declarative over scripted pipeline	Stages, `post`, and parallel blocks are enforced by the engine. The file reads top to bottom
Sidecar DB per stage	No shared test database, no cross-suite contamination. `post { always }` cleans up even on red
GitHub statuses, not just chat	The commit page shows which stage failed. Discord stays for the humans
`skipDefaultCheckout()` + `deleteDir()`	One explicit checkout, workspace wiped after every run. Nothing leaks between builds
SHA tags + captured `PREV_IMAGE`	Rollback is a re-tag and a recreate. No rebuild at the worst possible moment

What v2 adds

Designed, not shipped. Five upgrades, in order of leverage:

Multibranch Pipeline. Jenkins discovers every branch and PR that has a Jenkinsfile and builds each one. BRANCH_NAME replaces hand-rolled GIT_BRANCH parsing, and PR validation comes from the GitHub Branch Source plugin instead of a when clause.
Shared Library. About 110 lines of Discord and GitHub-status Groovy are duplicated across two repos. In vars/ they become discordNotify() and githubStatus(): versioned, reusable, imported with @Library.
Build once, promote the digest. Push app:<sha> to GHCR and deploy that exact digest to every environment. Rollback becomes re-pointing to the previous immutable tag.
Ephemeral Docker agents. Builds run in throwaway containers instead of on the controller, so a build cannot leave state behind on the box that serves prod.
A prod gate. Manual approval and a Trivy scan before anything ships.

Where it stands

v1 runs on the self-hosted controller and carries the same deploy path as the script it replaced. v2 is written up and waiting for the next window to implement.