Building & Deploying

How a merged commit becomes running containers. Before this: Developing & Committing. If a release goes wrong: Rollback.

	Status
Merge to master → images built → deployed to staging (deploy.yml)	CI/CD Phase 4
Production promoted by hand, same images (deploy-production.yml)	CI/CD Phase 4
Rollback and registry clean-up workflows	CI/CD Phase 4
Manual deploy from a laptop (infra/helm/deploy.sh): break-glass	Available
Migrations as a Helm hook on the api release	CI/CD Phase 2
Staging environment (--env staging)	CI/CD Phase 3
A database dump before any deploy with migrations	CI/CD Phase 5

Each service is a Helm release, playpalz-<service>, in the default namespace (see Kubernetes):

Service	Build script	Strategy
api	scripts/build-api.sh	Rolling, 3 replicas
web	scripts/build-web.sh	Rolling, 3 replicas
admin	scripts/build-admin.sh	Rolling, 2 replicas
docs	scripts/build-docs.sh	Rolling, 2 replicas
esu	scripts/build-esu.sh	Rolling, 2 replicas
ogun	scripts/build-ogun.sh	Rolling, HPA 2–8
anansi	scripts/build-anansi.sh	Recreate, singleton
igdb-heartbeat	scripts/build-igdb-heartbeat.sh	Recreate, singleton

Registry: registry.digitalocean.com/playpalzproduction/<service>.

The pipeline Live

PR ──► CI: lint, types, tests, migration safety, Helm check, Docker builds        (ci.yml)
merge to master
  └─► Deploy staging                                                              (deploy.yml)
        ├─ images: build what changed, re-tag the rest from the parent commit (same digest)
        ├─ deploy-all.sh staging <commit>: platform, api (migrate hook), then the rest
        └─ smoke test ──✗──► roll back every release this run changed
  └─► Deploy production — Actions → "Deploy production" → Run workflow           (deploy-production.yml)
        ├─ type "production" to confirm; optional sha (default: master)
        ├─ deploy-all.sh production <commit>: the same images staging ran, nothing rebuilt
        └─ smoke test ──✗──► roll back every release this run changed

How you ship:

1. Merge a PR to master.
2. Watch Deploy staging in the Actions tab, and try the change on *.staging.playpalz.gg.
3. Run Deploy production: Actions → Deploy production → Run workflow, type production, and leave sha blank for the current master.

What the pipeline guarantees:

• Every commit gets a full set of images, tagged with its 7-character hash. Services the commit didn't change are re-tagged from the parent (same digest, no rebuild), so Helm restarts only what actually changed.
• A deploy is a whole commit. Every release moves to the same commit, and each upgrade is --atomic.
• Failures roll back. If any release fails, or the smoke test does (infra/helm/smoke.sh: API and socket health, web, admin, docs), every release that run upgraded is rolled back to its previous revision.
• Production runs exactly what staging ran. It deploys the same digests staging did, never a rebuild.
• Migrations run once, in the api release's hook, before any pod changes. A database dump before production deploys with migrations arrives in Phase 5.
• Deploys queue, never cancel. One deploy per environment at a time; a newer push waits for the current one.
• An older commit can be deployed again. Run the deploy workflow with that commit's sha: the whole-system rollback.

What it needs (repository secrets; GitHub Free has no environment-scoped secrets):

Secret	What	Made with
DIGITALOCEAN_ACCESS_TOKEN	DigitalOcean token with registry read/write (push, retag, clean-up)	DO control panel → API → Tokens, custom scopes: registry
KUBECONFIG_STAGING	ci-deployer in namespace staging only	infra/k8s/ci/make-kubeconfig.sh staging <file>
KUBECONFIG_PRODUCTION	ci-deployer in namespace default only	infra/k8s/ci/make-kubeconfig.sh default <file>

The deployers are ServiceAccounts with the built-in admin role in their own namespace and nothing else (infra/k8s/ci/deployer.yaml). Anyone with write access to the repo can run the production workflow; GitHub Team would add a required approver.

Cost: a merge uses roughly 10–20 Actions minutes: about 1 per re-tagged image, 3–6 per rebuilt one, and a few for the deploy. The free plan includes 2,000 minutes a month.

Manual deploy (break-glass) Available

Use this when GitHub Actions is unavailable, or to deploy one service on its own. It uses the same releases as the pipeline, so either can deploy or roll back what the other did. Nothing but this checklist stands between your working tree and production, so follow it exactly.

Steps

# 1. Be on the commit you mean to ship
git switch master && git pull
git status                                  # must be clean
SVC=api
TAG=$(git rev-parse --short HEAD)

# 2. CI is green for this commit (Actions tab), or run the checks locally

# 3. Build and push the commit-tagged image
doctl registry login
./scripts/build-$SVC.sh "$TAG"

# 4. Deploy it: resolves the tag to its digest, asks for confirmation, then
#    helm upgrade --atomic (api: the migrate hook runs first)
infra/helm/deploy.sh "$SVC" "$TAG"

# 5. Verify
curl -sS https://api.playpalz.gg/health
kubectl logs -l app=playpalz-$SVC --tail=50

Try it on staging first. The same steps with --env staging deploy to *.staging.playpalz.gg. The images are shared, except web: build ./scripts/build-web.sh "$TAG" staging as well. See Environments → Staging.

What deploy.sh does:

• Pins the image by digest. It resolves the tag in the registry and deploys repository@sha256:…, so helm rollback returns to exactly the image that ran before, even if the tag moves.
• Undoes a failed rollout by itself. --atomic --wait: if the new pods don't become ready within 10 minutes, Helm rolls the release back.
• Records the deploy as a revision in helm history playpalz-<service>, with the tag in its description, and keeps the last 20 revisions.
• Only upgrades existing releases. It refuses to install one, because a failed first install under --atomic would delete the objects.

Don't use kubectl set image, kubectl apply or kubectl edit on these Deployments. The release's next deploy reverts them, and they don't appear in helm history. Change the values file instead.

Every build script builds with --platform linux/amd64 from the repo root. Never run docker build by hand without it: an arm64 image from an Apple Silicon Mac fails on the cluster with exec format error.

Migrations

Migrations run in a Helm hook on the api release. On every deploy.sh api …:

1. Before any pod changes, a Job (playpalz-api-migrate) runs prisma migrate deploy with the new image.
2. If it fails, the upgrade stops and the running API is untouched.
3. With nothing new, it applies nothing.

kubectl logs job/playpalz-api-migrate      # the last run's output (kept until the next deploy)

• If the hook fails, fix the migration in a new commit. A half-applied migration is marked failed in _prisma_migrations; see Migrations.
• Take a backup first if the migration does anything beyond adding (it shouldn't — see the migration rules):
  • doctl databases backups <cluster-id> lists the daily backups.
  • Point-in-time recovery covers the last 7 days.
  • For a contract migration, take a dump.
• helm rollback doesn't undo a migration. That's why migrations must stay compatible with the previous release.

Deploy order

Migration involved: deploy api first (its hook migrates), then the other services that use the new shape. A contract migration ships only after every service has stopped using the old shape.

Shared package change (@playpals/db, @playpals/types, @playpals/socket-client): rebuild and deploy every service that imports it, from the same commit. The package is compiled into each image, so a partial deploy leaves services disagreeing about a contract.

Realtime contract change: deploy esu and api together, and roll them back together.

anansi: don't deploy while a payout run is in progress. Check Jobs in the admin panel (the payout-calculate and payout-transfer rows). Recreate stops the old pod before the new one starts.

Secrets and config

Changing a secret doesn't restart anything:

kubectl apply -f infra/k8s/sealed-secrets/<name>.yaml
kubectl rollout restart deployment/<every-consumer>

Add new secret keys before the code that reads them, and remove old keys only after the code that read them is gone. A secret isn't part of a deploy's revision, so a rollback doesn't bring back a key you removed. See Secrets Management.

Infrastructure changes

cd infra/terraform
make plan            # read it — watch for "destroy and then create replacement"
make apply

See Terraform.

Checklist

• [ ] On master, tree clean, on the commit you mean to ship; CI green
• [ ] Migrations are additive; a backup exists if not
• [ ] New env vars and secret keys are in place before the deploy (env vars go in infra/helm/values/production/<service>.yaml)
• [ ] No payout run in progress (if deploying anansi)
• [ ] Commit-tagged image built and pushed
• [ ] infra/helm/deploy.sh <service> <tag> succeeded (for api, including the migrate hook)
• [ ] Health endpoint OK; logs show no new errors for 5 minutes
• [ ] You know the previous revision (helm history playpalz-<service> -n default) in case you need to roll back

Mobile releases

A separate pipeline: EAS Build, EAS Submit and EAS Update. See Builds & Releases.

The coupling: installed apps run old code for weeks, so backend changes must stay compatible with the app versions in the stores. Never ship a breaking API change together with the app that needs it.