Runbooks

Production incident procedures. For local problems see Troubleshooting.

First response

kubectl get pods                                     # anything not Running?
kubectl get events --sort-by=.lastTimestamp | tail -30
kubectl top nodes && kubectl top pods

curl -sS -o /dev/null -w "%{http_code}\n" https://api.playpalz.gg/health
curl -sS -o /dev/null -w "%{http_code}\n" https://socket.playpalz.gg/health
curl -sS -o /dev/null -w "%{http_code}\n" https://www.playpalz.gg

Then Grafana at grafana.playpalz.gg:

{app=~"playpalz-.*"} | json | level >= 50

The API is down

Symptom: api.playpalz.gg returns 5xx or does not respond. Everything in the app breaks.

kubectl get pods -l app=playpalz-api
kubectl describe pod <pod>
kubectl logs -l app=playpalz-api --tail=200

What you see	Cause	Action
CrashLoopBackOff	Bad config or a startup exception	Read the logs; check for a missing env var after a recent deploy
ImagePullBackOff	Registry auth or a bad tag	Check the registry-playpalzproduction pull secret
OOMKilled in describe	Memory limit hit	Raise limits, or find the leak
Pending	No schedulable node	kubectl top nodes; the pool autoscales 2–10
Pods Running, requests failing	Database or Redis unreachable	See below

Immediate mitigation: if a recent deploy caused it, roll back — check out the last good commit, rebuild, push, restart. See Deploying.

The database is unreachable

Symptom: every service logs Prisma connection errors.

doctl databases list
doctl databases get <database-id>
kubectl exec -it <api-pod> -- node -e "require('net').connect(25060,'<db-host>').on('connect',()=>{console.log('ok');process.exit(0)}).on('error',e=>{console.log(e.message);process.exit(1)})"

Check in order:

1. Database status in the DigitalOcean console — maintenance, failover, or an incident.
2. Firewall — digitalocean_database_firewall restricts access to the cluster. A new node pool or a recreated cluster can fall outside it.
3. Connection exhaustion — db-s-2vcpu-4gb has a bounded connection limit, and three API replicas plus four other services each hold a pool. This is the most likely cause under load.

SELECT count(*), state FROM pg_stat_activity GROUP BY state;
SELECT pid, state, query_start, left(query, 80) FROM pg_stat_activity
  WHERE state = 'active' ORDER BY query_start;

Scaling replicas down temporarily relieves connection pressure. The durable fix is a connection pooler (PgBouncer, or DigitalOcean's managed pooler).

Redis is down

Symptom: realtime stops delivering, media stops processing, feed latency rises.

kubectl get pods -l app=playpalz-redis
kubectl logs -l app=playpalz-redis --tail=100
kubectl exec -it <redis-pod> -- redis-cli -a <password> PING

Single replica with a 5 Gi PVC, so a restart loses in-flight state but not persisted data.

Impact	Recovery
BullMQ queue	Jobs persisted in Redis survive; in-flight jobs retry
Socket.IO adapter	Cross-pod fan-out stops entirely until it returns
Feed / trending cache	Cache misses fall through to Postgres — slower, still correct
anansi payout lock	A run cannot acquire the lock; it fails safe

After recovery, restart esu so its adapter reconnects cleanly:

kubectl rollout restart deployment/playpalz-esu

Realtime is broken

Symptom: messages arrive only on refresh; presence and typing do nothing.

kubectl get pods -l app=playpalz-esu
kubectl logs -l app=playpalz-esu --tail=200
curl https://socket.playpalz.gg/health

In Grafana:

{app="playpalz-esu"} | json | event = "socket:auth_failed"
{app="playpalz-esu"} |= "redis"

Symptom	Cause
Connect then immediate disconnect	JWT_SECRET mismatch between api-secrets and what esu reads
Every join AUTHZ_DENIED	API_BASE_URL wrong, or the API is failing the authz endpoints
Connects fine, no events	Redis adapter down, or the API cannot reach /admin/emit/*
Sporadic drops	Ingress timeout, or the sticky-session cookie not being honoured

Check the API side too — it is the producer:

{app="playpalz-api"} |= "REALTIME_SERVICE_URL"

Media is not processing

Symptom: uploads succeed but never get thumbnails; posts stay pending.

kubectl get pods -l app=playpalz-ogun
kubectl logs -l app=playpalz-ogun --tail=200

kubectl exec -it <redis-pod> -- redis-cli -a <pw> LLEN 'bull:media-processing:wait'
kubectl exec -it <redis-pod> -- redis-cli -a <pw> LRANGE 'bull:media-processing:failed' 0 -1

Queue state	Meaning
wait growing, worker logs silent	Worker is not consuming — restart ogun
failed growing	Processing is erroring — read the error and Media.errorMessage
Queue empty, media still pending	The API is not enqueuing — check API logs

Video specifically: Media.status only becomes ready when Mux calls the webhook. Confirm the webhook is registered and reaching api.playpalz.gg/api/v1/webhooks/mux.

Remember ogun has no health probe, so Kubernetes will not restart a wedged worker. Restart it yourself:

kubectl rollout restart deployment/playpalz-ogun

A payout run failed

Highest-severity class of incident — real money.

kubectl logs -l app=playpalz-anansi --since=48h

curl -H "Authorization: Bearer $ADMIN_SECRET" \
  http://localhost:4005/admin/payout/status?period=2026-01

The 25th run failed

1. Check the logs for the cause — a Stripe error, a database error, or a lock conflict.
2. GET /admin/payout/preview to see what would be calculated. It writes nothing.
3. If no Payout rows were created, trigger POST /admin/payout/calculate.
4. If rows were partially created, the idempotency check will skip a re-run. Reconcile manually before re-triggering.

The 1st run failed

1. GET /admin/payout/status for the period.
2. POST /admin/payout/retry re-attempts only status=failed transfers — safe to run.
3. Persistent failures are usually creator-side: incomplete Stripe Connect onboarding, or a restricted account. Check the Stripe dashboard for that connected account.

A creator was paid twice

Stop and investigate before doing anything else.

1. Find the duplicate Ledger entries and Payout rows.
2. Check the idempotencyKey values — if they differ, the guard was bypassed and that is the bug.
3. Reverse in Stripe, and record an ADJUSTMENT ledger entry. Never edit or delete an existing ledger row.
4. Verify anansi is still at replicas: 1.

TLS certificate expired or failing

kubectl get certificate
kubectl describe certificate playpalz-tls
kubectl logs -n cert-manager -l app=cert-manager --tail=100

Usual causes: DNS changed and the HTTP-01 challenge can no longer reach the ingress; port 80 blocked; or Let's Encrypt rate limiting after repeated failures. See DNS & TLS.

Everything is unreachable

kubectl cluster-info
kubectl get nodes
doctl kubernetes cluster list
kubectl get service -n ingress-nginx
dig api.playpalz.gg +short

Check, in order: the DigitalOcean status page, node health, the LoadBalancer's external IP, and whether DNS still resolves to it.

Restoring the database

doctl databases backups list <database-id>
doctl databases restore <database-id> --backup-created-at <timestamp>

Managed Postgres takes daily backups with point-in-time recovery. A restore creates a new cluster — you then have to update database-credentials and restart every service that uses it.

Before restoring, be certain: everything written since the restore point is lost.

Escalation

Situation	Do
Money moved incorrectly	Stop anansi, preserve the ledger, involve whoever owns finance
Suspected data breach	Rotate JWT_SECRET (logs everyone out), rotate all sealed secrets, preserve logs
Total outage > 15 min	Check the DigitalOcean status page; open a support ticket
Data loss	Stop writes, assess the blast radius, restore from a snapshot

After an incident

1. Write down the timeline and the root cause.
2. Add the failure mode to this page if it is not here.
3. Add the monitoring that would have caught it earlier — see Observability.
4. Fix the underlying cause, not just the symptom.