Kubernetes

Workloads are plain manifests in infra/k8s/, applied with kubectl apply. No Helm, no Kustomize, no GitOps — everything lives in the default namespace.

Manifests

File	Deploys
api-deployment.yaml	api Deployment + Service
esu-deployment.yaml	esu Deployment + Service
ogun-deployment.yaml	ogun Deployment + HPA
anansi-deployment.yaml	anansi Deployment + Service
igdb-heartbeat-deployment.yaml	igdb-heartbeat Deployment + Service
web-deployment.yaml	web Deployment + Service
redis-deployment.yaml	Redis StatefulSet + PVC + Service
ingress.yaml	Main, socket, and Grafana Ingress objects
sealed-secrets/	Encrypted secrets

Deployment shape

Service	Replicas	Strategy	Probes	Ingress
web	3	RollingUpdate, maxUnavailable: 0	GET /	✔
api	3	RollingUpdate, maxUnavailable: 0	GET /health	✔
esu	2	RollingUpdate	GET /health	✔
ogun	2 (HPA 2–8)	RollingUpdate	none	—
anansi	1	Recreate	GET /health	—
igdb-heartbeat	1	Recreate	GET /health	—
redis	1 (StatefulSet)	—	✔	—

maxUnavailable: 0 with maxSurge: 1 means a new pod becomes ready before an old one is removed — zero-downtime rollouts at the cost of briefly running one extra pod.

The two singletons use Recreate on purpose. Both are cron-driven, and overlapping pods during a rollout could mean a duplicate payout run or a duplicate catalog sync.

Autoscaling

Only ogun has an HPA:

minReplicas: 2
maxReplicas: 8
metrics:
  - type: Resource
    resource: { name: cpu,    target: { type: Utilization, averageUtilization: 70 } }
  - type: Resource
    resource: { name: memory, target: { … } }

That is the right service to autoscale — image processing is bursty and CPU-bound. The node pool itself autoscales 2–10 nodes, so a burst can add capacity underneath.

api is fixed at 3 replicas. An HPA on it would be a sensible addition, bounded by the Postgres connection limit.

Redis in-cluster

A StatefulSet with a 5 Gi PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: playpalz-redis-pvc }
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 5Gi } }

A StatefulSet gives stable network identity, and the PVC persists AOF/RDB snapshots across restarts.

Single replica, no failover

Redis backs the media queue, the Socket.IO adapter, the feed and trending caches, and anansi's payout lock. If the pod is rescheduled, realtime fan-out stops and queue operations fail until it comes back. The PVC means data survives, but availability does not.

For a queue and a cache this is a defensible cost trade. Revisit before Redis holds anything that cannot be rebuilt.

The manifest's header comment mentions apps/hermes — a service that no longer exists in this repository. Harmless, but stale.

Security context

Every service container:

securityContext:
  allowPrivilegeEscalation: false
  capabilities: { drop: [ALL] }

# pod level
securityContext:
  runAsNonRoot: true
  runAsUser: 1001
  fsGroup: 1001

This is consistently applied and worth preserving in anything new.

Placement

api and web use soft pod anti-affinity:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector: { matchExpressions: [{ key: app, operator: In, values: [playpalz-api] }] }
          topologyKey: kubernetes.io/hostname

preferred rather than required, so replicas spread across nodes when possible but a busy cluster never leaves pods unschedulable.

Images

image: registry.digitalocean.com/playpalzproduction/api:latest
imagePullPolicy: Always
imagePullSecrets:
  - name: registry-playpalzproduction

:latest with Always is why deploys need a restart

Because the tag never changes, kubectl apply sees no diff and does nothing. The deploy step is kubectl rollout restart, which creates new pods that re-pull latest.

It works, but it means you cannot tell which image a running pod has, and rolling back requires rebuilding. Tagging images with a version or git SHA and updating the manifest would make deploys declarative and rollbacks trivial. See Deploying.

Ingress

Three objects in ingress.yaml:

Main — playpalz.gg, www.playpalz.gg, api.playpalz.gg:

cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-protocols: "TLSv1.2 TLSv1.3"
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

Socket — socket.playpalz.gg, deliberately separate:

nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/affinity-mode: "persistent"
nginx.ingress.kubernetes.io/session-cookie-name: "ESUSESSION"

The file's own comments explain both choices: HTTP/1.1 and hour-long timeouts must not apply to REST routes, and Engine.IO long-polling issues several short-lived requests per connection that must land on the same pod, since session state is in that pod's memory.

Grafana — grafana.playpalz.gg.

A staging Ingress for staging.playpalz.gg / api-staging.playpalz.gg is present but commented out.

Operating

kubectl get pods
kubectl get deployments
kubectl get ingress
kubectl get hpa

kubectl logs -l app=playpalz-api --tail=100 -f
kubectl describe pod <pod>
kubectl exec -it <pod> -- sh

kubectl rollout restart deployment/playpalz-api
kubectl rollout status  deployment/playpalz-api
kubectl rollout undo    deployment/playpalz-api

kubectl scale deployment/playpalz-api --replicas=5

kubectl top nodes
kubectl top pods

Applying changes

kubectl apply -f infra/k8s/api-deployment.yaml
kubectl apply -f infra/k8s/ingress.yaml

Manifests are the source of truth. If you change something with kubectl edit or kubectl scale, put it in the file too — otherwise the next apply silently reverts it.

The default namespace

Everything is in default, which is workable at this size but has costs: no namespace-level resource quotas, no network policies between tiers, and no clean separation when staging arrives. The Helm plan addresses this by putting staging in playpalz-staging. See Environments.

Stale documentation nearby

infra/k8s/README.md predates the current setup: it references media-service-deployment.yaml (now ogun-deployment.yaml), .com domains rather than .gg, and kubectl create secret for credentials that are now sealed secrets. Treat this page and the manifests as authoritative.