Observability

Centralised logging with Grafana Loki. There is no metrics stack and no alerting.

Capability	Status
Structured logs	Loki + Promtail + Grafana
Mobile crash reporting	Bugsnag
Socket event tracing	esu, opt-out
Socket.IO live dashboard	Available, disabled by default
Metrics (Prometheus)	None
Alerting	None
Backend error tracking	None
Distributed tracing	None

Logging stack

Deployed via the grafana/loki-stack Helm chart into the monitoring namespace:

Rendering diagram…

Configuration in infra/helm/loki-stack-values.yaml:

loki:
  persistence: { enabled: true, storageClassName: do-block-storage, size: 20Gi }
  config: { auth_enabled: false }

promtail:
  config: { snippets: { pipelineStages: [{ cri: {} }] } }

grafana:
  admin: { existingSecret: grafana-secrets, userKey: admin-user, passwordKey: admin-password }
  ingress: { enabled: false }          # managed in infra/k8s/ingress.yaml instead
  datasources: [{ name: Loki, type: loki, url: http://loki-stack:3100, isDefault: true }]

auth_enabled: false is acceptable only because Loki is not exposed outside the cluster — access is through Grafana, which is behind its own credentials.

Full install steps: dev-notes/loki-setup.md.

Structured logging with pino

Every backend service uses pino. The object comes first, the message second:

logger.info({ port: config.port }, "Server started");
logger.error({ err: error, mediaId }, "image processing failed");

That ordering is what makes the fields queryable in Loki. String interpolation (logger.info(`port ${port}`)) produces a flat message you cannot filter on.

ogun goes further and namespaces by module:

const log = logger.child({ module: "processImage" });

so {app="playpalz-ogun"} | json | module = "processImage" isolates one stage of the pipeline, and mediaId traces one item end to end.

In development, pino-pretty renders logs readably. In production they stay JSON for Loki.

Useful queries

{app="playpalz-api"}                                    # everything from the API
{app="playpalz-api"} | json | level >= 50               # errors and above
{app="playpalz-ogun"} | json | module = "processImage"
{app="playpalz-ogun"} | json | mediaId = "clx…"         # one media item's history
{app="playpalz-esu"} | json | event = "socket:in"       # inbound socket events
{app="playpalz-esu"} | json | userId = "clx…"           # one user's realtime session
{app="playpalz-anansi"} |= "cron"                       # payout runs

pino levels: 10 trace, 20 debug, 30 info, 40 warn, 50 error, 60 fatal.

Socket event logging

esu logs every inbound and outbound socket event when SOCKET_EVENT_LOGGING=true (the default):

socket.onAny((event, ...args) => {
  logger.info({ event: "socket:in", name: event, userId, socketId, args: summarizeSocketArgs(args) },
    `[esu-realtime] <- ${event}`);
});

summarizeSocketArgs truncates payloads so message bodies do not end up in the log store. Set the variable to false if volume becomes a problem.

Socket.IO Admin UI

A live view of connections, rooms, and events. Disabled by default:

pnpm --filter @playpals/esu admin-ui:creds     # generates a bcrypt hash

ADMIN_UI_ENABLED=true
ADMIN_UI_USERNAME=admin
ADMIN_UI_PASSWORD_HASH=<hash>

Then open admin.socket.io and connect to socket.playpalz.gg. It is instrumented in production mode, so payloads are not exposed, and gated behind basic auth.

Enabling it adds https://admin.socket.io to the CORS allowlist and requires the polling transport — which is why the socket ingress pins clients with the ESUSESSION cookie.

Health endpoints

Service	Endpoint
api	GET /health → {"status":"ok"}
esu	GET /health → {"ok":true}
anansi	GET /health
igdb-heartbeat	GET /health → {"status":"ok","service":"igdb-heartbeat"}
web	GET /
ogun	none

Kubernetes liveness and readiness probes use these. ogun has neither, so a wedged worker looks healthy to the cluster.

Mobile crash reporting

Bugsnag, initialised in app/_layout.tsx outside development only. Source maps are uploaded by the eas-build-on-success hook, so production stack traces resolve to real file and line numbers.

Quick triage

kubectl get pods                                  # anything not Running?
kubectl top nodes && kubectl top pods             # resource pressure
kubectl logs -l app=playpalz-api --tail=100 -f
kubectl describe pod <pod>                        # events — OOMKilled, ImagePullBackOff
kubectl get events --sort-by=.lastTimestamp | tail -30

The gaps that matter

No metrics and no alerting

Nothing measures request latency, error rate, queue depth, or database connection usage, and nothing pages anyone when something breaks. Every incident is discovered by a user reporting it or by someone happening to look at Grafana.

The highest-value additions, roughly in order:

1. Alerting on what already exists. Grafana can alert on Loki queries — error rate above a threshold, anansi cron failing to log a run, a spike in socket auth failures. This needs no new infrastructure.
2. Backend error tracking. Bugsnag already covers the mobile app; adding the Node SDK to the API would give grouped, deduplicated backend exceptions with stack traces.
3. Prometheus + metrics. prom-client in the services, kube-state-metrics for the cluster, dashboards for latency and queue depth.
4. Uptime monitoring. An external check against api.playpalz.gg/health catches the failure mode where the whole cluster is unreachable and therefore not logging anything.

Retention

Loki has 20 Gi of block storage and no explicit retention policy in the values file, so it will fill and start dropping the oldest data. Set limits_config.retention_period (30 days is a reasonable default) rather than discovering the limit during an incident.