Media Pipeline

Uploads never block a request. The API takes the original bytes and returns immediately; the ogun worker does the expensive work off the request path.

Two upload paths

There are two ways media reaches the platform, and both converge on the same queue.

Direct-to-storage (preferred)

The client uploads straight to DigitalOcean Spaces with a presigned URL. The API never touches the bytes.

Rendering diagram…

The storage key is namespaced per user and year:

uploads/<userId>/<images|videos>/<year>/<uuid>.<ext>

Presigned URLs expire after 5 minutes.

Multipart through the API

POST /api/v1/posts accepts multipart/form-data and streams files to Spaces with multer-s3. It is simpler for the client, but the bytes pass through the API pod. Use the presigned path for anything large.

Both paths end the same way: a Media row with status: "pending" and a job on the media-processing queue.

The queue

Defined once in packages/queue/src/index.ts so producer and consumer cannot disagree:

export const PROCESS_MEDIA_QUEUE = "media-processing";

export const ProcessMediaQueue = new Queue(PROCESS_MEDIA_QUEUE, {
  connection,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: "exponential", delay: 2000 },
  },
});

Three attempts with exponential backoff starting at 2 s. The Redis client sets maxRetriesPerRequest: null, which BullMQ requires.

ogun consumes with concurrency: 5 per worker process.

Image processing

apps/ogun/src/jobs/processImage.ts:

1. GetObject the original from Spaces.
2. Read metadata with Sharp.
3. Generate one JPEG per variant for the parent post's type.
4. PutObject each variant to variants/<mediaId>/<type>.jpg.
5. Upsert a Variant row per output.
6. Set Media.status = "ready" with the source dimensions.
7. Emit media:processing:update to the author.

Variant matrix

Post type	Variant	Width	Height	Quality
story_image	story	1080	1920	85
story_image	thumb	320	568	78
photo, carousel, photos, default	thumb	320	400	78
photo, carousel, photos, default	medium	720	900	82
photo, carousel, photos, default	full	1080	1350	85

Every variant is:

• Resized fit: cover with sharp.strategy.attention cropping, so faces survive the crop
• withoutEnlargement: true — a small original is never upscaled
• Auto-rotated from EXIF (.rotate())
• Encoded as JPEG with mozjpeg
• Uploaded public-read with Cache-Control: public, max-age=31536000, immutable

The immutable cache header is safe because the key contains the media id, so a new upload is a new key.

Video processing

apps/ogun/src/jobs/processVideo.ts hands off to Mux and returns:

const asset = await mux.video.assets.create({
  inputs: [{ url: media.originalUrl }],
  playback_policies: ["public"],
  video_quality: "basic",
});

await prisma.media.update({
  where: { id: media.id },
  data: { muxAssetId: asset.id, muxPlaybackId },
});

The job completes immediately — Mux transcodes asynchronously. Completion arrives later as a webhook:

Rendering diagram…

Video status depends on a reachable webhook

If Mux cannot call POST /api/v1/webhooks/mux, video media stays pending forever even though transcoding succeeded. Locally, tunnel it (ngrok http 4000) and register the tunnel URL in the Mux dashboard.

Media states

Status	Meaning
pending	Row created, job queued or in flight
ready	Variants exist (image) or Mux reports the asset ready (video)
failed	Processing threw; errorMessage holds the reason

Image failures update the row to failed, emit the realtime update, and re-throw so BullMQ retries. After three attempts the job lands in the failed set.

How the client learns it is done

Two mechanisms, belt and braces:

1. Realtime — media:processing:update on the user:<id> room, consumed by useRealtimeMediaProcessing.
2. Polling — useMediaProcessingPoll re-fetches the post until media reports ready.

Realtime from ogun is currently broken in production

ogun's emit() helper does not prepend /admin, but the deployment sets REALTIME_SERVICE_URL=http://playpalz-esu:4010 — the same value the API uses, where the API's helper does add the prefix. So ogun POSTs to /emit/media/processing/update, which esu does not route, and the image-ready event is silently dropped. .catch(() => {}) on the call hides the failure.

Only polling is delivering image-ready updates today, which is why nobody noticed. The video path is unaffected — that emit comes from the API, which builds the URL correctly. See Service Topology for the fix.

Storage layout

Prefix	Contents
uploads/<userId>/images/<year>/<uuid>.<ext>	Originals (image)
uploads/<userId>/videos/<year>/<uuid>.<ext>	Originals (video)
variants/<mediaId>/<type>.jpg	Generated image variants

Originals are retained after processing. There is no lifecycle rule expiring them, so storage grows monotonically — worth revisiting as volume increases.

Operating the pipeline

# Is anything queued?
redis-cli -a <password> KEYS 'bull:media-processing:*'

# What failed?
redis-cli -a <password> LRANGE 'bull:media-processing:failed' 0 -1

# Worker logs
kubectl logs -l app=playpalz-ogun --tail=100 -f

Scaling throughput means adding ogun replicas — the work is CPU-bound in Sharp. Watch Mux rate limits before scaling video volume aggressively.