PgQue · Design Brief

Partition KeysOrdered, parallel consumption — the log-native way

slug partition-keys version v0.1 (draft) layer consumer-side engine untouched

Within one queue, events that share a partition key are consumed in order by a single worker at a time; events with different keys are consumed in parallel. Order within a key, parallelism across keys — Kafka's partition model, achieved by routing, with no per-event locks or mutable state.

01The problem

PgQue is an ordered, immutable log, not a job queue. The motivating workload (a multi-tenant storage service evaluating PgQue to replace pg-boss) emits millions of file-lifecycle events — FileCreated, FileDeleted, FileOverwritten. They must be processed in order per tenant, but order across tenants does not matter.

A single in-order consumer can't keep up; naive multi-worker consumption breaks per-tenant order. PgQue has cooperative consumers, but they distribute events without key affinity — so a key's order is not preserved. This brief closes that gap.

02Architecture

producers pgque.send(queue, partition_key, payload) key → ev_extra1 (free today) engine · sacred — UNCHANGED append-only event tables · global ev_id order next_batch / get_batch_events / rotation — no edits to batch_event_sql batch of events partition layer · NEW (routing only) slot = hash(partition_key) mod N rides on cooperative consumers · no per-event state, no lock slot 0 · worker A keys where h%N==0 in ev_id order tenants 0,3,6… slot 1 · worker B keys where h%N==1 in ev_id order tenants 1,4,7… slot N-1 · worker C keys where h%N==N-1 in ev_id order tenants 2,5,8… one key → one slot → per-key order preserved · distinct keys → parallel
The new logic lives only in the distribution step. The PgQ engine is not modified.

03Scope

In · v0.1

  • Partition key carried on an event
  • Stable hash(key) % N assignment
  • Per-key order across batches
  • Explicit failure policy (pause vs skip)

Out · later

  • Producer idempotency / dedup window
  • Dynamic rebalancing / elastic N
  • Cross-queue / cascaded partitioning
  • Automatic hot-partition mitigation

04Key decisions

IDDecisionChoice (v0.1)
D1Where the key livesev_extra1 — no schema change
D2Failure policy (head-of-line)Pause the partition (default); skip opt-in
D3ElasticityFixed N in v0.1
D4Assignment functionhashtext(key) % N, stable
D5Per-event stateNone — routing only, no lease, no lock

D2 is the contract-shaping choice: PgQ retry re-inserts a failed event with a later ev_id, which would reorder a key — so strict order must block the key until the failure resolves.

05Sprint plan

S1  producer + key plumbing S2  partition-aware assignment S3  pause-on-failure + skip mode S4  docs + throughput benchmark

Biggest open risk: a crash-safe pause-on-failure that holds a key "blocked" with no mutable table — likely re-derived at slot start from the presence of an unacked/retrying event for the key. Next concrete step is confirming where assignment hooks into cooperative consumers without touching the engine.