PgQue · Design Brief
Within one queue, events that share a partition key are consumed in order by a single worker at a time; events with different keys are consumed in parallel. Order within a key, parallelism across keys — Kafka's partition model, achieved by routing, with no per-event locks and no new mutable state.
PgQue is an ordered, immutable log, not a job queue. The motivating
workload (a multi-tenant storage service evaluating PgQue to replace pg-boss) emits
millions of file-lifecycle events — FileCreated,
FileDeleted, FileOverwritten. They must be processed
in order per tenant, but order across tenants does not matter.
One in-order consumer can't keep up; naive multi-worker consumption breaks per-tenant order.
Per-key affinity + FIFO. Every event of key K maps to one slot hashtextextended(K,0) % N; within it, non-retried events of K arrive in ev_id order, never to another slot.
Single processor per key. At most one worker holds an unacked event for K at any instant.
Failure boundary. pause: no later K event until the failed one is acked or dead-lettered. skip (v0.1 default): at-least-once, order not held after a failure.
Phase 1 · converged, build now
send()-sourced eventsskip failure policy (default)Phase 2 · follow-up / out
pause strict order — open mechanic (defer w/o retry-count)| ID | Decision | Choice (v0.5) |
|---|---|---|
| D1 | Where the key lives | ev_extra1, send()-sourced queues only |
| D2 | Failure policy | skip default (Phase 1); pause = Phase 2 |
| D3 | Slot count N | Fixed, persisted & enforced per (queue, consumer) |
| D4 | Assignment function | (hashtextextended(key,0) % N + N) % N (stable, sign-safe) |
| D5 | State budget | Happy path & skip: none. pause: a compact blocked-key marker (per failing key, not per event) |
| D6 | Producer signature | send(queue, type, payload, partition_key =>) |
| D7 | Slot & single-owner | slot = consumer "<c>#k/N"; receive lock |
| D8 | Worker→slot assignment | client-side claim via pg_try_advisory_lock; no leader/assignor. G2 receive lock = correctness; advisory lock = distribution |
Three review rounds against the real engine. Round 2 verified G1's ev_id ordering and the G2 single-owner lock are real. Round 3 declared Phase 1 (skip) converged / implementation-ready and split pause out as a follow-up: withholding a blocked key's later event has an unsolved mechanic — a server-side drop would lose the event, and event_retry would wrongly count it toward DLQ — so strict pause needs a defer-without-retry-increment primitive that doesn't exist yet. It also corrected the security model: the filter hook is admin-only, so the consumer wrappers reach it by co-ownership with the installer, not by a role grant.
Phase 1 (S1–S2) is the converged, build-now slice. Phase 2 (S3, pause)
is gated on two open items: a defer-without-retry-increment primitive (O1) and
bounding the hot-blocked-key cost (O2). Phase 1 alone covers the per-tenant
ordered, parallel consumption the motivating workload needs.
Slots are claimed, not assigned. There is no consumer-group
leader, no PartitionAssignor, and no stop-the-world rebalance —
Kafka needs all three because a partition is exclusive by protocol.
Here the database is the coordinator, so assignment is
pull-based and self-balancing: a worker doesn't get told which slot to
take, it claims a free one.
Claim loop · client-side
pg_try_advisory_lock per slot 0..N-1receive_partitionedScale 8 → 10 workers
Correctness — G2 receive lock (blocking for update): two workers targeting slot k → exactly one gets the batch; the other blocks. Ordering can never break.
Distribution — advisory slot lock (non-blocking try): steers workers onto distinct slots. Pure liveness — skipping it loses even spread, never G1/G2.
N is fixed (D3) and is the read-amplification multiplier (§6 — every slot scans the full stream), so N is chosen to match the parallelism you need, not inflated to dodge a future resize. Even spread under key skew (a hot slot stays hot) and online resize of N (which reshuffles keys across slots, breaking per-key order mid-flight) are deferred — R3, R4.