PgQue revives PgQ, a queue system battle-tested at Skype for over a decade handling hundreds of millions of users. It strips out the original’s C extension and daemon, rebuilding everything in pure PL/pgSQL. Install it with one SQL file on Postgres 14+, pair it with pg_cron for ticking, and it runs on managed services like RDS, Aurora, Cloud SQL, AlloyDB, Supabase, or Neon—no restarts, no approvals needed.
This matters because most in-database queues crumble under load. They use SKIP LOCKED with row-by-row DELETE or UPDATE, piling up dead tuples, bloating indexes, and spiking VACUUM costs. Performance decays over months. PgQue sidesteps this: it batches via snapshots and rotates tables with TRUNCATE. The hot path generates zero dead tuples, stays predictable, and scales for sustained heavy loads without drift.
Proven Roots, Modern Constraints
PgQ powered Skype’s messaging from the mid-2000s. Marko Kreen presented it at PGCon 2009; it processed billions of events on large self-hosted Postgres clusters. But standard PgQ needed a C extension (pgq) and daemon (pgqd), locking it out of managed Postgres. Providers block custom extensions for security and stability.
PgQue ports the core logic to PL/pgSQL, inheriting Postgres guarantees: ACID transactions, WAL durability, replication, backups, and SQL visibility. Enqueue and consume stay transactional. No extra distributed system like Kafka or RabbitMQ means simpler ops—no separate scaling, monitoring, or failure modes. For teams on managed DBs, this slashes infrastructure tax while keeping queue semantics inside your primary data store.
Skepticism check: PL/pgSQL is interpreted, so it won’t match the original C speeds at extreme TPS. But for most workloads—background jobs, event streaming, task queues—it prioritizes reliability over raw microsecond latency. Original PgQ handled Skype-scale; PgQue targets the same architecture sans bloat.
Performance Realities and Latency
PgQue batches events via snapshots, not row-locking. Consumers grab a consistent view of unprocessed items, process in bulk, then rotate tables. This enforces zero bloat but trades single-message latency for stability.
Default end-to-end latency hits 1-2 seconds: up to 1 second for the next cron tick, plus consumer poll time. Individual calls (enqueue/receive/ack) clock microseconds. Tune it lower by cranking tick frequency or queue thresholds—force_tick() pushes immediate batches for testing. Under load, it avoids the “slow tail” other queues suffer from tuple cleanup.
Benchmarks aren’t public yet, but the project promises comparisons soon. Expect it to shine in sustained throughput: think 10k+ events/sec without degradation, based on PgQ heritage. Test it yourself; the repo includes scripts. Why care? In production, queues fail from bloat, not peak speed. PgQue engineers for the long haul, matching Postgres’ strengths.
Getting Started and Caveats
Installation takes minutes. Run the single pgque.sql file as superuser. Grant roles to app users. Enable pg_cron (available on most managed Postgres) to tick queues every second or so. Client libs exist for Python, Node.js, Go—check the docs for enqueue/consume patterns.
Architecture splits queues into batch tables rotated hourly/daily. Consumers poll via pgque.receive_batch(), ack with done(). Supports partitioning, retries, dead-letter queues. Project status: alpha-ish, actively developed. MIT license, contributions welcome.
Implications? For startups on Supabase/Neon or enterprises on RDS, PgQue embeds queuing without vendor lock or ops overhead. Ditch external brokers for 80% of use cases—simpler stacks win. But if you need sub-100ms latency or billion-event bursts, layer Redis/Kafka on top. Fair trade: battle-tested design now portable, at the cost of PL/pgSQL’s overhead. Deploy it; Postgres queues just got viable.
