Welcome to PGQueuer’s documentation!

Logo

Introduction

PGQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PGQueuer uses PostgreSQL’s LISTEN/NOTIFY to manage job queues effortlessly.

The repository is hosted on github

Installation

Install PGQueuer using pip:

pip install pgqueuer

Features

  • Simple Integration: Easily integrate with existing Python applications using PostgreSQL.

  • Efficient Concurrency Handling: Utilizes PostgreSQL’s FOR UPDATE SKIP LOCKED for reliable and concurrent job processing.

  • Real-time Notifications: Leverages LISTEN and NOTIFY for real-time updates on job status changes.

  • Batch Processing: Handles large job batches efficiently for both enqueueing and dequeueing.

Example Usage

Here’s how you can use PGQueuer in a typical scenario processing incoming data messages:

Start a consumer

Start a long-lived consumer that will begin processing jobs as soon as they are enqueued by another process.

pgq run examples.consumer.main

Start a producer

Start a short-lived producer that will enqueue 10,000 jobs.

python3 examples/producer.py 10000