Need a lightweight job queue that runs across multiple machines without Redis, RabbitMQ, or cloud services? In this post, we’ll build a surprisingly capable distributed queue using SQLite, Python, and a little file-locking magic. This pattern works best when you need a shared queue across local or networked disks — think cron clusters, render farms, or batch processors — without spinning up infrastructure.

Why Use SQLite as a Queue?


  • No external services to maintain
  • Great for jobs with infrequent updates
  • Works well on shared network storage (NFS, SMB)

Step 1: Create the Queue Table


We’ll use a simple table with a claimed flag and a timestamp:


import sqlite3


def init_db():

  conn = sqlite3.connect("queue.db")

  conn.execute("""

    CREATE TABLE IF NOT EXISTS jobs (

      id INTEGER PRIMARY KEY,

      task TEXT,

      claimed_by TEXT,

      claimed_at DATETIME

    )

  """)

  conn.commit()

  conn.close()

Step 2: Enqueue Jobs


Add a job by inserting a row:


def enqueue(task):

  conn = sqlite3.connect("queue.db")

  conn.execute("INSERT INTO jobs (task) VALUES (?)", (task,))

  conn.commit()

  conn.close()

Step 3: Claim a Job with Locking


To safely claim jobs across machines, we’ll use a conditional update:


import datetime, socket


def claim_job():

  conn = sqlite3.connect("queue.db", isolation_level="IMMEDIATE")

  conn.row_factory = sqlite3.Row

  hostname = socket.gethostname()

  now = datetime.datetime.utcnow().isoformat()



cur = conn.execute("""

    UPDATE jobs

    SET claimed_by = ?, claimed_at = ?

    WHERE id = (

      SELECT id FROM jobs WHERE claimed_by IS NULL LIMIT 1

    )

    RETURNING *

  """, (hostname, now))



job = cur.fetchone()

  conn.commit()

  conn.close()

  return job

Step 4: Process and Delete the Job


def process(job):

  print(f"Processing: {job['task']}")


conn = sqlite3.connect("queue.db")

  conn.execute("DELETE FROM jobs WHERE id = ?", (job["id"],))

  conn.commit()

  conn.close()

Step 5: Worker Loop


This can run in a cron or systemd timer on multiple machines:


if name == "main":

  init_db()

  job = claim_job()

  if job:

    process(job)

Pros and Cons


✅ Pros


  • Zero external dependencies
  • Easy to inspect/debug
  • Works across NFS or shared volumes

⚠️ Cons


  • Not suitable for high-throughput workloads
  • Concurrent writes are serialized
  • SQLite file locking can behave differently per OS

🚀 Alternatives


  • Redis + RQ: Good for scalable Python job queues
  • Beanstalkd: Lightweight and dedicated for queuing
  • Celery: Overkill for local jobs, but full-featured

Summary


For scenarios where installing or maintaining queue infrastructure isn’t ideal, a SQLite-powered distributed queue offers a minimalist alternative. It’s perfect for home labs, edge nodes, or anything that needs durable, shared-state task processing with no server overhead.

If this was useful, you can support me here: buymeacoffee.com/hexshift