taskq: Distributed Job Queue
A distributed job queue in Go, backed by PostgreSQL and Redis and deployed on Kubernetes.
The problem
Background work like emails, reports and data pipelines must run exactly once, even when workers crash mid-job or the network drops. Losing or duplicating a job is a real production incident.
What I built
- Built a job queue in Go that distributes work across concurrent workers
- Used PostgreSQL as the durable record of every job and its state
- Used Redis for fast coordination between workers
- Deployed the system on Kubernetes
Key decisions
Postgres for durability, Redis for speed
Each store does what it's best at: Postgres guarantees no job is lost, and Redis keeps scheduling fast.
Go for the workers
Goroutines make many concurrent workers cheap, and a single static binary simplifies deployment.
Progress
0 of 5 milestones- Enqueue and dequeue API
- Durable job state in Postgres
- Retries and crash recovery
- Kubernetes deployment
- Throughput benchmarks
Tech stack
GoPostgreSQLRedisKubernetes