ragserve: Retrieval-Augmented Generation Service
A retrieval + generation service built on FastAPI and pgvector, with ONNX-optimized inference, deployed on Kubernetes.
The problem
Language models answer confidently even when they're wrong. Grounding every answer in retrieved documents makes it traceable, but only if retrieval and inference are fast enough to serve real traffic.
What I built
- Built a FastAPI service that retrieves relevant documents before generating an answer
- Stored embeddings in PostgreSQL with pgvector for similarity search
- Exported models to ONNX for faster inference
- Containerized the service and deployed it on Kubernetes
Key decisions
pgvector instead of a dedicated vector database
Vectors and metadata live in one Postgres instance, which means one system to run, back up and query.
ONNX-exported inference
It cuts latency and decouples the serving runtime from the training framework.
Progress
0 of 5 milestones- Retrieval with pgvector
- Generation endpoint
- ONNX inference
- Kubernetes deployment
- Latency and retrieval-quality benchmarks
Tech stack
PythonFastAPIpgvectorONNX RuntimeKubernetes