The challenge
Give developers access to capable models on private infrastructure within a practical memory and capacity budget.
Dutch critical-infrastructure organisation
A 72B coding model on one H100.
In-house quantisation reduced a 72B coding model from around 144 GB to 66 GB, fitting one H100 with a 90K-token context window.
What was built
In-house GPTQ quantisation, private model serving, an API gateway with keys and usage limits, and a shared chat interface. Embeddings, reranking and multimodal models fit the same access layer.
What the figures mean
The memory reduction describes this implementation. It is not a general quality claim against larger or external models.
VF / 01REFERENCE ARCHITECTURE
- People and applications: CHAT / API / DEVELOPER TOOLS
- Access · API gateway: IDENTITY / LIMITS / ROUTING
- AI agents, retrieval and models: TOOLS / SOURCES / MODELS
- GPU · Storage · MLOps: KUBERNETES / NVMe-oF / OBSERVABILITY
REFERENCE, NOT A NETWORK MAP
The pattern behind the solution.
A generic overview of the building blocks. Technical addresses, internal systems and client identities are excluded from publication.
Explore the architecture ↗