Serving Open Models in Production

Six chapters that take an open-weight checkpoint from a container that answers one request to a deployment that survives real traffic, a real budget, and more than one tenant.

Why serving a model to many users at once is a different engineering problem than running one on your own machine.

Turning weight count and context length into a real VRAM budget, instead of guessing and watching a container crash on startup.

One stable address in front of two correctly sized backends, with routing, failover, and health checks doing the work no client should have to.

Turning a vague sense that requests feel slow into a percentile target you can actually alert on, and tracing which hop is responsible when it is breached.

Why the GPU hourly rate is rarely the number that actually sets cost per token, and what a tight latency target quietly costs in reserved headroom.

Turning named-but-unconfigured virtual keys into an actual isolation guarantee, so one tenant's burst can never quietly spend another tenant's headroom.