Kubernetes, LLM and
Platform Engineer
walk into a bar
Sergey Pronin
Founder, Solanica Inc. · Maintainer, OpenEverest
Three walk into a bar
Platform engineer
wants reproducibility
LLM
wants 80 GB of VRAM
Kubernetes
wants an object
and a reconciler
Then the ticket arrives
PLAT-4242
assignee: you
We need to run LLMs
on our cluster.
Figure it out.
Why not just call an API?
data stays in
$
no token bill
zzz
idle GPUs
What we're building
chat app
search
notebook
one URL
/v1
qwen3-14b
llama-3.1-8b
mistral-7b
Day 1
What is a model, anyway?
A model is just files
a folder of files
from Hugging Face
14,000,000,000
numbers
like a database's data files
It writes one token at a time
Kubernetes is
model
a
Kubernetes is
a
model
container
Kubernetes is
a container
model
orchestrator
every token = math on all 14 billion numbers
Serving a model
weights
+
vLLM
engine
+
GPU
=
HTTP API
an endpoint
14B parameters ≈ 30 GB
Qwen3 4B
8 GB
Qwen3 14B
30 GB
Qwen3 32B
66 GB
Llama 70B
141 GB
20 GB GPU
80 GB GPU
What fills the GPU
GPU memory · 20 GB
weights · 8 GB
+ KV cache: memory for every open conversation
full → new requests wait
On one machine, it's one command
$ vllm serve Qwen/Qwen3-4B
loading weights… ready on :8000
$ curl localhost:8000/v1/chat/completions
→ "Kubernetes is a container orchestrator…"
Week 1
Put it on Kubernetes
The naive way
Deployment
image: vllm/vllm-openai
args: serve Qwen3-4B
nvidia.com/gpu: 1
pod
vLLM
it works!
Then day 2 happens
every restart downloads the model again
new model version = hand-made rollout
every model needs its own flags
ten teams, ten copies of this YAML
sound familiar?
An operator for models
LLMInferenceService
model: qwen3-14b
gpus: 1
replicas: 1
reconcile
pod
vLLM
Week 3
The other teams find out
One endpoint for every model
model: qwen3-14b
app
AI
gateway
qwen3-14b
llama-3.1-8b
One key per app
chat app
batch job
alice
one shared key
qwen3-14b
Count tokens, not requests
3,000 tokens per hour, per key
chat app
1,800
batch job
3,000
429
Too Many Requests
Month 2
Someone wants a bigger model
It doesn't fit
GPU · 20 GB
Qwen3 14B · 30 GB
OOM!
Split it across nodes
layers 1–14
layers 15–27
layers 28–40
head
worker
worker
Month 3
Is it working? Who's paying?
Two questions
Is the model healthy?
vLLM
time to first token
requests waiting
Who is using it?
AI gateway
tokens per key
tokens per model
CPU % is a lie
CPU
bored
requests waiting
GPU is full
scale on this
A new replica is a 30 GB download
traffic spike
schedule
pull image
download weights
load to GPU
minutes, not seconds
Look at what we just built
chat
search
batch
AI
gateway
keys · token limits
KServe
vLLM + GPU
vLLM + GPU
vLLM + GPU
metrics · dashboards · autoscaling
We've built this before
LLM
databases
weights
≈
data files
vLLM
≈
database server
KServe
≈
operator
AI gateway
≈
proxy
API key
≈
database user
split across GPUs
≈
sharding
A model is just another Instance
OpenEverest
PostgreSQL
MongoDB
Qwen3 14B
same UI · same API · same RBAC
Everyone got their order
models are Instances,
like databases
as many GPUs
as it needs
objects and
reconcilers
Try it on your cluster
1. install OpenEverest
2. add the KServe provider
3. deploy a model
4. one key per app
Thank you
openeverest.io
Sergey Pronin · solanica.io
#openeverest-users on CNCF Slack
