Kubernetes 1.37 “Garhwal”: 7 Treats for AI and Data Teams

For years Kubernetes treated accelerators, distributed training and fat data volumes as an afterthought. The scheduler counted GPUs like they were apples in a bag, a single flaky card could cordon a whole node, and anyone running gang-scheduled training bolted on a second scheduler and prayed the two never disagreed.

Kubernetes 1.37, codenamed Garhwal, is the release where a lot of that duct tape finally comes off. The interesting part isn’t the changelog length - it’s that the primitives AI and data platforms have been faking with external controllers are now in the upstream API. Here are the seven that matter, why the old way hurt, and what changes for you.

1. Dynamic Resource Allocation grows up

The old Device Plugin model advertised a GPU as an integer. Eight units of “gpu”, no topology, no health, no NUMA. The scheduler had no idea whether those eight cards shared a fast interconnect or sat on opposite sides of the PCIe bus, so collective operations in a training run could quietly fall off a cliff. Worse, one overheating card meant cordoning the entire eight-GPU box.

1.37 graduates a whole cluster of Dynamic Resource Allocation features to stable at once:

• Extended resources through DRA go GA, so your existing pods that request example.com/accelerator: 1 keep working unchanged while DRA does the real allocation underneath. No rewrite to the ResourceClaim API on day one.
• Device taints and tolerations go GA. You can now taint a single degraded card and evict just the pods that don’t tolerate it, leaving the seven healthy GPUs in service.
• A standardized resource.kubernetes.io/numaNode attribute gives every vendor a common language for NUMA alignment across GPU, memory and NIC.
• Claim status now exposes the physical bits - MAC addresses, IPs, interface handles - which is what network-heavy training actually needs.

Two more land in beta: workload-level resource claims lift the old 256-pod reservation ceiling so an entire PodGroup can share one claim, and the scheduler requeue path drops from a linear scan to an indexed lookup, which roughly doubles scheduling throughput when a big node pool comes online.

Quarantining a bad card looks like this:

apiVersion: resource.k8s.io/v1alpha3
kind: DeviceTaintRule
metadata:
  name: quarantine-faulty-accelerator
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: worker-gpu-node-12
  deviceSelector:
    matchAttributes:
      driver: accelerator.example.com
      deviceIndex: "3"
  taints:
    - key: hardware.example.com/thermal-degraded
      value: "true"
      effect: NoSchedule

One node keeps running. One card takes the day off. That was fantasy a release ago.

2. Gang scheduling that lives in the scheduler

Frameworks like PyTorch DDP, Megatron-LM and Ray need every rank to start together, or they deadlock waiting for a ring that never forms. The default scheduler places pods one at a time, so a 32-node job could get 30 pods placed, lose the last two to another workload, and sit there holding thirty idle GPUs hostage. The usual fix was a heavyweight batch scheduler shadowing the real one - and the two drifting apart at the worst possible moment.

Workload-Aware Scheduling is now baked into the core control plane. The Workload and PodGroup APIs move to v1beta1, and a new CompositePodGroup lets you describe multi-tier jobs - coordinators, parameter servers, workers - as a tree that schedules all-or-nothing. If any child group can’t meet its minimum, nothing binds. No partial reservations, no capacity deadlocks.

The part that removes the most yak-shaving is native Job integration. You express gang behavior and topology right on a plain batch/v1 Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-foundation-training
spec:
  parallelism: 8
  completions: 8
  completionMode: Indexed
  scheduling:
    schedulingPolicy: gang
    minCount: 8
    disruptionMode: all
    schedulingConstraints:
      topology:
        - key: topology.kubernetes.io/rack

minCount is mutable, so autoscalers can resize the gang without tearing the job down, and disruptionMode: all means the scheduler won’t evict a lone rank out of the middle of your communication ring.

3. In-place vertical scaling that can make room for itself

In-place pod resizing already let you grow a container’s CPU or memory without a restart - as long as the node had spare headroom. On a bin-packed AI cluster it usually didn’t, so the resize came back marked Deferred and just sat there. Your mission-critical inference pod would wait indefinitely while some low-priority batch job hogged the slack, and you’d end up manually hunting for a pod to kill.

1.37 adds an alpha feature gate, InPlacePodVerticalScalingSchedulerPreemption, that lets the scheduler notice a deferred resize and clear space for it. The twist is that it stays on a single node - it evicts lower-priority victims on that exact host to free the CPU or memory, respecting PodDisruptionBudgets and priority the whole way, then the kubelet adjusts cgroups without a restart. The container never blinks.

4. etcd RangeStream keeps the control plane off the OOM ledge

Big AI clusters beat on the API server relentlessly - thousands of pods, dense DRA metadata, constant status churn. When the watch cache warms up, it pulls entire resource collections out of etcd. The old paginated Range calls counted keys, not bytes, so a single page of fat objects could balloon to hundreds of megabytes, buffered in full on both sides. During a restart or failover, several caches warming at once was a reliable recipe for OOM panics cascading through etcd and the apiserver.

RangeStream, beta and on by default with etcd v3.7+, swaps the buffered request-response for chunked gRPC streaming. etcd cuts results into byte-bounded chunks and streams them; the apiserver decodes each chunk and drops the buffer before asking for the next. Neither side ever holds the whole collection. The result is a flat, predictable memory ceiling during exactly the moments that used to take clusters down. Run an old etcd and it quietly falls back to the paginated path.

5. Native scale-to-zero for GPUs that sit idle

Inference traffic is spiky. A specialized model can sit untouched for an hour between requests, and until now the HPA flatly refused to let minReplicas drop below one. On dedicated GPU nodes that meant paying, continuously, for a hot replica doing nothing - or bolting on KEDA or Knative to fake scale-to-zero with a non-standard control loop.

1.37 promotes native scale-to-zero to beta. Point an HPA at an external or object metric - queue depth, message lag, gateway latency - and you can set minReplicas: 0 directly. A new ScaledToZero status condition tells your monitoring the difference between “idle on purpose” and “someone tore this down”, so nobody panics at 3am.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-queue-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-gateway-worker
  minReplicas: 0
  maxReplicas: 16
  metrics:
    - type: External
      external:
        metric:
          name: kafka_consumergroup_unprocessed_records
        target:
          type: Value
          averageValue: "10"

The catch is sensible: CPU and memory metrics can’t drive it, because those signals vanish the moment the last pod dies. Scale on the queue, not the pod.

6. Checkpoint and restore, straight from the CRI

A large model server spends real time before it serves a single token - pulling weights from object storage, warming shared memory, compiling graphs through Triton or XLA, priming CUDA contexts. When a spot node gets reclaimed or a box goes down for patching, all of that warmup evaporates and the replacement pays it again from scratch.

1.37 introduces alpha CheckpointPod and RestorePod primitives in the CRI itself (KEP-5823). With a runtime like containerd 2.0+ and CRIU underneath, CheckpointPod freezes the whole pod - process tree, memory pages, file descriptors, sockets - into an on-disk archive, and RestorePod rehydrates it into a fresh sandbox. That unlocks a few genuinely new moves:

• Warm the model once, snapshot it, and stamp out pre-warmed replicas across the cluster in sub-second time.
• Checkpoint a long training job before node maintenance and resume it elsewhere instead of losing hours of compute.
• Freeze a misbehaving pod exactly as-is and copy it into a staging namespace to poke at the memory.

It’s early and low-level, but it’s the foundation the “instant replica” tooling has been waiting for.

7. The scheduler finally thinks about storage

Data engines - Spark, Trino, vector indexes, streaming stores - live and die on local disk, yet the scheduler historically placed pods on CPU and memory alone. So it would happily drop a storage-hungry pod onto a node with plenty of cores and almost no disk, and you’d eat PVC binding stalls and provisioning failures. Cloud volume-attach limits made it worse: a node could look free on compute and still refuse the volume because it hit its hypervisor disk-slot ceiling, leaving pods stuck in ContainerCreating.

1.37 promotes two storage-scheduling integrations to beta. Storage capacity scoring adds a scoring plugin that ranks nodes by remaining disk headroom, so heavy data workloads spread out instead of piling onto a node that’s about to fill up. And the cluster autoscaler now tracks CSI volume-attach limits during its simulation, so when a node is near its attach ceiling it provisions another one rather than scheduling a pod that can never attach its disk.

The cheat sheet

Not everything here is production-ready, and that’s the whole point of reading the maturity column before you get excited. Here’s where the seven land:

So, should you rush the upgrade?

The stable stuff is the easy yes. DRA consolidation and the storage-aware scheduling changes are the kind of thing that quietly stops 2am pages, and etcd RangeStream is on by default the moment you pair 1.37 with etcd v3.7 - you mostly just get to stop worrying about a failure mode you used to. Native gang scheduling on plain Jobs is the headline for anyone running distributed training, and it’s solid enough at beta to start planning migrations off your bolt-on scheduler.

The alpha features - resize preemption and pod checkpoint/restore - are the ones to prototype in a lab, not point production traffic at. They’re a clear signal of where things are heading, and checkpoint/restore in particular is worth understanding early, because “warm once, stamp out replicas” changes the economics of serving big models.

Taken together, Garhwal is less a pile of features and more a statement: the primitives for AI and data don’t live in a sidecar controller anymore. They live in Kubernetes. If you run this stuff on OpenEverest or the Solanica Platform, that’s the release you’ve been waiting for.