Nine Ways to OOM-Kill Valkey on Kubernetes

Every Valkey-on-Kubernetes tutorial tells you the same three things: set maxmemory, pick an eviction policy, and watch out for OOMKilled. Fair advice, and almost useless on its own, because it never says by how much, which failure mode you actually get, or what happens next.

So instead of writing another “best practices” list, we built 9 small Valkey Instances on a live managed Kubernetes cluster through OpenEverest, tried to break each one on purpose, and wrote down exactly what happened: the exit codes, the INFO snapshots, the Kubernetes events that did and didn’t show up, and the one scenario that quietly erased an entire dataset while the cluster kept insisting everything was fine.

If you want to reproduce any of this yourself, every manifest and load-generation script is in blog-data, alongside the raw write-up for each test.

TL;DR

The quick version: Valkey with no maxmemory has no idea it lives in a container, so the kernel ends up killing it instead of Valkey killing its own keys. Set maxmemory with real headroom, though, and it behaves exactly like the manual promises. The two results that actually changed how we’d configure a cluster are test 5 and test 7 below - a BGSAVE that OOM-killed two pods in under a second despite hundreds of megabytes of apparent headroom, and a replica full resync that quietly erased an entire dataset while kubectl kept reporting the cluster as perfectly healthy. Everything else - eviction policies, AOF overhead, fragmentation, QoS classes - mostly confirmed what you’d expect, with a few sharp edges worth knowing about before they find you in production.

The setup

Each test is an OpenEverest Instance (core.openeverest.io/v1alpha1), which the provider and valkey-operator turn into a ValkeyCluster/ValkeyNode, a StatefulSet, and the usual services and secrets. Nothing exotic - this is the same object you’d use to get a Valkey database in a few lines of YAML:

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: test3-mm-headroom
spec:
  engine: valkey
  engineVersion: "9.0"
  components:
    engine:
      resources:
        requests: { cpu: 500m, memory: 1Gi }
        limits: { cpu: 500m, memory: 1Gi }
      replicas: 0
  parameters:
    config:
      maxmemory: "850mb"
      maxmemory-policy: noeviction
    tls:
      mode: disabled

Traffic came from valkey-benchmark and plain valkey-cli, not synthetic shortcuts. That distinction mattered more than expected - more on that in test 2.

Test 1: no maxmemory at all

A 1Gi pod, default config, no persistence. We loaded it steadily and watched used_memory climb: 55MB, 519MB, 982MB, 1033MB - then the container was gone.

round 20: dbsize=101000  used_memory=1033.5MB rss=1057.1MB   (limit is 1Gi = 1073.7MB)
round 21: container OOMKilled. New container up within ~1s. dbsize back to 0.

Nothing here is surprising on its own - Valkey has no concept of a cgroup limit unless you tell it one via maxmemory, so it kept writing until the kernel stepped in. What is worth noting: kubectl get events never once says the word “OOMKilled.” The only hard evidence lives in status.containerStatuses[].lastState.terminated:

{"exitCode":137,"reason":"OOMKilled","startedAt":"...18:03:43Z","finishedAt":"...18:09:36Z"}

If your monitoring only watches Kubernetes events, it will never see this happen.

Tests 2-4: maxmemory, headroom, and eviction policy

This is where the DEBUG POPULATE issue showed up. Our first runs set maxmemory to 1000mb and 850mb under a 1Gi limit, loaded with DEBUG POPULATE, and… both got OOMKilled at the exact same point, regardless of the different maxmemory values. That’s not maxmemory failing - it’s the load generator skipping the check that would have stopped it. allkeys-lru wasn’t affected, because eviction runs as part of the server cron, independent of which command triggered it.

Re-running test 3 with real traffic told a completely different story. used_memory climbed to 890,949,032 bytes - essentially exactly maxmemory (891,289,600) - and then every further write failed with the same, real error:

(error) OOM command not allowed when used memory > 'maxmemory'.

DBSIZE settled at 86,081 keys. No restart, no drama. noeviction with genuine headroom (we used about 16% of the limit) does precisely what the manual says. allkeys-lru under the same headroom just quietly evicted the oldest keys instead and kept accepting writes, no errors at all.

As a rough starting point for your own maxmemory, under this kind of load we saw about 80% of the limit hold up fine for a cache-only pod, closer to 65% once persistence was involved, and around 55% once a replica entered the picture. Treat every one of those numbers as a starting point, not a guarantee - test 5, next, is exactly why.

Tests 5 & 6: the BGSAVE surprise

This is the one result we genuinely did not expect. Two pods, 2Gi limit, maxmemory at 1700mb, default save rules. We loaded each to about 1.5GB of live data, which left roughly 519MB of headroom under the container limit and 184MB under maxmemory. A second before we touched anything, both pods looked completely healthy: used_memory:1599132736 (about 1525MB), used_memory_rss:1628868608 (1553MB), mem_fragmentation_ratio:1.02.

Then we ran a plain BGSAVE. The very first sample we managed to pull back, one second later, told the whole story:

used_memory_rss:16384000

Sixteen megabytes. The container had already been OOM-killed and restarted before our one-second poll came back around - on both pods, on two different nodes, with no concurrent writes and hundreds of megabytes of apparent headroom on either side. Classic copy-on-write theory says this shouldn’t happen. Our best guess is page-table and allocator bookkeeping for a dataset made of a huge number of very small objects - an artifact of how we loaded the data - rather than payload copy-on-write itself, but we’re reporting this as a measured, reproduced anomaly, not a fully explained one.

The practical lesson doesn’t need the root cause to be useful: don’t trust “headroom under maxmemory” as a safety margin for snapshotting. Validate BGSAVE/BGREWRITEAOF under realistic memory pressure specifically, not just under normal traffic.

AOF, measured cleanly with real traffic this time, was much tamer. used_memory landed at 525,508,776 bytes (about 501MB) and used_memory_rss at 557,436,928 (about 532MB) - roughly 6% higher, a normal and predictable AOF tax. The AOF file itself came out to about 706MB, bigger than used_memory, which makes sense: it’s an append-only command log, not a compact snapshot. No OOM, nothing dramatic.

Test 7: the one that should worry you

Here’s the scenario, reconstructed from hard evidence end to end, no inference.

We set up a primary with one replica, 1Gi limit each, maxmemory 850mb, default save rules left on - nothing exotic. We loaded the primary to 75,000 keys (about 776MB) and confirmed the replica was fully caught up, then forced one full resync by hand just to watch it work. It did, cleanly, nothing unusual.

About 11 minutes later, with no write load running and nobody touching anything, the primary was OOM-killed anyway. It restarted in under a second, still claiming role:master, still showing connected_slaves:1 - but used_memory:3212960. Roughly 3MB. Empty. The replica never noticed a thing; it had stayed connected the entire time.

So we did the next logical thing: deleted the replica pod, forcing it to ask for a brand-new full resync on reconnect - the closest real-world stand-in for an actual replica restart. The primary served that resync without complaint, straight from its now-empty dataset:

Full resync requested by replica ... Synchronization with replica ... succeeded

And that was it. Both DBSIZEs read 0. All 75,000 keys, gone - and through every single step of this, kubectl get valkeycluster test7-replica-sync kept reporting STATE: Ready REASON: ClusterHealthy. Before the kill, during the empty restart, during the resync, and after all the data was gone.

We don’t have a fully confirmed cause for why the primary tipped over 11 minutes into an idle period - the two live candidates are a periodic background save under the default save rules, or cumulative overhead from repeated diskless resyncs - and we’re saying that plainly rather than guessing with false confidence. What we can say with full confidence is the consequence: a transient, self-healing-in-under-a-second pod restart cascaded into permanent, total data loss across the entire replica set, and nothing in the cluster’s own health reporting would have caught it. The only thing that would have is diffing DBSIZE before and after.

Test 8: fragmentation and a defrag footgun

Load 190k keys of 2KB values, then delete a scattered half of them - not a contiguous block, because contiguous deletes tend to free whole slabs back to the OS on their own - and watch what jemalloc does with the holes it leaves behind. used_memory dropped to about 205MB, correctly reflecting the live data left behind. used_memory_rss barely moved, parked at 396MB, for a mem_fragmentation_ratio of 1.93: jemalloc was still holding on to every page it had ever touched.

Turning on activedefrag looked like it worked, for about fifteen seconds: RSS fell from 396MB to 335MB, ratio from 1.93 to 1.63 - and then it just stopped. active_defrag_hits sat at zero the entire time it was supposedly working, which is the part that looked broken. MEMORY STATS had the answer: the remaining waste had settled at about 102MB, just over the default active-defrag-ignore-bytes floor of 100MB. Active defrag treats anything under that floor as not worth the CPU cycles, so once it got close, it quietly gave up.

Lower that floor to 1MB and widen the per-cycle budget, and the exact same process finishes almost instantly: RSS down to 233MB, ratio down to 1.14, and active_defrag_hits jumping from zero to 88,451 in about ten seconds. The default isn’t broken, it’s just sized for a much bigger server - on a 2Gi pod, 100MB is a large enough fraction of the whole process that activedefrag yes can quietly do almost nothing while looking exactly like it’s stuck.

Test 9: Kubernetes decides who dies, and it isn’t always about memory

We pinned a Valkey pod and a memory-hog neighbor pod onto the same node, then ran it twice in opposite configurations.

Run 1 - Burstable Valkey (64Mi request / 4Gi limit, using 977MB) next to a Guaranteed neighbor (using 4.5GB):

The node was low on resource: memory. Threshold quantity: 100Mi, available: 93696Ki. Container server was using 977060Ki, request is 64Mi, has larger consumption of memory.

The kubelet’s soft eviction manager killed the Valkey pod - twice - while the neighbor using more than 4x as much absolute memory was never touched. It ranks Burstable/BestEffort pods by how far usage exceeds request, not by absolute usage.

Run 2, reversed - Guaranteed Valkey (1.5Gi request = limit, using 894MB) next to a Burstable neighbor (64Mi request / 6Gi limit, using 4.5GB):

The neighbor was kernel-OOM-killed (exitCode: 137). Valkey, now Guaranteed, kept running untouched - no restart, DBSIZE unchanged.

The rule that falls out of this: for anything you don’t want casually evicted under node pressure, set requests == limits. A generous limit with a token request looks fine right up until the node gets tight - and then it’s often the first thing killed, regardless of how much memory it’s actually using compared to its neighbors.

What we’d actually change

Four small, concrete changes, not vague advice:

1. Default maxmemory. Derive it from the memory limit, with headroom, fully overridable. Today it defaults to 0, so every Instance starts out as test 1. We opened valkey-operator issue #560 for this.
2. Requests below limits. Warn, or validate against it, for the engine container. That mismatch is exactly what turned test 9’s Valkey pod Burstable.
3. Surface restarts in the Instance status. Not just on the pod. Test 7 is the reason: a cluster can pass every health check it exposes and have silently lost all its data.
4. Pick a default for “no persistence.” save "" makes a pod a pure cache - no forks, no stale reloads. The current default keeps periodic saves on, which is part of what made test 7 ambiguous.

None of this replaces reading your own INFO output under your own workload. But it’s shorter and more specific than “set maxmemory and watch out for OOMKilled,” and every line of it came from something we watched actually happen.

Try it yourself

If you’re running Valkey on Kubernetes and want a second pair of eyes on your maxmemory, replica, or QoS setup, book a call with our team. We’re happy to run these same tests against your own configuration.