Semantic Caching for LLMs on Kubernetes: Faster, Cheaper, and Sometimes Wrong

In Model-as-a-Service with OpenEverest we put a model behind an OpenAI-compatible endpoint with API keys and token quotas. The next question we kept hearing was: now that the GPU is the expensive part, how do we call it less?

The common answer is a semantic cache. Embed every question, keep the answers, and when a new question is close enough to one you have already answered, return the stored answer instead of calling the model. “How do I turn on AOF?” and “How can I enable append-only persistence?” should cost one generation, not two.

It sounds like a free lunch, so we built one and measured it: a small retrieval-augmented generation (RAG) service over the Valkey documentation, with a Valkey semantic cache in front of the LLM. Every database and every model in it is an OpenEverest Instance. Then we replayed the same 600 questions at six similarity thresholds and counted hits, tokens, latency and wrong answers.

TL;DR

• The stack: Qwen2.5-7B on vLLM (one GPU), MiniLM embeddings on CPU, Milvus for the documents, Valkey with valkey-search as the cache, a second Valkey for the gateway’s token quotas, and Envoy AI Gateway in front of the models. Deployed with OpenEverest and the Valkey, Milvus and KServe providers on a managed Kubernetes cluster.
• Fast and cheap: a cache hit answered in 88 ms at the median instead of 4.7 s, and the cache cut LLM tokens by 72-81%. The vector lookup in Valkey took 0.8 ms; most of a hit was the embedding call.
• But mostly exact: in our traffic, 59-71% of all questions were exact repeats of a cached question. Reworded questions that got the right answer added only 0.3-16 points on top.
• And sometimes wrong: every extra point of semantic hit rate came with wrong answers - from 0 at a strict threshold to 30 in 600 at a loose one. “What does the ’no’ option do?” is 0.107 away from “What does the ‘yes’ option do?”, closer than most real rewordings.
• A bigger embedding model didn’t fix it. bge-base separated rewordings from different questions no better than MiniLM; it just moved everything closer together.
• Memory: about 5.4 KB per cached answer with a 1 KB answer; 200,000 answers took 1 GB, and the lookup stayed under 1 ms.

What we built

One request can take three paths. Pick one to see which parts it touches:

Six Instances, three providers, one cluster:

Without the documents, Qwen doesn’t know what Valkey is. Asked “In two sentences: what is Valkey?”, it replied that “Valkey is not a widely recognized term or concept”. With the four retrieved chunks it answers from the docs. So every miss in this setup really does need retrieval plus generation, which is what makes a cache attractive.

A vector index in the cache you already run

The cache is a regular Valkey Instance. The valkey-bundle image ships the search module, and one config line loads it:

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: vk-cache
spec:
  providerRef:
    name: valkey
  version: "9.0"
  topology:
    type: replication
  components:
    engine:
      type: valkey
      image: valkey/valkey-bundle:9.0.6
      replicas: 1
      resources:
        limits: {cpu: "2", memory: 4Gi}
      storage:
        size: 10Gi
      parameters:
        config:
          loadmodule: /usr/lib/valkey/libsearch.so
          maxmemory: 3gb
          maxmemory-policy: allkeys-lru
    monitoring:
      type: exporter

The whole cache is three commands. One index over hashes with an sc: prefix:

FT.CREATE semcache ON HASH PREFIX 1 sc: SCHEMA
    emb VECTOR HNSW 6 TYPE FLOAT32 DIM 384 DISTANCE_METRIC COSINE

A lookup for the single nearest cached question, returning its answer and the cosine distance (0 = identical, 2 = opposite):

FT.SEARCH semcache "*=>[KNN 1 @emb $v AS dist]" PARAMS 2 v <384 float32s>
    RETURN 2 a dist DIALECT 2

And on a miss, a write that is indexed as it lands:

HSET sc:<id> q "<question>" a "<answer>" emb <384 float32s>
EXPIRE sc:<id> 86400

If dist is at or below a threshold, the stored answer is returned. Choosing that number is the whole game, as the rest of this post shows.

How we tested

The questions. We sampled 60 chunks from 60 different documentation pages and asked Qwen to write, for each:

• a question that the chunk answers,
• three rewordings of it with the same meaning, and
• a near-miss: a question with much of the same vocabulary that asks for something else.

That is 120 question groups: 60 groups of four equivalent questions, and 60 single near-misses. A cached answer served to a question from a different group is a wrong answer.

The traffic. 600 questions, 16 in flight at a time. Group popularity follows a Zipf distribution, the long tail that real FAQ-like traffic tends to have, and each request picks a random wording from its group. Each run starts with an empty cache.

The runs. One run without the cache, then the same traffic at thresholds 0.05, 0.10, 0.15, 0.20, 0.25 and 0.30.

The app, the scripts, the Instance manifests and the raw results of every run are in spron-in/blog-data/semantic-cache-llm-valkey-kubernetes, with a README on how to reproduce them.

Results

Each cached-run row is 600 questions. The no-cache run used 1,042 tokens per question, so the same 600 questions would have cost about 625k tokens. Even the strictest threshold cut that by 72%.

A hit is cheap, and Valkey is the cheap part of it

The cache made the service about 50 times faster per hit. The vector lookup in Valkey is under a millisecond; nearly all of a hit is the embedding call. A single embedding request on an idle model took about 15 ms; the 85 ms is two CPU replicas of MiniLM serving 16 concurrent clients. If you want faster hits, scale the embedding model, not the cache.

Throughput went from 4.4 to 10-15 questions per second. That is not a ceiling: with 16 clients, the few slow misses hold most of the client slots. The GPU now did a quarter of the work.

Most hits were exact repeats

The first column of the chart is the uncomfortable one. At a threshold of 0.05, where only near-identical questions match, 70.5% of all questions were already served from the cache. Loosening it to 0.30 added 15.5 points of correct semantic hits - and 30 wrong answers.

Two reasons. Zipf traffic repeats itself: our 600 questions were only 149 distinct strings. And our rewordings were deliberately varied, which is honest for humans but hard for a small embedding model: the median distance between a question and its rewording was 0.118, and one in ten was further than 0.29.

That doesn’t mean semantic caching is useless. It means a plain exact-match cache on the normalized question captures most of the savings with no risk, and the semantic part is a trade-off you tune.

What the wrong answers looked like

A wrong answer is a cached answer that belonged to a different question group. Here is what they were.

Same words, opposite meaning. Embedding models are very good at topic and bad at negation and direction. In a quick check before the runs, “How do I disable AOF persistence in Valkey?” landed 0.057 away from the cached “How do I enable AOF persistence in Valkey?”, closer than almost nine in ten real rewordings. The user asking how to disable AOF got:

To enable AOF persistence in Valkey, you would typically configure the appropriate settings in the Valkey configuration file… set the appendonly option to yes.

Meanwhile, a true rewording, “How can I turn on append-only file persistence?”, was 0.598 away and missed. From the test set: “What does the ’no’ option in the benchmarking tool do?” was 0.107 away from the cached “What does the ‘yes’ option…?”. Same story.

Same topic, different question. The most common wrong answer, four times in the 0.15 run: users asked “What does the maxmemory directive do?” (distance 0.100) and got the answer to “What happens if the maxmemory setting is exceeded?”. Close in topic, different in what the user wants. Similarly, “How do I set up ldap.bind_dn_prefix…?” got the answer to “What are the default values for ldap.bind_dn_prefix…?” at 0.063.

The first question wins. Look at the maxmemory example again: which answer gets served depends on which wording happened to be asked first. The cache doesn’t hold “the answer about maxmemory”; it holds the answer to whatever question arrived first, and lends it to every neighbor. In a different order, the other group would have been the one getting wrong answers.

Some “wrong” ones are fine. Our labels come from an LLM, and they aren’t perfect. “Why does Valkey crash frequently and how can I prevent it?” was labeled a near-miss of “How can I prevent Valkey server crashes or instability?”. A human would happily accept the cached answer. So the wrong-answer counts above are an upper bound, and the misses near the threshold include some answers that would have been acceptable. The direction of the trade-off doesn’t change.

Try the threshold yourself

Every dot below is one question from the test set, placed at its distance from the closest cached question. Everything left of the line is answered from the cache. Drag the threshold, switch the embedding model, and hover a dot to see the pair.

A bigger embedding model didn’t fix it

The obvious next step is a better embedding model, so we ran the same test with bge-base-en-v1.5: twice the dimensions and about five times the parameters of MiniLM. Switch the model in the chart above and the dots move left. Everything got closer, both rewordings and near-misses.

What matters is whether the two groups separate, and they didn’t, any better:

Two lessons. First, a threshold belongs to a model: 0.15 is strict for MiniLM and loose for bge-base. Change the embedding model and the threshold you tuned means something else. Second, general-purpose embeddings measure what a question is about, and a cache needs to know what it asks. A different model moves the trade-off along the same curve rather than off it.

How much memory a cache takes

To size a cache, we filled a fresh index with synthetic entries: a 1,000-byte answer, an 80-byte question, and a 384-dimension vector each. Then we timed 300 nearest-neighbor lookups from a pod in the same cluster, over TLS.

About 5.4 KB per answer, of which 2.2 KB is the HNSW index and the rest is the hash itself (the 1.5 KB vector is stored in both). The replica holds a second copy. Lookups stayed under a millisecond at 200,000 entries, the round trip included. The real answers in our runs were shorter, about 2.5 KB per hash.

Two settings mattered more than the numbers. maxmemory with allkeys-lru turns the cache into a cache: under memory pressure Valkey evicts the least recently used answers, and valkey-search drops them from the index. The TTL keeps answers from outliving the documents they were generated from.

The second Valkey: token quotas

The KServe provider can enforce an hourly token quota per API key and model at the gateway. Envoy’s rate limiter needs a Redis-compatible store for the counters, so that is vk-quota, a second, small Valkey Instance with TLS off. We kept it apart from the cache on purpose: quotas are small, hot and should never be evicted; cache entries are large and evicted all the time.

To check it, we lowered the LLM’s quota to 3,000 tokens per hour. The gateway answered straight away:

HTTP/1.1 429 Too Many Requests
x-ratelimit-limit: 3000, 3000;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 3405

rag-app passes that on as a 429 with Retry-After. Cache hits never reach the model, so they kept working: with a cache in front, running out of quota slows a service down instead of taking it down.

What we’d tell anyone adding a semantic cache

1. Start with an exact cache. Normalize the question (case, whitespace, punctuation) and cache on that. In our traffic it delivered most of the savings with zero wrong answers.
2. Then add the semantic part, and measure the wrong answers, not just the hit rate. A hit rate that goes up can mean more right answers or more wrong ones. You need labeled pairs from your own domain to tell them apart.
3. Your threshold belongs to your embedding model. Re-tune it whenever the model changes.
4. Embeddings don’t understand negation. “Enable” and “disable” are neighbors. If your questions have direction, check for it, or don’t serve those from a fuzzy match.
5. The first question wins. A semantic cache lends one answer to every nearby wording. Consider caching only answers that have been confirmed (asked several times, or rated), and expire them with the source data.
6. Valkey is not your bottleneck. The lookup took under a millisecond at 200,000 entries; the embedding call took 85 ms. Scale the embedding model.
7. Keep the cache and the quota store apart. One evicts constantly, the other must never lose a key.

Try it

Everything here runs on OpenEverest with the Valkey, Milvus and KServe providers. Model-as-a-Service with OpenEverest walks through the gateway and model setup step by step; the cache is one more Valkey Instance and about 200 lines of Python.

If you are building retrieval, caching or model serving on Kubernetes, book a call with our team. We are happy to look at your setup, or run the same measurements on your data.