Semantic Caching for LLMs on Kubernetes: Faster, Cheaper, and Sometimes Wrong
We put a Valkey semantic cache in front of a RAG pipeline on Kubernetes - Qwen on vLLM, MiniLM embeddings, Milvus, all deployed as OpenEverest Instances - and measured it. Cache hits answered in 88 ms instead of 4.7 s and cut LLM tokens by three quarters. Most of that came from exact repeats, and every bit of extra fuzziness bought wrong answers.