The Cloud-Native Data Manual

1. Introduction: Kubernetes doesn't understand databases

Kubernetes is very good at keeping processes running. It knows whether a container started, whether it answers a probe, and which node it runs on. It doesn't know which replica is the primary, how far behind a replica is, whether the last backup worked, or whether a slow startup is a crash recovery that must not be interrupted.

For stateless applications, that is fine: if a pod misbehaves, restart it. For databases, restarting is often the wrong answer, and the right one depends on the engine. Promoting a replica in PostgreSQL works differently from electing a primary in MongoDB or recovering a node in a MySQL Group Replication cluster.

Operators fill the gap

A Kubernetes operator is a controller that knows how to run one database. You describe the cluster you want in a custom resource, and the operator creates the pods, volumes, and services, sets up replication, schedules backups, and handles upgrades, scaling, and recovery. Mature operators exist for every major open source database:

• PostgreSQL: CloudNativePG, the Percona Operator, and others. See Running PostgreSQL on Kubernetes.
• MySQL: Percona XtraDB Cluster, the Percona Operator for MySQL, Oracle's MySQL Operator, and others. See Running MySQL on Kubernetes.
• MongoDB: the Percona Operator for MongoDB and MongoDB Controllers for Kubernetes. See Running MongoDB on Kubernetes.
• SQL Server: commercial operators such as the Solanica MSSQL Operator, DH2i DxOperator, and KubeDB. See Running SQL Server on Kubernetes.

An operator does the engine-specific work, but it can't fix a bad platform underneath. The rest of this manual covers the decisions that are yours: how pods are placed, how much memory they get, what storage they sit on, where backups go, what idle databases cost, and who can reach them.

2. Availability: beyond the Running pod

A database pod can be Running while the engine is replaying its log after a crash, or while a replica is hours behind. High availability on Kubernetes comes from the database's own replication and failover, with Kubernetes making sure the copies are physically apart.

Why rescheduling is too slow for failover

Databases usually run in StatefulSets, which give each pod a stable name and its own volume. A StatefulSet also guarantees that at most one pod with a given identity runs at a time, so two pods never write to the same volume. That guarantee makes rescheduling after a node failure slow:

1. The node stops reporting. After node-monitor-grace-period, 50 seconds by default in current Kubernetes versions and 40 in older ones, it is marked NotReady.
2. Pods tolerate the resulting unreachable taint for 300 seconds by default before they are evicted.
3. Eviction can't complete, because the kubelet on the dead node can't confirm that the containers stopped. The pod stays in Terminating.
4. The StatefulSet won't create a replacement until the old pod is gone. That takes a force delete, deleting the Node object, or the node.kubernetes.io/out-of-service taint.

A single-instance database that relies on rescheduling is therefore down for at least six minutes, and often until someone acts. The fix is to not depend on rescheduling at all: run replicas, each with its own volume, and let the database promote one when the primary fails. The failed pod comes back later as a replica.

Anti-affinity: one replica per node

Replicas only help if they fail independently. Use required pod anti-affinity so that no two members of the same database cluster land on the same node. Most operators set this for you. Check that it is required, not preferred, in production.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: orders-db

Multi-AZ distribution

Spread replicas across availability zones so that losing a whole zone still leaves a quorum. Three replicas in three zones is the usual setup. With two zones, the zone holding the majority takes the cluster down with it.

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app.kubernetes.io/instance: orders-db

Zones protect against a data center failure, not a region failure. For that, you need a replica cluster or restorable backups in another region. Also note that cloud block volumes are zonal: a pod can't move to another zone and keep its volume, which is one more reason to fail over to a replica instead of rescheduling.

Disruption budgets and probes

Planned maintenance causes more restarts than hardware failure. When nodes are drained for a Kubernetes upgrade, a PodDisruptionBudget with maxUnavailable: 1 makes sure only one member of a cluster is evicted at a time. Most operators create one. Check it exists before your first node upgrade.

Be careful with liveness probes. A database that is replaying its log after a crash may not answer for minutes, and a strict liveness probe will kill it and start the recovery over again. Use a startup probe or generous thresholds for that phase, and use readiness probes to control which pods receive traffic.

3. Compute: the Guaranteed tier

The most common cause of databases restarting at random on Kubernetes is how CPU and memory requests and limits are set.

Guaranteed QoS (requests equal limits)

For stateless apps, low requests and high limits are a standard way to pack more pods onto each node. For databases, this causes problems. If the memory limit is higher than the request, the Linux OOM killer targets the database pod first when the node runs low on memory.

Set requests equal to limits for CPU and memory. This puts the pod in the Guaranteed QoS class, the last to be killed when a node is under pressure. It applies to every container in the pod, including sidecars such as backup agents and metrics exporters. A single sidecar without limits drops the whole pod to Burstable.

resources:
  requests:
    memory: 2Gi
    cpu: 500m
  limits:
    memory: 8Gi
    cpu: 4000m

resources:
  requests:
    memory: 8Gi
    cpu: 4000m
  limits:
    memory: 8Gi
    cpu: 4000m

Size the database cache inside the limit

Guaranteed QoS protects a pod from other pods. It doesn't protect it from itself: a container that goes over its own memory limit is killed regardless of its class. Each database has a main memory cache, and it has to fit inside the container limit together with connections, queries, background work, and sidecars.

Most operators calculate these settings from the container limit. If you change the memory limit, check that the cache setting followed.

CPU limits and throttling

A CPU limit is enforced by throttling. A database that hits it in short bursts gets slower without showing high average usage, which looks like random latency spikes. Watch container_cpu_cfs_throttled_periods_total. For latency-sensitive databases, give Guaranteed pods whole CPUs and enable the kubelet's static CPU Manager policy, which pins them to dedicated cores.

4. Storage and backups: three tiers

Storage is usually the biggest cost driver of a database, and the most likely performance bottleneck. It helps to think of it as three tiers, each with a different job.

Tier 1: object storage for backups

Persistent volumes are not backups. A volume can be corrupted, deleted by mistake, or lost with its zone or cluster. The foundation of any production setup is a continuous copy of the data in S3-compatible object storage outside the cluster, following the 3-2-1 rule: three copies, on two kinds of storage, with one off-site.

Every major database can do this with a base backup plus a continuous stream of its change log, which also gives you point-in-time recovery:

Keep the bucket in a separate account or region, so an incident in the cluster's account doesn't reach it. Turn on object lock or versioning to protect backups from deletion and ransomware. Restore regularly, because an untested backup is not a backup.

Tier 2: network-attached block storage

Network volumes such as AWS EBS, Google Persistent Disk, Azure Disk, or Ceph are the right default for most databases. They are decoupled from the node: if the node fails, the volume can be attached to a new pod with its data intact. They can also be resized online and snapshotted.

The trade-offs are latency and cost. Every I/O goes over the network, and performance beyond the baseline is paid for separately. On AWS, for example, gp3 volumes include 3,000 IOPS and 125 MiB/s, and anything more is provisioned and billed on top. For busy databases, provisioned IOPS can cost more than the volume itself. Volumes are also zonal, so they can't follow a pod to another zone.

Volume snapshots are useful for cloning and fast restores, but they don't replace tier 1. They usually live in the same account, and they don't give you point-in-time recovery.

Tier 3: local NVMe

Local NVMe drives on the node give the lowest latency and the highest IOPS, without paying for provisioned performance. They need instance types with local storage, and a provisioner that turns the drives into persistent volumes, such as the Local Static Provisioner, TopoLVM, or OpenEBS LocalPV.

The trade-off is that the data lives and dies with the node. A pod with a local volume can only run on that node, so when the node is gone, its volume claim has to be deleted before the replacement can be scheduled elsewhere. The new member then copies all its data from the other replicas. That is safe only with at least three replicas and current backups, and resyncing a large dataset takes time.

StorageClass settings that matter

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: db-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "6000"
  throughput: "250"
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

• volumeBindingMode: WaitForFirstConsumer creates the volume in the zone where the pod is scheduled. Without it, a volume can end up in a zone where its pod can't run.
• allowVolumeExpansion: true lets you grow a full volume without moving data. Databases rarely give disk space back, so you will need it.
• reclaimPolicy: Retain keeps the volume when its claim is deleted by mistake. Clean up retained volumes deliberately, or they become a hidden cost.

Storage decision matrix

5. Cost: eliminating idle infrastructure

Databases are always-on by default. In development, test, and staging environments, most of them sit idle for most of the week while their compute and provisioned storage keep billing.

Suspend non-production databases

Most operators can stop a database cluster's pods while keeping its volumes, configuration, and credentials. CloudNativePG calls it hibernation, and the Percona operators use spec.pause. Run it on a schedule, from a CronJob or your CI system, so that development databases stop in the evening and start again in the morning.

The savings are simple to estimate. A database used ten hours a day on weekdays runs 50 of the 168 hours in a week, so suspending it the rest of the time cuts its compute cost by about 70%. Storage keeps billing while it is suspended, and backups and log archiving stop until it resumes.

Scaling to zero when idle and starting on the first connection sounds attractive, but databases take time to start, and something has to hold the client's connection while they do. For most teams, a schedule is simpler and more predictable.

Rightsize with VPA recommendations

Instead of guessing memory and CPU, run the Vertical Pod Autoscaler in recommendation-only mode (updateMode: "Off") and compare its suggestions with what you have allocated. Apply the changes through the operator's custom resource, not by letting VPA edit the pods. The operator owns the pod spec and will revert changes it didn't make, and the database cache settings have to change along with the memory limit.

In-place pod resizing, generally available since Kubernetes 1.35, lets operators change CPU and memory without restarting a pod. Databases still need their configuration updated to use more memory, and shrinking memory below what the cache already uses gets the pod OOM-killed.

Common sources of waste

6. Security: zero-trust data access

Inside a Kubernetes cluster, any pod can try to connect to any other pod by default, and pod IPs change all the time. Database security has to be based on identity, labels, and encryption, not IP addresses.

Namespaces and RBAC

Put databases in their own namespaces, separated by environment or team, and use RBAC to control who can manage them and who can read their secrets. Namespaces also carry ResourceQuotas, which stop one team from using up a shared cluster. On their own, namespaces don't isolate network traffic. That takes NetworkPolicies.

Network exposure

Expose databases with ClusterIP Services by default, so they are only reachable inside the cluster. When applications outside the cluster need access, prefer an internal load balancer that stays inside your private network.

Then use NetworkPolicies to allow only the namespaces that need the database:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders-db-ingress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: orders-db
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: orders-app

NetworkPolicies are only enforced if your CNI plugin supports them. When you lock down ingress, remember the traffic the database itself needs: replication between its own pods, the operator, monitoring, and backup agents.

Encryption in transit

Encrypt both client connections and replication traffic between members. A compromised pod inside the cluster can otherwise read traffic it shouldn't see. Most operators can issue certificates through cert-manager and rotate them automatically. Enabling TLS isn't enough on its own, though: configure the database to reject unencrypted connections.

Encryption at rest and secrets

Encrypt volumes through the StorageClass, and encrypt backups in object storage with server-side or client-side encryption. Kubernetes Secrets, which hold database passwords, are only base64-encoded in etcd by default. Enable encryption at rest for Secrets with a KMS provider, or keep credentials in an external store such as HashiCorp Vault, and limit who can read Secrets in database namespaces.

Putting it together with OpenEverest

OpenEverest is a CNCF Sandbox project for running databases on Kubernetes through one control plane. It builds on proven operators such as CloudNativePG and the Percona operators, so failover, replication, and backups are handled by the engines and operators that know them best. On top, it gives you one UI and API for every database, with several of the practices in this manual built in:

• Replicated clusters with database-native failover, plus pod scheduling policies for anti-affinity and zone spread
• Resource presets and a choice of storage class, with volume expansion
• Backups to S3-compatible storage with point-in-time recovery
• Suspend and resume for databases you don't need around the clock
• ClusterIP Services by default, with reusable load balancer configurations for external access
• Role-based access control across namespaces, with single sign-on

OpenEverest is modular, so the same approach covers PostgreSQL, MySQL, MongoDB, and a growing list of other databases. You learn one API and one way of operating, whichever operator or database runs underneath.

Related whitepapers

• Running PostgreSQL on Kubernetes: streaming replication, Patroni HA, PgBouncer, pgBackRest backups, and operator comparison.
• Running MongoDB on Kubernetes: replica sets, sharded clusters, WiredTiger and oplog sizing, PBM backups, and operator comparison.
• Running MySQL on Kubernetes: InnoDB Cluster, Group Replication, XtraBackup, and operator selection.
• Running SQL Server on Kubernetes: availability groups without a Windows cluster, memory and licensing-aware sizing, native backups to S3, and operator comparison.
• Running Valkey on Kubernetes: replication and Valkey Cluster, the persistence trap, maxmemory and container limits, RDB and AOF, and operator comparison.

Run production databases on Kubernetes with Solanica

Solanica builds on OpenEverest to give you the automation of a managed database service on infrastructure you own, in any cloud or in your own data center. Talk to us about your setup, or try OpenEverest yourself.