CloudPlatformLabs
GuidesContact
Kubernetes12 min readPublished: 2026-09-28

Kubernetes in Production: Essential Best Practices for Reliability and Security

A production-tested blueprint for configuring Kubernetes workloads: pod disruption budgets, resource limits, liveness and readiness probes, and network security policies.

CPL

CloudPlatformLabs Engineering

Platform Architecture Team

Back to all guides

1. Precise Resource Requests and Limits Configuration

Running workloads in Kubernetes without properly defined CPU and memory constraints is the single most common cause of cluster instability. When CPU limits are misconfigured, CFS throttling severely degrades latency; when memory limits are omitted, memory leaks trigger node-wide OOM (Out of Memory) kills that randomly evict critical system pods.

Always set requests based on your 95th percentile baseline utilization during steady load, and set limits to prevent unbounded resource starvation. For memory, we recommend setting requests equal to limits to guarantee that the pod is placed in the 'Guaranteed' Quality of Service (QoS) tier, drastically reducing unexpected evictions.

deployment-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
      - name: app
        image: registry.cloudplatformlabs.dev/payment-service:v2.4.1
        resources:
          requests:
            cpu: "250m"
            memory: "512Mi"
          limits:
            cpu: "1000m"
            memory: "512Mi"

Avoid Premature CPU Throttling

Setting CPU limits too tight causes Linux Completely Fair Scheduler (CFS) quota throttling even when cluster CPU usage is low. Monitor 'container_cpu_cfs_throttled_periods_total' in Prometheus.

2. Structuring Health Probes (Liveness, Readiness, Startup)

Health probes govern how Kubernetes handles application lifecycles. Conflating liveness and readiness probes is a frequent anti-pattern: if your database goes down and your liveness probe checks database connectivity, Kubernetes will reboot all your application pods simultaneously, precipitating a cascading failure.

Use Startup probes for services with slow initialization (such as JVM-based microservices or apps running database migrations). Use Readiness probes to signal whether the pod can accept client traffic right now, and keep Liveness probes strictly isolated to detecting internal deadlocks and unrecoverable threads.

  • Never perform external dependency checks (DB, Redis, downstream microservices) inside a Liveness probe.
  • Ensure readiness probes recover quickly once downstream caches or connections stabilize.
  • Tune timeoutSeconds to avoid false-positive restarts under transient latency spikes.
probes-spec.yaml
startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 5
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 2
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 15
  failureThreshold: 3

3. PodDisruptionBudgets (PDBs) and Anti-Affinity

During routine node upgrades, autoscaling scale-down events, or voluntary cluster maintenance, node drains evict pods. Without a PodDisruptionBudget, an entire deployment could be evicted at once, causing customer-facing downtime.

Pairing PDBs with pod anti-affinity ensures replicas are spread across distinct availability zones and worker nodes, surviving zone outages without manual intervention.

pdb-anti-affinity.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-service
---
# Pod Anti-Affinity Spec
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values: [payment-service]
        topologyKey: "topology.kubernetes.io/zone"

4. Enforcing Hardened Pod Security Standards

Containers must never run as root in production. Running non-root containers with read-only root filesystems and dropped Linux capabilities blocks over 90% of container escape exploits before they can even start.

Ensure your container image creates and targets an unprivileged user (UID 10001 or higher), and explicitly configure `allowPrivilegeEscalation: false`.

security-context.yaml
securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  seccompProfile:
    type: RuntimeDefault

Temporary Writable Directories

When running with readOnlyRootFilesystem: true, mount an emptyDir memory volume at /tmp or /var/cache if your application framework requires write access for temporary files.

Related Topics & Tags

KubernetesDevOpsReliabilityContainer SecurityProduction
CPL

Written by CloudPlatformLabs Engineering

Platform Architecture Team

CloudPlatformLabs publishes independent, peer-reviewed cloud and DevOps architecture blueprints, hands-on tutorials, and infrastructure case studies. Our mission is to make advanced infrastructure engineering transparent, resilient, and developer-friendly.

Recommended Architecture Guides