Choose Docker Compose Over Kubernetes

I’ve deployed hundreds of services in the Docker Compose vs Kubernetes debate. The industry narrative says Kubernetes is the only “production-ready” choice, but my logging platform processing 500,000 events daily on Docker Compose proves otherwise.

The truth is uncomfortable: most teams adopt Kubernetes because it looks good on their architecture diagrams, not because their workloads actually need it.

Understand the Kubernetes Complexity Tax

Kubernetes is brilliant engineering, but it’s engineering for problems most companies don’t have. I’ve watched teams spend three months setting up cluster autoscaling, Pod Security Admission and Gatekeeper policies, and network policies before deploying their first actual workload.

Here’s what a minimal Kubernetes deployment actually requires:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
      - name: api
        image: myapp:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  selector:
    app: api-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

That’s 40 lines of YAML for a single service with three replicas. And this is just the beginning—you’ll need Ingress controllers, persistent volume claims, ConfigMaps, Secrets, and service mesh considerations.

Compare that to Docker Swarm (which uses the Docker Compose file format):

# docker-compose.yml (for Swarm mode)
services:
  api:
    image: myapp:latest
    ports:
      - "80:8080"
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
    restart: unless-stopped

Same functionality. 15 lines. Use docker stack deploy for production features like replicas and resource limits. No learning curve for your junior developers.

Deploy 500K Events/Day with Docker Compose

I run a centralized logging infrastructure that processes half a million log entries daily. It’s not a trivial workload—we’re handling real-time ingestion, parsing, indexing, and querying across multiple services.

The stack runs on three Docker hosts in Swarm mode:

• Fluentd for log aggregation (3 replicas)
• Elasticsearch cluster (3 nodes)
• Kibana for visualization
• Grafana + Prometheus for metrics
• Nginx reverse proxy

Total infrastructure cost: $180/month on DigitalOcean droplets.

I prototyped this same architecture on AWS EKS. The managed control plane alone was $73/month, plus EC2 worker nodes, load balancers, and data transfer. Total: $640/month minimum, and that’s before you factor in the CloudWatch costs for monitoring the cluster itself.

Here’s the deployment script I use:

#!/bin/bash
# deploy.sh - Zero-downtime rolling updates

set -e

STACK_NAME="logging-infra"
COMPOSE_FILE="docker-compose.prod.yml"

echo "Pulling latest images..."
docker compose -f $COMPOSE_FILE pull

echo "Deploying stack..."
docker stack deploy -c $COMPOSE_FILE --prune --resolve-image always $STACK_NAME

echo "Waiting for services to stabilize..."
sleep 10

# Health check loop
for i in {1..30}; do
  HEALTHY=$(docker service ls --filter "name=${STACK_NAME}" --format "{{.Replicas}}" | \
    grep -Ec '^([1-9][0-9]*)/\1$' || true)
  TOTAL=$(docker service ls --filter "name=${STACK_NAME}" --format "{{.ID}}" | wc -l)
  
  if [ "$HEALTHY" -eq "$TOTAL" ]; then
    echo "All services healthy!"
    exit 0
  fi
  
  echo "Waiting for services... ($i/30)"
  sleep 5
done

echo "Deployment verification timeout!"
exit 1

This script handles rolling updates, health checks, and rollback capability in 30 lines of bash. The equivalent in Kubernetes requires Helm charts, custom health check configurations, and deployment strategies that span multiple YAML files.

Evaluate When You Actually Need Kubernetes

I’m not anti-Kubernetes. I use it in production for clients who genuinely need it. Here are the actual signals that you’ve outgrown Docker Compose:

Multi-Cloud Requirements

If you’re running workloads across AWS, GCP, and Azure simultaneously, Kubernetes’ abstraction layer is invaluable. Docker Swarm doesn’t handle this well.

Massive Scale (1000+ Containers)

When you’re orchestrating thousands of containers across dozens of hosts, Kubernetes’ scheduling algorithms shine. I’ve seen it efficiently pack workloads that would require manual intervention in Swarm.

Complex Service Mesh Needs

If you need mutual TLS between every service, circuit breakers, and sophisticated traffic splitting, Istio on Kubernetes is the proven choice. Docker Compose with Traefik gets you 80% there but hits limits.

Advanced Autoscaling

Horizontal Pod Autoscaling based on custom metrics (not just CPU/memory) is where Kubernetes excels. I’ve built systems that scale based on queue depth, and Kubernetes made it straightforward.

But here’s the key: if you’re not checking multiple boxes above, you’re paying the Kubernetes tax for capabilities you don’t use.

Migrate from Kubernetes to Docker Compose

I’ve helped three companies migrate from Kubernetes back to Docker Compose after they realized they’d over-engineered their infrastructure. Here’s the approach that worked:

Step 1: Audit Your Actual Needs

List every Kubernetes feature you actually use. Most teams discover they’re only using Deployments, Services, and ConfigMaps—all available in Docker Compose.

Step 2: Start with Non-Critical Services

Move your staging environment first. Document the migration process. Measure the operational overhead reduction.

Step 3: Implement Health Checks Properly

Docker Compose health checks aren’t as robust as Kubernetes probes, but they work for 95% of cases:

services:
  api:
    image: api:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Step 4: Use Docker Swarm for Production Features

Secrets management, rolling updates, and service discovery all work in Swarm mode without the Kubernetes complexity.

Compare Docker Compose vs Kubernetes Performance

I ran identical load tests against both setups for my logging infrastructure to settle the Docker Compose vs Kubernetes performance question:

Kubernetes on EKS (t3.medium nodes)

• Log ingestion: 850 events/second
• Query latency (p95): 240ms
• Deploy time: 4.5 minutes
• Cost: $640/month

Docker Compose on DigitalOcean (similar specs)

• Log ingestion: 920 events/second
• Query latency (p95): 210ms
• Deploy time: 45 seconds
• Cost: $180/month

The performance difference comes from reduced network hops and simpler overlay networking. Kubernetes’ Service abstraction adds latency that doesn’t exist in Docker Compose’s direct container-to-container communication.

Reduce Operational Burden with Simpler Tools

The hidden cost of Kubernetes isn’t in your cloud bill—it’s in your team’s cognitive load. I’ve seen senior engineers spend entire sprints debugging CNI networking issues, RBAC permission problems, and cert-manager failures.

With Docker Compose, problems are tractable. When a container isn’t starting, you docker logs it. When networking is broken, you check your compose file’s network definitions. The debugging surface area is 10x smaller.

My incident response time dropped from an average of 45 minutes on Kubernetes to 12 minutes on Docker Compose. That’s not because I got better at debugging—it’s because there are fewer abstraction layers hiding the actual problem.

Monitor Docker Compose Infrastructure Effectively

Kubernetes advocates point to its extensive metrics and observability. But here’s what actually matters: time to insight.

My Prometheus + Grafana setup on Docker Compose gives me everything I need:

# Custom exporter for application metrics
from prometheus_client import Counter, Histogram, Gauge
from prometheus_client import start_http_server
import time

# Metrics
log_ingestion_total = Counter('logs_ingested_total', 'Total logs processed')
processing_duration = Histogram('log_processing_seconds', 'Log processing duration')
active_connections = Gauge('active_connections', 'Active log shipper connections')

def process_log(log_entry):
    start = time.time()
    
    # Your log processing logic here
    
    log_ingestion_total.inc()
    processing_duration.observe(time.time() - start)

if __name__ == '__main__':
    # Expose metrics
    start_http_server(9090)
    # Your application loop

This exports to Prometheus running in the same Docker Compose stack. No service meshes, no sidecar containers, no Kubernetes-specific exporters. Just Python code and a single port binding.

Apply the Docker Compose vs Kubernetes Decision Framework

Here’s how I evaluate whether to use Kubernetes for a new project:

Choose Docker Compose when:

• You have fewer than 50 services
• Your team is under 20 engineers
• You’re running on 10 or fewer hosts
• Deploy frequency is less than 10 times per day
• Your infrastructure budget is under $5K/month

Choose Kubernetes when:

• You’re deploying across multiple cloud providers
• You need sub-minute autoscaling response
• You’re managing 100+ microservices
• You have dedicated platform engineering teams
• Compliance requires specific isolation guarantees

Notice that “production” isn’t on either list. Production-ready is about reliability engineering, not orchestration platform choice.

Optimize Your Docker Compose Setup

If I were starting my logging platform today, I’d still choose Docker Compose. But I’d make these changes:

1. Implement proper secrets rotation using Docker Swarm secrets with a cron job that regenerates them monthly
2. Add structured logging earlier to make debugging even faster
3. Set up automated backups of Docker volumes from day one
4. Document the disaster recovery process before the first production incident

These improvements would have saved me time regardless of orchestration platform. The fundamentals of reliability engineering matter more than which YAML syntax you use.

Make the Right Orchestration Choice

Kubernetes solved Google’s problems. It might not solve yours.

In the Docker Compose vs Kubernetes debate, I’m not advocating for technological regression. I’m advocating for appropriate technology choices. Docker Compose isn’t “good enough for now”—it’s often the right long-term choice for companies processing millions of requests per day.

The next time someone tells you Kubernetes is required for production, ask them to quantify what “production-ready” means for your specific workload. You might find that simpler infrastructure is the sophisticated choice.

My logging platform proves it every day: 500,000 events processed, 99.95% uptime, $180/month. No Kubernetes required.

Need help evaluating Docker Compose vs Kubernetes for your infrastructure? Let’s discuss your specific requirements and find the right balance between simplicity and scale.