Shadow (Dark) Deployment

Shadow (Dark) deployment mirrors live production traffic to a new version running in parallel. The shadow version processes real requests but its responses are discarded -- users always get responses from the production version. This provides zero-risk validation with real-world data.

How It Works

A service mesh or proxy (Istio, Envoy) duplicates incoming requests and sends them to both production and the shadow version. Production responses go to users; shadow responses are logged and compared. You can diff response bodies, compare latency, and validate correctness without any user impact.

When to Use

• ML model updates (compare prediction quality)
• Search ranking algorithm changes
• Financial calculation engine updates
• Performance benchmarking with real traffic patterns

When NOT to Use

• Write-heavy services (risk of duplicate writes)
• Services that trigger external side effects (email, SMS, payments)
• Budget constraints (requires 2x compute for shadow)
• Simple CRUD applications where shadow adds no value

Real-World Examples

Step-by-Step Implementation (Istio)

1. Deploy shadow version

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pricing-shadow
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pricing
      version: shadow
  template:
    metadata:
      labels:
        app: pricing
        version: shadow
    spec:
      containers:
      - name: pricing
        image: myregistry/pricing:2.0.0-shadow
        env:
        - name: SHADOW_MODE
          value: "true"    # Disable writes to external systems

2. Configure traffic mirroring with Istio

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: pricing-vs
spec:
  hosts:
  - pricing
  http:
  - route:
    - destination:
        host: pricing
        subset: production
      weight: 100
    mirror:
      host: pricing
      subset: shadow
    mirrorPercentage:
      value: 100.0   # Mirror 100% of traffic

3. Compare responses

# Diff shadow vs production responses
# Log shadow responses and compare with production output
kubectl logs -l version=shadow --tail=100

# Check shadow error rates independently
# Shadow errors do NOT affect users but indicate issues with new version

Common Pitfalls