Executive Summary

Non-prod EKS clusters burn 168 hours of compute per week while developers work 40–50 hours. This article shows how to chain KEDA CronScaler → HPA → Karpenter to reach zero EC2 worker nodes during nights and weekends — dropping 65–85% of your non-prod compute bill with no application changes and no custom controllers.

Business case in four sentences: A standard 3-cluster non-prod setup (dev/staging/QA) running 5× m5a.xlarge Spot nodes costs ~$600/month in EC2 alone. This pattern cuts that to ~$216/month — saving ~$4,600/year per cluster, ~$13,800/year across three. Implementation takes one platform engineer 2–3 days. The only end-user impact is a 60–90-second cold-start delay during the morning scale-up, mitigated by a 10-minute warm-up trigger.

Article Map — Jump to Your Persona

1. Business Case

The Problem

Non-prod EKS clusters — dev, staging, QA, performance, security — run 168 hours per week. Developers use them roughly 40–50 hours per week (Mon–Fri, 7 AM–8 PM). The remaining 120+ hours per week, EC2 worker nodes sit idle: no deployments, no test runs, no traffic. You pay full price.

Typical non-prod setup:
  3 clusters × 5 nodes × m5a.xlarge

Cost at Spot pricing ($0.058/hr):
  3 × 5 × 0.058 × 8,760 hrs/year = $7,627/year  (Spot)
  3 × 5 × 0.192 × 8,760 hrs/year = $25,228/year (On-Demand)

Usable hours (Mon-Fri 7AM-8PM):
  ~2,860 hrs/year  (33% of total)
You pay for 8,760. You use 2,860. Waste = 67%.

The Fix and Its ROI

Scale worker nodes to zero during off-hours. Pay only for the EKS control plane ($72/mo flat) when nobody is working.

3-cluster saving vs Spot 24/7: ~$2,232/year 3-cluster saving vs On-Demand 24/7: ~$19,548/year

Risk Profile

Organizational Prerequisites

Before implementing, confirm:

1. CI/CD pipelines run only during business hours — or use a dedicated namespace excluded from scale-down
2. Developers aware clusters go offline at 8 PM — communicate the schedule
3. Runbook updated — on-call knows how to manually wake the cluster
4. No stateful workloads in app namespaces without EBS PVC (emptyDir data is lost on scale-down)

2. Architecture Overview

How the Three Layers Chain Together

Architecture Diagram

Three-layer chain (text summary):

KEDA CronTrigger (8PM)
  └─ patches HPA.desiredReplicas = 0
       └─ HPA scales Deployments → 0 pods
            └─ Karpenter: node empty → consolidateAfter 30s
                 └─ EC2 TerminateInstances
                      └─ $0/hr EC2 cost (off-hours)

KEDA CronTrigger (7AM)
  └─ patches HPA.desiredReplicas = 2
       └─ HPA: pods Pending → Karpenter provisions Bottlerocket node (~60s)
            └─ Pods Running → devs can work

Scale-Down Sequence (8 PM)

8:00 PM — KEDA CronTrigger fires
  └─ KEDA patches HPA.desiredReplicas = 0
       └─ HPA stabilizationWindowSeconds: 60 expires
            └─ HPA sends pod DELETE (terminationGracePeriodSeconds respected)
                 └─ Pods fully Terminated
                      └─ Karpenter consolidation loop (evaluates every ~10s)
                           └─ Node: 0 schedulable pods
                                └─ consolidateAfter: 30s timer starts
                                     └─ Karpenter cordons node
                                          └─ Karpenter drains remaining pods (DaemonSets)
                                               └─ EC2 TerminateInstances API call
                                                    └─ EBS /dev/xvdb deleted (deleteOnTermination=true)
                                                         └─ Node gone. $0/hr.

Scale-Up Sequence (7 AM)

6:50 AM — Warm-up CronTrigger fires → desiredReplicas: 1
  └─ 1 pod Pending → Karpenter provisions Bottlerocket node (~60s)
       └─ Image pulled from ECR (cached in region) (~10s)

7:00 AM — Business hours CronTrigger fires → desiredReplicas: 2
  └─ Node already warm → pod starts in <10s
       └─ Full capacity reached before developers log in

Decision Framework: Fargate vs EC2+Karpenter

Use this table to choose the right pattern for each cluster:

This article covers EC2+Karpenter. For pure Fargate (zero EC2 ever), see companion article: Zero EC2 Nodes Forever with EKS Fargate + KEDA.

3. Module Overview: SPHTech-Platform/eks/aws

Registry coordinates

module "eks" {
  source  = "SPHTech-Platform/eks/aws"
  version = "0.22.20"
}

Version 0.22.20 (published 2026–05–08, 314,942 downloads) ships with:

• Karpenter v1.12.0 (karpenter.sh/v1 API — not v1beta1)
• EKS 1.35 default
• AWS provider ≥ 6.0 (breaking change from 5.x)
• Terraform ≥ 1.5

What the module provisions automatically

When autoscaling_mode = "karpenter" and fargate_cluster = false:

IAM Layer:
  aws_iam_role.cluster       → "non-prod"
    ├── AmazonEKSClusterPolicy
    └── AmazonEKSVPCResourceController

aws_iam_role.workers       → "non-prod-workers"   ← use in EC2NodeClass role:
    ├── AmazonEKSWorkerNodePolicy
    ├── AmazonEC2ContainerRegistryReadOnly
    ├── AmazonSSMManagedInstanceCore             ← SSM Session Manager (no SSH needed)
    └── AmazonEKS_CNI_Policy

Encryption Layer:
  module.kms_secrets         → KMS key: Kubernetes secrets envelope encryption
  module.kms_ebs             → KMS key: EBS volume encryption
    └── output: ebs_kms_key_arn

Cluster:
  aws_eks_cluster            → EKS 1.35 | auth=API | IMDSv2 enforced
  aws_eks_access_entry       → access entries (API auth mode)

Karpenter (submodule: modules/karpenter):
  helm_release.karpenter_crd → CRDs v1.12.0
  helm_release.karpenter     → Controller v1.12.0
  aws_iam_role.controller    → IRSA for karpenter:karpenter SA
  kubectl_manifest           → EC2NodeClass (from karpenter_nodeclasses)
  kubectl_manifest           → NodePool (from karpenter_nodepools)
Addons (most_recent = true):
  kube-proxy | vpc-cni | aws-ebs-csi-driver | eks-pod-identity-agent | CoreDNS

Default Bottlerocket node group:
  aws_eks_node_group         → count = !var.fargate_cluster ? 1 : 0
  AMI: BOTTLEROCKET_x86_64

Key module defaults to know

# These are MODULE DEFAULTS — shown explicitly to highlight what we override

kubernetes_version     = "1.35"
autoscaling_mode       = "karpenter"
karpenter_chart_version = "1.12.0"

# karpenter_nodeclasses default = []  ← EMPTY! Must configure or no nodes ever
karpenter_nodeclasses  = []

# karpenter_nodepools default (single "default" pool):
karpenter_nodepools = [{
  consolidation_policy     = "WhenEmptyOrUnderutilized"
  consolidate_after        = "10m"    # ← We override to "30s" for zero-node
  expire_after             = "168h"
  termination_grace_period = "5h"
  capacity_type            = ["on-demand"]  # ← We add "spot"
  disruption_budgets       = [{ nodes = "10%" }]  # ← We set "100%"
  # Bottlerocket updater label: "bottlerocket.aws/updater-interface-version" = "2.0.0"
  # ← REQUIRED in every nodepool — omitting breaks Bottlerocket auto-updater
}]

4. Implementation

Step 1: Terraform — Provision the Cluster

# terraform/main.tf
module "eks" {
  source             = "SPHTech-Platform/eks/aws"
  version            = "0.22.20"

  name               = "non-prod"
  kubernetes_version = "1.35"
  vpc_id             = var.vpc_id
  subnet_ids         = var.subnet_ids   # private subnets
  
  authentication_mode     = "API"
  endpoint_private_access = true
  endpoint_public_access  = true
  # force_imdsv2 = true  ← module default
  # force_irsa   = true  ← module default

  fargate_cluster  = false
  autoscaling_mode = "karpenter"

  # EC2NodeClass — defines the Bottlerocket template for Karpenter-launched nodes
  # Module karpenter_nodeclasses default is [] — MUST configure this
  karpenter_nodeclasses = [
    {
      nodeclass_name = "non-prod".

      # Subnet/SG discovery: module tags these with karpenter.sh/discovery automatically
      karpenter_subnet_selector_maps = [
        { tags = { "karpenter.sh/discovery" = "non-prod" } }
      ]
      karpenter_security_group_selector_maps = [
        { tags = { "karpenter.sh/discovery" = "non-prod" } }
      ]
   
      # Worker IAM role — module creates this as "${var.name}-workers"
      # Use terraform output: worker_iam_role_name
      karpenter_node_role = "non-prod-workers"
   
      # AMI: empty = Bottlerocket @latest (module default)
      karpenter_ami_selector_maps = []
      
      # IMDSv2 — httpPutResponseHopLimit=2 allows pods to call IMDS
      karpenter_node_metadata_options = {
        httpEndpoint            = "enabled"
        httpTokens              = "required"
        httpPutResponseHopLimit = 2
      }
      
      karpenter_node_tags_map = {
        Environment              = "non-prod"
        ManagedBy                = "karpenter"
        "karpenter.sh/discovery" = "non-prod"
      }
      
      # Bottlerocket requires TWO EBS volumes (unlike AL2/AL2023 which uses one)
      # /dev/xvda → OS root (read-only, dm-verity) — keep small
      # /dev/xvdb → container image layers + emptyDir — size this for your workload
      karpenter_block_device_mapping = [
        {
          deviceName = "/dev/xvda"
          ebs = { encrypted = true, volumeSize = "10Gi", volumeType = "gp3", deleteOnTermination = true 
}
        },
        {
          deviceName = "/dev/xvdb"
          ebs = { encrypted = true, volumeSize = "50Gi", volumeType = "gp3", deleteOnTermination = true 
}
        }
      ]
      karpenter_node_kubelet   = {}
      karpenter_node_user_data = ""
    }
  ]

  # NodePool — zero-node configuration
  karpenter_nodepools = [
    {
      nodepool_name  = "non-prod-general"
      nodeclass_name = "non-prod"

      karpenter_nodepool_node_labels = {
        # REQUIRED: Bottlerocket auto-updater label — omitting breaks BR updater silently
        "bottlerocket.aws/updater-interface-version" = "2.0.0"
        "environment"    = "non-prod"
        "managed-by"     = "karpenter"
      }
      
      karpenter_nodepool_annotations    = {}
      karpenter_nodepool_node_taints    = []
      karpenter_nodepool_startup_taints = []

      karpenter_requirements = [
        { key = "karpenter.sh/capacity-type",            operator = "In", values = ["spot", "on-demand"] 
},
        { key = "karpenter.k8s.aws/instance-category",   operator = "In", values = ["m", "t"] },
        { key = "karpenter.k8s.aws/instance-cpu",        operator = "In", values = ["2", "4"] },
        { key = "karpenter.k8s.aws/instance-memory",     operator = "Gt", values = ["2048"] },
        { key = "karpenter.k8s.aws/instance-generation", operator = "Gt", values = ["2"] },
        { key = "kubernetes.io/arch",                    operator = "In", values = ["amd64"] },
        { key = "kubernetes.io/os",                      operator = "In", values = ["linux"] }
      ]

      karpenter_nodepool_disruption = {
        consolidation_policy     = "WhenEmptyOrUnderutilized"
        consolidate_after        = "30s"     # Override module default 10m → 30s
        expire_after             = "168h"    # Weekly node rotation (security patches)
        termination_grace_period = "5h"      # Drain window for long batch jobs
      }

      # Override module default 10% → 100%: allow full simultaneous disruption
      karpenter_nodepool_disruption_budgets = [{ nodes = "100%" }]
      karpenter_nodepool_weight = 10
    }
  ]

  enabled_log_types                      = ["api", "audit", "authenticator"]
  cloudwatch_log_group_retention_in_days = 7
  addons                                 = {}
}

# KEDA — not part of the SPHTech module, deployed separately
resource "helm_release" "keda" {
  name             = "keda"
  repository       = "https://kedacore.github.io/charts"
  chart            = "keda"
  namespace        = "keda"
  create_namespace = true
  version          = "2.14.0"

  set { name = "resources.operator.requests.cpu",      value = "50m"   }
  set { name = "resources.operator.requests.memory",   value = "100Mi" }
  set { name = "resources.metricServer.requests.cpu",  value = "50m"   }
  set { name = "resources.metricServer.requests.memory", value = "100Mi" 

  depends_on = [module.eks]
}
# Apply
terraform init && terraform plan -out=tfplan
terraform apply tfplan
# Configure kubectl
aws eks update-kubeconfig --name non-prod --region ap-southeast-1
# Verify NodePool deployed with correct settings
kubectl get nodepool non-prod-general \
  -o jsonpath='{.spec.disruption}' | jq .
# Expected: consolidationPolicy=WhenEmptyOrUnderutilized, consolidateAfter=30s

Step 2: KEDA ScaledObjects

Each deployment that should scale to zero needs its own ScaledObject. KEDA creates and owns the HPA — do not create a separate HPA for the same deployment.

# manifests/scaledobject-api.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-service-scaler
  namespace: non-prod
  annotations:
    # Restore original replica count when ScaledObject is deleted
    autoscaling.keda.sh/restore-to-original-replica-count: "true"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  
  pollingInterval: 30     # Seconds between trigger evaluations
  cooldownPeriod:  60     # Seconds to wait before allowing scale-down again
  minReplicaCount: 0      # 0 overrides HPA's built-in minimum of 1
  maxReplicaCount: 10

  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60   # Wait 60s with zero desired before acting
          policies:
            - type: Percent
              value: 100
              periodSeconds: 60            # All pods gone in one shot
        scaleUp:
          stabilizationWindowSeconds: 0    # No delay on scale-up
          policies:
            - type: Percent
              value: 100
              periodSeconds: 30
  triggers:
    # 6:50 AM warm-up: 1 pod starts early to warm Bottlerocket image cache
    # Bottlerocket cold start: EC2 boot (~45s) + image pull (~15s) = ~60s total
    # Schedule 10 min early so full capacity is ready exactly at 7 AM
    - type: cron
      metadata:
        timezone: "Asia/Singapore"
        start: "50 6 * * 1-5"
        end:   "0 7 * * 1-5"
        desiredReplicas: "1"
    
    # Business hours: Mon-Fri 7AM-8PM
    - type: cron
      metadata:
        timezone: "Asia/Singapore"
        start: "0 7 * * 1-5"
        end:   "0 20 * * 1-5"
        desiredReplicas: "2"
    
    # Weeknight scale-down: 8PM to next morning
    - type: cron
      metadata:
        timezone: "Asia/Singapore"
        start: "0 20 * * 1-5"
        end:   "0 7 * * 2-6"
        desiredReplicas: "0"

    # Weekend: Friday 8PM through Monday 7AM
    - type: cron
      metadata:
        timezone: "Asia/Singapore"
        start: "0 20 * * 5"
        end:   "0 7 * * 1"
        desiredReplicas: "0"

KEDA trigger precedence: when multiple cron triggers overlap, KEDA picks the maximum desiredReplicas across all active triggers. Keep your scale-down and scale-up windows non-overlapping to avoid conflict.

kubectl apply -f manifests/scaledobject-api.yaml
# Verify KEDA created the HPA and owns it
kubectl get hpa -n non-prod
kubectl describe scaledobject api-service-scaler -n non-prod \
  | grep -A5 "Condition\|Active"

Step 3: Exclude CI/CD Namespaces

Pipelines that run after 8 PM (nightly integration tests, overnight builds) must be excluded from scale-down. Do this by simply not creating a ScaledObject for those deployments, or by using a dedicated namespace with no KEDA resources:

# ci-system namespace: NO ScaledObjects applied here
# Jobs run independently of business hours
---
apiVersion: v1
kind: Namespace
metadata:
  name: ci-system
  labels:
    keda-excluded: "true"
    environment: non-prod
# Optionally enforce via OPA policy: deny ScaledObjects in ci-system namespace
# See your platform OPA policies for implementation

Step 4: Manual Override for Developers

Developers sometimes need the cluster after hours (late deployment, incident). Provide a simple override ScaledObject they can temporarily modify:

# manifests/override-scaledobject.yaml
# Developer use: kubectl patch scaledobject api-service-scaler -n non-prod
#   --type=merge -p '{"spec":{"minReplicaCount":1}}'
# Reverts automatically at next cron trigger evaluation (pollingInterval: 30s)

# Platform-provided override script (put in team runbook)
# Wake cluster manually for 2 hours:
kubectl scale deployment api-service worker frontend \
  --replicas=1 -n non-prod

# Check when next KEDA cron fires (will override your manual scale)
kubectl describe scaledobject api-service-scaler -n non-prod \
  | grep "Last Active"

5. GitOps Integration

ArgoCD Sync Wave Ordering

ScaledObjects must be applied after the target Deployment exists. Use ArgoCD sync waves to enforce ordering:

# Wave 0: Namespaces
# Wave 1: CRDs (KEDA CRDs, Karpenter CRDs)
# Wave 2: System components (KEDA operator, Karpenter)
# Wave 3: Application Deployments
# Wave 4: ScaledObjects (must reference existing Deployments)

# manifests/scaledobject-api.yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "4"   # After Deployment in wave 3

ArgoCD Application for KEDA Resources

# argocd-app-keda-scalers.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: non-prod-keda-scalers
  namespace: argocd
spec:
  project: non-prod
  source:
    repoURL: https://github.com/your-org/platform-gitops
    targetRevision: main
    path: clusters/non-prod/keda
  destination:
    server: https://kubernetes.default.svc
    namespace: non-prod
  syncPolicy:
    automated:
      prune: true     # Remove ScaledObjects no longer in git
      selfHeal: true  # Revert manual kubectl changes
    syncOptions:
      - CreateNamespace=true
      - RespectIgnoreDifferences=true  # Ignore KEDA status fields
  ignoreDifferences:
    - group: keda.sh
      kind: ScaledObject
      jsonPointers:
        - /status   # KEDA writes status — ignore in drift detection

Flux Alternative

# flux/kustomization-keda-scalers.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: keda-scalers
  namespace: flux-system
spec:
  interval: 5m
  path: ./clusters/non-prod/keda
  prune: true
  sourceRef:
    kind: GitRepository
    name: platform-gitops
  dependsOn:
    - name: keda-operator        # KEDA must be running first
    - name: app-deployments      # Deployments must exist first
  wait: true
  timeout: 2m

6. Observability & Alerting

How to Confirm Zero-Node at 8:30 PM

# Quick check: should see zero non-system pods after 8PM
kubectl get pods -A \
  --field-selector 'metadata.namespace!=kube-system,metadata.namespace!=keda' \
  | grep -v "Completed\|No resources"
# Check actual node count
kubectl get nodes | wc -l
# Expected after scale-down: 0 (or 1 if system NodePool exists)
# Karpenter consolidation events
kubectl get events -n kube-system \
  --field-selector reason=Disrupted \
  --sort-by='.lastTimestamp' | tail -10

Karpenter Metrics in CloudWatch

# Karpenter emits metrics to CloudWatch when enabled in module
# module/karpenter/karpenter.tf: serviceMonitor enabled by default
# Access via Container Insights or direct CloudWatch Logs Insights:

# CloudWatch Logs Insights query — detect zero-node at wrong time
fields @timestamp, @message
| filter @logStream like /karpenter/
| filter @message like /disrupting|terminated|deleted node/
| sort @timestamp desc
| limit 50
# Terraform: CloudWatch alarm — nodes still running at 8:30 PM
resource "aws_cloudwatch_metric_alarm" "nodes_after_hours" {
  alarm_name          = "non-prod-nodes-after-hours"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "cluster_node_count"
  namespace           = "ContainerInsights"
  period              = 300
  statistic           = "Average"
  threshold           = 0
  alarm_description   = "EC2 nodes running outside business hours — check Karpenter consolidation"
  
  dimensions = {
    ClusterName = "non-prod"
  }
  # Trigger between 8:30 PM and 6:45 AM Mon-Fri
  # Use EventBridge scheduled rule to enable/disable this alarm
}

KEDA ScaledObject Health Check

# Check all ScaledObjects are READY and ACTIVE correctly
kubectl get scaledobjects -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,ACTIVE:.status.conditions[?(@.type=="Active")].status,REPLICAS:.status.currentReplicas'

# Expected at 9PM:
# NAMESPACE   NAME                  READY   ACTIVE   REPLICAS
# non-prod    api-service-scaler    True    False    0
# non-prod    worker-scaler         True    False    0
# non-prod    frontend-scaler       True    False    0
# Expected at 8AM:
# non-prod    api-service-scaler    True    True     2

Prometheus Alert Rules (if using kube-prometheus-stack)

# prometheus/alerts/keda-zero-node.yaml
groups:
  - name: keda-zero-node
    rules:
      - alert: ClusterNotScaledDownAfterHours
        expr: |
          (kube_node_info{node!~".*fargate.*"} * on(node) group_left()
           (hour() >= 20 or hour() < 7)) > 0
        for: 30m
        labels:
          severity: warning
          cluster: non-prod
        annotations:
          summary: "EC2 nodes still running after business hours"
          description: "{{ $value }} EC2 nodes running at {{ $labels.node }}. Expected 0 after 8PM."
      - alert: ClusterNotScaledUpMorning
        expr: |
          (kube_node_info{node!~".*fargate.*"} * on(node) group_left()
           (hour() >= 7 and hour() < 9) and (day_of_week() >= 1 and day_of_week() <= 5)) == 0
        for: 15m
        labels:
          severity: critical
          cluster: non-prod
        annotations:
          summary: "Non-prod cluster has no EC2 nodes during business hours"
          description: "Zero EC2 nodes at 7-9AM on a weekday — KEDA scale-up may have failed."

Cost Verification

# Tag all Karpenter EC2 instances via EC2NodeClass tags
# Then query Cost Explorer:
aws ce get-cost-and-usage \
  --time-period Start="$(date -v-7d +%Y-%m-%d)" End="$(date +%Y-%m-%d)" \
  --granularity DAILY \
  --filter '{
    "And": [
      {"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}},
      {"Tags":{"Key":"Cluster","Values":["non-prod"]}}
    ]
  }' \
  --metrics UnblendedCost \
  --output table

# Expected pattern: weekend days should show near-zero EC2 cost
# Weekday nights (after 8PM before midnight) lower than daytime

7. Failure Modes & Mitigations

Failure 1: KEDA down — morning scale-up never fires

What happens: KEDA operator pod crashes or is evicted. CronTrigger at 7 AM never fires. HPA stays at 0 replicas. Developers get no cluster at 7 AM.

Blast radius: Full cluster unavailable until KEDA recovers (auto-restarts in <2min usually).

Mitigations:

# 1. Run KEDA on a tainted system node pool (never on zero-scale pool)
# System nodes use WhenEmpty policy + no budget disruption during biz hours

# 2. KEDA pod anti-affinity across nodes
# keda-operator and keda-metrics-apiserver on different nodes

# 3. HPA fallback: set HPA's own minReplicas=1 independently
# If KEDA fails, HPA's own policy kicks in and maintains 1 replica
# Trade-off: cluster never goes fully to 0 (use only for critical namespaces)

# 4. Runbook: manual wake command
kubectl scale deployment api-service worker frontend --replicas=1 -n non-prod

Failure 2: Karpenter consolidation stuck — nodes don't terminate

What happens: Pods hit 0 but nodes stay. EC2 cost continues.

Root causes and fixes:

# Check 1: PodDisruptionBudget blocking drain
kubectl get pdb -n non-prod
# PDB with minAvailable=1 on 1-replica deployment blocks drain forever
# Fix: ensure KEDA scales to 0 BEFORE Karpenter tries to drain
# (KEDA/HPA acts first, Karpenter sees empty node after)

# Check 2: karpenter.sh/do-not-disrupt annotation on node
kubectl get nodes -o json \
  | jq '.items[].metadata.annotations["karpenter.sh/do-not-disrupt"]'
# If "true" — something set this. Remove:
kubectl annotate node <node> karpenter.sh/do-not-disrupt-
    
# Check 3: Karpenter CRD version mismatch
# v1beta1 NodePool with v1 controller = silent failure
kubectl get nodepool non-prod-general \
  -o jsonpath='{.apiVersion}'
# Must be: karpenter.sh/v1

Failure 3: 7 AM scale-up too slow — devs frustrated

What happens: Nodes take 60–90 seconds to provision. Devs experience 502s.

Solution — warm-up trigger (already in ScaledObject above):

# 10-minute warm-up trigger: 1 pod at 6:50 AM
# Forces Karpenter to provision node before full scale-up
# By 7:00 AM node is warm, remaining pods start in <10s
- type: cron
  metadata:
    timezone: "Asia/Singapore"
    start: "50 6 * * 1-5"   # 6:50 AM
    end:   "0 7 * * 1-5"
    desiredReplicas: "1"

Failure 4: Overnight CI/CD job killed at 8 PM

What happens: A long-running integration test or nightly build is running. KEDA ScaledObject fires at 8 PM and kills it.

Mitigation:

# Option A: No ScaledObject in CI namespace (simplest)
# CI jobs run in ci-system namespace — no ScaledObjects applied there

# Option B: PodDisruptionBudget protecting CI jobs
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ci-job-pdb
  namespace: ci-system
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: integration-test
# Combined with terminationGracePeriod: 5h — gives CI jobs up to 5h to finish

# Option C: KEDA ScaledObject with later scale-down for CI
# ci-system gets different cron: scale down at 2AM instead of 8PM

Failure 5: Spot interruption during business hours

What happens: AWS reclaims a Spot instance. Running pods evicted.

Karpenter handles this automatically (module configures SQS + EventBridge):

# Verify Spot interruption handler is configured
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter \
  --since=1h | grep -i "spot\|interruption"
# Expected: "spot interruption warning received, cordoning node"
# Karpenter gets 2-min warning, drains node, provisions replacement

8. Troubleshooting

Nodes not terminating after pods = 0

# DaemonSet pods: Karpenter ignores them for WhenEmptyOrUnderutilized
# but they count for WhenEmpty. Verify policy:
kubectl get nodepool non-prod-general \
  -o jsonpath='{.spec.disruption.consolidationPolicy}'

# Check Karpenter disruption events
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter \
  --since=5m | grep -E "disruption|consolidat|blocker"

# Check for stuck PDB
kubectl get pdb -A
kubectl describe pdb <name> -n non-prod

KEDA not scaling to 0 (pods stay at 1)

# Verify KEDA owns the HPA (not created separately)
kubectl get hpa -n non-prod -o yaml | grep ownerReferences -A5

# Verify minReplicaCount is 0 in ScaledObject
kubectl get scaledobject api-service-scaler -n non-prod \
  -o jsonpath='{.spec.minReplicaCount}'

# Check KEDA operator logs for errors
kubectl logs -n keda -l app=keda-operator --tail=50 | grep -i error

Karpenter provisions wrong/oversized instances

# See what Karpenter evaluated
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter \
  | grep "selected instance" | tail -10

# Check NodePool requirements actually match available Spot capacity
aws ec2 describe-spot-price-history \
  --instance-types m5a.large m5.large m5a.xlarge m5.xlarge \
  --product-descriptions "Linux/UNIX" \
  --max-items 20 \
  --query 'SpotPriceHistory[*].{Type:InstanceType,Price:SpotPrice,AZ:AvailabilityZone}' \
  --output table
# If no Spot capacity: add more instance types to karpenter_requirements

Bottlerocket /dev/xvdb full — container pulls failing

# Symptom: "no space left on device" even though cluster looks fine
# /dev/xvdb (container storage) is separate from /dev/xvda (OS)

kubectl debug node/<node> -it --image=busybox -- df -h | grep xvdb

# Short-term fix: drain and delete node (Karpenter auto-replaces)
kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets \
  --delete-emptydir-data && kubectl delete node <node>

# Long-term: increase /dev/xvdb volumeSize in karpenter_nodeclasses

9. Recommendation Guide for Technical Advisors

When to Recommend This Pattern

Strong yes when client has:

• ≥ 2 non-prod EKS clusters with predictable usage schedules
• Developers working standard business hours (consistent timezone)
• No DaemonSet-dependent observability tools (or willing to migrate to alternatives)
• Platform team with Terraform and Helm experience (2–3 days implementation)
• Monthly EC2 bill > $300/month for non-prod

Qualified yes when:

• CI/CD pipelines run after hours (add namespace exclusion — extra half-day work)
• Mixed timezones (use the timezone with the latest end-of-day as scale-down trigger)
• Stateful workloads in cluster (ensure EBS PVCs used, not emptyDir)

Recommend against when:

• Production workloads in same cluster as non-prod
• Cluster requires DaemonSets for security (Falco) or compliance monitoring 24/7
• < $150/month EC2 bill — ROI doesn't justify implementation complexity
• Team lacks Terraform skills — operational risk outweighs savings

Vendor Lock-in Assessment

For multi-cloud clients: KEDA is portable; replace Karpenter with cluster-autoscaler for GKE/AKS (equivalent zero-node via node pool autoscaling to zero).

Implementation Effort Estimate

Executive Talking Points

1. Cost: "$4,600/year per cluster saved, 2-day implementation — 1-month payback"
2. Risk: "Morning cold start 60–90s, mitigated with warm-up trigger. Not an outage."
3. Operations: "No custom controllers. KEDA is CNCF. Karpenter is EKS-official. Module handles IAM, KMS, Bottlerocket."
4. Rollback: "Delete ScaledObjects → cluster scales up normally. 5-minute rollback."

Conclusion

Zero-node non-prod EKS is production-ready when built correctly:

• KEDA CronScaler handles time-based intent — no custom controller needed
• Karpenter (consolidateAfter: 30s + WhenEmptyOrUnderutilized) reacts to empty nodes in 30 seconds
• SPHTech-Platform/eks/aws v0.22.20 handles Karpenter deployment, IAM, dual KMS, Bottlerocket AMI, and IMDSv2 — reducing platform configuration to a well-documented Terraform module call
• Warm-up trigger at 6:50 AM absorbs Bottlerocket cold-start latency so developers never notice the zero-node cycle

One platform engineer, two days, $4,600/year/cluster saved.