Building a Production-Ready Kubernetes Health Check Script

Sun, 20 Oct 2024 00:00:00 +0000

“Can you write a script to monitor our K8s cluster health and send email alerts if something breaks?”

This was my task as a DevOps intern. Simple request, but I learned a lot about production monitoring patterns.

The Requirements

Check cluster health - nodes, pods, services
Monitor resource usage - CPU, memory across nodes
Score the health - 0-100 points system
Email alerts - When score drops below threshold
No external dependencies - Just bash and kubectl
Production-ready - Run via cron every 30 minutes

The Architecture

┌─────────────┐
│ Cron Job │
│ (*/30 * *) │
└──────┬──────┘
 │
 v
┌─────────────────────────────────────────┐
│ Health Check Script (latest.sh) │
│ │
│ 1. Check K8s Cluster Status │
│ 2. Check Resource Utilization │
│ 3. Check Platform Services │
│ 4. Check Network/DNS │
│ 5. Calculate Health Score (0-100) │
│ 6. Send Email if Score < 70 │
└─────────────────────────────────────────┘
 │
 v
┌─────────────┐
│ msmtp │ → Email to DevOps team
└─────────────┘

The Implementation

Part 1: Cluster Status Check

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


#!/bin/bash

check_cluster_status() {
 echo "=== KUBERNETES CLUSTER STATUS ==="
 
 # Check if all nodes are Ready
 NOT_READY_NODES=$(kubectl get nodes --no-headers | grep -v "Ready" | wc -l)
 
 if [ "$NOT_READY_NODES" -gt 0 ]; then
 echo "CRITICAL: $NOT_READY_NODES nodes not ready"
 HEALTH_SCORE=$((HEALTH_SCORE - 40))
 else
 echo "OK: All cluster nodes ready"
 fi
 
 # Check for failed pods
 FAILED_PODS=$(kubectl get pods -A --no-headers | \
 grep -E "Error|CrashLoopBackOff|ImagePullBackOff|OOMKilled" | wc -l)
 
 if [ "$FAILED_PODS" -gt 0 ]; then
 echo "CRITICAL: $FAILED_PODS pods in failed state"
 HEALTH_SCORE=$((HEALTH_SCORE - 30))
 
 # List the failed pods
 kubectl get pods -A | grep -E "Error|CrashLoop|ImagePull|OOMKilled"
 else
 echo "OK: No pods in failed state"
 fi
}

Part 2: Resource Utilization

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


check_resource_utilization() {
 echo "=== CLUSTER RESOURCE UTILIZATION ==="
 
 # Check memory usage per node
 kubectl top nodes --no-headers | while read node cpu mem; do
 # Extract percentage from memory (e.g., "65%" → 65)
 mem_percent=$(echo $mem | sed 's/%//')
 
 if [ "$mem_percent" -gt 85 ]; then
 echo "CRITICAL: Node $node memory at $mem_percent%"
 HEALTH_SCORE=$((HEALTH_SCORE - 25))
 elif [ "$mem_percent" -gt 70 ]; then
 echo "WARNING: Node $node memory at $mem_percent%"
 HEALTH_SCORE=$((HEALTH_SCORE - 10))
 fi
 done
}

Part 3: Platform Services

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


check_platform_services() {
 echo "=== PLATFORM SERVICES STATUS ==="
 
 # Check critical services
 SERVICES=("containerd" "kubelet")
 
 for service in "${SERVICES[@]}"; do
 if systemctl is-active --quiet $service; then
 echo "OK: $service service running"
 else
 echo "CRITICAL: $service service not running"
 HEALTH_SCORE=$((HEALTH_SCORE - 20))
 fi
 done
}

Part 4: Health Scoring System

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Start with perfect score
HEALTH_SCORE=100

# Run all checks (they deduct points)
check_cluster_status
check_resource_utilization 
check_platform_services
check_network_connectivity
check_infrastructure_status

# Determine overall health
if [ "$HEALTH_SCORE" -ge 90 ]; then
 HEALTH_STATUS="HEALTHY"
 STATUS_COLOR="GREEN"
elif [ "$HEALTH_SCORE" -ge 70 ]; then
 HEALTH_STATUS="WARNING" 
 STATUS_COLOR="YELLOW"
else
 HEALTH_STATUS="CRITICAL"
 STATUS_COLOR="RED"
fi

echo "PLATFORM STATUS: $HEALTH_STATUS ($HEALTH_SCORE/100)"

Part 5: Email Alerts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


send_alert() {
 local subject=$1
 local body=$2
 
 # Only send if score is below threshold
 if [ "$HEALTH_SCORE" -lt 70 ]; then
 echo "$body" | msmtp \
 --from="no-reply.nsuno@solytics-partners.com" \
 --subject="$subject" \
 devops-team@solytics-partners.com
 fi
}

# Generate alert
ALERT_SUBJECT="[${HEALTH_STATUS}] Platform Health Alert (Score: ${HEALTH_SCORE}/100)"
ALERT_BODY=$(cat <<EOF
CLUSTER & PLATFORM HEALTH CHECK - $(date +%Y-%m-%d_%H-%M-%S)
Cluster Node: $(hostname)
Scope: ${NAMESPACE}
===============================================

$(cat /tmp/health_check_results.txt)

PLATFORM STATUS: ${HEALTH_STATUS} (${HEALTH_SCORE}/100)
EOF
)

send_alert "$ALERT_SUBJECT" "$ALERT_BODY"

The Full Script

The complete script checks:

Monitoring on Munish Thakur | DevOps Engineer