Kubernetes on Munish Thakur | DevOps Engineer

Building My Portfolio: From HTML to Hugo

Sun, 01 Dec 2024 00:00:00 +0000

When I decided to build my portfolio, I had a choice: keep it simple with plain HTML, or use it as an opportunity to demonstrate my DevOps skills. I chose the latter.

Why Hugo?

Hugo is a static site generator written in Go. Here’s why it made sense for a DevOps engineer:

Speed: Hugo builds are blazing fast (milliseconds, not minutes)
No runtime dependencies: The output is just HTML, CSS, and JS
DevOps-friendly: The Kubernetes documentation itself uses Hugo
Markdown-based: Write content in Markdown, focus on writing

The Architecture

┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Markdown Files │────▶│ Hugo Build │────▶│ Static Files │
└─────────────────┘ └──────────────┘ └─────────────────┘
 │
 ┌───────────────────────┼───────────────────────┐
 │ │ │
 ▼ ▼ ▼
 ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
 │ GitHub Pages │ │ Docker/Nginx │ │ Kubernetes │
 │ (Production) │ │ (Local Dev) │ │ (Proof of Work)│
 └─────────────────┘ └─────────────────┘ └─────────────────┘

The DevOps Artifacts

This repository isn’t just a portfolio—it’s a showcase of DevOps practices:

Kubernetes Cost Optimization: How I Saved 81% CPU and 68% Memory

Fri, 15 Nov 2024 00:00:00 +0000

During my time at Solytics Partners, I was asked to analyze our Kubernetes cluster resource usage. The CFO wanted to know: “Are we wasting money on over-provisioned infrastructure?”

Spoiler: We were. By a lot.

The Investigation

I was given access to our pre-production cluster and asked to conduct a comprehensive resource utilization analysis. The goal: identify optimization opportunities without risking service stability.

The Methodology

I wrote a script to collect resource metrics across all namespaces:

Debugging OOMKilled Pods in Production: A Step-by-Step Guide

Sun, 10 Nov 2024 00:00:00 +0000

It’s Monday morning, and Slack is blowing up. The backend service is down. Again.

I SSH into the node and run kubectl get pods:

1
2


NAME READY STATUS RESTARTS AGE
backend-django-7d8f9c-abcd 0/1 OOMKilled 5 10m

If you’ve worked with Kubernetes, you’ve seen this. OOMKilled means your pod tried to use more memory than allowed and got terminated by the kernel.

Here’s how I debug it.

Azure Authentication in Kubernetes: A Troubleshooting Guide

Mon, 28 Oct 2024 00:00:00 +0000

Django migrations were failing in our Kubernetes cluster. The error looked innocent enough:

1
2


azure.identity._exceptions.ClientAuthenticationError: 
DefaultAzureCredential failed to retrieve a token from the included credentials.

But this single error was blocking all database migrations, meaning no deployments.

The Problem

Our Django app used Azure Blob Storage for file uploads. During migration, Django initializes all storage backends - including Azure. But the pod had no credentials.

Building a Production-Ready Kubernetes Health Check Script

Sun, 20 Oct 2024 00:00:00 +0000

“Can you write a script to monitor our K8s cluster health and send email alerts if something breaks?”

This was my task as a DevOps intern. Simple request, but I learned a lot about production monitoring patterns.

The Requirements

Check cluster health - nodes, pods, services
Monitor resource usage - CPU, memory across nodes
Score the health - 0-100 points system
Email alerts - When score drops below threshold
No external dependencies - Just bash and kubectl
Production-ready - Run via cron every 30 minutes

The Architecture

┌─────────────┐
│ Cron Job │
│ (*/30 * *) │
└──────┬──────┘
 │
 v
┌─────────────────────────────────────────┐
│ Health Check Script (latest.sh) │
│ │
│ 1. Check K8s Cluster Status │
│ 2. Check Resource Utilization │
│ 3. Check Platform Services │
│ 4. Check Network/DNS │
│ 5. Calculate Health Score (0-100) │
│ 6. Send Email if Score < 70 │
└─────────────────────────────────────────┘
 │
 v
┌─────────────┐
│ msmtp │ → Email to DevOps team
└─────────────┘

The Implementation

Part 1: Cluster Status Check

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


#!/bin/bash

check_cluster_status() {
 echo "=== KUBERNETES CLUSTER STATUS ==="
 
 # Check if all nodes are Ready
 NOT_READY_NODES=$(kubectl get nodes --no-headers | grep -v "Ready" | wc -l)
 
 if [ "$NOT_READY_NODES" -gt 0 ]; then
 echo "CRITICAL: $NOT_READY_NODES nodes not ready"
 HEALTH_SCORE=$((HEALTH_SCORE - 40))
 else
 echo "OK: All cluster nodes ready"
 fi
 
 # Check for failed pods
 FAILED_PODS=$(kubectl get pods -A --no-headers | \
 grep -E "Error|CrashLoopBackOff|ImagePullBackOff|OOMKilled" | wc -l)
 
 if [ "$FAILED_PODS" -gt 0 ]; then
 echo "CRITICAL: $FAILED_PODS pods in failed state"
 HEALTH_SCORE=$((HEALTH_SCORE - 30))
 
 # List the failed pods
 kubectl get pods -A | grep -E "Error|CrashLoop|ImagePull|OOMKilled"
 else
 echo "OK: No pods in failed state"
 fi
}

Part 2: Resource Utilization

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


check_resource_utilization() {
 echo "=== CLUSTER RESOURCE UTILIZATION ==="
 
 # Check memory usage per node
 kubectl top nodes --no-headers | while read node cpu mem; do
 # Extract percentage from memory (e.g., "65%" → 65)
 mem_percent=$(echo $mem | sed 's/%//')
 
 if [ "$mem_percent" -gt 85 ]; then
 echo "CRITICAL: Node $node memory at $mem_percent%"
 HEALTH_SCORE=$((HEALTH_SCORE - 25))
 elif [ "$mem_percent" -gt 70 ]; then
 echo "WARNING: Node $node memory at $mem_percent%"
 HEALTH_SCORE=$((HEALTH_SCORE - 10))
 fi
 done
}

Part 3: Platform Services

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


check_platform_services() {
 echo "=== PLATFORM SERVICES STATUS ==="
 
 # Check critical services
 SERVICES=("containerd" "kubelet")
 
 for service in "${SERVICES[@]}"; do
 if systemctl is-active --quiet $service; then
 echo "OK: $service service running"
 else
 echo "CRITICAL: $service service not running"
 HEALTH_SCORE=$((HEALTH_SCORE - 20))
 fi
 done
}

Part 4: Health Scoring System

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Start with perfect score
HEALTH_SCORE=100

# Run all checks (they deduct points)
check_cluster_status
check_resource_utilization 
check_platform_services
check_network_connectivity
check_infrastructure_status

# Determine overall health
if [ "$HEALTH_SCORE" -ge 90 ]; then
 HEALTH_STATUS="HEALTHY"
 STATUS_COLOR="GREEN"
elif [ "$HEALTH_SCORE" -ge 70 ]; then
 HEALTH_STATUS="WARNING" 
 STATUS_COLOR="YELLOW"
else
 HEALTH_STATUS="CRITICAL"
 STATUS_COLOR="RED"
fi

echo "PLATFORM STATUS: $HEALTH_STATUS ($HEALTH_SCORE/100)"

Part 5: Email Alerts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


send_alert() {
 local subject=$1
 local body=$2
 
 # Only send if score is below threshold
 if [ "$HEALTH_SCORE" -lt 70 ]; then
 echo "$body" | msmtp \
 --from="no-reply.nsuno@solytics-partners.com" \
 --subject="$subject" \
 devops-team@solytics-partners.com
 fi
}

# Generate alert
ALERT_SUBJECT="[${HEALTH_STATUS}] Platform Health Alert (Score: ${HEALTH_SCORE}/100)"
ALERT_BODY=$(cat <<EOF
CLUSTER & PLATFORM HEALTH CHECK - $(date +%Y-%m-%d_%H-%M-%S)
Cluster Node: $(hostname)
Scope: ${NAMESPACE}
===============================================

$(cat /tmp/health_check_results.txt)

PLATFORM STATUS: ${HEALTH_STATUS} (${HEALTH_SCORE}/100)
EOF
)

send_alert "$ALERT_SUBJECT" "$ALERT_BODY"

The Full Script

The complete script checks: