← Back to Sessions

Session 3: Kubernetes Scheduling

Master Node Affinity & Pod Placement

Hands-On Lab Guide

Lab Overview

What You'll Learn

This comprehensive lab covers Kubernetes pod scheduling mechanisms, from basic manual scheduling to advanced node affinity rules. You'll learn how the Kubernetes scheduler makes decisions and how to control pod placement across your cluster.

Prerequisites:
  • Running Kubernetes cluster (3 nodes recommended)
  • kubectl configured and connected to your cluster
  • Basic understanding of Pods and Deployments

Part 1: Understanding Static Pods

What are Static Pods?

Static pods are managed directly by the kubelet daemon on a specific node, without the API server observing them. Master components (kube-apiserver, kube-scheduler, kube-controller-manager, etcd) run as static pods.

Location: /etc/kubernetes/manifests/
Static pods are defined in this directory and the kubelet automatically starts them on node boot and restarts them if they crash.
View static pods
kubectl get pods -n kube-system
💡 Tip: You'll see master components with node names appended (e.g., kube-scheduler-master-node).

Part 2: Manual Scheduling with nodeName

Concept

The simplest form of scheduling is using the nodeName field to assign a pod to a specific node.

busybox-manual.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox-manual
spec:
  nodeName: node-22  # Hard-coded node assignment
  containers:
  - name: busybox
    image: busybox
    command: ["sleep", "3600"]
Apply and verify
kubectl apply -f busybox-manual.yaml
kubectl get pods -o wide
⚠️ Limitations:
  • Static assignment - pod is locked to that specific node
  • No flexibility - if node is unavailable, pod won't schedule
  • Not recommended for production deployments

Part 3: Node Selectors (Label-Based Scheduling)

Labeling Nodes

First, add labels to your nodes to categorize them:

Label nodes
# Label nodes by color
kubectl label node node-20 color=blue
kubectl label node node-21 color=green
kubectl label node node-22 color=red

# Label nodes by size
kubectl label node node-20 size=small
kubectl label node node-21 size=small
kubectl label node node-22 size=medium

# Verify labels
kubectl get nodes --show-labels

Using Node Selector

backend-nodeselector.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 5
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      nodeSelector:
        size: small  # Only schedule on nodes with size=small
      containers:
      - name: backend
        image: nginx:latest
        ports:
        - containerPort: 80
Apply and observe
kubectl apply -f backend-nodeselector.yaml
kubectl get pods -o wide
Expected: All pods should be on node-20 and node-21 (size=small)

What Happens When Labels Don't Exist?

Test with non-existent label
# Edit deployment to use non-existent label
kubectl edit deployment backend
# Change: size: small -> size: large

# Scale to trigger rescheduling
kubectl scale deployment backend --replicas=0
kubectl scale deployment backend --replicas=5

# Check pod status
kubectl get pods
⚠️ Result: Pods will be in Pending state! Use kubectl describe pod <pod-name> to see: "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector"

Part 4: Node Affinity (Advanced Scheduling)

Types of Node Affinity

  • requiredDuringSchedulingIgnoredDuringExecution
    • Required: Mandatory rule, pod won't schedule without match
    • IgnoredDuringExecution: If node labels change after pod is running, pod continues running
  • preferredDuringSchedulingIgnoredDuringExecution
    • Preferred: Soft rule, scheduler tries to match but will schedule anyway
    • Weight: 1-100 priority for this preference

Required Node Affinity

backend-required-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 5
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: size
                operator: In
                values:
                - small
                - medium
      containers:
      - name: backend
        image: nginx:latest
Operators Available:
  • In: Label value must be in the list
  • NotIn: Label value must not be in the list
  • Exists: Label key must exist (any value)
  • DoesNotExist: Label key must not exist
  • Gt: Greater than (numeric values)
  • Lt: Less than (numeric values)

Preferred Node Affinity with Weights

backend-preferred-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 5
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 50
            preference:
              matchExpressions:
              - key: size
                operator: In
                values:
                - small
          - weight: 30
            preference:
              matchExpressions:
              - key: size
                operator: In
                values:
                - medium
      containers:
      - name: backend
        image: nginx:latest
💡 How Weights Work:
  • Scheduler calculates scores for each node
  • Higher weight = higher priority
  • If multiple preferences match, weights are added
  • Scheduler picks node with highest total score
  • Fallback: If no preferences match, scheduler still places pod somewhere

Part 5: Understanding Pod Lifecycle

Pod Lifecycle Stages

┌─────────┐     ┌───────────┐     ┌─────────┐     ┌─────────────┐
│ Pending │ --> │ Scheduled │ --> │ Running │ --> │ Terminated  │
└─────────┘     └───────────┘     └─────────┘     └─────────────┘
     │                                                     │
     └─────────────────────────────────────────────────────┘
                    (Image Pull / Errors)
                
  • Pending: Pod created, waiting for scheduler
  • Scheduled: Scheduler assigned pod to a node
  • Running: Container(s) are running
  • Succeeded/Failed: Pod completed (for Jobs)
  • Unknown: Node communication lost

Scheduling vs Execution

  • During Scheduling: Affinity rules are evaluated
  • During Execution: Pod is running on the node
  • "IgnoredDuringExecution": If node labels change while pod is running, pod is NOT evicted
Test label change impact
# Create pod with required affinity for color=red
kubectl apply -f pod-with-affinity.yaml

# Verify pod is running on node-22 (color=red)
kubectl get pods -o wide

# Change node label
kubectl label node node-22 color=blue --overwrite

# Pod continues running! (IgnoredDuringExecution)
kubectl get pods -o wide

# Delete and recreate pod
kubectl delete pod <pod-name>

# New pod will be Pending (no node with color=red)
kubectl get pods

Part 6: YAML Syntax - Arrays vs Lists

Understanding YAML Structure

List (multiple items)

YAML List Example
nodeSelectorTerms:
- matchExpressions:  # Item 1
  - key: color
    operator: In
    values:
    - red
- matchExpressions:  # Item 2
  - key: size
    operator: In
    values:
    - small

Array (single element with multiple values)

YAML Array Example
values:
- small
- medium
- large

Part 7: Practical Scenarios

Scenario 1: Environment-Based Scheduling

Label nodes by environment
kubectl label node node-20 env=dev
kubectl label node node-21 env=uat
kubectl label node node-22 env=prod
Deployment for UAT
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: env
            operator: In
            values:
            - uat

Scenario 2: High-Priority Workload

Label high-performance nodes
kubectl label node node-22 performance=high
Critical application
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: performance
            operator: In
            values:
            - high

Part 8: Debugging Pending Pods

Check Pod Status

Debugging commands
# List all pods with status
kubectl get pods

# Detailed pod information
kubectl describe pod <pod-name>
Common Issues:
  • No matching nodes: "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector"
  • Insufficient resources: "0/3 nodes are available: 3 Insufficient cpu"
Solutions
# Check node labels
kubectl get nodes --show-labels

# Check node resources
kubectl describe node <node-name>

# Modify affinity rules
kubectl edit deployment <deployment-name>

# Or update labels
kubectl label node <node-name> <key>=<value>

Part 9: Best Practices

1. Use Preferred Over Required When Possible

  • Provides flexibility
  • Prevents pods from being stuck in Pending
  • Allows graceful degradation

2. Use Meaningful Label Names

  • env: dev, uat, prod
  • tier: frontend, backend, database
  • zone: availability zones
  • disktype: ssd, hdd
  • size: small, medium, large

3. Test Affinity Rules

Always test with:

  • Scale to 0 and back
  • Delete and recreate pods
  • Remove labels and observe behavior

Quick Reference

Useful kubectl commands
# Label operations
kubectl label node <node> <key>=<value>
kubectl label node <node> <key>-  # Remove label
kubectl get nodes --show-labels

# Pod operations
kubectl get pods -o wide
kubectl describe pod <pod-name>
kubectl delete pod <pod-name>

# Deployment operations
kubectl scale deployment <name> --replicas=<count>
kubectl edit deployment <name>
kubectl rollout restart deployment <name>

# Debugging
kubectl get events --sort-by='.lastTimestamp'
kubectl logs <pod-name>

Summary

In this lab, you learned:

  • ✅ Static pods and their role in Kubernetes
  • ✅ Manual scheduling with nodeName
  • ✅ Label-based scheduling with nodeSelector
  • ✅ Advanced scheduling with Node Affinity
  • ✅ Required vs Preferred affinity rules
  • ✅ Weight-based scheduling priorities
  • ✅ Pod lifecycle and scheduling phases
  • ✅ YAML syntax for complex affinity rules
  • ✅ Debugging pending pods
  • ✅ Best practices for production use
Next Steps:
  • Explore Pod Affinity and Anti-Affinity
  • Learn about Taints and Tolerations
  • Study DaemonSets for node-level scheduling
  • Practice CKA/CKAD scheduling scenarios