← Back to Sessions

Session 2: Pods, Services & DaemonSets

Deep Dive into Networking & Service Discovery

Hands-On Lab Guide

Lab Overview

Application Architecture

In this lab, we'll deploy a full-stack MERN application to Kubernetes. The application consists of three microservices:

  • Frontend: React application (Port 80)
  • Backend: Node.js/Express API (Port 5000)
  • Database: MongoDB (Port 27017)

Initial Setup

1. Verify Cluster Access

First, verify you can access the Kubernetes cluster:

Check nodes
kubectl get nodes
Expected Output:
NAME              STATUS   ROLES           AGE   VERSION
ip-10-1-1-10      Ready    control-plane   1d    v1.28.0
ip-10-1-1-20      Ready    worker          1d    v1.28.0
Check all pods
kubectl get pods -A

2. Clone the Repository

Clone the demo application repository:

Clone repository
git clone git@github.com:sasidatta/docker_demo.git
cd docker_demo/k8s
💡 Tip: If you need to set up SSH keys for GitHub, generate a key with ssh-keygen and add it to your GitHub account.

Namespaces

Understanding Namespaces

Namespaces provide a virtual cluster within a physical cluster. They're useful for:

  • Isolating resources between teams or projects
  • Applying resource quotas and policies
  • Organizing resources logically
⚠️ Important: Namespaces are a virtual construct - pods from different namespaces still share the same network space by default!

Create Your Namespace

Create namespace
kubectl create namespace <your-name>
Set default namespace
kubectl config set-context --current --namespace=<your-name>
💡 Pro Tip: Setting the default namespace saves you from typing --namespace with every command!
Verify namespace
kubectl get namespaces

Deploy Application

Deploy MERN Stack

Let's get our application running so we can explore Pods, Services, and Networking.

Apply deployments
kubectl apply -f frontend.yaml
kubectl apply -f backend.yaml
Scale backend
kubectl scale deployment backend --replicas=2
Verify Pods:
NAME                        READY   STATUS    RESTARTS   AGE
backend-6b5f7b8889-abc12    1/1     Running   0          20s
backend-6b5f7b8889-def34    1/1     Running   0          10s
frontend-7c9d4s8881-xyz78   1/1     Running   0          30s

Pods

Understanding Pods

A Pod is the smallest deployable unit in Kubernetes. Key concepts:

  • Pods hold one or more containers (typically one)
  • Containers in a pod share network and storage
  • Pods are ephemeral - they can be created and destroyed
  • Each pod gets its own IP address

Declarative vs Imperative

Declarative (YAML manifest)

Create pod from YAML
kubectl apply -f busybox.yaml

Imperative (Command line)

Create pod imperatively
kubectl run nginx --image=nginx:latest --restart=Never

Pod Lifecycle & Restart Policy

Check restart policy
kubectl get pod busybox-pod -o yaml | grep -i restart
Output:
restartPolicy: Always
restartCount: 1
Restart Policies:
  • Always - Always restart the container (default)
  • OnFailure - Restart only if container exits with error
  • Never - Never restart the container

View Pod Details

Get pods with details
kubectl get pods -o wide
Output shows: Pod name, status, IP address, node placement
NAME           READY   STATUS    IP            NODE
frontend-abc   1/1     Running   10.244.1.30   ip-10-1-1-20
Delete pod
kubectl delete pod busybox-pod nginx

Pod vs Deployment

Feature Pod Deployment
Self-healing ❌ No ✅ Yes
Scaling ❌ Manual ✅ Automatic
Rolling updates ❌ No ✅ Yes
Use case Testing, debugging Production workloads

Services

Why Services?

Pods are ephemeral and their IPs change. Services provide:

  • Stable endpoint for accessing pods
  • Load balancing across pod replicas
  • Service discovery via DNS
  • Abstraction layer over pods

Service Types

ClusterIP (Default)

Purpose: Internal cluster communication

Access: Only within cluster

Use case: Backend ↔ Database

NodePort

Purpose: External access via node IP

Access: <NodeIP>:<NodePort>

Use case: Exposing frontend to users

LoadBalancer

Purpose: Cloud load balancer integration

Access: External IP from cloud provider

Use case: Production external access

Deploy Services

Apply backend service (ClusterIP)
kubectl apply -f backend-service.yaml
Apply frontend service (NodePort)
kubectl apply -f frontend-service.yaml
List services
kubectl get services -o wide
Expected Output:
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
backend    ClusterIP   10.103.45.67    <none>        5000/TCP       2m
frontend   NodePort    10.107.12.34    <none>        80:30678/TCP   2m

Access the Application

NodePort Access: Access the frontend using any node's IP and the NodePort:
http://<node-ip>:30678
Example: http://44.204.96.183:30678

Networking & Policies

Understanding Pod Networking

Pod-to-Pod

All pods can communicate with all other pods on any other node without NAT.

Service Discovery

Services provide stable IPs and DNS names for sets of pods.

Deep Dive: Manual Pod-to-Pod Communication

Let's verify that pods can talk to each other directly using their IPs, bypassing services.

1. Get Target Pod IP

First, find the IP address of your backend or frontend pod.

Get Pod IPs
kubectl get pods -o wide
Note the IP:
NAME      READY   IP            NODE
backend   1/1     10.244.1.45   ip-10-1-1-20

2. Launch Debug Pod

Create a temporary busybox pod to run network tests.

Start Debug Shell
kubectl run debug --image=busybox --restart=Never -it -- sh

3. Test Port Connectivity (Netcat)

Once inside the debug shell, use nc (netcat) to check if the port is open.

Netcat Check
# Syntax: nc -zv <POD_IP> <PORT>
nc -zv 10.244.1.45 5000
Expected Output:
10.244.1.45 (10.244.1.45:5000) open

4. Verify Response (Wget)

Fetch the HTML/JSON response from the application manually.

Curl/Wget Website
# Syntax: wget -qO- <POD_IP>:<PORT>
wget -qO- 10.244.1.45:5000
💡 Commands:
  • -q: Quiet (no loading bar)
  • -O-: Output to standard out (console)

Network Policies

Network Policies control traffic flow between pods. By default, all traffic is allowed.

Scenario: Restrict database access so only the backend can reach it.
Create Network Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-only
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend

Ingress

What is Ingress?

Ingress manages external access to the services in a cluster, typically HTTP/HTTPS.

  • Provides Load Balancing
  • Terminates SSL/TLS
  • Name-based virtual hosting

Ingress Resource

Apply Ingress
kubectl apply -f ingress.yaml
💡 DNS Mapping: Ensure your local usage points k8slab.local to the Ingress Controller IP given by: kubectl get ingress

DaemonSets

Understanding DaemonSets

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod.

  • Use Cases: Log collection (Fluentd), Monitoring (Prometheus Node Exporter)
  • Automatically adds Pods to new Nodes
  • Garbage collects Pods when Nodes are removed

Deploy a DaemonSet

Create DaemonSet (Logging Agent)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
Check DaemonSet Pods
kubectl get daemonsets -n kube-system
Expected Output: One pod per node
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE 
fluentd-elasticsearch   2         2         2       2            2

Debugging & Troubleshooting

Common kubectl Commands

Get pod logs
kubectl logs <pod-name>
Describe pod (detailed info)
kubectl describe pod <pod-name>
Execute command in pod
kubectl exec <pod-name> -- /bin/sh
Interactive shell in pod
kubectl exec -it <pod-name> -- /bin/sh
⚠️ Note: Some containers don't have /bin/bash, use /bin/sh instead!

Container Runtime (crictl)

On worker nodes, use crictl instead of docker:

List containers
crictl ps
Stop container
crictl stop <container-id>
💡 Self-Healing: When you stop a container, Kubernetes automatically restarts it based on the restart policy!

Common Issues

ImagePullBackOff

Cause: Cannot pull container image

Solution: Check image name, registry access, and credentials

CrashLoopBackOff

Cause: Container keeps crashing

Solution: Check logs with kubectl logs

Pending Pods

Cause: Cannot schedule pod (resources, node selector, etc.)

Solution: Use kubectl describe pod to see events

Best Practices

Security & Network Policies

By default, all pods can communicate with each other. Use Network Policies to restrict traffic:

  • Implement least-privilege access
  • Restrict ingress/egress traffic
  • Use namespace isolation
  • Apply pod security standards
⚠️ Production Tip: Never allow unrestricted pod-to-pod communication in production!

Resource Management

  • Always set resource requests and limits
  • Use horizontal pod autoscaling (HPA)
  • Monitor resource usage
  • Implement proper health checks (liveness/readiness probes)

Deployment Strategies

  • Rolling Update: Gradual replacement (default)
  • Recreate: Stop all, then start new (downtime)
  • Blue/Green: Full environment switch
  • Canary: Gradual traffic shift