Skip to main content

kubernetes

Intro​

The history of application deployments​

  • 1990s and 2000s - single physical host: in the early days people used to hire a sysadmin to take the application code from the developers and then upload it to a physical host machine they owned.
    • Since there was no virtualization back then, the server could only run one single server-facing process so you had one tiny application running on a physical server that probably cost $20,000.
    • It was a complete waste of resources and very time-consuming.
  • 2000s - 2013 - VMs: with the innovation of virtualization came virtual machines, which allowed you to run multiple guest operating systems on one single physical host server. This meant you could take advantage of all the CPU and resources of a single giant host machine and also deploy more applications (since each VM was secured in its own way to prevent application memory in one VM leaking over to another VM)
  • 2013 - now (containers): containers are sort of like virtual machines except they don't need a guest operating system or a hypervisor to run their code, making it very lightweight since they just use the underlying container engine operating system, like Docker desktop on Mac, Windows, or Linux.
    • Containers allow for creating lightweight microservices, and are more lightweight than VMs.

NOTE

The combination of distributed computing systems and packing containers on a host makes for the most efficient use of computer CPU and memory.

Because containers are the most efficient way to use a computer's CPU and memory, it's the de facto way to deploy microservices today.

Why Kubernetes​

The issue of container deployments arises from four main problems:

  • container lifecycle management: it's a difficult problem to replace crashed containers
  • container scaling: it's a difficult problem to try and scale the number of containers proportionally to traffic or some other metric.
  • traffic distribution: there should be rules concerning ingress traffic distribution and how to distribute that across container instances.
  • server hardening: you still have to harden server configuration, update server hardware, and harden security for the server you're running the containers on.

Kubernetes solves the first three problems through a cloud-provider agnostic framework that applies to any infra, but you're still left with maintaining the infra that kubernetes runs on.

The 4th problem of server hardening and maintenance is solved via a cloud-managed kubernetes cluster like EKS.

No vendor lockin​

Kubernetes is a standard that is provider-agnostic, meaning you're never locked in

Kubernetes provides one API for every environment (laptop, cloud, on-prem). You can declare resources once, and each environment knows how to run them. The same manifests work in a Kind cluster locally and a cloud cluster in production, eliminating the need for separate configurations like ECS, Google Compute, or Azure-specific setups.

Kubernetes makes multi-cloud deployment easy by providing a consistent interface across cloud providers. Instead of maintaining separate configurations for Amazon ECS, Google Compute, Azure, etc., you can deploy Kubernetes clusters in each cloud environment and manage everything identically using the same resource declarations.

The same API, different infra:

  • local infra: In a local cluster (like on a laptop), the control plane, worker nodes, storage, and gateway all run on one machine, typically as containers.
  • cloud infra: In a cloud cluster, the control plane is managed and highly available, worker nodes run as true isolated VMs rather than containers, and components are distributed across multiple cloud resources for reliability and scalability.

How kubernetes works​

Kubernetes exists because when you're dealing with a container orchestration system where you orchestrate many containers among many different hosts, you have to deal with things like termination, graceful failover, and auto-scaling. Those things are extremely difficult to manually implement because there are so many things that can go wrong when creating your own auto-scaling microservice system between containers.

Kubernetes fixes exactly that problem, where you declaratively describe the containers and how they should connect to each other and then it just deploys them and does all the scaling up and graceful failover for you.

Here is what K8S does in the happy path

  1. an API request to the K8S cluster arrives at the master control plane
  2. The master control plane orchestrates the request to be forwarded to a certain node (container host) in the cluster.
  3. The individual container host forwards the request to the individual container or containers via load balancing, and the container handles that request.

Here is how K8S orchestrates node failure and recovery

  1. The master control plane decides what is the desired state for the cluster
  2. The master scheduler is continuously monitoring for failures and scaling requests to process them and notifies the master control plane if there are any changes that must be made to achieve the desired state.

Terminology​

  • node : a single virtual machine that can deploy and run multiple pods and replicas of those pods. Think of docker desktop being a node.
    • worker node: A specific type of node whose sole purpose is to run pods. It can be local (docker desktop) or remote (EC2 instance)
    • master node: A single node that is also called the control plane, used to manage all worker nodes.
  • pod: a system that manages and runs multiple containers, all running on the same IP address. You can think of a pod as a single server that runs each container inside it on a different port. Pods can have replicas, which is where scaling and availability comes into play
  • cluster: A cluster is a control plane and the worker node(s) it manages. All the scaling up and down of resources happens in the context of a cluster.
  • control plane: The control plane manages all worker nodes, scaling them up or down. It has 5 components to it:
    • Cloud Controller Manager: Connects a Kubernetes cluster to a cloud provider's API, managing cloud-specific resources and ensuring proper integration with the underlying infrastructure
    • etcd: A key-value store that saves all data about the state of the cluster; only the kube-apiserver can communicate directly with etcd
    • kube-apiserver: The kube-apiserver is a key component of Kubernetes that exposes the Kubernetes API, handles most requests, and manages interactions with the cluster by processing and validating API requests, making it essential for the cluster's operation
    • kube-controller-manager: Monitors the Kubernetes cluster's state, running processes to ensure the current state matches the desired state
    • kube-scheduler: Identifies a newly created pod that has not been assigned a worker node and assigns it to a specific node

NOTE

K8S in a nutshell

When deploying Kubernetes apps, you deploy a single cluster, which is made from many nodes that can be scaled horizontally to adapt to server traffic.

Kubernetes deploys nodes, which you can think of as each node being a single virtual machine, and in each node you can have multiple pods, where a pod can run multiple containers at once together.

The loop​

K8S follows a self-healing loop where you describe the desired state, and then K8S performs actions to arrive at that state, and that loop continues forever.

  • Desired state: what you declare you want
  • Actual state: what the cluster reads as real
  • Reconciliation: it closes the gap, forever

K8S architecture​

The control plane is what handles the orchestration of pods across worker nodes and other things like networking between nodes.

The control plane​

Think of the Kubernetes control plane (also called master node) as kind of an air traffic controller node that determines and directs where pods are created on different worker nodes.

Here are the different control plane components:

  • API server: a server that exposes a REST API to control K8S resources, only component that communicates with etcd
  • etcd: Highly-available key value store that lives in the control plane to store all data about the current state of the cluster.
  • kube scheduler: runs in a loop, identifies newly created pods which have not been assigned a worker node and then assigns them a node for the pod to run on.
  • kube controller manager: manages worker nodes and ensures that they are up and running and deploys self-healing operations to keep them up.
  • cloud controller manager: just the kube controller manager but retrofitted for each individual cloud provider so you can use kubernetes on cloud infrastructure. Your cloud provider will take care of providing this component.

NOTE

Think of the API server as the only component that's actually doing any action, while the other components (like the cube scheduler and the cube controller manager) all they do is watch for changes and intelligently request the API server to create resources and allocate them to specific nodes.

All components of the control plane run as containerized pods within the cluster, which live in the kube-system built-in namespace.

You can see all control plane components running as pods with this command:

kubectl -n kube-system get pods
API server​

All K8S resources like pods, deployments, and horizontal pod autoscaler have API endpoints.

The API server component on K8S exposes a REST API interface to control all of these resources, and CLI tools like kubectl use the API server REST API under the hood to control k8s resources via HTTP requests.

Here are the key takewaways:

  • REST interface
  • only component that saves state directly to etcd
  • All clients interact with the API server, never directly to etcd

To see all the K8S resources that have available API endpoints to control their provisioning, run this command:

kubectl api-resources
etcd​

Highly-available key value store that lives in the control plane to store all data about the current state of the cluster.

NOTE

Only the Kube API server can communicate directly with etcd.

Kube controller manager​

A component that runs in a loop and continuously checks the status of the cluster to see what the desired state is, to make sure things are running properly.

For example if a worker node gets broken, then through its continuous checks it will see that and then queue up an operation to replace the broken worker node with a newly working one.

Cloud controller manager​

This component offers a provider-agnostic way to connect your cluster to a cloud provider like AWS, GCP, or any other platform that supports kubernetes

worker nodes​

All K8S clusters run with a minimum of three worker nodes to be highly available.

Each worker node has three components:

  1. kubelet: this is an agent that runs on every worker node and it makes sure that containers in a pod are running and healthy. It communicates directly with the API server component in the control plane to monitor newly created pods.
  2. container runtime: Once kubelet verifies a new pod is ready to go, it builds and starts the containers in the pod spec using the Container runtime interface (CRI), which enables the Kubelet to create containers with supported container engines.
    • Pulls container images, creates and manages containers, and ensures they run properly and securely as directed by the Kubernetes control plane
  3. kube proxy: offers a proxy between pods and services running in a worker node to the API server in the control plane so that pods and services within the same cluster can communicate with each other.
    • A network proxy that runs on each node in a Kubernetes cluster, maintaining network rules and enabling communication between pods and services within the node and the control plane, while also communicating directly with the kube-apiserver
Kubelet​

Kubelet is an agent that runs on each node in a Kubernetes cluster, ensuring containers in a pod are running and healthy while communicating with the API server in the control plane to maintain the desired state of the node.

Key takeaways:

  • manages a pod's lifecycle
  • ensures that the containers described in the pod are running and healthy
Container Runtime​

The container runtime uses a container engine to run containers within a pod, and the kubelet communicates with the CRI (container runtime interface) to control the container runtime and request container management operations to be executed.

Kube proxy​

The kube-proxy component is a network proxy on each node. It maintains network rules to allow communication to your pods, which is the basis of the service resource in Kubernetes.

Key takeaways:

Core objects​

  • Pod: a pod is the smallest and most fundamental deployable unit, which represents a single instance of a running process.
    • A pod encapsulates one or more tightly coupled containers that share storage and network resources.
    • The most common pattern is one container per pod.
  • replica set: its purpose is to ensure that a specified number of identical pods, called replicas, are running at any given time.
  • deployment: a deployment is an abstraction over replica sets, making it a higher-level object that manages replica sets and their life cycles:
    • Provides declarative updates to pods
    • Handles rolling updates and rollbacks
    • Deployments are the standard way to manage stateless applications.
  • service: a service provides a stable endpoint to access a logical set of pods, which means it can target pods via a label or selector and then provide a stable DNS name that they can refer to statically without the fragility of IP addresses changing.
    • Since pods are ephemeral and their IP addresses change, we use services to provide a stable endpoint to forward traffic to those pods.
    • Services provide a stable virtual IP address called a cluster IP and also a DNS name that is static and unchanging.
    • Traffic to a service is automatically load-balanced to the backend pods.
  • namespace: namespaces provide a mechanism for isolating groups of resources within a single cluster.
    • They are a way to divide cluster resources between multiple namespaces, ensuring that resource names must be unique within a namespace but not across them because resources are always sectioned into the namespace that they are in

The full flow​

  1. create kubernetes resources: You either imperatively or declaratively, create Kubernetes resources, which then make a request to the API server component in the control plane.
  2. save state: the API server receives the request, creates the resources, and saves the new state of the cluster in etcd.
  3. controller-manager checks for changes: the controller manager component continuously checks for changes to see if it should add a new worker node.
  4. scheduler gets triggered: because a new deployment was created the scheduler gets triggered and pings the API server to ask if there are any unassigned, newly created pods.
  5. API server notifies scheduler: the API server notifies the scheduler that there are newly created unassigned pods.
  6. scheduler chooses node for a pod: the scheduler chooses a node for the pod and sends its request to the API server to place the newly created pods within those specific nodes.
  7. API server saves state: API server saves state of cluster in etcd
  8. kubelet in worker node checks for newly assigned pod: kubelet inside the worker node that got the newly created pod assigned to it wakes up and pings API server to receive the pod spec.
  9. API server sends pod spec: the API server sends the pod spec to kubelet in the worker node.
  10. kubelet triggers container runtime: Kubelet receives the pod spec, triggers container runtime to create container and start it
  11. kubelet monitors pod status and sends to API server
  12. API server saves state to etcd
CONTROL PLANE (Brain)
┌────────────────────────────────────────────────────────────────────────┐
│                                                                        │
│   kubectl apply -f manifest.yaml                                       │
│          │                                                             │
│          ▼                                                             │
│   [kube-apiserver] <─────────> [etcd] (Raft KV store: state/specs)    │
│     ▲          ▲                                                       │
│     │          │                                                       │
│     ▼          ▼                                                       │
│ [kube-scheduler] [kube-controller-manager]                             │
│ (Assigns nodes)   (Reconciliation loops: Deployment, ReplicaSet, Node) │
└─────┬──────────────────────────────────────────────────────────────────┘
      │
      │ gRPC / TLS
      ▼
┌────────────────────────────────────────────────────────────────────────┐
│ WORKER NODE (Muscle)                                                   │
│                                                                        │
│   [kubelet] ──(CRI / gRPC)──> [containerd]                             │
│   (Node agent & static pods)        │                                  │
│                                     ▼                                  │
│   [kube-proxy]               ┌───────────────┐ ┌───────────────┐       │
│   (iptables / IPVS rules)    │ Express App   │ │ Redis Pod     │       │
│                              │ 10.244.0.15   │ │ 10.244.0.16   │       │
│                              └───────┬───────┘ └───────▲───────┘       │
│                                      │   ClusterIP SVC │ :6379         │
│   [CoreDNS / CNI (Flannel/Calico)]   └─────────────────┘               │
│   (Flat /16 Pod Overlay Network)                                       │

  • kube-apiserver (00:07:30): The stateless REST gateway. Validates payloads, handles authentication/RBAC, and is the only component allowed to read/write to etcd.

  • etcd (00:07:45): A consistent, highly available distributed key-value store using the Raft consensus algorithm. It holds the entire cluster state.

  • kube-scheduler (00:08:12): Matches unassigned pods to suitable nodes by evaluating affinity rules, resource requests/limits, taints, and tolerations.

  • kube-controller-manager (00:08:35): A collection of reconciliation processes (Node Lifecycle Controller, ReplicaSet Controller, EndpointSlice Controller) running continuous control loops.

  • kubelet (00:09:08): The local node supervisor. It watches the API server for PodSpecs assigned to its node and drives the container runtime via CRI (Container Runtime Interface) to spawn containers and monitor health probes.

  • kube-proxy (00:09:30): Manages Layer 4 transport rules (iptables or IPVS) on the node host, translating stable virtual Service IPs (ClusterIP) to individual Pod IP endpoints.

  • CNI Plugin (Container Network Interface) (00:13:40): Allocates IP subnets to nodes and creates a flat overlay network where every Pod gets a unique routable IP without host NAT.

Minikube basics​

Minikube allows you to run a kubernetes cluster on your local computer for practice.

Here are the defaults:

  • single worker node: a single worker node that contains the control plane and data plane all within that node

You can configure all these defaults and change them to provision multiple nodes, different compute settings, etc.

Minikube and Kubectl Setup​

  1. Install on WSL
curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64
  1. Install kubectl
brew install kubectl
kubectl version --client
  1. Start a cluster with minikube
minikube start
  1. View cluster info using kubectl
kubectl cluster-info # view cluster info
kubectl get nodes # view nodes
kubectl get namespaces # view namespaces
kubectl get pods -A # view pods from all namespaces
kubectl get services -A # get all services

Minikube management​

Minikube CLI reference​

  • minikube start: creates a local cluster you can use
  • minikube tunnel: proxies internet traffic from your locally running minikube cluster to your local machine on localhost, making services able to run on localhost.
  • minikube delete: deletes the cluster.
  • minikube status: returns the status of the cluster
  • minikube ip: returns the IP of the primary worker node of the cluster.

starting and deleting clusters​

You can start a cluster with the minikube start command

minikube start

You can delete the default cluster with the minikube delete command.

minikube delete

minikube start​

CPU and memory

You can define how much CPU cores and memory to give to the cluster.

minikube start --cpus=4 --memory=8192

This configuration provides 4 CPUs and 8GB of memory to your Minikube cluster, which can significantly improve performance for resource-intensive applications.

profiles

The -p/--profile flag lets you create a "profile" for a minikube cluster that you can then reload later to load the cluster with all your settings.

NOTE

This approach allows you to maintain separate development environments for different projects or testing scenarios without interference between them.

minikube start -p <profile-name>

You can list all your profiles to see what profiles are available:

minikube profile list

To switch between profiles, you can use the profile command:

minikube profile minikube

This command switches the active profile to "minikube," reconfiguring kubectl to interact with that cluster.

multi-node mode

This is how you can create multiple virtual nodes within your minikube cluster:

minikube start --nodes=3

So for a node port service, you'll have something like this available:

minikube       192.168.49.2
minikube-m02   192.168.49.3
minikube-m03   192.168.49.4

A NodePort service can be reached through any of those node IPs:

192.168.49.2:30080
192.168.49.3:30080
192.168.49.4:30080

Minikube addons​

You can add addons to minikube which include plugins like networking plugins, metrics server, ingress, etc.

To see which add-ons are available and their status:

minikube addons list

Metric server​

Here is how to enable the metrics server add on

minikube addons enable metrics-server

Minikube with deployments and pods​

Since minikube runs locally and you may not have access to a container registry, minikube offers a local container registry you can push to, so that you can create local images with docker and instead of having to push them up to a registry, you can just build the images locally from a Dockerfile in your codebase.

Minikube allows you to reuse its internal Docker daemon. By building your image directly inside Minikube's environment, Kubernetes can instantly access it

  1. Point your terminal's shell to Minikube's Docker daemon:
eval $(minikube docker-env)

  1. Build the image
docker build -t my-image:latest .

# or if you already built image

minikube image load my-image:latest
  1. Deploy the K8S resource of pod or deployments, make sure imagePullPolicy is set to Never so that it pulls images locally from the minikube registry
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: user
  template:
    metadata:
      labels:
        app: user
    spec:
      containers:
        - name: user-container
          image: amallick-users:latest
          imagePullPolicy: Never

Minikube dashboard visualizations​

The minikube dashboard command lets you visualize a GUI over your K8S resources in your minikube cluster

minikube dashboard

This command runs the local dashboard server in the cluster as a Pod, talking to the local control plane.

Minikube for services​

If you want to test services like a LoadBalancer service locally without provisioning an actual load balancer infra, you can do so with minikube.

LoadBalancer service​

  1. Create the service
apiVersion: v1
kind: Service
metadata:
  name: users-service
spec:
  selector:
    app: users
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  1. Apply the service
kubectl apply -f kubernetes/users-service.yaml
  1. Give the service a loopback IP address on your local machine via minikube, which can accept ingress traffic for the cluster.
minikube service users-service

  1. Now request it from the new private URL:
curl -d '{"email" : "test@test.com", "password": "testers"}'  http://192.168.49.2:31367/login

You can also pass the --url option to the minikube service command to non-interactively get the public service URL:

curl $(minikube service hello-minikube --url)

NodePort service​

In minikube, the cluster VM acts itself as a node. The control plane components and any K8S resources you create are all on the same node, which is the one VM minikube provisions to act as the entire local cluster.

The minikube ip command. returns the IP of the primary Minikube node. It is not a load-balanced virtual IP representing the whole cluster.

So for a NodePort service that exposes a port on a node and runs the service as an origin on all nodes within the cluster, you can query the service like so:

curl http://$(minikube ip):$NODEPORT

The reason it works is:

  1. The IP belongs to a node.
  2. NodePort opens the port on every node.
  3. kube-proxy redirects traffic from that node to one of the Service's pod endpoints.
  4. The chosen pod may be on the same node or a different node.
Client
  ↓
Primary node IP (from minikube ip)
  ↓
NodePort
  ↓
kube-proxy rules
  ↓
Service endpoint selection
  ↓
Target pod

kind​

kind is a CLI tool for managing local k8s clusters just like Minikube.

Cluster management​

creating clusters

Cluster management is handled declaratively and imperatively, where to create a cluster, you apply kind to a cluster yaml definition like so:

  1. Create a cluster yaml manifest
kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080
        hostPort: 30080
        protocol: TCP
  - role: worker
  - role: worker
  1. Imperatively create the cluster
kind create cluster --config kind-cluster.yaml --name <cluster-name>

kubectl basics​

Declarative vs imperative​

You can create resources in kubernetes either imperatively or declaratively:

  • imperative: running CLI commands to create k8 resources
  • declarative: describing resources in YAML and then creating them with the kubectl apply -f <yaml-file> command to create a resource from its YAML description.

NOTE

Declarative vs Imperative

  • imperative: using kubectl CLI to create resources
  • declarative: configuring resources in yaml files and then using two basic commands to manage their creation and deletion.

Declarative: YAML​

When using K8S declaratively, you create manifest YAML files that have the spec for describing how a resource should be created:

Once you describe the outline for a resource in a yaml file, you have these two basic commands to bring them into and out of existence:

  • kubectl apply -f <yaml-file> : creates the resource from the specified yaml file by making a request to the API server
  • kubectl delete -f <yaml-file> : deletes the resource that was created from the yaml specification a request to the API server

Imperative: CLI​

The CLI for kubernetes is based on CRUD functionality for deployments, services, namespaces, and pods, making using the CLI extremely predictable. It always follows this syntax:

kubectl CRUD_METHOD RESOURCE

The crud methods are as follows:

  • CREATE: the apply keyword creates a deployment or service
  • READ: the describe keyword inspects a single resource, or use get to list many resources.
  • DELETE: the delete keyword deletes a resource

getting multiple resources

NOTE

You can get a bird’s eye view of everything running with the kubectl get all command.

You can also request multiple resources at once via comma separation:

kubectl get deployments,pods,services

kubectl YAML basics​

All k8s YAML files must have these two keys:

  • apiVersion: should be set to v1
  • kind: the type of resource this YAML is

Besides that, you have these rules:

  • can have multiple resources in one YAML: it's possible to have multiple resources defined in one YAML by separating the resources with a --- fence.

Kubectl CLI​

kubectl commands to YAML​

You have two important flags that lets you run any kubectl command and get the output in declarative YAML:

  • -o yaml: outputs the result of the resource created in YAML
  • --dry-run: doesn't actually execute the command, just performs a dry run of it.

TIP

--dry-run is useful when paired with -o yaml to see what a certain kubectl command would actually output.

The below command works with all resources and lets you view all the info of a specific provisioned resource within your cluster in YAML format:

kubectl get <resource> <resource-name> -o yaml

Here's an example

kubectl get pod my-pod -o yaml
kubectl create configmap my-configmap --from-file=index.html -o yaml --dry-run

Explain a resource​

kubectl explain <resource>

This command explains everything you can do with pods

kubectl explain pod

Get diff of YAML file​

Shows a diff of a YAML file from the current unapplied changes compared to the previously applied changes.

kubectl diff -f blue-green.yaml

Watch a command​

The watch kubectl command allows you to watch any kubectl command in real time and get live updates:

watch kubectl get pods

Waiting for a condition​

The kubectl wait command allows you to wait for a specific condition or property value on a K8S resource.

NOTE

This is effective when combined with label selectors via the -l flag.

kubectl wait --for=condition=Ready pod -l app=webhook

kubectl completion​

Run this command to get bash shell completion:

source <(kubectl completion bash)

Kubectl Config​

The ~/.kube/config file contains configuration about all the k8s clusters and contexts available that you can connect to:

# metadata defining config file
apiVersion: v1
kind: Config

# metadata that applies to currently connected cluster
current-context: docker-desktop
preferences: {}

# lists all clusters
clusters:
- cluster:
    certificate-authority-data: some-bullshit
    server: https://127.0.0.1:6443
  name: docker-desktop
- cluster:
    server: https://rancher-test.hhmi.org/k8s/clusters/c-m-bf8pt7pg
  name: test-janelia
  
# lists all contexts
contexts:
- context:
    cluster: docker-desktop
    user: docker-desktop
  name: docker-desktop
- context:
    cluster: test-janelia
    user: test-janelia
  name: test-janelia

# lists all the users
users:
- name: docker-desktop
  user:
    client-certificate-data: some-bullshit
    client-key-data: some-bullshit
- name: test-janelia
  user:
    token: some-bullshit

Kubectl krew​

kubectl krew is a k8s package manager that ships with kubectl. It provides additional commands/packages you can download to do extra things on top of the kubectl CLI.

Here's an example:

kubectl krew install tree
kubectl tree deployment blue-green

Networking in Kubernetes basics​

Intro​

You can consider a Kubernetes cluster as its own LAN network with private IP addresses that it assigns to pods, services, and nodes.

Kubernetes networking model has four main requirements:

  1. intra-pod communication: Containers must be able to communicate with other containers in the same pod.
    • Achieved by creating a docker network within the pod
  2. inter-pod communication: Pods must be able to communicate with other pods, whether within the same node or in different nodes.
    • Achieved by giving each pod within a node its own private IP address
  3. intra-cluster communication: Pods must be able to communicate with services.
  4. internet communication: There must be a way for traffic from the internet to communicate with services inside a Kubernetes cluster.

Since a node represents a physical machine, you can think of a node as a subnet within an LAN and a cluster, which consists of many nodes, as the LAN itself. Here are the analogies:

  • cluster: consider this like a LAN with its cluster IP being a default gateway being some private IP address, but via a local cloud controller manager like minikube, we can make the default gateway's public IP address forwarding to 127.0.0.1.
  • node: consider this as a subnet, where a node has its own private IP address that the cluster can connect to, but clusters can't connect to the pods within the nodes.
  • pod: consider this as an individual machine within a subnet, having its own private IP address within th enode.

The first three requirements are satisfied by different cluster components assigning IP addresses to pods, nodes, and services:

  • How CNIs solve inter-pod communication within the same node: CNIs (cloud network interfaces) assign unique private IP addresses to pods so that they can communicate with each other in the same node, satisfying inter-pod communication.
  • How kubeapi solves intra-cluster communication: the kubeapi component assigns services unique, private IP addresses
  • How kube controller manager solves inter-pod communication: The kubcontroller manager component in the control plane assigns unique, private IP addresses the node.

Inter-pod communication​

Kubernetes is designed so that every pod can communicate with every other pod by default, regardless of which node they are running on. This is achieved through its default flat networking model, which has several key principles:

All pods and nodes are assigned unique IP addresses within a single, flat network. This means there is no need for Network Address Translation (NAT) between pods, and each pod’s IP is visible to all other pods.

This leads to three core consequences:

  1. Pod-to-Pod Communication: The network is set up so that any pod can reach any other pod directly using its IP address. This is handled by Kubernetes networking plugins (like Kubenet or Azure CNI), which manage routing and ensure connectivity across nodes.

  2. No NAT Between Pods: Since all pods are on the same network, there’s no translation of addresses. The IP a pod sees itself as is the same IP others use to reach it.

  3. Node-to-Pod Communication: Nodes can also communicate with any pod inside it, and vice versa, without NAT.

NOTE

Pods can have one or more containers, but all containers in a pod share the same network namespace (same IP and ports).

Intra-pod communication​

Multi-container pods have inter-container communication within a pod very trivially through communication via localhost.

Containers within a multi-container pod can communicate with each other through localhost targeting the exposed ports of each container, since Containers only run on ports, and the pod is what has the IP address.

CNI and CNI plugins​

CNI plugins in Kubernetes are software packages that set up and manage the cluster's network. They create a private network that allows containers within the same pod to communicate, pods to talk to each other, pods to connect with services, and external traffic to reach services inside the cluster.

Essentially, CNIs assign unique IP addresses to pods and ensure smooth network communication within Kubernetes.

NOTE

This pluggable design lets Kubernetes support different networking solutions depending on the cluster's needs.

Because CNIs are so pluggable and only have to implement the 4 requirements of the Kubernetes Network Model, there are many third party CNI providers.

NOTE

All CNIs must implement the 4 parts of the Kubernetes Network Model, but some CNIs add additional features.

Calico CNI plugin​

  1. Delete any previous clusters you have
minikube delete
  1. Recreate the minikube cluster with the CNI set to calico:
minikube start --network-plugin=cni --cni=calico

Network policies​

A network policy in K8S is a set of rules that allow you to control traffic flow at the IP address or port level for a pod, basically like a stateless firewall for a pod.

NOTE

 By default, all pods can freely communicate, but network policies let you restrict this traffic to enforce security principles like least privilege and zero trust.

When no network policy is in place, all pods within the same node are able to communicate with each other via inter-pod communication and if services are live, intra-cluster communication as well, without any filtering or checking. This violates the principle of least privilege and zero trust.

NOTE

One thing to always keep in your mind is that network policies are overkill for many applications, and as such, GKE and minikube don't support network policies and thus applying NetworkPolicy resources won't do anything.

How network policies work

They specify whether to allow or deny ingress (incoming) or egress (outgoing) traffic based on pod selectors, namespaces, IP addresses, and ports.

When defining a network policy for a pod, for each firewall rule you need to define two components:

  • ingress or egress: whether the traffic type the rule applies to is ingress traffic or egress traffic.
  • allow or deny: whether to allow to deny the traffic of the specific network type.

using network policies

  1. To use network policies, your cluster must have a CNI plugin that supports them, such as Calico.

  2. As soon as you add a network policy to your cluster, you've opted into the granular control that network policies provide, and all traffic gets denied by default.

  3. Then you have to configure your specific network policies to actually change that behavior.

creating network policies​

There is a special NetworkPolicy resource in K8S that allows you to define a network policy for a pod or for multiple pods via the spec.podSelector object:

ingress example​

For example, what the below network policy is saying is that for the pod with selector app: echo-server, only allow ingress traffic to that pod from the pod with selector app: learning-resources and on port 80.

---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-from-learning-resources
spec:
  podSelector:
    matchLabels:
      app: echo-server
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: learning-resources
      ports:
        - port: 80
egress example​

The example below allows egress traffic from the pod with selector app: frontend-ui to the subnet 172.11.0.0/20, which is probably a node and all the pods within it.

---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-egress
  namespace: frontend
spec:
  podSelector:
    matchLabels:
      app: frontend-ui
  egress:
    - to:
        - ipBlock:
            cidr: 172.11.0.0/20
Deny-all policy​

This deny all policy basically blocks all traffic from one pod to another pod:

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Kubectl constructs​

Context​

Contexts are a way to refer to the current kubernetes cluster you are running on. You can have a local cluster managed by Docker Desktop, or you can run on a remote cluster and create deployments there.

Here are the commands for the context:

  • kubectl config current-context: gets the current context
  • kubectl config get-contexts: lists all contexts
  • kubectl config use-context [contextName]: set the current context to a different context
  • kubectl config use-context [contextName]: delete the specified context

Setting namespace for current context​

Here is how you can set the default namespace for the current context:

kubectl config set-context --current --namespace=team1

You can also use the kubectx third-party package for easy context and namespace management:

  1. Install kubectx
brew install kubectx
  1. Switch current namespace with kubens
kubens # opens TUI to select namespace
kubens <namespace-name> # non-interactively switch default namespace

Namespaces​

Kubernetes namespaces let you organize and isolate your workloads and also offers granular RBAC for accessing resources scoped to a namespace.

NOTE

Once belonging to a namespace, if the namespace gets deleted, all resources belonging to that namespace also get deleted, which is good for cleanup purposes.

Using namespaces in Kubernetes offers several benefits:

  1. Organization: Namespaces help organize resources by grouping related pods, services, and other objects together. This makes it easier to manage and locate resources, especially in large clusters.

  2. Conflict Reduction: Namespaces prevent naming conflicts. Resources only need to have unique names within their own namespace, not across the entire cluster. This is useful when multiple teams or projects use the same cluster.

  3. Access Control: Namespaces allow you to apply role-based access control (RBAC) to restrict user access to specific resources within a namespace, improving security and management.

  4. Environment Separation: You can use namespaces to separate environments (like development, testing, and production) within the same cluster, reducing the risk of accidental interference.

  5. Easier Management: With namespaces, you can list, monitor, and manage resources for a specific team or application without affecting others.

Speical namespaces and default namespace​

In a K8S cluster you have several namespaces that come built-in default to k8s, which contain important resources used to control the cluster:

  • kube-system: contains pods of the components of the control plane. It contains:
    • CoreDNS (DNS resolution)
    • etcd (state storage)
    • kindnet (networking)
    • API server
    • Controller manager
    • kube-proxy
    • Scheduler
    • Local path provisioner (for local storage)
  • default: the default namespace to work in, created by k8s.

What if you want to see a resource type across all namespaces? You can do so with the --all-namespaces option:

kubectl get pods --all-namespaces
Default service​

The default service in a Kubernetes cluster is called "Kubernetes." It acts as the internal API endpoint for the Kubernetes Control Plane.

This service is always present, even in an empty cluster, and it allows pods to communicate with the control plane's API.

  • Essentially, it serves as the central point where cluster management commands are handled.
  • While all pods can send requests to this service, they don't have permission to make changes unless explicitly granted, which helps keep the cluster secure.

Creating namespaces​

Here is the declarative way to create a namespace

  1. Create the YAML to make a namespace called "development"
namespace-development.yaml
---
apiVersion: v1
kind: Namespace
	metadata:
		name: development
  1. Apply the YAML to make it a live k8s resource:
kubectl apply -f namespace-development.yaml

Namespace management​

The namespace resource (shorthand ns) is used for controlling namespaces via CLI

  • kubectl create namespace <namespace-name>: creates a namespace
  • kubectl delete namespace <namespace-name>: deletes a namespace
  • kubectl get namespaces : returns all namespaces in the current context

Tying resources to a namespace​

There are two types of resources in K8S:

  • namespaced resources: things like pods, deployments, and configmaps, which are arbitrary constructions and can be sectioned off via namespaces
  • non-namespaced resources: things like nodes or ingress resources, which are real infra and thus cannot use namespaces.
    • Also namespaces cannot be namespaced

You can tie resources to namespaces in two different ways:

  • imperatively: Use the -n <namespace-name> option to refer to a namespace. Without the -n option, all the commands for creating resources will be run in the default namespace.
  • declaratively: After creating a namespace either declaratively (with yaml) or imperatively (with CLI), you can specify that namespace in other yaml files’ metadata key like the example below to tie resources to a namespace.
    • Otherwise, those resources will be created in the default namespace.

The default namespace is named default by default, but you can change that.

If you’re tired of using -n all the time, you can change the default namespace for the current context, like so:

kubectl config set-context --current --namespace <namespace-name>

Nodes​

Nodes are individual VMs that run multiple pods at once. Nodes are scaled up and down by the control plane, as you wish. On your local machine, docker desktop provides you only with a single node.

When a node is added to the cluster, these three tools are installed, which are necessary to run pods:

  • kubelet: manages the pods’ lifecycles

  • container runtime: supports docker to create and run containers

  • kube-proxy: manages network traffic

  • kubectl get nodes : gets all nodes

  • kubectl describe node <nodename> : describes the node

Labels and selectors​

Labels and selectors within kubernetes are ways to tag resources and then reference them in other resources.

Each resource in kubernetes can have multiple labels, which you specify through the spec.template.metadata.labels object

spec:
  template:
    metadata: 
      labels:
        app: second-app
        tier: backend

Then you target specific resources by their labels via the spec.selector.matchLabels object:

spec:
  selector:
    matchLabels:
      app: second-app
      tier: backend

Here is the full example of a deployment targeting pods to include in its deployment via the label-selector method:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: second-app-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: second-app
      tier: backend
  template:
    metadata: 
      labels:
        app: second-app
        tier: backend
    spec: 
      containers:
        - name: second-node
          image: academind/kub-first-app:2

Targeting labels with CLI​

We can target resources using the --selector option or the --label option

kubectl get pods --selector=myapp
kubectl logs --label app=blue-green
kubectl logs -l app=blue-green

Labels​

On any resource yaml, you define the labels as key-value pairs under the labels key like so:

  1. Create a label of app with value myapp
  2. Create a label of type with value front-end

Selectors​

Here are the different selector keys you can have on a resource:

  • selector: selects a pod based on the label and value pair, searching across all current pod instances to do so.
  • nodeSelector : selects a node based on the label and value pair, searching across all current node instances to do so.

  1. We set a label of disktype=superfast on Node A
  2. When we want to select a node to assign the pod to using the nodeSelector key, we select on the disktype label and select the node with the value of that disktype label being equal to superfast, thus selecting Node A.

K8S Resources​

Pods​

Pods are a shell around a grouping of containers, essentially a single server with a single IP address that runs each container inside the pod on its dedicated port. Here are the advantages of working with pods:

  • shared data: All pods in a container can access the same volumes
  • shared network: All pods in a container live on the same server and thus the same network, allowing for simple inter-container communication without having their traffic be exposed to the real world.
  • replicable: pods are the main scalable unit in kubernetes, allowing you to deploy multiple copies of the same pod and run them on a worker node.

Pods can be in one of 6 lifecycle states:

  • pending: pending for creation
  • running: running within a node
  • succeeded: ran and exited with status 0
  • failed: all containers within the pod exit and at least one exited with a non-zero exit status code.
  • unknown
  • CrashLoopBackOff: started, crashed, again and again.

Declarative POD YAML​

A pod resource is declared in yaml through the Pod kind of resource.

specifying container behavior​

The main key to specify when creating a pod resource is the containers key, where you specify all the containers that will run in a pod. Here are the options in each container:

  • image: the image to build the container from. By default, it pulls from dockerhub or any other external registry, but if you want to build a container from your local image, you need to also include the imagePullPolicy: never key.
  • imagePullPolicy: describes the pulling behavior of images. You can supply these values:
    • always : always pull the image from an external registry, always pulling on pod creation.
    • never: pull the image from your local images only.
    • IfNotPresent: pull the image only if you don't have it
  • ports : runs the container on the specified port. You have these keys to supply:
    • containerPort: required, the port number to run the container on.
    • protocol : the layer 4 protocol to run on, TCP or UDP
    • name: the layer 7 protocol, like http, https, etc.
  • command : a string array of the split string command to override the main container’s entrypoint. Useful for debugging.
  • env : provides key value pairs of environment variables to load into the container.
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
    type: front-end
spec:
  containers:
  - name: nginx-container
    image: nginx
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    env:
    - name: DBCON
      value: connectionstring
    command: ["/bin/sh", "-c"]
    args: ["echo ${DBCON}"]

example 1: local image

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: user
  template:
    metadata:
      labels:
        app: user
    spec:
      containers:
        - name: user-container
          image: amallick-users:latest
          imagePullPolicy: Never
compute request​

It's important to specify available compute limits for a pod to avoid the noisy neighbor problem of a pod hogging too much RAM on a node.

You can specify the amount of compute (memory and CPU) a container within a pod gets under the spec.containers.resources key, specifying both requests and resources

  • spec.containers.resources.requests: the requests for the minimum amount of compute that should be available for use.
  • spec.containers.resources.limits: the requests for the maximum amount of compute that a container can use up.

For both of those objects, you have these fields:

  • memory: the amount in megabytes of gigabytes to allocate for the RAM of the pod.
    • Examples are 64Mi or 1Gi.
  • cpu: the amount of CPU cores of a node to reserve for the pod.
    • Examples are 1, which refers to a full CPU core. Another example is something like 100m, which is 100 millicores

NOTE

If a pod has compute requests far above what is available at the time, that pod will not be scheduled onto a node and it will continue to sit in the queue until space frees up so it can fit onto a node that can accommodate its compute requests.

--- 
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
    type: front-end
spec:
  containers:
  - name: pod-info-container
	image: kimschles/pod-info-app:latest
	resources:
	  requests:
		memory: "64Mi"
		cpu: "250m"
	  limits:
		memory: "128Mi"
		cpu: "500m"
	securityContext:
	  allowPrivilegeEscalation: false
	  runAsNonRoot: true
	  capabilities:
		drop:
		  - ALL
	  readOnlyRootFilesystem: true
	ports:
	- containerPort: 3000
	env:
	  - name: POD_NAME
		valueFrom:
		  fieldRef:
			fieldPath: metadata.name
	  - name: POD_NAMESPACE
		valueFrom:
		  fieldRef:
			fieldPath: metadata.namespace
	  - name: POD_IP
		valueFrom:
		  fieldRef:
			fieldPath: status.podIP

NOTE

How to measure CPU compute

In Kubernetes, CPU resources are measured in cores. When you see 1 CPU, it means the pod can use one full CPU core — that's a lot of processing power. On the other hand, 100m means 100 millicores, or 0.1 of a CPU core (10%). So, 100m is just a fraction of a full core.

This lets you allocate CPU very precisely. For example, if your app only needs a small amount of CPU, you might request 100m. If it needs more power, you might request 1 or more cores. Also, pods can burst above their requested CPU if limits allow, but they can never exceed the set limit.

Think of it like reserving a slice of CPU time: 1 is a full slice, and 100m is a tenth of that slice.

Multi-container pods​

A pod can deploy multiple containers, which has the main advantage of letting all containers within that pod access the same shared resources like if they were on the same physical machine, leading to two key benefits:

  • intra-pod communication: containers within the same pod can communicate with each other through localhost.
    • Each pod runs on its own individual IP address, and thus each container in a pod runs on a separate process and thus separate port.
  • shared volumes: Volumes are declared at the pod level, thus containers within the same pod can share volumes.

NOTE

Containers in the same pod are processes running on the same IP address and can communicate with each other through localhost, therefore.

For example, this would be the mongo URI to use when dealing with a multi-container setup with an express app in one container and a mongodb database running on port 27017 in another container:

PORT=3000
MONGO_INITDB_ROOT_USERNAME=mongo
MONGO_INITDB_ROOT_PASSWORD=mongo
MONGO_INITDB_DATABASE=db
SERVICE_NAME=localhost 

MONGO_URI="mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@$SERVICE_NAME:27017?authSource=admin"

Pod observability​

You can perform wellness checks on your pods to see if they are responding correctly, which is good for making sure certain pods are up and running before trying to communicate with them.

These checks are called probes and there are three of them:

  1. startup probe: to know when a container has started
  2. readiness probe: to know when a container is ready to accept traffic.
    • A failing readiness probe will stop the application from receiving traffic.
  3. liveness probe: indicates whether the code is running or not
    • A failing liveness probe will restart the container.
Pod lifecycle​

Kubernetes knows to restart a pod when it detects that the pod's container has crashed or exited with a nonzero status code.

  1. It continuously monitors the pod's state, and if the container stops unexpectedly, Kubernetes automatically restarts it to keep your application running.
  2. Additionally, Kubernetes uses liveness probes—health checks you define that periodically test if the pod is healthy. If a liveness probe fails, Kubernetes treats the pod as unhealthy and restarts it.

Kubernetes needs to know when a pod is ready to accept traffic. By default, it declares that a pod is unable to receive traffic if:

  • the container process of a pod crashes
  • the container process of a pod exits, whether in failure or successfully

If you need more granular control over when to determine if a pod is ready or not (maybe the pod needs a database connection or health check), then you should use a readiness probe.

Probes​

Each of these probes has a suitable, appropriate test you can configure, which can be of these three types:

  1. ExecAction: running a command in a container

  1. TCPSocketAction: continuously pinging and checking if a container is listening on a port via the TCP protocol

  1. HTTPGetAction: fetching a route with HTTP

Are here are the recommended test types for each different type of probe:

  • startupProbe: Usually, running a command through the exec test is a good indicator for a container being started.
    • test type: use ExecAction
    • function: used for custom logic to check if a pod is in a correct state to start doing its intended purpose.
  • readinessProbe: the readiness probe test should try seeing if a container is listening/running on a certain port before declaring that the container is ready to receive network traffic
    • test type: use TCPSocketAction
    • function: Used to check when a pod is ready to handle traffic. Readiness probes set the pod's ready condition. Service only send traffic to ready pods.
  • livenessProbe: the liveness probe test runs a continuous test to see if the container is still running.
    • test type: use HTTPGetAction
    • function: detects when a pod enters a broken state with the purpose of alerting Kubernetes about it. Kubernetes will restart the pod for you if a pod is broken.

Each probe has these same properties that influence the frequency of the probe and when to declare failure:

  • failureThreshold: the amount of attempts the probe is allowed to fail before declaring failure on the probe and thus giving up.
  • periodSeconds: the amount of time to wait before reattempting the probe.
  • initialDelaySeconds: the number of seconds to wait before starting the probe test.

There are different behaviors that K8 takes whenever you fail one of the probes:

  • failing readinessProbe: if you fail a readiness probe, the pod stops accepting traffic.
  • failing startupProbe: if you fail a startup probe, the entire pod restarts.
  • failing livenessProbe: if you fail a liveness probe, the pod restarts.

IMPORTANT

All container probes must pass for the pod to pass and be considered ready to receive traffic.

Probes are declared at the container level.

apiVersion: v1
kind: Pod
metadata:
  name: goproxy
  labels:
    app: goproxy
spec:
  containers:
  - name: goproxy
    image: k8s.gcr.io/goproxy:0.1
    ports:
    - containerPort: 8080
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20
    startupProbe:
      httpGet:
        path: /healthz
        port: 80
      failureThreshold: 3
      periodSeconds: 10
apiVersion: apps/v1 # apps API group
kind: Pod
metadata:
    name: data-tier
	spec: # Pod spec
	  containers:
	  - name: redis
		image: redis:latest
		imagePullPolicy: IfNotPresent
		ports:
		  - containerPort: 6379
			name: redis
		livenessProbe:
		  tcpSocket:
			port: redis # named port
		  initialDelaySeconds: 15
		readinessProbe:
		  exec:
			command:
			- redis-cli
			- ping
		  initialDelaySeconds: 5
livenessProbe:
  httpGet:
	path: /probe/liveness
	port: server
  initialDelaySeconds: 5
readinessProbe:
  httpGet:
	path: /probe/readiness
	port: server
  initialDelaySeconds: 3
init containers​

Init containers are containers that run every time a pod is registered for creation, before its containers are created via containerd.

Init containers are a way to spin up containers and deal with dependencies, like checking database connection is ready before starting your express app.

You specify your init containers under the initContainers key at the same level as containers, and specify them the same way. There’s only a few differences:

  • If an init container fails, it restarts repeatedly until it succeeds, unless restartPolicy: never is set on these containers.
  • Probes like livenessProbe, readinessProbe, and startupProbe are not supported on init containers.

Here are the rules of init containers:

  1. Pods can declare any number of init containers
  2. If an init container fails, the default behavior is that it restarts repeatedly until it succeeds, but you can change that through the restartPolicy property on an init container.
  3. Init containers run sequentially in the order they are defined in the YAML
  4. As soon as all the init containers finish, the normal containers specified under containers are allowed to run.
  5. init containers cannot contain probes, since init containers are ephemeral, meant to be deleted after they successfully run once.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-tier
  labels:
    app: microservices
    tier: app
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: app
  template:
    metadata:
      labels:
        app: microservices
        tier: app
    spec:
      containers:
	      - name: server
	        image: lrakai/microservices:server-v1
	        ports:
	          - containerPort: 8080
	            name: server
      initContainers:
        - name: await-redis
          image: lrakai/microservices:server-v1
          env:
          - name: REDIS_URL
            value: redis://$(DATA_TIER_SERVICE_HOST):$(DATA_TIER_SERVICE_PORT_REDIS)
          command:
            - npm
            - run-script
            - await-redis

Imperative pods​

Pod management​

create pods

You can create pods imperatively with the kubectl run command:

kubectl run <podname> --image=<imagename>

fetching pods

You can get pods with kubectl get pod command, making sure to specify a namespace if your pods are in a deployment that belongs to a namespace.

kubectl describe pod # gets all pods in default namespace
kubectl describe pod -o wide # get more info on pods

You can then get info about a single pod with the kubectl describe pod command, specifying the namespace if necessary.

kubectl describe pod POD_NAME # describes the pod by pod name
kubectl describe pod nginx-deployment-d556bf558-5jtpv -n "nginx"

deleting pods

kubectl delete pod $POD
kubectl delete pod $POD -n $NAMESPACE
Pod-container interaction​

interactive mode

You can check out the logs of a pod or execute commands in it interactively with this command, to go into its shell:

kubectl exec -it <podname> -- sh
  • -i: interactive
  • -t: with tty

checking logs

You can get the logs of a pod with the kubectl logs command:

kubectl logs <podname>

multi-container pods

When dealing with multi container pods, you often have to specify the desired container you want to work with using the -c <container-name> option:

kubectl logs <podname> -c <container_name> # get logs from container

And this is how you can get inside the shell of the specified container running inside the specified pod:

kubectl exec -it <podname> -c <container_name> -- /bin/sh

Workloads​

Workloads are an abstraction over controlling how a pod runs and its lifecycle behavior or execution behavior.

There are 5 types of workloads:

  • replica set: maintain a set of pod copies and provide self-healing capabilities to replace crashing pods with healthy replicas.
  • deployments: An abstraction over replica sets that allow you to manage a single pod template and also updates and rollbacks to pods and any replica sets

Deployments and replica sets​

ReplicaSets are workloads that manage pod replicas and provide self-healing capabilities to replace crashing pods with healthy replicas.

NOTE

Obviously pods in a replica set will have different IP address since the rule is one IP address per pod.

Their main job is to always ensure the desired number of pods are always running.

NOTE

You can create replica sets imperatively or declaratively through the rs resource, but it’s recommended use deployments instead, since they are abstractions over replica sets.

  • kubectl get rs: list replica sets
  • kubectl describe rs <rs-name>: get info of a specific replica set
  • kubectl delete rs <rs-name>: delete a specific replica set

Deployments manage a single pod template and can also manage replica sets, abstracting that resource away. They manage updates and rollbacks to pods and any replica sets

Here is how you can declaratively define replica set behavior:

  • replicas : the number of pod replicas to manage in a replica set
  • revisionHistoryLimit : sets the number of previous iterations to keep
  • strategy : the type of behavior for determining how updates and rollbacks work in the deployment. You have these two types you can pass:
    • RollingUpdate : cycle through updating pods. The default for the below strategies are 25%.
      • maxSurge : the number of additional pods to have as backup in addition to the desirecd number of pods. This is an expressed as a percentage from 0 to 1.
      • maxUnavailable : the number of pods that are allowed to unavailable from the desired number of pods. This is expressed as a percentage between 0 and 1.
    • Recreate : kills all existing pods before creating new ones
  • template : you create the pod declaratively here, specifying the containers to run, the env variables, the volumes, etc. It’s just a pod.yaml essentially.

Here's an example of a deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blue-green
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blue-green
  template:
    metadata:
      labels:
        app: blue-green
    spec:
      containers:
        - name: blue-green
          image: mtinside/blue-green:blue

Deployment know which pods to manage via label-selector syntax:

  1. Specify deployment template to have pods with label app: blue-green
  2. Make deployment target pods with label app: blue-green in the spec.selector.matchLabels object.

Deployment scaling​

Deployment imperative scaling​

As long as pods are stateless (don't have anything stateful like websockets), you can scale them horizontally via replicas.

You can scale an individual deployment by adding more replicas, via the kubectl scale deployment command:

kubectl scale deployment <deployment-name> --replicas=<n>

kubectl scale --namespace <namespace> deployment <deployment-name> --replicas=<n>

Let's create deployments:

apiVersion: v1
kind: Service
metadata:
  name: data-tier
  labels:
    app: microservices
spec:
  ports:
  - port: 6379
    protocol: TCP # default 
    name: redis # optional when only 1 port
  selector:
    tier: data 
  type: ClusterIP # default
---
apiVersion: apps/v1 # apps API group
kind: Deployment
metadata:
  name: data-tier
  labels:
    app: microservices
    tier: data
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: data
  template:
    metadata:
      labels:
        app: microservices
        tier: data
    spec: # Pod spec
      containers:
      - name: redis
        image: redis:latest
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 6379

We can scale this deployment as much as we want, since a server is stateless.

apiVersion: v1
kind: Service
metadata:
  name: app-tier
  labels:
    app: microservices
spec:
  ports:
  - port: 8080
  selector:
    tier: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-tier
  labels:
    app: microservices
    tier: app
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: app
  template:
    metadata:
      labels:
        app: microservices
        tier: app
    spec:
      containers:
      - name: server
        image: lrakai/microservices:server-v1
        ports:
          - containerPort: 8080
        env:
          - name: REDIS_URL
            # Environment variable service discovery
            # Naming pattern:
            #   IP address: <all_caps_service_name>_SERVICE_HOST
            #   Port: <all_caps_service_name>_SERVICE_PORT
            #   Named Port: <all_caps_service_name>_SERVICE_PORT_<all_caps_port_name>
            value: redis://$(DATA_TIER_SERVICE_HOST):$(DATA_TIER_SERVICE_PORT_REDIS)
            # In multi-container example value was
            # value: redis://localhost:6379
Autoscaling with HPA​

Autoscaling adjusts the number of deployment replicas based on CPU usage or custom metrics, with specified minimum and maximum limits for the number of replicas that you can configure.

The Horizontal Pod Autoscaler (HPA) decides to add more pods by monitoring metrics like CPU usage across your current pods.

  1. It looks at the average CPU utilization compared to the CPU requests you've set for your pods.
  2. When this average goes above a target threshold (commonly 80%) of the resource requests of a pod, the HPA automatically increases the number of pod replicas to handle the load and keep performance stable.

Here is an example of setting the limit with CPU utilization:

  1. set metric threshold: Set target threshold CPU utilization to 70%
  2. set replica bounds: set minimum and maximum number of replicas to maintain
    • minimum replicas should be 1
    • minimum replicas should be 5

IMPORTANT

It's important to always understand that the metric threshold observability is based on the resource requests of a pod, not the limits or the average, therefore it's mandatory to set compute requests on a pod if you want to autoscale it.

Now if the CPU utilization threshold of 70% is breached on any of the pods in the deployment, the number of replicas increases up to the maximum 5 replicas gradually.

Here is the example declaratively using the HorizontalPodAutoscaler resource:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: app-tier
  labels:
    app: microservices
    tier: app
spec:
  # 1. threshold and bounds
  maxReplicas: 5
  minReplicas: 1
  targetCPUUtilizationPercentage: 70
  
  # 2. define what deployment to autoscale for
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-tier

This is the full workflow to achieve horizontal scaling:

  1. Define resource request limits on each container within a pod, which is mandatory to limit memory and CPU.
  2. Create a HorizontalPodAutoscaler resource that targets the deployment whose pods you want to scale, as well as specifying these three components:
    • minimum replicas: the minimum amount of replicas the targeted deployment must maintain
    • maximum replicas: the maximum amount of replicas the targeted deployment must not go over.
    • metric: a custom metric threshold value based on the resource request limits, which determines when to increase or decrease replica count.

Here is an example of using a custom metric which is average CPU utilization across the pods.

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: app-tier
  labels:
    app: microservices
    tier: app
spec:
  # 1. threshold and bounds
  maxReplicas: 5
  minReplicas: 1
  metrics:
	  - type: Resource
	    resource:
		    name: cpu
		    target:
			    type: Utilization
			    averageUtilization: 80

  # 2. define what deployment to autoscale for
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-tier

NOTE

You can also configure it to scale based on memory or custom metrics if you use advanced monitoring tools like Prometheus

You can also do this imperatively:

kubectl autoscale deployment app-tier --max=5 --min=1 --cpu-percent=70

Deployment rollouts and rollbacks​

  • rollout: updating a deployment and its infra to a new version
  • rollback: rolling back a deployment and its infra to a previous version

A rollout in Kubernetes is the process of updating or replacing replicas in a deployment with new replicas that match an updated deployment template.

This can involve changes to configurations, such as environment variables or labels, or code changes like updating the image version.

Any change to the deployment’s template triggers a rollout, and there are different rollout strategies you can set on the deployment:

  • rolling update strategy: replicas are updated in groups or one at a time rather than all at once, allowing the service to remain available during the update process. This is the default.
  • recreate strategy: kills all old deployments and then immediately creates the new version, incurring brief downtime but ensuring only the newest version of the app is running at any one time.
kubectl rollout CLI​

You can roll back deployments imperatively like so:

  • check status of rollout: since rollouts are triggered automatically on a deployment template update, we can check a rollout's live status with the kubectl rollout status deployment CLI
kubectl rollout status deployment <deployment-name>
  • pause rollout: You can pause a rollout with the kubectl rollout pause deployment command, whereby afterwards deployments will not be automatically updated, meaning no new replicas will be created after a paused rollout
kubectl rollout pause deployment <deployment-name>
  • resume rollout: You can resume a paused rollout with the kubectl rollout resume deployment command
kubectl rollout resume deployment <deployment-name>

These next commands deal with rollbacks, where you can rollback to the immediately previous deployment revision, or look at the deployment history to get a list of all deployment revision history for a specific deployment.

  • rollback: You can roll back to the previous deployment state with the kubectl rollout undo deployment command.
  • history: You can get the history of all deployment revisions with the kubectl rollout history deployment command.
rollouts with RollingUpdate strategy​

 By default, Kubernetes uses a rolling update strategy, where replicas are updated in groups rather than all at once, allowing the service to remain available during the update process.  

  • pro: Keeps application available while gradually rolling out new version of the deployment in groups.
  • con: since both old and new versions of the deployment will be running simultaneously for a brief period of time, you need to gracefully handle different application versions in your codebase.

NOTE

A rolling update in Kubernetes is a way to update your application pods gradually, one at a time, without causing downtime.

Instead of stopping all old pods and starting new ones at once—which could make your service unavailable—Kubernetes replaces pods incrementally.

This means it terminates one old pod and starts a new one before moving on to the next, ensuring your application stays available throughout the update.

You can also pause or roll back the update if something goes wrong, making deployments safer and smoother.

Here is the spec for a rolling update strategy on a deployment:

  • strategy.rollingUpdate.maxSurge: This defines how many extra pods (above the desired number of replicas) can be created during a rollout.

    • For example, if you have 10 replicas and maxSurge is set to 25%, up to 2 or 3 extra pods can be created temporarily while updating.
    • This helps ensure new pods are ready before old ones are removed, minimizing downtime.

  • strategy.rollingUpdate.maxUnavailable: This specifies how many pods can be unavailable during the update.

    • If set to 25%, up to 2 or 3 pods can be taken down at once while new ones are being created.
    • This setting helps balance availability and rollout speed.
apiVersion: apps/v1 # apps API group
kind: Deployment
metadata:
  name: data-tier
  labels:
    app: microservices
    tier: data
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: data
  # override RollingUpdate strategy
  strategy:
	  type: RollingUpdate
	  rollingUpdate:
		  maxSurge: 25%
		  maxUnavailable: 25%
  template:
    metadata:
      labels:
        app: microservices
        tier: data
    spec: # Pod spec
      containers:
      - name: redis
        image: redis:latest
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 6379
            name: redis

Going more in depth into the rolling update strategy, let’s paint a picture of having a replica set of 3 pods, and the following stretegy values:

  • max surge = 33%: have one additional pod as backup ready to substitute in at any time.
  • max unavailable 66%: Allow two pods (2/3 = 66%) out of the desired 3 pods to be killed or unavailable before you ask for the rolling update feature to trigger.

DaemonSet​

A DaemonSet workload is used to ensure all nodes run exactly one instance of a pod, basically ensuring maximum replicability and availability.

It makes sure that as new nodes are added to the cluster, specified pods are automatically replicated and added to those nodes and run.

  • use case: This is useful for running services like caches or monitoring agents that need to be present on all nodes to provide fast, local access.
  • main property: Unlike other Kubernetes resources that manage pods more abstractly, DaemonSets directly tie pods to nodes, making sure each node has its own instance of the pod.

Here are the rules:

  • one pod per node: daemonsets place one pod per node for all nodes in a cluster,
    • EXAMPLE: if you have four nodes then the daemonset will produce four pods, placing one pod in each node.
  • run containers as background processes: runs containers in the pod spec of a daemon set as background processes.

NOTE

DaemonSets allow you to run one pod per node, which works well for running pods implementing background processes such as agents.

Here is how to declare a DaemonSet declaratively:

  • spec.tolerations : any specified nodes that you do not want to run the pods on, like the master node or control plane
  • spec.containers: the containers that will make up the pod belonging to a daemon set.
Imperative DaemonSet control​

You can also create DaemonSets imperatively through the ds resource:

  • kubectl get ds: lists all daemonsets
  • kubectl describe ds <daemonset-name>: get info of a specific daemonset
  • kubectl delete ds <daemonset-name>: delete a specific daemonset

StatefulSet​

Containers are stateless by design, but StatefulSets offer stateful approaches.

A StatefulSet in Kubernetes is a resource designed to manage stateful applications, like databases, where each pod needs a unique identity and stable storage.

Unlike Deployments that create identical, interchangeable pods for stateless apps, StatefulSets treat each pod individually, allowing them to coordinate with each other—such as managing leaders and followers in a database cluster.

NOTE

This makes StatefulSets essential for running applications that require persistent state and ordered, stable deployment.

Here are the rules:

  1. If a pod dies, it is replaced with a new one with the exact same identifier and IP address.
  2. Pods are added in sequence, and deletes pods in sequence
  3. New pods after old ones die will share the same attached volumes, meaning data persists across pod deaths.
Imperative stateful set control​

Here are the CLI commands to create stateful sets imperatively using the sts resource:

  • kubectl get sts: lists all stateful sets
  • kubectl describe sts <sts-name>: get info of a specific stateful set
  • kubectl delete sts <sts-name>: delete a specific stateful set

Job​

Jobs are workloads for short-lived tasks that run pods, get a termination status, and then cleanup themselves.

IMPORTANT

We cannot use pods for one-off containers that execute a command and then exit, because if a pod exits a process and stops executing and does not have a container that runs on a port, Kubernetes thinks that the pod is in CrashLoopBackoff and will continuously try to restart the pod, leading to infinite restarts.

NOTE

Think of them like jobs from github actions - just there to do some task and then end.

  • job success criteria: You control the metrics for whether the job should succeed or fail.
  • job execution behavior: Jobs are run sequentially by default. You have to specify parallelism.

Here are the important keys concerned with declaratively creating a job:

  • restartPolicy: since jobs are one-time short-lived tasks, you should always set the restart policy to never for a job resource.
  • parallelism: if this key is specified, you need to provide the number of containers in the pod you want to run in parallel.
  • completions: The number of pods that should successfully run and exit in order for a job to be considered successfully completed.
  • template.spec: the pod template that should be run in a job, where you define your short-lived containers that do some command and then exit.
    • restartPolicy: set this to Never to make sure the job only tries to run once.
Imperative job control​

This is how you create and use jobs imperatively using the job resource:

  • kubectl create job <jobName> --image=<docker-image>: create a job
  • kubectl get job: list jobs
  • kubectl describe job <jobName>: print out info of a specific job
  • kubectl delete job <jobName>: delete a specific job

CronJob​

CronJobs are just Job workloads that run on a schedule using the cron syntax, and specify cron syntax for the spec.schedule property.

Here is how you can use cron jobs imperatively using the cj resource.

Services​

Services have two core features:

  • stable IP address: Services are resources that have persistent DNS names and IP addresses which are designed for facilitating inter-pod communication and even extra-cluster communication.
    • Pods have ephemeral IP addresses, meaning that if you want to connect to another pod in your application logic or through your local machine, you must use a service to have stable port forwarding.
  • allow ingress traffic from internet: services are the only component that allows ingress traffic to come in to your k8s cluster from the internet.

Let’s go over some basic use cases of services:

  • connecting to localhost: you have a pod running a server and you want a service that forwards that pod’s IP address to localhost on your laptop.
  • inter-pod communication: You have one server pod and one database pod, and you want to expose the database pod on a service for a persistent communication so that the server pod can access the database.

NOTE

For a pod to communicate with another pod, they have to go through a service with port forwarding and DNS forwarding, because each pod has 1 IP address.

There are five types of services you can have:

  • ClusterIP : the default, which gives the service a public IP address that is accessible only from within the cluster.
    • Only other pods within the cluster can communicate with the service, but not from localhost.
  • LoadBalancer : load balances requests from the service to all matching pods it has from its selector property. It operates at layer 4, meaning it uses the TCP protocol.
  • NodePort :  exposes pods to external traffic by allocating an exposed port accepting ingress traffic on each node in the cluster, then the service uses the node's IP address for receiving ingress traffic.
  • Headless: service without an IP address, meant for internal use for multi-container pods performing intra-cluster communication.
  • ExternalName: provide a DNS alias for a pod

Here's a table comparing all these services:

Service TypeForwards traffic to internet?Has private cluster IP?
ClusterIPNo, only accessible within cluster.Yes
LoadBalancerYes, also creates external IP address using cloud provider to provision a load balancer.Yes
NodePortYes, forwards traffic directly to localhost (the loopback) on the node.Yes

services to pod connection​

A basic service YAML is like so:

---
apiVersion: v1
kind: Service
metadata:
  name: demo-service
  namespace: development
spec:
  selector:
    app: pod-info
  ports:
    - port: 80
      targetPort: 3000
  type: LoadBalancer

Each service should have a selector that points to a corresponding label on a pod so the service targets the pod for directing traffic to it.

  1. On a pod resource YAML, provide a value for metadata.labels.app
  2. On a service YAML, use that same pod label value on spec.selector.app

IMPORTANT

The most important thing to understand about pod-targeting with services is that there can be more than one eligible target, and thus traffic is load balanced to all of them for any type of service.

Here are the spec properties:

  • spec.ports: the ports to expose, which have these properties:
    • port: the port that the selected pod is listening on
    • targetPort: the port that the selected pod is listening on. This value is not applied if the service type is NodePort, so you can just omit this.
    • nodePort: the port to map to on your localhost. This must be a large value between 30000 - 32767, as to not interfere with important ports.
    • protocol: the layer 4 protocol to use, either TCP or UDP
  • spec.type: the service type, which can be of these types:
    • ClusterIP: service gains a private IP address in cluster, facilitates intra-cluster communication.
    • LoadBalancer: service provisions a load balancer infra and gains a public IP address, facilitating both intra-cluster and internet communication
    • NodePort: service communicates with pods by reserving origins for one port on each node, forwards traffic to pods by running pods as an origin on a node.

NOTE

port vs target port

All services act as process (origins) that then forward to other origins, so you must supply the spec.port and spec.targetPort properties for all services.

  • port: In a Kubernetes Service, port is the port number that the service listens on — think of it as the front door of the service. When other parts of your cluster want to communicate with the service, they connect to this port.
  • targetPort, on the other hand, is the port on the actual pod where the application is running — the back door where the service forwards the traffic.

This setup lets you expose a standard port (like 80) on the service while the pod might be listening on a different port (like 8080), providing flexibility in how your app is accessed inside the cluste

Imperative​

You can do the same thing imperatively

You can use a single command to expose a pod's port for traffic by creating a service under the hood that forwards to that target port.

Here is an example:

kubectl expose pod <podname> --port <target-port> --name <service-name>

kubectl expose pod mypod --port 8080 --name mypod-service
Declarative​

The equivalent of that declaratively is the following:

  1. Create a pod that exposes port 8080 and has label app: blue-green
  2. Create a ClusterIP service with spec.selector set to app: blue-green to target all pods with that label.
    • port: the service port being 80 means that the service runs as a process on port 80.
    • targetPort: the exposed port on the target pods to forward traffic to.

Now traffic will be load balanced from the service to all pods with the label app: blue-green, to the target port 8080.

ClusterIP​

Gives a private fixed IP address for the pod within the cluster, enabling intra-cluster communication between pods in different nodes.

NOTE

This is the base service that all other service types inherit from and add onto.

NOTE

Since the cluster IP is the most basic service and all it does is enable intra-cluster communication between pods and different nodes, it gives an IP address to the pod that is only available within the cluster (meaning that, by default, no internet traffic can reach the cluster IP).

To expose traffic to the internet, you look towards other services.

  • ports: the ports to expose
    • port: the port that the server will expose
    • targetPort: the port that the selected pod is listening on.

NodePort​

The NodePort extends the ClusterIP service.

Here's how it works behind the scenes:

  1. node port assignment: When you create a NodePort service, Kubernetes allocates a static port (from a range, usually 30,000–32,767) on every node in the cluster.
  2. access via IP address: The NodePort service can be accessed using the IP address of any node in the cluster along with the allocated port. This means that users can connect to your application using any worker node’s IP address and the NodePort.
  3. NodePort service translation: Any request sent to any node’s IP address on the allocated port will be forwarded to the service, which then routes the request to one of the target pods.
    • Kubernetes uses kube-proxy to route incoming requests on the NodePort to the appropriate pods. When a request hits the node on the NodePort, kube-proxy forwards the traffic to one of the pods that are part of the service.

For example, if your webserver pod is exposed via a NodePort service on port 30001, you can access it by sending a request to any node’s IP at port 30001:

  1. Define a Pod: Let's say you have a simple web application running in a pod defined by the following YAML, where the exposed port of the container in the pod is port 80.
apiVersion: v1
kind: Pod
metadata:
  name: my-web-app
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - containerPort: 80

NOTE

Kubernetes automatically manages the mapping between the NodePort and the pods, even if pods are added or removed.

  1. Create a NodePort Service: Next, you create a NodePort service like this:
apiVersion: v1
kind: Service
metadata:
  name: my-web-app-service
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30001  # Optional; Kubernetes can choose an available port
  selector:
    app: my-web-app
  1. Access the Application: Once the service is created, you can access your application by going to http://<NodeIP>:30001.
    • <NodeIP> is the IP address of any node in your cluster.

Here are the disadvantages:

  1. multiple IP addresses: since NodePort allows redirecting traffic to any node in the cluster, that means the IP address can change to different nodes' IP addresses depending on the which pod you want to request ingress traffic for, and which node that pod lives in.
  2. No Load Balancing: NodePort does not provide any built-in load balancing. Traffic directed to the NodePort is sent to one of the pods, but it does not distribute incoming requests evenly across multiple pod replicas. This can lead to uneven traffic distribution and potential performance issues, especially under heavy load.
  3. Manual Management: Exposing services via NodePort can require more manual management. You must know the IP of the nodes in your cluster and the NodePort to access the service, which can complicate the client experience and require workarounds for dynamic situations.
  4. Security Concerns: NodePorts open ports on every node in the cluster, which can pose a security risk. If these ports are exposed to the internet, they may be vulnerable to attacks if not properly secured.

NOTE

In summary, while NodePort is straightforward for exposing services, it has limitations regarding port management, traffic distribution, security, and flexibility that make it less ideal for more complex use cases and production environments.

Loadbalancer​

The load balancer service calls upon a cloud provider the cluster is hosted on to provision a real load balancer to handle external internet traffic being directed to the target pod.

The load balancer service tries to acquire an external IP address for the provisioned load balancer, allowing it to accept internet traffic immediately.

  • If using a local cluster management service like minikube, you can use minikube tunnel command to actually expose the external IP address as being an origin on localhost IP address.
  • If using a cloud provider like Google Cloud, Google Cloud will actually provision a load balancer for you and attach to it an external IP address immediately available on the internet.

NOTE

Behind the scenes if you use a load balancer service, a cloud provider like Google Cloud or AWS EKS will actually create a load balancer for you and provision that resource for you with the IP, the port, and traffic rules that you decide on.

A load balancer service has two main use cases:

  1. accept ingress traffic from the internet: provisions a load balancer infra with a stable, public IP address that can accept ingress traffic and forward it to the rest of the cluster.
  2. distributes traffic across pods: automatically distributes incoming ingress traffic across all available pods no matter which node they live in.

Here are the steps to create a basic load balancer service

  1. Run minikube tunnel to expose your kubernetes cluster to localhost internet
  2. Create this yaml of a service, targeting the specific pod you want to proxy traffic to via spec.selector.<selector-tag> property.
---
apiVersion: v1
kind: Service
metadata:
  name: demo-service
  namespace: development
spec:
  selector:
    app: pod-info
  ports:
    - port: 80
      targetPort: 3000
  type: LoadBalancer
  1. Create the associated pod and apply the service
  2. Visit localhost:80 to see your pod running via a service

ExternalName​

The ExternalName type service redirects traffic from a target pod to an external DNS address that's typically outside your server.

apiVersion: v1
kind: Service
metadata:
  name: database-service
  namespace: prod
spec:
  externalName: ://database.com
  type: ExternalName

Connecting to services​

Accessing ClusterIP services​

ClusterIP services have private IP addresses within the cluster, so they are not accessible unless you are within the cluster itself.

You can do this by SSHing into the cluster, or if using minikube, just use this command:

minikube ssh
Intra-pod / Inter-container communication​

Containers within the same pod can request each other through localhost on their respective exposed ports since a pod acts like a VM in this case.

Pod to pod communication​

If you create a ClusterIP service for a deployment or pod, that Kubernetes will automatically generate environment variables which let you dynamically request the service DNS/IP for connecting to that pod.

The environment variable for the cluster-facing internal service IP is in this format:

<SERVICE_NAME>_SERVICE_HOST

For connecting to a service DNS, you can use this naming syntax instead:

http://<service-name>.<namespace>
http://<service-name>.<namespace>.svc.cluster.local

NOTE

Forwarding ingress traffic to a service via DNS is much more stable than the auto-generated environment variables.

  • service-name: the name of the service
  • namespace: the namespace the service belongs to
  • svc.cluster.local: represents a service being created in a local cluster.

CLI​

Services help expose your containers to the outer world via port forwarding, doing either forwarding to your localhost or to a DNS mapping to a registered domain name.

  • kubectl get services: lists all services
  • kubectl delete service <service-name> : deletes the specified service
  • kubectl describe service <service-name> : gets detailed info about the specified service.
port forwarding​

You can do port forwarding from a service to a port on your laptop if you didn’t already do nodePort forwarding on your service:

kubectl port-forward service/<service-name> <nodeport>:<serviceport>
  • service-name: the name of the service to port forward
  • nodeport: the port number on your local machine you want to forward to
  • serviceport : the port on the service that is currently being exposed and that you want to forward.

Ingress for cluster​

By default, pods can only respond to requests that come from other pods within the same cluster, meaning that by default, the highest amount of networking power is intra-cluster communication.

If you want external internet traffic to be able to request resources in your cluster like pods via a DNS or IP address, then you need to add ingress to the cluster, which can be achieved in two possible ways:

An Ingress is simply a Kubernetes object that describes:

"How should external traffic reach my application?"

  • Method 1 - Ingress with ingress controller: ingress defines the routing reverse proxy rules, ingress controller provisions load balancer or public IP infra to handle public ingress and then distributing it via the ingress rules.
  • Method 2 - Load balancer service: provisions a load balancer with a public IP address that then accepts internet ingress traffic and forwards that traffic to targeted pods.

An ingress acts as a reverse proxy for your K8S cluster resources, which provides these benefits:

  • granular routing rules: forward traffic depending on matching route paths
  • prevent publicly exposing services: you can route to cluster-internal services, which in turn forward traffic to private K8S resources.
  • easier than multiple load balancers: You should use ingress if you have more than one public load balancer service in your cluster, which makes forwarding ingress traffic to the cluster better by acting as a single entrypoint for the cluster, removing the need for multiple load balancer services within your cluster.

Services vs ingress​

  • services: used primarily for internal load balancing between pods, providing stable IP and DNS addressing for inter-pod and intra-cluster communication.
Pod1
Pod2
Pod3
  ^
  |
Service
  • ingress: Handles external ingress traffic and then routing that via reverse proxy. It sits above services.
Internet
   |
   v
Ingress
   |
   v
Service
   |
   v
Pods
  • load balancer service: a special type of service which provisions a load balancer infra with a public IP, enabling it to receive ingress traffic and then reroute it via intracluster traffic to pods that are targeted by that service.
Pod1
Pod2
Pod3
  ^
  |
Load balancer Service <---- internet ingress traffic

Therefore there is a key difference between ingress and services:

  • ingress is a reverse proxy: One important thing to note is that an Ingress is NOT a load balancer service. It simply is a collection of routing rules, like a reverse proxy. It does not do any load balancing across target services.
  • services load balance: the only purpose of services is to forward traffic to target pods and load balance the traffic across all targeted pods.

Another key difference:

  • ingresses target services: Ingresses forward traffic to IP addresses or DNS names, which is suited for targeting services.
  • services target pods: services target pods via labels and selectors and load-balance distribute the traffic to those pods.

Ingress and services work together.

  1. Ingresses act as a reverse proxy to redirect the traffic granularly to target services.
  2. Services act as a load balancer to distribute traffic among the targeted pods.

Creating an ingress and ingress controller​

We can achieve the ingress method by adding an Ingress and an Ingress controller K8S resource:

  • Ingress: A Kubernetes ingress object is a resource that defines rules for routing external HTTP or HTTPS traffic to services within your cluster. It essentially specifies how requests should be directed based on hostnames or paths.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: game-ingress
spec:
  rules:
  - host: game.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service-2048
            port:
              number: 80
  • Ingress Controller: The ingress controller, on the other hand, is the software that enforces these rules. It provisions the infra to act as a reverse proxy and load balancer, receiving incoming traffic and routing it according to the ingress object’s rules.

NOTE

While ingress objects are built into Kubernetes, ingress controllers are separate, pluggable components that you need to install (like Ingress-Nginx or Traefik).

NOTE

So, the ingress object sets the rules, and the ingress controller makes those rules happen by managing the traffic flow into your cluster.

NOTE

An Ingress controller in Kubernetes is a reverse proxy deployed inside your cluster that manages and routes HTTP traffic coming into the cluster. It acts like a gatekeeper, directing requests based on rules.

The Ingress resource, on the other hand, is a Kubernetes API object where you define those routing rules—such as which paths or hostnames should route to which services.

Essentially, the Ingress resource specifies the desired traffic routing, and the Ingress controller enforces those rules by handling the actual traffic flow. This setup allows you to manage complex HTTP routing for multiple services through a single external entry point.

Here's how an ingress request to your cluster works at a high level

  1. An external HTTP request sends ingress traffic to a cluster IP address the Ingress object made.
  2. The ingress object sends that traffic to the ingress controller.
  3. The ingress controller checks the list of rules that you set up on your ingress, and routes traffic to the appropriate pod.
Declarative ingress basics​

Let's create a real production examples based on three microservices: 1) frontend 2) orders 3) payments, trying to achieve this routing scheme:

https://app.example.com
        |
        +--> /
        |      -> frontend-service
        |
        +--> /orders
        |      -> orders-service
        |
        +--> /payments
               -> payments-service

Create an ingress with the appropriate routing rules:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: microservices-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          service:
            name: frontend

      - path: /orders
        backend:
          service:
            name: orders

      - path: /payments
        backend:
          service:
            name: payments
WIth NGINX​

The Ingress object in kubernetes allows you to define the ingress rules as well as the specific third-party ingress controller to use for the reverse proxy functionality of the ingress controller.

  1. Use the NGINX ingress controller
  2. Define the available DNS host for ingress as lil-microservices.com
  3. On HTTP requests, accept all /* matching routes and redirect that to the frontend-ui service.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: lil-microservices.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-ui
                port:
                  number: 8080

Deployments in depth​

A deployment is a single deployment unit of a microservice, which creates all the pods necessary for that microservice.

Declarative deployments​

In this example below, we create a deployment named pod-info-deployment in the namespace development which contains one pod built from the aadilmallick/pod-info-app:latest docker image.

--- 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-info-deployment
  namespace: development
  labels:
    app: pod-info
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pod-info
  template:
    metadata:
      labels:
        app: pod-info
    spec:
      containers:
      - name: pod-info-container
        image: aadilmallick/pod-info-app:latest
        ports:
        - containerPort: 3000
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name # pod-info-deployment
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace  # development
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP # resolved at runtime

Imperative deployments​

Here is the basic crud:

  • kubectl get deployments : gets all deployments
  • kubectl describe deployment <deployment-name> : gives more info on the specified deployment
  • kubectl delete deployment <deployment-name> : deletes the specified deployment

create deployments

Deployments are a grouping of pods, and in deployments you describe how to create the pods through a yaml file, but you can also create them imperatively (not recommended)


kubectl create deployment <deployment-name> \
--image=<image-name> \  # image to create container (1-container pod)
--replicas=3 \    # the number of pods in the replica set
--port=80         # the port to run the container on

fetching deployments

You can get deployments with the kubectl get deployment command, which by default will look in the default namespace. To specify the namespace to search in, use the -n command:

kubectl get deployment # gets all deployments in the default namespace
kubectl get deployment -n "nginx" # gets deployments in the "nginx" namespace

Once you get a deployment, you can describe it with the kubectl describe deployment command, making sure to specify the namespace if the deployment belongs to a namespace

kubectl describe deployment DEPLOYMENT_NAME_HERE

deleting deployments

To delete a deployment, use the kubectl delete deployment command, which will automatically delete and stop all pods within that deployment. This command also needs to be namespaced if the deployment is tied to a namespace.

kubectl delete deployment DEPLOYMENT_NAME_HERE

Volumes​

There are three types of volume resources that are used in kubernetes:

  • volumes: these are normal pod-specific volumes that are tied to a pod and their lifecycle, meant for storing data across containers within a pod and surviving pod and container restarts.
  • PersistentVolumes: volumes that are available throughout the cluster and can be attached to any pod but only through a Persistent Volume Claim.
  • PersistentVolumeClaims: PVCs for short, these are one-to-one mappings of PersistentVolumes to filepaths on a container. Pods use PVCs in order to use a volume in their container(s).

Volumes​

Volumes are attached to a single pod, coupled with the pod lifecycle, and are intended as persistent data storage for all containers within the same pod.

Here are the rules:

  1. Volumes live at the pod-level, so they survive container restarts and removals, but not pod destruction. Once a pod is deleted, the volume also gets deleted.

Kubernetes volumes are different than Docker volumes in terms of their flexibility and use cases:

Kubernetes volumesDocker volumes
Storage typesSupports many different drivers and types, like cloud storageSupports only local storage on a host filesystem
PersistenceVolumes persist for pod lifetimeVolumes persist until manually cleared
Container storageVolumes survive container restarts and removalsVolumes survive container restarts and removals

These are the available volume types:

  • emptyDir: the default. This creates an empty directory on the node the pod lives in, and all data is persisted to that folder on the node, but only for the lifecycle of the pod.
    • tradeoff: is simple, but is temporary storage that is node-specific and pod-specific (lives only during the lifetime of a single pod)
  • csi: the CSI (container storage interface) is an abstraction over concrete storage classes like AWS EBS, EFS, etc.
  • hostPath: The hostPath volume option mounts a file or directory from the host node's filesystem into your pod
    • tradeoff: provides persistent storage that multiple pods can access if they live in the same node, but data is node-specific, and is not shared across pods in different nodes.

NOTE

emptyDir vs hostPath

  • Use emptyDir for temporary storage that is only needed for the lifecycle of a single pod.
  • Use hostPath for accessing and sharing specific files or directories from the host machine, useful for multiple pods using the same volume.
emptyDir​
  1. Definition: An emptyDir volume is created when a pod is assigned to a node and lasts for the duration of that pod's lifecycle.
  2. Storage Location: It is stored on the node’s filesystem, but it is temporary and only exists as long as the pod is running.
  3. Use Case: Ideal for data that needs to be shared among containers in a pod or temporary data that should not persist beyond the pod's lifecycle (e.g., scratch space for container processes).
apiVersion: apps/v1
kind: Deployment
metadata:
  name: story-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: story
  template:
    metadata:
      labels:
        app: story
    spec:
      containers:
        - name: story-container
          image: academind/kub-first-app:2
          # 2. mount volume to container at path
          volumeMounts:
	        # 3. set this to the WORKDIR of your docker file
		    - mountPath: /app/story
	# 1. attach volume to pod
      volumes:
        - name: story-volume
          emptyDir: {}
hostPath​

The hostPath volume option mounts a file or directory from the host node's filesystem into your pod.

  1. Definition: The hostPath volume mounts a file or directory from the host node’s filesystem into the pod.
  2. Storage Location: It directly uses the path on the host's filesystem, allowing pods to access or store data there.
  3. Use Case: Useful for scenarios where you want to share data between the host and the pod, or if you need access to specific files on the host system. However, using hostPath can lead to portability issues if the paths differ across environments.

NOTE

You can think of hostPath as a bind mount from a node's filesystem into your container.

WARNING

hostPath only stores data on a single node, so be aware of that, that this approach will break on multi-node clusters, because if your pod gets reschedule to a different node, it will read data from a different node filesystem and thus different volume.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: story-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: story
  template:
    metadata:
      labels:
        app: story
    spec:
      containers:
        - name: story-container
          image: academind/kub-first-app:2
          # 2. mount volume to container at path
          volumeMounts:
	        # 3. set this to the WORKDIR of your docker file
		    - mountPath: /app/story
	# 1. attach volume to pod
      volumes:
        - name: story-volume
          hostPath:
	          path: /data # persist to /data path on node's filesystem
	          type: DirectoryOrCreate # upsert directory
nfs​

The nfs storage type lets you use a separate network file system as a volume.

csi​

The CSI (container storage interface) is an abstraction over storage volume types that you can implement a concrete provider for.

NOTE

CSI was created to streamline the integration of various storage providers into Kubernetes. Instead of continually adding new built-in volume types for every storage solution, Kubernetes exposes a standard interface that any storage provider can implement.

With CSI, you can use any storage system that has a CSI driver developed for it. This includes cloud storage solutions like Amazon Elastic File System (EFS), Google Cloud Persistent Disks, and many others.

Developers can create their own CSI drivers to integrate their specific storage solutions with Kubernetes, enhancing the system's versatility.

Persistent volumes​

Persistent volumes are independent of a pod's lifetime.

PersistentVolumes are the actual volume resources you have to create before you can use PVCs to actually claim storage from those volumes and use them in your pods.

There are two ways to create/user persistent volumes:

  • Static provisioning for Persistent Volumes (PVs) means that a cluster administrator manually creates and configures the storage resources ahead of time.
  • Dynamic provisioning, on the other hand, allows Kubernetes to automatically create a new PV when a PVC is made, if there isn’t already a suitable PV available.
Declarative​

Here are the possible keys you have in the yaml definition:

  • accessModes: determines the behavior of the pods reading and writing to the volume: This must match the PVC access mode behavior.
    • ReadWriteMany: the volume can be mounted by many pods, and all of them have read and write access to the volume
    • ReadMany: the volume can be mounted by many pods, and all of them have read-only access to the volume
    • ReadWriteOnce: the volume can be mounted by many pods, but only one of them will have read and write access to the volume while the rest will have read-only access.
  • persistentVolumeReclaimPolicy: determines the behavior of what happens to the data upon the PersistentVolume K8S resource deletion
    • Retain: even if the PersistentVolume K8S resource
  • storageClassName: the StorageClass resource to target to specify the storage driver.
  • volumeMode: the data storage type of the volume
    • FileSystem: stores data in a hierarchical filesystem manner.

Then you have these different volume types:

  • node volume storage: persisting data to the node's filesystem (the node that the pod lives in)
    • hostPath: allows you to host the volume in your local laptop through docker desktop, although this will nto work in production, and instead you should use a cloud-provided storage plugin.

PVCs​

Besides just being a way for pods to use PersistentVolumes, PVCs are claims on certain amounts and thresholds of storage, and can be released.

PVCs describe a pod's request for a persistent volume storage, which includes details like storage amount, type of storage, and access mode.

There are 4 possible states a PVC can be in related to the PV it wants to claim (via labels):

  • Available: the volume is not yet bound to a claim
  • Bound: the volume is bound to a claim
  • Released: the claim is deleted, but the resource is not yet reclaimed by the cluster
  • Failed: PV has failed its automatic reclamation

NOTE

If dynamic provisioning isn’t enabled and no suitable static PV exists, the PVC will remain in a pending state until a matching PV is available.

There are all policies for what happens to the volume data mounted in a pod when a PVC is released

  • delete: deletes the data from the pod once the PVC is released. This is the default, so be careful.
  • retain: this retains the data in the pod even if the PVC is released.

Here is how to use PVCs declaratively:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-health-dashboard-pvc
  namespace: health-dashboard
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-local-path
  resources:
    requests:
      storage: 1Gi
  • accessModes: determines the behavior of the pods reading and writing to the underlying volume of the claim:
    • ReadWriteMany: the volume can be mounted by many pods, and all of them have read and write access to the volume
    • ReadMany: the volume can be mounted by many pods, and all of them have read-only access to the volume
    • ReadWriteOnce: the volume can be mounted by many pods, but only one of them will have read and write access to the volume while the rest will have read-only access.
  • storageClassName: the storage class name (if exists) to bind the PVC to
Connecting Pods to PVC​

Once you create a PVC, a PVC is useless without allowing pods to access that PVC and by extension the underlying PV the PVC has claimed. Once pods gain access to a PVC, the containers within those pods will be able to use the underlying volume for storage.

To attach a persistent volume for use to a pod, follow these steps:

  1. Create a PV with static provisioning or dynamic provisioning
  2. Create a PVC that targets a PV for use

  1. To connect a PVC to a pod so its containers can use the volume, add the volumes key on the pod spec so you get this flow:
POD uses PVC uses PV

In summary, here are the steps to allow pods to access a PVC:

  1. Create a persistent volume claim with a unique name
  2. In the pod template in your deployment, create a volumes key, where you can specify volumes based on two keys:
    • name: the name of the volume you are going to create.
    • persistentVolumeClaim: takes in a claimName property, which should be set to the name of the PVC you want to use.
  3. In the containers specification, you can mount volumes in each container through the volumeMounts property, which takes in these two keys to mount a volume:
    • name: the volume name to use, defined in the volumes key in a deployment.
    • mountPath: where to mount the volume in the container’s filesystem.

Here's the YAML:

  1. Create a persistent volume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-tier-volume
spec:
  capacity:
    storage: 1Gi # 1 gibibyte
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore: 
    volumeID: INSERT_VOLUME_ID # replace with actual ID
  1. Create a persistent volume claim for that volume
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-tier-volume-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 128Mi # 128 mebibytes
  1. Create a deployment whose pods request that PVC for use
apiVersion: apps/v1 # apps API group
kind: Deployment
metadata:
  name: data-tier
  labels:
    app: microservices
    tier: data
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: data
  template:
    metadata:
      labels:
        app: microservices
        tier: data
    spec: # Pod spec
      containers:
      - name: redis
        image: redis:latest
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 6379
            name: redis
        # 2. mount volume from pod to /data on container
        volumeMounts:
          - mountPath: /data
            name: data-tier-volume
      # 1. attach PVC to pod
      volumes:
      - name: data-tier-volume
        persistentVolumeClaim:
          claimName: data-tier-volume-claim

Storage classes​

Storage classes are things that your admin or cloud provider creates that are abstractions over persistent volumes, allowing you to connect your PVC to a storage class directly rather than having to create a PersistentVolume.

Here is the basic flow of using storage classes with PVCs instead of PVs with PVCs:

  1. Define a storage class or use one created by your provider
  2. In your PVC definition, use storageClassName to request storage from the storage class rather than the PV.
  3. Use the PVC as normal in your pods.

Declarative storage classes​

Here's an example of using an AWS EFS CSI storage class:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

Once you create a storage class, that now provides a common interface for creating PVs and PVCs with the same syntax. Here's an example:

  • creating PersistentVolume resources with a storage class: You can create a storage class on the fly via this approach through the spec.csi object.
    • This specific example uses the EFS CSI driver with the specific EFS storage block with id fs-59d14521 to create a storage class called efs-sc, then creates a persistent volume claiming data from that storage class.
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity: 
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-59d14521
  • creating PersistentVolume resources with a storage class: you first create a storage class resource then you can create a PVC resource that directly uses that storage class
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi
Imperative storage classes​

You can also imperatively manage storage classes through the sc resource:

Config maps, secrets, environment variables​

Instead of hard coding environment variables inside each of your deployments, it is best practice to separate configuration from implementation. For these purposes we have configMaps and secrets resources in kubernetes:

  • configMaps: store non-sensitive data that can be referenced in pods to extract the the data into environment variables that can be used in containers in a pod.
  • secrets: store sensitive data that can be referenced in pods to extract the the secret data into environment variables that can be used in containers in a pod. The secrets can be encrypted on the server side, or obfuscated at the very least.

Here is how you can give pods access to configuration values and secrets:

  1. create config maps and secrets: create config map and secret k8s resources, which store data in key-value pairs
  2. connect pods to config maps and secrets: pods must reference config maps and secrets to use their data
  3. inject key-value pairs into the containers: you can inject and store the key-value pairs into the containers via different two ways:
    • volume: attach config map and secrets as volume onto pod, then mount that volume to a container.
      • how it works: Each key-value pair becomes a file (key becomes filename, value becomes file content) in the mountpath folder.
    • environment variables: set environment variables on the individual containers of the pod that use the key-value pairs from config maps and secrets to convert them into environment variables.
      • how it works: Each key-value pair becomes an env var and value pair

Config maps​

You specify all key-value pairs under the data key in a config map.

apiVersion: v1
kind: ConfigMap
metadata:
	name: color-config
data:
	color: pink

In a pod, you can extract data from a specified config map by going into the env key on a container, and using the valueFrom key to specify that you want to extract the environment variable from another K8 resource.

You specify configMapKeyRef as telling you want to pull from a config map, which needs two properties:

  • name: the name of the config map resource you want to pull from
  • key: the name of the specific key from the key-value pairs in the config map that you want to pull the value of.

Once you connect a config map resource to a pod, there are two methods to actually injecting the configuration values into the containers:

  • env method: set config map values as environment variables on a container

apiVersion: v1
kind: Pod
metadata:
  name: pink
spec:
  containers:
    - name: pink
      image: mtinside/blue-green:blue
      env:
        - name: COLOUR
          valueFrom:
            configMapKeyRef:
              name: colour-config
              key: colour
  • volume method: first 1) attach the config map as a volume onto the pod, then 2) mount that volume to a container
apiVersion: v1
kind: Pod
metadata:
  name: pink
spec:
  containers:
    - name: pink
      image: mtinside/blue-green:blue
      volumeMounts:
	      - name: color-volume
	        mountPath: /usr/share/nginx/html
  volumes:
    - name: color-volume
      configmap:
	      name: colour-config
Imperative use​

create a config map

kubectl create configmap website --from-file=index.html --dry-run -o yaml

Secrets​

Secrets are used the exact same way as config maps except that they obfuscate plain text to base64, intended for storing sensitive data. Here are the rules:

  1. The control plane encrypts secrets: Kubernetes enhances the security of data in Secrets by encrypting the data at rest within its control plane database, so the sensitive information isn't stored in plain text.
  2. Secrets are never written to disk on the worker node: When Secrets are mounted as files in pods, the data is never written to the worker node's disk, which helps prevent unauthorized access.
  3. secrets are obfuscated with base64: Additionally, Secrets are stored and retrieved in Base64 encoding, adding a layer of obfuscation.

In Kubernetes Secrets, both stringData and data are used to store key-value pairs, but they work differently:

  • stringData: This field allows you to provide secret values as plain (unencoded) strings. When you apply the manifest, Kubernetes automatically encodes these values to base64 and stores them in the data field.
    • It’s a convenient way to write secrets without manually encoding them.
  • data: Here, you must provide values already encoded in base64. Kubernetes stores these values as-is.

NOTE

In summary, stringData is for human-friendly input (plain text), while data is for base64-encoded values. Both end up as base64 in the actual Secret, but stringData saves you the manual encoding step.

Imperative use​

Here is how you can create a secret with an unencoded key=value pair:

kubectl create secret generic <secret-name> --from-literal=<KEY>=<VALUE>

Advanced K8S​

Affinity and pod scheduling​

Pod scheduling​

Pod scheduling refers to how we can control pods being placed in different nodes.

K8S does this automatically via the kube-scheduler component, but we might want extra control as in these use cases:

  • we want a pod to be on a specific node: some pods running ML work would prefer to be on a node with a GPU.
  • pod compute requests: we have specific compute resource requests for a pod that need to be fulfilled by a compatible node that can accomodate that pod's requests.
  • availability: we can choose to place pod replicas in separate worker nodes or even availability zones to maximize availability
  • latency: we can choose to place pods that communicate with each other in the same worker node or availability zone

To control pod scheduling, we use pod affinity and node affinity

Intro affinity​

  • pod affinity: when one pod has an "affinity" for another pod; it wants to live close to that pod.
  • node affinity: when you enable node affinity for a pod, it wants to be scheduled on a specific worker node.

Pod affinity matters if two pods communicate with each other over the cluster network.

Pod affinity has two possible use cases:

  • low latency, different nodes: two pods are scheduled within the same availability zone in different nodes, lowering network latency.
  • same node: two pods are scheduled in the same node, lowering network latency.

The Scheduling Rule Type (Strict vs. Preferred)

You must choose how strictly Kubernetes should enforce your rule using one of these two fields:

  • requiredDuringSchedulingIgnoredDuringExecution (Strict): This is a hard rule. The scheduler must separate the pods. If no separate infrastructure is available, the pod will remain stuck in a Pending state.
  • preferredDuringSchedulingIgnoredDuringExecution (Soft): This is a best-effort rule. The scheduler will try to separate the pods. If it cannot (e.g., you only have one node left), it will break the rule and schedule the pod anyway.

topologyKey (The Boundary Layer)

The topologyKey defines the domain or boundary where you don't want pods to co-exist. Kubernetes looks at the labels on your cluster's nodes to determine this.

  • topology.kubernetes.io/zone: Keeps pods in different Availability Zones (highly recommended for production clouds like EKS).
  • kubernetes.io/hostname: Keeps pods on different physical/virtual nodes. If you have 3 replicas, each will go to a different server.

in summary

Feature PropertyTraditional Pod Anti-AffinityTopology Spread Constraints
Primary GoalTotal separation; ensuring pods do not live together on a domain.Even distribution; balancing workloads proportionally across domains.
Granularity ControlNone. It is binary (allowed vs. not allowed).High. Uses maxSkew to set the exact maximum difference in pod counts between zones.
Unsatisfiable BehaviorStrict failure (Pending) or complete fallback choice.ScheduleAnyway balances as much as possible before dropping constraints safely.
Typical Production FitBest for small single-instance setups (e.g., ensuring a primary database and secondary database run on different hardware).Best for large autoscaling stateless services running dozens of nodes across public clouds.
Pod affinity example​

Here is an example of manually setting a pod's affinity for another:

  • topology key: defines the affinity level
    • kubernetes.io/hostname to put the pods in the same worker node
    • kubernetes.io/zone to put the pods in the same availability zone, but in different worker nodes.

  1. Label the pods you want to relate: Assign a label to the pod(s) you want other pods to be close to. For example, label your Redis cache pod with app=cache.

  2. Define the affinity in your pod spec: In the pod that should be scheduled near the labeled pod, add an affinity section specifying podAffinity.

  3. Use a label selector to find the target pods: Inside podAffinity, use labelSelector with matchLabels to select pods with the label you assigned (e.g., app=cache).

  4. Set the topology key: Specify a topologyKey such as kubernetes.io/hostname to ensure both pods are scheduled on the same node, or use kubernetes.io/zone to place them in the same availability zone.

  5. Apply the configuration: Deploy your pods with these affinity rules. Kubernetes will then try to schedule the pods close together based on your affinity settings.

Pod anti-affinity​

Pod anti-affinity refers to placing pods as far away from each other, where the main use case is to place one pod per availability zone for maximum availability.

To set up pod anti-affinity declaratively, follow these steps:

  1. Label the pods you want to avoid colocating: Assign a label to the pods that you want other pods to stay away from, for example, app=web.

  2. Add an affinity section with podAntiAffinity in your pod spec: This tells Kubernetes that the pod should avoid being scheduled near pods with certain labels.

  3. Use a labelSelector inside podAntiAffinity to select the pods to avoid: For example, select pods with the label app=web.

  4. Set the topologyKey: Use keys like kubernetes.io/hostname to avoid scheduling on the same node, or kubernetes.io/zone to avoid the same availability zone.

  5. Apply the configuration: Deploy your pods with these anti-affinity rules. Kubernetes will try to schedule pods so they are spread out according to your rules.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: web
          image: nginx:alpine
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: frontend # Avoid other instances of myself
              # Distribute across separate availability zones
              topologyKey: "topology.kubernetes.io/zone" 
              maxSkew: 1
  • maxSkew: 1: even out the pod replicas in each availability zone, make sure they all have an equal amount of replicas.

  • topologyKey: topology.kubernetes.io/zone: organize pods across availability zones

Here's another example:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule # Alternative to ScheduleAnyway
      labelSelector:
        matchLabels:
          app: web

Here's another example:

spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          # 1. Enforcement Level: Strict/Hard rule
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  # 3. Label Selector: Target other "web" pods
                  - key: app
                    operator: In
                    values:
                      - web
              # 2. Topology Key: Avoid sharing the same individual machine
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: nginx
          image: nginx:latest

Node affinity​

Node affinity in Kubernetes is a scheduling feature that lets you specify which nodes your pods should run on based on node labels.

NOTE

It allows you to give Kubernetes hints or strict rules about pod placement, especially useful when certain pods need special hardware like GPUs or fast SSDs.

  • For example, you can tell Kubernetes to schedule machine learning pods only on nodes with GPUs by using node affinity.
  • This helps optimize resource usage by matching pods to the right nodes, breaking the usual abstraction where Kubernetes treats all nodes as a uniform pool.
Node affinity examples​

If you have nodes in your company network explicitly labeled with hardware-tier=high-mem to handle heavy resource pools, you can mandate that a database pod only run there.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache-db
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cache-db
  template:
    metadata:
      labels:
        app: cache-db
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
      affinity:
        nodeAffinity:
          # Hard rule enforcement
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: hardware-tier
                    operator: In
                    values:
                      - high-mem

To save costs on non-critical web apps, you can tell the scheduler to favor cheap EC2 Spot nodes, but fall back to baseline On-Demand nodes if Spot capacity is empty.

spec:
  affinity:
    nodeAffinity:
      # Soft rule enforcement
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100 # Highest preference weight (1-100)
          preference:
            matchExpressions:
              - key: ://amazonaws.com
                operator: In
                values:
                  - SPOT

Node anti-affinity​

Node anti-affinity is useful when you want pods to stay of certain nodes, like wanting a web server pod to not be on a ML node.

To prevent pods from being scheduled to a node, we use taints and tolerations

  1. Put a taint on a node, which prevents any pod from being scheduled to that node
  2. Put a toleration on a pod to tolerate a specific taint, allowing the pod to be scheduled to that node.

Service meshes​

A service mesh in Kubernetes is software you install in your cluster that manages all internal service-to-service communication.

  • It handles service discovery, encrypts traffic between pods for security, and provides authentication and authorization options.
  • Additionally, it offers observability tools to monitor the health of your microservices.

NOTE

While it simplifies managing complex microservices architectures by taking over network management tasks, it also introduces a new system you need to learn and manage.

The service mesh interface (SMI) defines a spec of how a service mesh should eb implemented.

Popular service mesh implementations include Istio, Linkerd, and HashiCorp Consul.

RBAC​

A Kubernetes service account is a machine user identity that pods run as inside the cluster.

A service account lives at the pod level, and acts as an IAM role equivalent for pods, containing permissions that decide granularly what a pod process can or can't do when querying the K8S api.

All pods have a default service account with basic RBAC permissions, but you can create new service accounts and attach permissions to those.

NOTE

By default, all pods use the "default" service account, but you can create custom service accounts to give specific permissions only to certain pods.

However, by default, this service account has minimal or no permissions to perform actions on the Kubernetes API. This means pods running with the default service account generally cannot create, modify, or delete Kubernetes resources unless explicit permissions are granted.

Here are what pods with the default service account are able to do:

  • self pod CRUD: able to view its own pod info
  • environment info: able to view environment info like control plane version and the namespace of the pod.

Creating custom service accounts + basic RBAC​

NOTE

Creating a custom service account instead of using the default one for all pods is important for security and control.

  • The default service account is shared by all pods that don't specify one, so granting permissions to it would give those permissions to every pod in your cluster — which can be risky.
  • By creating a custom service account, you can assign only the specific permissions that a particular pod or application needs, following the principle of least privilege.

To assign RBAC permissions to a service account, we need to create these K8S resources:

  • Roles: roles specify a combination of allowed CLI verbs and resources (nouns) to create a list of allowed permissions for that role
  • RoleBinding: role bindings attach roles to service accounts

These are the steps to create a service account and assign RBAC permissions to it:

  1. Create a service account resource
apiVersion: v1
kind: ServiceAccount
metadata:
	name: myserviceaccount
  1. Assign a pod that service account resource you created:
apiVersion: v1
kind: Pod
metadata:
  name: my-service-account
spec:
  serviceAccount: my-service-account
  containers:
    - name: envbin-sa
      image: mtinside/envbin:latest

  1. Define a Role with specific permissions:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
# this role gives kubectl get/watch/list pods permissions from within the pod
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
  1. Bind the role to the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: my-service-account
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

AWS IAM with K8S​

Sidecar​

For decoupled and performant microservices, often multi-container pods are the way to go.

Sidecar with service mesh​

Adding Linkerd as a sidecar to a pod enables advanced networking features by intercepting all network traffic in and out of that pod. This lets you monitor detailed metrics like request rates, success rates, and latency through dashboards such as Linkerd's own and Grafana.

  1. Install linkerd
linkerd install | kubectl apply -f
  1. Check if worked
linkerd check

Kubernetes security​

Basic tips​

Tip 1 - use securityContext​

In terms of pods running containers, there is a huge attack surface on how the containers are running themselves, usually on two fronts:

  • root access: any container that allows root access means that it can be compromised since root can basically do anything.
  • write filesystem: allowing the container to have its file system written to and having the user be root means that root can basically delete the entire container. We want to prevent writing on the container file system, even if we're root.

So here is how we do that on YAML via the spec.containers.securityContext object, which has these boolean flags.

  • allowPrivilegeEscalation: allow users to use sudo to assume root access.
  • runAsNonRoot: if set to true, does not allow running container as a root user to start off with.
  • readOnlyRootFilesystem: if set to true, the root user can only read files in the container filesystem, not being able to write anything.
--- 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-info-deployment
  namespace: development
  labels:
    app: pod-info
spec:
  replicas: 3
  selector:
    matchLabels:
      app: pod-info
  template:
    metadata:
      labels:
        app: pod-info
    spec:
      containers:
      - name: pod-info-container
        image: kimschles/pod-info-app:latest
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          capabilities:
            drop:
              - ALL
          readOnlyRootFilesystem: true
        ports:
        - containerPort: 3000

Tip 2 - use snyk CLI​

snyk is a static vulnerability analysis tool that also offers a CLI that lets you find out any vulnerabilities of code files.

snyk iac test <k8s-yaml-file>

Kubernetes and cloud native​

What is Cloud Native?​

Cloud native is a solution that allows organizations to run scalable applications in the cloud, using these components:

  • microservices and containers: microservices running as containers for horizontal scaling and decoupling
  • service meshes: reverse proxy infra to control traffic flow between different services
  • declarative APIs and immutable infrastructure: IaC to declaratively create containers and microservices.

12-factor app​

The 12-factor app is a methodology for creating robust cloud-native solutions.

Here is the 12-factor app with kubernetes:

  1. codebase: one codebase, tracked in version control, with all IaC tracked as well.
  2. dependencies: explicitly declare and isolate dependencies of pods with probes and init containers.
  3. config: store configuration variables in secrets, environment variables, and config maps
  4. backing services: treat backing services like pluggable networking as plugins
  5. build, release, run: strictly separate build and run stages
  6. processes: execute the app as one or more stateless processes: microservices or containers
  7. port binding: export services via port binding
  8. concurrency and parallelism: use replica sets to run multiple copies of the same process
  9. disposability: maximize robustness with fast startup and graceful shutdown. Use probes.
  10. dev/prod parity: keep development, staging, and production as similar as possible via containers and environment variables
  11. logs: treat logs as event streams and store them in the orchestration layer, not the application itself.
  12. admin processes: run admin/management tasks as one-off processes (ansible playbooks, kubernetes jobs)

Cloud native standards​

These interface allow abstraction over networking in orchestration systems:

  • SMI (service mesh interface): describes how to implement service mesh in orchestrated systems.

These interfaces allow abstraction over container plugin implementation details

  • CNI (container network interface): defines how to implement networking for containers
  • CRI (container runtime interface): defines how to implement container runtimes in orchestration systems like Kubernetes
  • CSI (container storage interface): defines how to implement storage for containers in orchestration systems like Kubernetes

Check out [[01-Docker#Container runtimes]] for more info.

Cloud native DNS​

  • service: a networking component with a stable IP address and DNS name
  • service registry: stores the mapping of service DNS name to the IP addresses the service is supposed to forward traffic to

Cloud native systems implementing cloud native DNS should include these components to create a robust service solution:

  • API that makes it easy for services to auto-register.
  • Key-value store that is used to store essential system information and which may be used for service discovery, like Etcd.

Cloud native storage​

Cloud native storage has two different methods for storing data on containers:

  • ephemeral read-writable layer: a container is ephemeral, but has a read-writable layer in the working directory specified by the WORKDIR directive in a container image.
    • All data written to a container within this read-writable layer will be destroyed after a container dies
  • decoupled volume: mounting an external volume onto a container and then writing to that volume will persist the data, since the volume is decoupled from the container.

K8S practice​

Level 1 - Basic microservices​

Create the deployments​

  1. Create a deployment that has a web server pod running on port 3000, and a service that proxies the traffic for that pod forwarding port 3000 on that pod to port 80 on the cluster IP address, via the ClusterIP service type.
--- 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: learning-resources
  labels:
    app: learning-resources
spec:
  replicas: 3
  selector:
    matchLabels:
      app: learning-resources
  template:
    metadata:
      labels:
        app: learning-resources
    spec:
      containers:
      - name: learning-resources-container
        image: kimschles/learning-resources:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 3000
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: learning-service
  labels:
    app: learning-resources
spec:
  selector:
    app: learning-resources
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: ClusterIP
  1. Create deployment that has a long running pod running on port 80, and a NodePort service that proxies traffic for the pod, and forwards traffic from port 80 on the pod to port 30076 on your localhost.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
      - image: kimschles/echo-server:latest
        imagePullPolicy: Always
        name: echo-server
        ports:
        - containerPort: 80
        env:
        - name: PORT
          value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: echo-service
spec:
  selector:
    app: echo-server
  type: NodePort
  ports:
    - name: echo
      port: 80
      targetPort: 80
      nodePort: 30076
      protocol: TCP
  1. Create a deployment that runs a VITE frontend app on port 4173 across a replica set of pods and creates a LoadBalancer service that distributes traffic equally to those pods and forwards traffic on those pods from port 4173 to port 80 on the Cluster IP.
---
apiVersion: v1
kind: Namespace
metadata:
  name: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: frontend
  labels:
    app: frontend-ui
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend-ui
  template:
    metadata:
      labels:
        app: frontend-ui
    spec:
      containers:
      - name: frontend-container
        image: kimschles/frontend:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 4173
        env:
        - name: PUBLIC_K8S_SERVICE_URL
          value: "http://learning-service.default.svc.cluster.local"
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  namespace: frontend
spec:
  selector:
    app: frontend-ui
  ports:
    - port: 80
      targetPort: 4173
  type: LoadBalancer

Create the services​

ClusterIP example

For a ClusterIP service like this:

apiVersion: v1
kind: Service
metadata:
  name: learning-service
  labels:
    app: learning-resources
spec:
  selector:
    app: learning-resources
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: ClusterIP

This is how you connect to it:

  1. Create a BusyBox Pod: Deploy a BusyBox pod in your Kubernetes cluster. This pod will be used to run commands inside the cluster.
  2. Verify Pod is Running: Use kubectl get pods to ensure the BusyBox pod is up and running.
  3. Access the BusyBox Pod: Run kubectl exec -it <busybox-pod-name> -- sh to open an interactive shell inside the BusyBox pod.
  4. Find Pod IP Addresses: In another terminal tab, run kubectl get pods -o wide to list pods and their IP addresses.
  5. Make HTTP GET Request to a Pod: Inside the BusyBox shell, use wget -O- http://<pod-ip>:<port> to make a GET request directly to a pod's IP and port.
  6. Get Service Information: Run kubectl get services to see the available services and their ClusterIP addresses.
  7. Make HTTP GET Request to Service Name: Inside BusyBox, use wget -O- http://<service-name> to make a request to the service by its name.
  8. Make HTTP GET Request to Service DNS Name: Use the full DNS name pattern http://<service-name>.<namespace>.svc.cluster.local with wget inside BusyBox to access the service.

NodePort example

  1. Create a pod that has a container running on port 80 and exposes it
  2. Create a node port service that targets the pod definition and reroutes traffic from port 80 on that pod to port 30076 on localhost.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
      - image: kimschles/echo-server:latest
        imagePullPolicy: Always
        name: echo-server
        ports:
        - containerPort: 80
        env:
        - name: PORT
          value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: echo-service
spec:
  selector:
    app: echo-server
  type: NodePort
  ports:
    - name: echo
      port: 80
      targetPort: 80
      nodePort: 30076
      protocol: TCP
  1. Create a BusyBox Pod: Deploy a BusyBox pod in your Kubernetes cluster to run commands inside the cluster.
  2. Verify Pod is Running: Use kubectl get pods to ensure the BusyBox pod is up and running.
  3. Access the BusyBox Pod: Run kubectl exec -it <busybox-pod-name> -- sh to open an interactive shell inside the BusyBox pod.
  4. Find Pod IP Addresses: In another terminal tab, run kubectl get pods -o wide to list pods and their IP addresses.
  5. Make HTTP GET Request to a Pod: Inside the BusyBox shell, use wget -O- http://<pod-ip>:<port> to make a GET request directly to a pod's IP and port.
  6. Get Node IP Addresses: Run kubectl get nodes -o wide to find the IP addresses of your Kubernetes nodes.
  7. Make HTTP GET Request to Node IP with NodePort: Use wget -O- http://<node-ip>:<nodeport> to access the service exposed via NodePort.
  8. Get Service Information: Run kubectl get services to see the available services and their types.
  9. Make HTTP GET Request to Service Name: Inside BusyBox, use wget -O- http://<service-name> to make a request to the service by its name.
  10. Make HTTP GET Request to Service DNS Name: Use the full DNS name pattern http://<service-name>.<namespace>.svc.cluster.local with wget inside BusyBox to access the service.

load balancer service

Here are the generic steps to replicate the process shown in the video for learning how to deploy and test a LoadBalancer service in Kubernetes:

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  namespace: frontend
spec:
  selector:
    app: frontend-ui
  ports:
    - port: 80
      targetPort: 4173
  type: LoadBalancer
  1. Create a Namespace: Define and apply a YAML file to create a new namespace for your service.
  2. Deploy Backend Pods: Create a deployment YAML to spin up your backend pods with the appropriate container image.
  3. Deploy Frontend Pods: Similarly, create and apply a deployment YAML for your frontend pods.
  4. Create a LoadBalancer Service: Define a service YAML of type LoadBalancer that fronts the frontend pods by matching their labels.
  5. Apply All Configurations: Use kubectl apply -f <filename.yaml> to deploy the namespace, deployments, and service.
  6. Verify Service Creation: Run kubectl get services -n <namespace> to check the service status and note the ClusterIP and External IP (which may be pending in local clusters).
  7. Create a BusyBox Pod in the Same Namespace: Deploy a BusyBox pod to use as a client for testing service connectivity.
  8. Exec into BusyBox Pod: Use kubectl exec -n <namespace> -it <busybox-pod-name> -- sh to open a shell inside the pod.
  9. Test Service Connectivity: Use commands like wget -O- http://<service-name> or the full DNS name http://<service-name>.<namespace>.svc.cluster.local to query the service and verify it returns expected data.
  10. Port Forward to Access Service Externally: Find a pod name with kubectl get pods -n <namespace>, then run kubectl port-forward -n <namespace> <pod-name> <local-port>:<container-port> to forward a local port to the pod.
  11. Open Browser to Localhost: Access the service via http://localhost:<local-port> to see the combined frontend and backend response.

Level 2 - containerizing frontend​

To containerize a Vite React app using Nginx and deploy it as a pod in Kubernetes, you can follow these general steps:

  1. Use a Multi-Stage Build:
    • Build your application with Vite in one stage.
    • Use a second stage to serve the built files with Nginx.
# Stage 1: Build the React app
FROM node:14 AS build
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install
COPY . .
RUN yarn build

# Stage 2: Serve with Nginx
FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
  1. nginx configuration: Create an nginx.conf file in the frontend directory to configure Nginx to serve your React application.
    • Via the dockerfile, this will get copied to the system nginx config in the container, thus serving your frontend on port 80, rewriting all /* matched routes to the frontend.
server {
    listen 80;
    location / {
        root /usr/share/nginx/html;
        try_files $uri $uri/ /index.html;
    }
}
  1. build the docker image and push it to a registry: In order for K8S to use your image, you must build it and then push it to a registry.
docker build -t your-image-name .
docker push your-image-name
  1. create a deployment for the image: Create a deployment for the frontend pod.
    • nginx runs on port 80, so let that be the exposed port.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: your-image-name
        ports:
        - containerPort: 80
  1. make the frontend public: to make the frontend publicly accessible from the internet, create a LoadBalancer service that targets your frontend deployment
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: frontend
  1. Apply the resources
kubectl apply -f frontend-deployment.yaml
kubectl apply -f frontend-service.yaml

Adding reverse proxy​

If you want to create a fullstack app instead of just frontend with NGINX, you should use the proxy_pass reverse proxy functionality in NGINX, and give that the highest priority.

For example, an example nginx.conf could be this:

  1. api rules: anything that matches /api* should be given the highest priority, and proxy pass to a service.
  2. catch all rules: anything else that is not an API request should be redirected to the react frontend app.
server {
    listen 80;
    
    location /api/ {
	    proxy_pass "http://<service_name>.<namespace>:80/"
    }
    
    location / {
        root /usr/share/nginx/html;
        try_files $uri $uri/ /index.html =404;
    }
}

Kustomize and advanced K8S Yaml​

The kustomize tool is a way to combine multiple yaml files describing k8s resources into one so you don’t have to think about individual resources. You simply deploy one kustomize file, and to delete all resources, you simply delete that kustomize file.

You can create a kustomization file which must be named kustomization.yaml like so, specifying the resources you want to include:

kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: health-dashboard
resources:
  - mongo-deployment.yaml
  - server-deployment.yaml

Here are the general steps for integrating kustomize into your workflow:

  1. Run the kubectl kustomize <dir> command, which looks for a kustomization.yaml file in the directory, and if found, outputs the contents of all the yaml files concatenated.
  2. Pipe the output of the kustomize command into a file.
  3. Run that file manually with kubectl apply -f to create resources, and to delete resources, use kubectl delete -f

You can also just do it the declarative way where by naming the file as kustomization.yaml and then using the -k option with kubectl apply:

  • kubectl apply -k <dir>: Looks in the specified directory for a kustomization.yaml, and if found, creates or updates all k8s resources specified in that file.
  • kubectl delete -k <dir>: Looks in the specified directory for a kustomization.yaml, and if found, deletes all k8s resources specified in that file.

Base and overlays​

All yaml should be in a top-level folder called kustomize, and that should have two subfolders:

  • kustomize/base: stores YAML that defines core resources like pods
  • kustomize/overlay: The overlay directory contains environment-specific configurations that build upon the base configurations.

The overlay folder should be divided into local and prod environments with two subfolders:

  • overlay/local
  • overlay/prod

Tilt​

Intro​

Tilt is a program that simplifies k8 development on your local machine. It provides benefits such as restarting deployments, rebuilding containers, doing everything automatically in watch mode, and much more.

Install tilt like so:

curl -fsSL https://raw.githubusercontent.com/tilt-dev/tilt/master/scripts/install.sh | bash
  • tilt version: prints the version of tilt
  • tilt up: runs the TiltFile file in the current directory, running all the k8 resources you specify in the tiltfile.
  • tilt down: deletes all the deployments and resources you created when running tilt up.

Basics​

A TiltFile code is run in Starlark, which is a simplified dialect of python. It looks like this:

# 1. specify which contexts to allow
allow_k8s_contexts([
  'docker-desktop',
]);

# 2. specify which context to set as current context
k8s_context("docker-desktop")

# 3. builds an image
docker_build(
        "health-dashboard-server",  # tag name
        context="server",  # cwd to run `docker build` in
        
        # dockerfile contents
        dockerfile_contents=""" 
        ARG NODE_VERSION=24.0.2

        ######################################
        # Use node image for base image for all stages.
        FROM node:${NODE_VERSION}-alpine

        # install bash
        RUN apk add --no-cache bash

        # Set working directory for all build stages.
        RUN mkdir -p /usr/src/app
        RUN chown -R node:node /usr/src/app
        USER node
        WORKDIR /usr/src/app


        COPY --chown=node:node package.json ./
        COPY --chown=node:node package-lock.json ./

        RUN npm install

        COPY --chown=node:node . ./

        RUN npm run build

        CMD ["npm", "start"]
        """
)

# runs the k8s yaml files declaratively
k8s_yaml("kustomize/base/local/mongo-deployment.yaml")
k8s_yaml("kustomize/base/local/server-deployment.yaml")

Here are functions concerned with the context:

  • allow_k8s_contexts(contexts: [str]): Allows the specified list of context names to be used as valid contexts for running the kubernetes resources.
  • k8s_context(context: str): specify which context to set as current context

And here are other functions:

  • docker_build(image_name: str, **kwargs) : Builds the image with the image name based on either dockerfile contents you provide or the path to the dockerfile.
  • k8s_yaml(yaml_path : str) : runs the k8s resource specified in the yaml filepath. It’s basically just calling kubectl apply -f <yaml-file>, and it watches for changes to that file.

Examples​

Watching files​

env_path = "./server/.env"


allow_k8s_contexts([
  'docker-desktop',
  'test-janelia',
])

k8s_context("docker-desktop")

run("export IN_DEV_MODE=true")

# whenever file at env_path changes, run command
local_resource(
    name="env-file",
    cmd="bash scripts/tasks/initk8resources.sh",
    deps=[env_path]
)
watch_file(env_path)

docker_build(
        "health-dashboard-server", 
        context="server", 
        dockerfile="server/Dockerfile.prod",
)

# use kustomize to combine multiple YAML files into one.
combined_yaml_file = kustomize(
  "kustomize/overlay/local",
)

# run kubectl apply -f on the kustomize file
k8s_yaml(combined_yaml_file)

# define port forwarding for a service
k8s_resource(
  workload="health-dashboard-server-deployment",
  port_forwards="30000:3000"
)

k8s_resource(
  workload="mongodb",
  port_forwards="31017:27017"
)
  • run(command: str): lets you run a linux command that will persist in the shell session.

Kubernetes add-ons​

K8S Lens IDE​

The K8S Lens IDE offers a dashboard GUI way to manage a k8s cluster, whether you're running it locally or on the cloud.

You can install like so:

# windows
choco install lens

K9s​

K9s is a TUI for visualizing your k8s cluster and managing it.

Install like so:

brew install k9s

Helm​

Intro​

Helm is the package manager for kubernetes. It simplifies managing complex deployments by letting you define your app and its dependencies in reusable, versioned packages.

Helm is a tool used in Kubernetes to package, configure, and deploy applications using charts, which are collections of Kubernetes YAML manifests.

NOTE

Helm is the package manager for k8s, which lets you install a chart - package of k8s manifests - to create resources with a single command from a single chart.

Installation​

  1. install
brew install helm
  1. View verison
helm version

CLI​

Here is a brief reference:

  • helm search: search for charts
  • helm pull: download a chart to your local directory to view the manifests of that chart.
  • helm install: install a specific chart and apply it to your cluster
  • helm list: list the charts you have
  • helm delete

Most commands will deal with a specific chart, which you can specify in different ways:

  • OCI url: a unique URL with a OCI protocol that lets you install a chart via a URL
  • repo with chart: specified uniquely in a <repo>/<chart> syntax where:
    • repo refers to the specific github or artifact hub repo
    • chart refers to the chart name

helm install​

Installs and applies a chart, giving it a custom release name if you want.

helm install [RELEASE_NAME] <chart>

Here are the flags:

  • --namespace/-n <namespace>: specify the namespace to apply helm to.
  • --version <version>: specify the version of the chart you want to install

helm delete​

To uninstall/delete the cert-manager deployment:

helm delete cert-manager --namespace cert-manager

helm upgrade​

helm upgrade [RELEASE_NAME] <chart>

This is how you upgrade a chart from artifact hub to a specific version:

helm upgrade cert-manager <repo-name>/<chart-name> --version <version>

helm env​

Helm has a bunch of environment variables prefixed with HELM_ which configure the behavior of Helm.

You can list the current values of these env vars with the helm env command:

helm env

helm lint​

Used for linting chart files

helm lint <chart-name>

helm template​

The helm template command shows you the changes that have been made in your files.

helm template <chart-name>

You can also add these additional flags:

  • --debug: adds debugging
  • --validate: validates that chart config is correct

Chart management​

Installation quick guide​

Here's a quick guide:

helm install \
  cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.21.2 \
  --set crds.enabled=true

Here's a step-by-step guide to install cert-manager using Helm:

  1. Add the Jetstack Helm repository:
    Run helm repo add jetstack https://charts.jetstack.io to add the repository that contains the cert-manager chart.

  2. Update your Helm repositories:
    Run helm repo update to ensure you have the latest charts.

  3. Create a namespace for cert-manager (optional but recommended):
    Run kubectl create namespace cert-manager.

  4. Install the cert-manager Helm chart:
    Run helm install cert-manager jetstack/cert-manager --namespace cert-manager --version <version>
    Replace <version> with the desired cert-manager chart version or omit it to use the latest.

  5. Verify the installation:
    Use helm list -n cert-manager to see the installed release.
    Also, check the pods with kubectl get pods -n cert-manager to ensure they are running.

editing values​

The values.yaml file in a Helm chart is a YAML-formatted configuration file that holds default settings and parameters for the application you want to deploy on Kubernetes. It defines values that Kubernetes will use to create objects like pods, services, and configurations.

  • For example, it can include settings like enabling RBAC, priority class names, and probes for liveness and readiness.
  • This file is the main way to tailor a Helm chart to your specific Kubernetes environment.

How to see the values of a chart you installed:

helm show value <repo-name>/<chart-name>

rolling back charts​

What if you want to roll back a chart to a specific version or release?

First, check the history of that chart:

helm history <chart-name>

Now you have these options for actually rolling back the chart:

  • rollback to the immediately previous revision
helm rollback  <chart-name>
  • rollback to a specific revision: specify a revision number to roll back to.
helm rollback  <chart-name> <revision-number>

Creating helm charts​

You can create custom helm charts and then upload them to artifact hub.

First, create boilerplate helm chart files:

helm create <chart-name>

This creates a new directory with the following files and folders:

<chart-name>
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── httproute.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml
  • Chart.yaml: This is the package.json equivalent for helm, where you can configure the version and name of the package
  • values.yaml: configure the resources the chart creates
    • templates: stores K8S resources that make up the chart, and will be processed through the Helm templating engine to inject values from the values.yaml at chart installation time.

Chart.yaml​

apiVersion: v2
name: blue-green
description: my first chart
version: 1.0.0 # chart version

Values​

You can create semantically-named values-dev.yaml and values-prod.yaml for environment-specific injecting different values for the same variables.

values-dev.yaml
host: dev.example.com
tls:
	enabled: false

Then you can create the helm template YAML with the specific values file you select injecting those variables and baking them into the YAML:

helm template -f values-dev.yaml <chart-dirpath>

Then after that, you can install the chart with the baked in values:

helm install -f values-dev.yaml [RELEASE_NAME] <chart-dirpath>

Templates​

You can delete all the K8S resources from the templates/ folder, and then to start fresh, you can then add the K8S resources you want to add to the chart within the templates/ folder.

templates/config_map.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: first-chart-configmap
data:
  port: "8080"
  allowTesting: "true"

Local testing​

You can then perform local testing by installing the chart directory:

helm install [RELEASE_NAME] <path-to-chart-dir>

You can then update for local testing using helm upgrade in the same way:

helm upgrade [RELEASE_NAME] <path-to-chart-dir>

Kubeflow​

Kubeflow simplifies the deployment of machine learning workloads on Kubernetes, offering a complete stack for model building, training, and serving.

KNative​

Knative is a platform for building and managing serverless workloads on Kubernetes, providing portability and avoiding vendor lock-in, supported by major industry players.

KOPS​

kOps is a cloud-agnostic way to deploy a kubernetes cluster to different cloud providers, like EKS, AKS, or GKE.

Instead of manually setting up virtual machines and installing Kubernetes, kOps automates much of this process, creating the cluster configuration and launching the necessary cloud resources for you

Installation and setup​

  1. Install
brew update && brew install kops

Kubernetes admin​

Kubernetes admin refers to installing a Kubernetes cluster and managing it on a generic Linux device or VM.

Because Kubernetes is pretty heavy, each machine that runs a Kubernetes cluster must have these prerequisites:

  • 2 GB RAM per machine
  • 2 vCPUs for the control-plane node

Machine setup​

To run both the control plane and workloads, provision an EC2 instance meeting the video’s minimum requirements:

  • AMI: Ubuntu Server 24.04 LTS or 22.04 LTS

  • Instance Type: t3.medium minimum (2 vCPUs, 4 GiB RAM; kubeadm will abort on fewer than 2 vCPUs)

  • Storage: 20–30 GiB gp3

  • Security Group Inbound Rules:

    • 22/TCP (SSH from your IP)

    • 6443/TCP (Kubernetes API server)

    • 80/TCP & 443/TCP (HTTP/HTTPS web traffic)

    • 30000-32767/TCP (NodePort range, if exposing directly via NodePort)

Enable modules on boot​

SSH into your instance (ssh -i key.pem ubuntu@<EC2_PUBLIC_IP>). Kubernetes requires bridge networking packet traversal and the overlay filesystem

# Enable modules on boot
sudo tee /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF

# Load immediately into running kernel
sudo modprobe overlay
sudo modprobe br_netfilter

# Enable bridge netfilter and IPv4 packet forwarding for CNI routing
sudo tee /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

disable swap​

kubelet requires swap memory to be disabled so resource limits and QoS classes remain strictly deterministic

sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab