KIA CH03 Pods: running containers in Kubernetes

Hi there.

Today, let us read Chapter 03: Pods - running containers in Kubernetes (Part II: Core Concepts) of Kubernetes in Action.

1. Creating, running, and stopping pods
2. Organizing pods and other resources with labels
3. Performing an operation on all pods with a specific label
4. Using namespaces to split pods into non-overlapping groups
5. Scheduling pods onto specific types of worker nodes

3.1 Introducing pods

a pod is a co-located group of containers and represents the basic building block in Kubernetes.

• a pod may include one or more containers
• multiple containers in a pod must remain in the SAME worker node; a pod never spans multiple worker nodes.

3.1.1 Understanding why multiple containers are better than one container

Running each process in its own container (multiple containers) is better than combining processes in one container.

• (easy recovery) only restart the crashed process rather than all processes
• (easy management) logs are only kept for the single process in each container

3.1.2 Understanding pods

A pod allows you to run closely related processes together, and provide them with almost the same environment (as if they were all running in a single container) while keeping them partially isolated.

1. the partial isolation among containers inside the same pod.

   • all containers of same pod share the same set of Linux namespaces

2. containers in the same pod share the same IP and Port space

   • containers in same pod run in the same Network namespace; containers in different pods run in different Network namespaces
   • containers in same pod may NOT to bind to the same port numbers, to avoid port conflicts

3. the flat inter-pod network

   • All pods (in a Kubernetes cluster) reside in a single flat, shared, network-address space; every pod can access other pod by the IP address of the target pod.
     • achieved through a SDN layer on top of like computers on a LAN
     • no NAT gateways exist
   • Regardless of which worker nodes the pods are scheduled

@startuml
caption "Figure 3.1  Pods in a flat inter-pod network"
nwdiag {
    group {
        description = "Node 1";
        color = "#BAC3E2";
        Pod1;
        Pod2;
    }
    group {
        description = "Node 2";
        color = "#BAC3E2";
        Pod3;
        Pod4;
    }
    network inter_pod_network {
        address = "10.1.0.0/16";
        description = "Flat inter-pod network";
        Pod1 [address = "10.1.1.6"];
        Pod2 [address = "10.1.1.7"];
        Pod3 [address = "10.1.2.5"];
        Pod4 [address = "10.1.2.7"];
    }
}
@enduml

3.1.3 Organizing containers across pods

1. for tightly related components or processes that must work as a single whole, organize them into the same pod

   • E.g., a main process and other complementary processes

2. for multi-tier app that can run on different hosts, organize apps into multiple pods

   • different pods, deployed on different worker nodes, can run on different worker nodes, thereby improving the utilization of computational resources.
     • E.g., a multi-tier application, with a frontend server and a backend database, should be configured as different pods so that they can be deployed on different worker nodes.

3. for individual scaling of independent components that have different scaling requirements, organize apps into multiple pods.

   • A pod is the basic unit of scaling.

3.2 Creating pods from YAML/JSON descriptors

To create pods (and other Kubernetes resources), post a JSON or YAML manifest to Kubernetes REST API endpoint.

3.2.1 Examining a YAML descriptor of an existing pod

Use "kubectl get pods" with flag -o yaml to get the whole YAML definition of the pod.

1kubectl get pods kubia-zxzij -o yaml

• Metadata: name, namespace, labels, etc.
• Spec: actual description of the pod’s contents, e.g., containers, volumes, etc.
• Status: current information about the running pod, e.g., condition the pod, description and status of each container, internal IP, etc.

Note: status is not required during creating a new pod.

3.2.2 Creating a simple YAML descriptor for a pod

Kubernetes reference documentation and "kubectl explain" command list possible fields in a manifest.

 1#  kubia-manual.yaml
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5  name: kubia-manual
 6spec:
 7  containers:
 8  - image: ${yourDockerRegistryID}/kubia
 9    name: kubia
10    ports:
11    - containerPort: 8080
12      protocol: TCP

Note: use two spaces, NOT tab.

We need to explicitly defining ports in YAML file:

• let others know which ports your cluster is exposing
• assign a name to each port

3.2.3 Using "kubectl create" to create the pod

Use "kubectl create" with flag -f to create pod from YAML/JSON file.

1kubectl create -f kubia-manual.yaml

Use "kubectl get" to get full info about the pod, in the format of YAML/JSON:

1kubectl get pods kubia-manual -o YAML
2kubectl get pods kubia-manual -o JSON

3.2.4 Viewing application logs

In container runtime, i.e., Docker, use "docker logs" to redirect stdout and stderr streams to files

1docker logs ${ContainerID}

use "kubectl logs" to get logs of all containers or of the specific container of the pod:

1# get all logs in pod `kubia-manual`
2kubectl logs kubia-manual
3
4# get logs only in container `kubia` of pod `kubia-manual` 
5kubectl logs kubia-manual -c kubia 

3.2.5 Sending requests to the pod

Use "kubectl port-forward" command to forward a local network port to a port in the pod.

1kubectl port-forward kubia-manual 8888:8080

Node: Port forwarding is a convenient way of testing single pod.

• "kubectl port-forward" command forwards local port 8888 to port 8080 in pod
• use "curl localhost:8888" command to access the service in pod.

@startuml
caption "Figure 3.2  Kubectl Port-forwarding"
left to right direction
node "Local Machine" as local {
    rectangle curl
    rectangle "kubectl port-forwarding" as kubectl
}
node "Kubernetes Cluster" {
    rectangle "Pod: kubia-manual" as pod
}
curl-->kubectl: Port 8888
kubectl-->pod:  Port 8080
pod ..> kubectl
kubectl ..> curl
@enduml

3.3 Organizing pods with labels

Background: When running multiple microservices, pods structure can be complicated:

• multiple pods
• multiple replicas of each pod
• different releases of the same microservice

Kubernetes uses labels to organize all sorts of Kubernetes resources: pods and other objects.

3.3.1 Introducing labels

A label is an arbitrary key-value pair attached to a resource, which is then utilized when selecting resources using label selectors.

A resource can have more than one (unique) labels.

A typical practice for labelling:

• label app: which app, component, or microservice the pod belongs to
• label rel: which release the pod is - E.g., stable, beta, and canary (only a small fraction of user hit the new version, before rolling new version out to all users)

3.3.2 Specifying labels when creating a pod

Add "labels" to YAML to add labels to the pod

 1# kubia-manual-with-labels.yaml
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5  name: kubia-manual-v2
 6  # add labels to the pod
 7  labels: 
 8    creation_method: manual 
 9    env: prod
10spec:
11  containers:
12  - image: ${yourDockerRegistryID}/kubia
13    name: kubia
14    ports:
15    - containerPort: 8080
16      protocol: TCP

Note: this YAML added two labels

• creation_method
• env

To create pods, run with "kubectl create"

1kubectl create -f kubia-manual-with-labels.yaml

Use "kubectl get" to show labels:

• kubectl get pods --show-labels: show all labels
• kubectl get pods -L creation_method,env: only show labels named creation_method and env

3.3.3 Modifying labels of existing pods

To add a new label to existing pod, use "kubectl label"

1kubectl label pods kubia-manual-v2 creation_date_format=yyyyMMdd
2kubectl label pods kubia-manual-v2 creator=admin

To overwrite an existing label of the pod, use "kubectl label" with flag "--overwrite"

1kubectl label po kubia-manual-v2 env=debug --overwrite

3.4 Picking pods with label selectors

Use label selectors to filter resources:

• does/doesn't contain a label and a certain key
• contains a label with a certain key and value
• contains a label with a certain key, but with different value from your specification

3.4.1 Listing pods using a label selector

Use "kubectl get pods" with flag -l to filter pods:

• chooses pods with label env

1kubectl get pods -l 'env'

• choose pods without label env

1kubectl get pods -l '!env'

• choose pods whose label env is prod

1kubectl get pods -l 'env=prod'

• choose pods whose label env is not dev

1kubectl get pods -l 'env!=dev'

• choose pods whose label env is dev, test, or debug

1kubectl get pods -l env in '(dev, test, debug)'

• choose pods whose label env is neither dev nor debug

1kubectl get pods -l env notin '(dev, test, debug)'

3.4.2 Using multiple conditions in a label selector

use "kubectl get pods -l" with comma to separate multiple conditions

1kubectl get pods -l 'env notin (prod, dev)','creation_method=manual'

3.5 Using labels and selectors to constrain pod scheduling

Generally(homogenous hardware infrastructure), we would NOT care about which worker node a pod is scheduled to, when each pod gets the exact amount of computational resources, because we want to decouple applications from infrastructure.

However, when hardware infrastructure is NOT homogenous, e.g., some nodes with GPU-specialized hardware or high-speed SSDs, we want to schedule certain pods to one group of worker nodes.

Kubernetes uses node labels and node label selectors to select a node that matches those requirements.

3.5.1 Using labels to categorize worker nodes

Use "kubectl label" to specify the type of hardware the node provides, for later scheduling pods:

1# 1) mark the node `kind-worker2`, with label `ssd` is `true`
2kubectl label nodes kind-worker2 ssd=true
3
4# 2) mark the node `kind-worker3`, with labels both `ssd` and `gpu` are `true`
5kubectl label nodes kind-worker3 ssd=true
6kubectl label nodes kind-worker3 gpu=true

Use "kubectl get nodes" with flag -l (lowercase) to filter nodes:

1# choose the node with both `ssd` and `gpu` are `true`
2#   which, will turn out to be the node named `kind-worker3` 
3kubectl get nodes -l ssd=true,gpu=true

Use "kubectl get nodes" with flag -L (uppercase) to list labels in table columns:

3.5.2 Scheduling pods to specific nodes

In the "spec" section, add a "nodeSelector" field to let scheduler choose only among the worker nodes that contain the label with a specific value.

 1apiVersion: v1 
 2kind: Pod 
 3metadata: 
 4  name: kubia-gpu
 5spec: 
 6  nodeSelector: 
 7    gpu: "true" 
 8  containers: 
 9  - image: ${yourDockerRegistryID}/kubia
10    name: kubia

In this case, the pod kubia-gpu will only be scheduled to the nodes whose label gpu is true, i.e., kind-worker3 - This statement can be verified by command "kubectl get pods" with flag -o wide:

1kubectl get pods -o wide
2# result:
3# NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
4# kubia-gpu         1/1     Running   0          93s   10.244.3.7   kind-worker3   <none>           <none>

Now we can see the pod kubia-gpu is scheduled to kind-worker3.

3.5.3 Scheduling to one specific node

1. use "kubectl get nodes" command with flag -L "kubernetes.io/hostname" to list all posts with label "kubernetes.io/hostname":

1kubectl get nodes -L "kubernetes.io/hostname"
2# NAME                 STATUS   ROLES           AGE   VERSION   HOSTNAME
3# kind-control-plane   Ready    control-plane   17h   v1.29.2   kind-control-plane
4# kind-worker          Ready    <none>          17h   v1.29.2   kind-worker
5# kind-worker2         Ready    <none>          17h   v1.29.2   kind-worker2
6# kind-worker3         Ready    <none>          17h   v1.29.2   kind-worker3

2. write a new YAML file to designate the target node

Note: This is not recommended: the pod may become unschedulable if the node is offline.

 1# kubia-specific-node.yaml
 2apiVersion: v1 
 3kind: Pod 
 4metadata: 
 5  name: kubia-specific-worker2
 6spec: 
 7  nodeSelector: 
 8    kubernetes.io/hostname: "kind-worker2" 
 9  containers: 
10  - image: ${yourDockerRegistryID}/kubia
11    name: kubia

3. create pod with "kubectl create" command, and verify its info

1kubectl create -f kubia-specific-node.yaml
2kubectl get pods -o wide
3# NAME                           READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
4# kubia-specific-worker2         1/1     Running   0          9m42s   10.244.2.4   kind-worker2   <none>           <none>

3.6 Annotating pods

Annotations adds description to each pod or other API object, making collaboration on the cluster easier.

3.6.1 Looking up an object’s annotations

Use "kubectl describe" or "kubectl get" with flag -o yaml look up annotations:

1kubectl describe pods ${podName}
2kubectl get pod ${podName} -o yaml

3.6.2 Adding and modifying annotations

Use "kubectl annotate" to add/modify an annotation:

1kubectl annotate pod ${podName} ${yourPrefix}/${keyName}="${value}"
2
3# to verify the annotation, use `kubectl describe` command
4kubectl describe pod ${podName}

3.7 Using namespaces to group resources

Kubernetes utilizes namespaces to divide objects into distinct and non-overlapping groups, particularly when multiple objects possess identical labels.

3.7.1 Understanding the need for namespaces

Kubernetes uses namespaces to split complex systems with numerous components into smaller distinct groups, each containing the same unique resource name.

• multi-tenant environments
• different development stages
  • production
  • development
  • QA environment And by keeping resources in separate namespaces:
• isolating resource
  • keeps everything organized
  • prevents inadvertently deleting others' resources.
• access control: allowing only certain users access to particular resources
• quota limiting: limiting the amount of computational resources available to individual users

NOTE: Some resources cannot be namespaced, like Worker Nodes, are thus global.

3.7.2 Discovering other namespaces and their pods

Use "kubectl get namespaces" to list all namespaces in the cluster

1kubectl get namespaces
2kubectl get ns

Use "kubectl get pods" with flag --namespace to list pods with a specific namespace

1kubectl get pods --namespace ${namespaceName}
2kubectl get pods -n          ${namespaceName}

3.7.3 Creating a namespace

There are two ways of creating a namespace:

1. Use "kubectl create namespace" command to create a custom namespace

   1kubectl create namespace ${namespaceName}
   

2. Use "kubectl create" with flag -f to create namespace from YAML file:

   1kubectl create -f custom-namespace.yaml
   

   And the content of custom-namespace.yaml is shown below:

   1# custom-namespace.yaml
   2apiVersion: v1
   3kind: Namespace
   4metadata:
   5  name: custom-namespace
   

NOTE: namespace names could contain: letters, digits, and dashes; namespace names cannot contain: dots.

3.7.4 Managing objects in other namespaces

Use "kubectl create" with flag -n to assign the namespace at creation:

1# 1)
2kubectl create -f kubia-manual.yaml --namespace custom-namespace
3
4# 2)
5kubectl create -f kubia-manual.yaml --namespace custom-namespace

Use "kubectl config" to switch namespaces:

1kubectl config set-context $(kubectl config current-context) --namespace ${newNamespaceName}

best practice: create an alias for quickly changing namespaces

1alias kcd='kubectl config set-context $(kubectl config current-context) --namespace '
2
3# use `kcd` to quickly change namespaces to `custom-namespace`
4kcd custom-namespace

3.7.5 Understanding the isolation provided by namespaces

Namespaces only logically isolates objects into distinct groups, and do NOT prevent inter-namespace network communication, such as sending HTTP request traffic via the IP address of a pod in another namespace.

3.8 Stopping and removing pods

3.8.1 Deleting a pod by name

Use "kubectl delete" to delete any number of pods by name:

1kubectl delete pods kubia-gpu kubia-manual-v2

Kubernetes sends "SIGTERM" requests to all pods in the list, i.e., kubia-gpu and kubia-manual-v2 , waiting for them to shut down gracefully.

3.8.2 Deleting pods using label selectors

Use "kubectl delete pods" with flag -l to delete pods by label selector

1kubectl delete pods -l rel=canary

This will delete all pods in current namespace, with label rel is canary

3.8.3 Deleting pods by deleting the whole namespace

Use "kubectl delete namespace" to delete the whole namespace along with all pods inside it.

1kubectl delete namespace custom-namespace

This command will delete all pods inside namespace custom-namespace

3.8.4 Deleting all pods in a namespace, while keeping the namespace

Simply using "kubectl delete pods" with flag --all will not absolutely delete all pods as expected, because the Kubernetes will restart some pods; certain resources are preserved and need to be deleted explicitly, for example, a deployments.app will automatically create a new pod in the current namespace:

 1# 1) create deployments.app 
 2kubectl create deployment kubia --image=${yourDockerRegistryID}/kubia --port=8080
 3# deployment.apps/kubia created
 4
 5# 2) get all pods in current namespace
 6kubectl get pods
 7# NAME                    READY   STATUS    RESTARTS   AGE
 8# kubia-87dff5498-mhkzs   1/1     Running   0          4s
 9
10# 3) delete all pods in the current namespace
11kubectl delete pods --all
12# pod "kubia-87dff5498-mhkzs" deleted
13
14# 4) check if all pods are deleted
15kubectl get pods
16# NAME                    READY   STATUS    RESTARTS   AGE
17# kubia-87dff5498-c5btd   1/1     Running   0          35s

And this test shows that, after deleting the pod kubia-87dff5498-mhkzs, the deployment.app automatically creates a new pod named kubia-87dff5498-c5btd.

3.8.5 Deleting (almost) all resources in a namespace

Use "kubectl delete all" with flag --all to delete all resource instances, such as: deployments.app, services, pods, etc.

1kubectl delete all --all
2# pod "kubia-87dff5498-c5btd" deleted
3# service "kubernetes" deleted
4# deployment.apps "kubia" deleted
5# replicaset.apps "kubia-87dff5498" deleted

NOTE: the service "kubernetes" will be recreated automatically later.

3.9 Summary

This chapter talks about pods:

• in a pod, whether certain containers should be grouped together or not
• pods can run multiple processes, just as host computer does
• YAML and JSON are used to create and describe pods
• Labels together with label selectors make organizing and scheduling pods easier
• Node label and label selector can schedule pods only to the specific nodes; however, if the nodes were offline, the pods would be unschedulable.
• Annotations attach larger blob of data (than labels) to the pods, by people, or tool and library
• Namespaces allow different teams to use the same cluster as though they were using separate Kubernetes clusters.
• "kubectl explain" command is used to quickly look up info on any Kubernetes resources

3.10 Appendix: best practice

best practice about alias in Linux terminal:

1# use `k` instead of `kubectl`
2alias k=kubectl
3
4# use `kcd` to quickly change current namespace, e.g., `kcd default`
5alias kcd='kubectl config set-context $(kubectl config current-context) --namespace '