KIA CH03 Pods: running containers in Kubernetes

Hi there.

Today, let us read Chapter 03: Pods - running containers in Kubernetes (Part II: Core Concepts) of Kubernetes in Action.

1. Creating, running, and stopping pods
2. Organizing pods and other resources with labels
3. Performing an operation on all pods with a specific label
4. Using namespaces to split pods into non-overlapping groups
5. Scheduling pods onto specific types of worker nodes

3.1 Introducing pods

a pod is a co-located group of containers and represents the basic building block in Kubernetes.

• a pod may include one or more containers
• multiple containers in a pod must remain in the SAME worker node; a pod never spans multiple worker nodes.

3.1.1 Understanding why multiple containers are better than one container

Running each process in its own container (multiple containers) is better than combining processes in one container.

• (easy recovery) only restart the crashed process rather than all processes
• (easy management) logs are only kept for the single process in each container

3.1.2 Understanding pods

A pod allows you to run closely related processes together, and provide them with almost the same environment (as if they were all running in a single container) while keeping them partially isolated.

1. the partial isolation among containers inside the same pod.

   • all containers of same pod share the same set of Linux namespaces

2. containers in the same pod share the same IP and Port space

   • containers in same pod run in the same Network namespace; containers in different pods run in different Network namespaces
   • containers in same pod may NOT to bind to the same port numbers, to avoid port conflicts

3. the flat inter-pod network

   • All pods (in a Kubernetes cluster) reside in a single flat, shared, network-address space; every pod can access other pod by the IP address of the target pod.
     • achieved through a SDN layer on top of like computers on a LAN
     • no NAT gateways exist
   • Regardless of which worker nodes the pods are scheduled

3.1.3 Organizing containers across pods

1. for tightly related components or processes that must work as a single whole, organize them into the same pod

   • E.g., a main process and other complementary processes

2. for multi-tier app that can run on different hosts, organize apps into multiple pods

   • different pods, deployed on different worker nodes, can run on different worker nodes, thereby improving the utilization of computational resources.
     • E.g., a multi-tier application, with a frontend server and a backend database, should be configured as different pods so that they can be deployed on different worker nodes.

3. for individual scaling of independent components that have different scaling requirements, organize apps into multiple pods.

   • A pod is the basic unit of scaling.

3.2 Creating pods from YAML/JSON descriptors

To create pods (and other Kubernetes resources), post a JSON or YAML manifest to Kubernetes REST API endpoint.

3.2.1 Examining a YAML descriptor of an existing pod

Use “kubectl get pods” with flag -o yaml to get the whole YAML definition of the pod.

kubectl get pods kubia-zxzij -o yaml

• Metadata: name, namespace, labels, etc.
• Spec: actual description of the pod’s contents, e.g., containers, volumes, etc.
• Status: current information about the running pod, e.g., condition the pod, description and status of each container, internal IP, etc.

Note: status is not required during creating a new pod.

3.2.2 Creating a simple YAML descriptor for a pod

Kubernetes reference documentation and “kubectl explain” command list possible fields in a manifest.

apiVersion: v1
kind: Pod
metadata:
  name: kubia-manual
spec:
  containers:
    - image: ${yourDockerRegistryID}/kubia
      name: kubia
      ports:
        - containerPort: 8080
          protocol: TCP

Note: use two spaces, NOT tab.

We need to explicitly defining ports in YAML file:

• let others know which ports your cluster is exposing
• assign a name to each port

3.2.3 Using “kubectl create” to create the pod

Use “kubectl create” with flag -f to create pod from YAML/JSON file.

kubectl create -f kubia-manual.yaml

Use “kubectl get” to get full info about the pod, in the format of YAML/JSON:

kubectl get pods kubia-manual -o YAML
kubectl get pods kubia-manual -o JSON

3.2.4 Viewing application logs

In container runtime, i.e., Docker, use “docker logs” to redirect stdout and stderr streams to files

docker logs ${ContainerID}

use “kubectl logs” to get logs of all containers or of the specific container of the pod:

# get all logs in pod `kubia-manual`
kubectl logs kubia-manual

# get logs only in container `kubia` of pod `kubia-manual`
kubectl logs kubia-manual -c kubia

3.2.5 Sending requests to the pod

Use “kubectl port-forward” command to forward a local network port to a port in the pod.

kubectl port-forward kubia-manual 8888:8080

Node: Port forwarding is a convenient way of testing single pod.

• “kubectl port-forward” command forwards local port 8888 to port 8080 in pod
• use “curl localhost:8888” command to access the service in pod.

3.3 Organizing pods with labels

Background: When running multiple microservices, pods structure can be complicated:

• multiple pods
• multiple replicas of each pod
• different releases of the same microservice

Kubernetes uses labels to organize all sorts of Kubernetes resources: pods and other objects.

3.3.1 Introducing labels

A label is an arbitrary key-value pair attached to a resource, which is then utilized when selecting resources using label selectors.

A resource can have more than one (unique) labels.

A typical practice for labelling:

• label app: which app, component, or microservice the pod belongs to
• label rel: which release the pod is - E.g., stable, beta, and canary (only a small fraction of user hit the new version, before rolling new version out to all users)

3.3.2 Specifying labels when creating a pod

Add “labels” to YAML to add labels to the pod

apiVersion: v1
kind: Pod
metadata:
  name: kubia-manual-v2
  # add labels to the pod
  labels:
    creation_method: manual
    env: prod
spec:
  containers:
    - image: ${yourDockerRegistryID}/kubia
      name: kubia
      ports:
        - containerPort: 8080
          protocol: TCP

Note: this YAML added two labels

• creation_method
• env

To create pods, run with “kubectl create”

kubectl create -f kubia-manual-with-labels.yaml

Use “kubectl get” to show labels:

• kubectl get pods --show-labels: show all labels
• kubectl get pods -L creation_method,env: only show labels named creation_method and env

3.3.3 Modifying labels of existing pods

To add a new label to existing pod, use “kubectl label”

kubectl label pods kubia-manual-v2 creation_date_format=yyyyMMdd
kubectl label pods kubia-manual-v2 creator=admin

To overwrite an existing label of the pod, use “kubectl label” with flag “--overwrite”

kubectl label po kubia-manual-v2 env=debug --overwrite

3.4 Picking pods with label selectors

Use label selectors to filter resources:

• does/doesn’t contain a label and a certain key
• contains a label with a certain key and value
• contains a label with a certain key, but with different value from your specification

3.4.1 Listing pods using a label selector

Use “kubectl get pods” with flag -l to filter pods:

• chooses pods with label env

kubectl get pods -l 'env'

• choose pods without label env

kubectl get pods -l '!env'

• choose pods whose label env is prod

kubectl get pods -l 'env=prod'

• choose pods whose label env is not dev

kubectl get pods -l 'env!=dev'

• choose pods whose label env is dev, test, or debug

kubectl get pods -l env in '(dev, test, debug)'

• choose pods whose label env is neither dev nor debug

kubectl get pods -l env notin '(dev, test, debug)'

3.4.2 Using multiple conditions in a label selector

use “kubectl get pods -l” with comma to separate multiple conditions

kubectl get pods -l 'env notin (prod, dev)','creation_method=manual'

3.5 Using labels and selectors to constrain pod scheduling

Generally(homogenous hardware infrastructure), we would NOT care about which worker node a pod is scheduled to, when each pod gets the exact amount of computational resources, because we want to decouple applications from infrastructure.

However, when hardware infrastructure is NOT homogenous, e.g., some nodes with GPU-specialized hardware or high-speed SSDs, we want to schedule certain pods to one group of worker nodes.

Kubernetes uses node labels and node label selectors to select a node that matches those requirements.

3.5.1 Using labels to categorize worker nodes

Use “kubectl label” to specify the type of hardware the node provides, for later scheduling pods:

# 1) mark the node `kind-worker2`, with label `ssd` is `true`
kubectl label nodes kind-worker2 ssd=true

# 2) mark the node `kind-worker3`, with labels both `ssd` and `gpu` are `true`
kubectl label nodes kind-worker3 ssd=true
kubectl label nodes kind-worker3 gpu=true

Use “kubectl get nodes” with flag -l (lowercase) to filter nodes:

# choose the node with both `ssd` and `gpu` are `true`
#   which, will turn out to be the node named `kind-worker3`
kubectl get nodes -l ssd=true,gpu=true

Use “kubectl get nodes” with flag -L (uppercase) to list labels in table columns:

3.5.2 Scheduling pods to specific nodes

In the “spec” section, add a “nodeSelector” field to let scheduler choose only among the worker nodes that contain the label with a specific value.

apiVersion: v1
kind: Pod
metadata:
  name: kubia-gpu
spec:
  nodeSelector:
    gpu: "true"
  containers:
  - image: ${yourDockerRegistryID}/kubia
    name: kubia

In this case, the pod kubia-gpu will only be scheduled to the nodes whose label gpu is true, i.e., kind-worker3 - This statement can be verified by command “kubectl get pods” with flag -o wide:

kubectl get pods -o wide
# result:
# NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
# kubia-gpu         1/1     Running   0          93s   10.244.3.7   kind-worker3   <none>           <none>

Now we can see the pod kubia-gpu is scheduled to kind-worker3.

3.5.3 Scheduling to one specific node

1. use “kubectl get nodes” command with flag -L "kubernetes.io/hostname" to list all posts with label “kubernetes.io/hostname”:

kubectl get nodes -L "kubernetes.io/hostname"
# NAME                 STATUS   ROLES           AGE   VERSION   HOSTNAME
# kind-control-plane   Ready    control-plane   17h   v1.29.2   kind-control-plane
# kind-worker          Ready    <none>          17h   v1.29.2   kind-worker
# kind-worker2         Ready    <none>          17h   v1.29.2   kind-worker2
# kind-worker3         Ready    <none>          17h   v1.29.2   kind-worker3

2. write a new YAML file to designate the target node

Note: This is not recommended: the pod may become unschedulable if the node is offline.

apiVersion: v1
kind: Pod
metadata:
  name: kubia-specific-worker2
spec:
  nodeSelector:
    kubernetes.io/hostname: 'kind-worker2'
  containers:
    - image: ${yourDockerRegistryID}/kubia
      name: kubia

3. create pod with “kubectl create” command, and verify its info

kubectl create -f kubia-specific-node.yaml
kubectl get pods -o wide
# NAME                           READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
# kubia-specific-worker2         1/1     Running   0          9m42s   10.244.2.4   kind-worker2   <none>           <none>

3.6 Annotating pods

Annotations adds description to each pod or other API object, making collaboration on the cluster easier.

3.6.1 Looking up an object’s annotations

Use “kubectl describe” or “kubectl get” with flag -o yaml look up annotations:

kubectl describe pods ${podName}
kubectl get pod ${podName} -o yaml

3.6.2 Adding and modifying annotations

Use “kubectl annotate” to add/modify an annotation:

kubectl annotate pod ${podName} ${yourPrefix}/${keyName}="${value}"

# to verify the annotation, use `kubectl describe` command
kubectl describe pod ${podName}

3.7 Using namespaces to group resources

Kubernetes utilizes namespaces to divide objects into distinct and non-overlapping groups, particularly when multiple objects possess identical labels.

3.7.1 Understanding the need for namespaces

Kubernetes uses namespaces to split complex systems with numerous components into smaller distinct groups, each containing the same unique resource name.

• multi-tenant environments
• different development stages - production - development - QA environment And by keeping resources in separate namespaces:
• isolating resource
  • keeps everything organized
  • prevents inadvertently deleting others’ resources.
• access control: allowing only certain users access to particular resources
• quota limiting: limiting the amount of computational resources available to individual users

NOTE: Some resources cannot be namespaced, like Worker Nodes, are thus global.

3.7.2 Discovering other namespaces and their pods

Use “kubectl get namespaces” to list all namespaces in the cluster

kubectl get namespaces
kubectl get ns

Use “kubectl get pods” with flag --namespace to list pods with a specific namespace

kubectl get pods --namespace ${namespaceName}
kubectl get pods -n          ${namespaceName}

3.7.3 Creating a namespace

There are two ways of creating a namespace:

1. Use “kubectl create namespace” command to create a custom namespace

   kubectl create namespace ${namespaceName}

2. Use “kubectl create” with flag -f to create namespace from YAML file:

   kubectl create -f custom-namespace.yaml

   And the content of custom-namespace.yaml is shown below:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: custom-namespace

NOTE: namespace names could contain: letters, digits, and dashes; namespace names cannot contain: dots.

3.7.4 Managing objects in other namespaces

Use “kubectl create” with flag -n to assign the namespace at creation:

# 1)
kubectl create -f kubia-manual.yaml --namespace custom-namespace

# 2)
kubectl create -f kubia-manual.yaml --namespace custom-namespace

Use “kubectl config” to switch namespaces:

kubectl config set-context $(kubectl config current-context) --namespace ${newNamespaceName}

best practice: create an alias for quickly changing namespaces

alias kcd='kubectl config set-context $(kubectl config current-context) --namespace '

# use `kcd` to quickly change namespaces to `custom-namespace`
kcd custom-namespace

3.7.5 Understanding the isolation provided by namespaces

Namespaces only logically isolates objects into distinct groups, and do NOT prevent inter-namespace network communication, such as sending HTTP request traffic via the IP address of a pod in another namespace.

3.8 Stopping and removing pods

3.8.1 Deleting a pod by name

Use “kubectl delete” to delete any number of pods by name:

kubectl delete pods kubia-gpu kubia-manual-v2

Kubernetes sends “SIGTERM” requests to all pods in the list, i.e., kubia-gpu and kubia-manual-v2 , waiting for them to shut down gracefully.

3.8.2 Deleting pods using label selectors

Use “kubectl delete pods” with flag -l to delete pods by label selector

kubectl delete pods -l rel=canary

This will delete all pods in current namespace, with label rel is canary

3.8.3 Deleting pods by deleting the whole namespace

Use “kubectl delete namespace” to delete the whole namespace along with all pods inside it.

kubectl delete namespace custom-namespace

This command will delete all pods inside namespace custom-namespace

3.8.4 Deleting all pods in a namespace, while keeping the namespace

Simply using “kubectl delete pods” with flag --all will not absolutely delete all pods as expected, because the Kubernetes will restart some pods; certain resources are preserved and need to be deleted explicitly, for example, a deployments.app will automatically create a new pod in the current namespace:

# 1) create deployments.app
kubectl create deployment kubia --image=${yourDockerRegistryID}/kubia --port=8080
# deployment.apps/kubia created

# 2) get all pods in current namespace
kubectl get pods
# NAME                    READY   STATUS    RESTARTS   AGE
# kubia-87dff5498-mhkzs   1/1     Running   0          4s

# 3) delete all pods in the current namespace
kubectl delete pods --all
# pod "kubia-87dff5498-mhkzs" deleted

# 4) check if all pods are deleted
kubectl get pods
# NAME                    READY   STATUS    RESTARTS   AGE
# kubia-87dff5498-c5btd   1/1     Running   0          35s

And this test shows that, after deleting the pod kubia-87dff5498-mhkzs, the deployment.app automatically creates a new pod named kubia-87dff5498-c5btd.

3.8.5 Deleting (almost) all resources in a namespace

Use “kubectl delete all” with flag --all to delete all resource instances, such as: deployments.app, services, pods, etc.

kubectl delete all --all
# pod "kubia-87dff5498-c5btd" deleted
# service "kubernetes" deleted
# deployment.apps "kubia" deleted
# replicaset.apps "kubia-87dff5498" deleted

NOTE: the service “kubernetes” will be recreated automatically later.

3.9 Summary

This chapter talks about pods:

• in a pod, whether certain containers should be grouped together or not
• pods can run multiple processes, just as host computer does
• YAML and JSON are used to create and describe pods
• Labels together with label selectors make organizing and scheduling pods easier
• Node label and label selector can schedule pods only to the specific nodes; however, if the nodes were offline, the pods would be unschedulable.
• Annotations attach larger blob of data (than labels) to the pods, by people, or tool and library
• Namespaces allow different teams to use the same cluster as though they were using separate Kubernetes clusters.
• “kubectl explain” command is used to quickly look up info on any Kubernetes resources

3.10 Appendix: best practice

best practice about alias in Linux terminal:

# use `k` instead of `kubectl`
alias k=kubectl

# use `kcd` to quickly change current namespace, e.g., `kcd default`
alias kcd='kubectl config set-context $(kubectl config current-context) --namespace '