Introductions

• Hello!

• On stage: Jérôme (@jpetazzo)

• Backstage: Alexandre, Amy, Antoine, Aurélien (x2), Benji, David, Julien, Kostas, Nicolas, Thibault

• The training will run from 9:30 to 13:00

• There will be a break at (approximately) 11:00

• You should must ask questions! Lots of questions!

• Use Mattermost to ask questions, get help, etc. logistics.md

Exercises

• At the end of each day, there is a series of exercises

• To make the most out of the training, please try the exercises!

  (it will help to practice and memorize the content of the day)

• We recommend to take at least one hour to work on the exercises

  (if you understood the content of the day, it will be much faster)

• Each day will start with a quick review of the exercises of the previous day

logistics.md

A brief introduction

• This was initially written by Jérôme Petazzoni to support in-person, instructor-led workshops and tutorials

• Credit is also due to multiple contributors — thank you!

• You can also follow along on your own, at your own pace

• We included as much information as possible in these slides

• We recommend having a mentor to help you ...

• ... Or be comfortable spending some time reading the Kubernetes documentation ...

• ... And looking for answers on StackOverflow and other outlets

k8s/intro.md

Accessing these slides now

• We recommend that you open these slides in your browser:

  https://2022-02-enix.container.training/

• Use arrows to move to next/previous slide

  (up, down, left, right, page up, page down)

• Type a slide number + ENTER to go to that slide

• The slide number is also visible in the URL bar

  (e.g. .../#123 for slide 123)

shared/about-slides.md

Accessing these slides later

• Slides will remain online so you can review them later if needed

  (let's say we'll keep them online at least 1 year, how about that?)

• You can download the slides using that URL:

  https://2022-02-enix.container.training/slides.zip

  (then open the file 2.yml.html)

• You will find new versions of these slides on:

  https://container.training/

shared/about-slides.md

These slides are open source

• You are welcome to use, re-use, share these slides

• These slides are written in Markdown

• The sources of these slides are available in a public GitHub repository:

  https://github.com/jpetazzo/container.training

• Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ...

👇 Try it! The source file will be shown and you can view it on GitHub and fork and edit it.

shared/about-slides.md

Chat room

• We've set up a chat room that we will monitor during the workshop

• Don't hesitate to use it to ask questions, or get help, or share feedback

• The chat room will also be available after the workshop

• Join the chat room: Mattermost

• Say hi in the chat room!

shared/chat-room-im.md

Pre-requirements

• Be comfortable with the UNIX command line

  • navigating directories

  • editing files

  • a little bit of bash-fu (environment variables, loops)

• Some Docker knowledge

  • docker run, docker ps, docker build

  • ideally, you know how to write a Dockerfile and build it
    (even if it's a FROM line and a couple of RUN commands)

• It's totally OK if you are not a Docker expert!

shared/prereqs.md

Hands-on sections

• The whole workshop is hands-on

• We are going to build, ship, and run containers!

• You are invited to reproduce all the demos

• All hands-on sections are clearly identified, like the gray rectangle below

shared/prereqs.md

Doing or re-doing the workshop on your own?

• Use something like Play-With-Docker or Play-With-Kubernetes

  Zero setup effort; but environment are short-lived and might have limited resources

• Create your own cluster (local or cloud VMs)

  Small setup effort; small cost; flexible environments

• Create a bunch of clusters for you and your friends (instructions)

  Bigger setup effort; ideal for group training

shared/connecting.md

For a consistent Kubernetes experience ...

• If you are using your own Kubernetes cluster, you can use jpetazzo/shpod

• shpod provides a shell running in a pod on your own cluster

• It comes with many tools pre-installed (helm, stern...)

• These tools are used in many demos and exercises in these slides

• shpod also gives you completion and a fancy prompt

• It can also be used as an SSH server if needed

shared/connecting.md

We will (mostly) interact with node1 only

These remarks apply only when using multiple nodes, of course.

• Unless instructed, all commands must be run from the first VM, node1

• We will only check out/copy the code on node1

• During normal operations, we do not need access to the other nodes

• If we had to troubleshoot issues, we would use a combination of:

  • SSH (to access system logs, daemon status...)

  • Docker API (to check running containers and container engine status)

shared/connecting.md

Terminals

Once in a while, the instructions will say:
"Open a new terminal."

There are multiple ways to do this:

• create a new window or tab on your machine, and SSH into the VM;

• use screen or tmux on the VM and open a new window from there.

You are welcome to use the method that you feel the most comfortable with.

shared/connecting.md

Tmux cheat sheet

Tmux is a terminal multiplexer like screen.

You don't have to use it or even know about it to follow along.
But some of us like to use it to switch between terminals.
It has been preinstalled on your workshop nodes.

• Ctrl-b c → creates a new window
• Ctrl-b n → go to next window
• Ctrl-b p → go to previous window
• Ctrl-b " → split window top/bottom
• Ctrl-b % → split window left/right
• Ctrl-b Alt-1 → rearrange windows in columns
• Ctrl-b Alt-2 → rearrange windows in rows
• Ctrl-b arrows → navigate to other windows
• Ctrl-b d → detach session
• tmux attach → re-attach to session

shared/connecting.md

Exercise — Deploy Dockercoins

• Deploy the dockercoins application to our Kubernetes cluster

• Connect components together

• Expose the web UI and open it in a web browser to check that it works

exercises/k8sfundamentals-brief.md

Exercise — Local Cluster

• Deploy a local Kubernetes cluster if you don't already have one

• Deploy dockercoins on that cluster

• Connect to the web UI in your browser

• Scale up dockercoins

exercises/localcluster-brief.md

Exercise — Healthchecks

• Add readiness and liveness probes to a web service

  (we will use the rng service in the dockercoins app)

• See what happens when the load increses

  (spoiler alert: it involves timeouts!)

exercises/healthchecks-brief.md

Part 1

• Our sample application

• Kubernetes concepts

• First contact with kubectl

(auto-generated TOC)

Part 2

• Running our first containers on Kubernetes

• Kubernetes network model

• Exposing containers

• Shipping images with a registry

• Exercise — Deploy Dockercoins

• Running our application on Kubernetes

(auto-generated TOC)

Part 3

• Labels and annotations

• Revisiting kubectl logs

• Accessing logs from the CLI

• Namespaces

• Deploying with YAML

• Declarative vs imperative

(auto-generated TOC)

Part 4

• Authoring YAML

• Setting up Kubernetes

• Running a local development cluster

• Controlling a Kubernetes cluster remotely

• Accessing internal services

• Accessing the API with kubectl proxy

• Exercise — Local Cluster

(auto-generated TOC)

Part 5

• Scaling our demo app

• Daemon sets

• Labels and selectors

• Rolling updates

(auto-generated TOC)

Part 6

• Healthchecks

• The Kubernetes dashboard

• Security implications of kubectl apply

• k9s

• Tilt

• Exercise — Healthchecks

(auto-generated TOC)

Part 7

• Exposing HTTP services with Ingress resources

• Ingress and TLS certificates

(auto-generated TOC)

Part 8

• Volumes

• Managing configuration

• Managing secrets

• Executing batch jobs

(auto-generated TOC)

shared/toc.md

Our sample application

• We will clone the GitHub repository onto our node1

• The repository also contains scripts and tools that we will use through the workshop

(You can also fork the repository on GitHub and clone your fork if you prefer that.)

shared/sampleapp.md

Downloading and running the application

Let's start this before we look around, as downloading will take a little time...

Compose tells Docker to build all container images (pulling the corresponding base images), then starts all containers, and displays aggregated logs.

shared/sampleapp.md

What's this application?

DockerCoin in the microservices era

• The dockercoins app is made of 5 services:

  • rng = web service generating random bytes

  • hasher = web service computing hash of POSTed data

  • worker = background process calling rng and hasher

  • webui = web interface to watch progress

  • redis = data store (holds a counter updated by worker)

• These 5 services are visible in the application's Compose file, docker-compose.yml

shared/sampleapp.md

How dockercoins works

• worker invokes web service rng to generate random bytes

• worker invokes web service hasher to hash these bytes

• worker does this in an infinite loop

• every second, worker updates redis to indicate how many loops were done

• webui queries redis, and computes and exposes "hashing speed" in our browser

(See diagram on next slide!)

shared/sampleapp.md

Service discovery in container-land

How does each service find out the address of the other ones?

Example in worker/worker.py

redis = Redis("redis")
def get_random_bytes():
    r = requests.get("http://rng/32")
    return r.content
def hash_bytes(data):
    r = requests.post("http://hasher/",
                      data=data,
                      headers={"Content-Type": "application/octet-stream"})

(Full source code available here)

shared/sampleapp.md

Show me the code!

• You can check the GitHub repository with all the materials of this workshop:
  https://github.com/jpetazzo/container.training

• The application is in the dockercoins subdirectory

• The Compose file (docker-compose.yml) lists all 5 services

• redis is using an official image from the Docker Hub

• hasher, rng, worker, webui are each built from a Dockerfile

• Each service's Dockerfile and source code is in its own directory

  (hasher is in the hasher directory, rng is in the rng directory, etc.)

shared/sampleapp.md

Our application at work

• On the left-hand side, the "rainbow strip" shows the container names

• On the right-hand side, we see the output of our containers

• We can see the worker service making requests to rng and hasher

• For rng and hasher, we see HTTP access logs

shared/sampleapp.md

Connecting to the web UI

• "Logs are exciting and fun!" (No-one, ever)

• The webui container exposes a web dashboard; let's view it

A drawing area should show up, and after a few seconds, a blue graph will appear.

shared/sampleapp.md

Stopping the application

• If we interrupt Compose (with ^C), it will politely ask the Docker Engine to stop the app

• The Docker Engine will send a TERM signal to the containers

• If the containers do not exit in a timely manner, the Engine sends a KILL signal

Clean up

• Before moving on, let's remove those containers

shared/composedown.md

Kubernetes concepts

• Kubernetes is a container management system

• It runs and manages containerized applications on a cluster

What can we do with Kubernetes?

• Let's imagine that we have a 3-tier e-commerce app:

  • web frontend

  • API backend

  • database (that we will keep out of Kubernetes for now)

• We have built images for our frontend and backend components

  (e.g. with Dockerfiles and docker build)

• We are running them successfully with a local environment

  (e.g. with Docker Compose)

• Let's see how we would deploy our app on Kubernetes!

k8s/concepts-k8s.md

Basic things we can ask Kubernetes to do

Other things that Kubernetes can do for us

• Autoscaling

  (straightforward on CPU; more complex on other metrics)

• Resource management and scheduling

  (reserve CPU/RAM for containers; placement constraints)

• Advanced rollout patterns

  (blue/green deployment, canary deployment)

k8s/concepts-k8s.md

More things that Kubernetes can do for us

• Batch jobs

  (one-off; parallel; also cron-style periodic execution)

• Fine-grained access control

  (defining what can be done by whom on which resources)

• Stateful services

  (databases, message queues, etc.)

• Automating complex tasks with operators

  (e.g. database replication, failover, etc.)

k8s/concepts-k8s.md

Kubernetes architecture

k8s/concepts-k8s.md

Kubernetes architecture

• Ha ha ha ha

• OK, I was trying to scare you, it's much simpler than that ❤️

k8s/concepts-k8s.md

Credits

• The first schema is a Kubernetes cluster with storage backed by multi-path iSCSI

  (Courtesy of Yongbok Kim)

• The second one is a simplified representation of a Kubernetes cluster

  (Courtesy of Imesh Gunaratne)

k8s/concepts-k8s.md

Kubernetes architecture: the nodes

• The nodes executing our containers run a collection of services:

  • a container Engine (typically Docker)

  • kubelet (the "node agent")

  • kube-proxy (a necessary but not sufficient network component)

• Nodes were formerly called "minions"

  (You might see that word in older articles or documentation)

k8s/concepts-k8s.md

Kubernetes architecture: the control plane

• The Kubernetes logic (its "brains") is a collection of services:

  • the API server (our point of entry to everything!)

  • core services like the scheduler and controller manager

  • etcd (a highly available key/value store; the "database" of Kubernetes)

• Together, these services form the control plane of our cluster

• The control plane is also called the "master"

k8s/concepts-k8s.md

Interacting with Kubernetes

• We will interact with our Kubernetes cluster through the Kubernetes API

• The Kubernetes API is (mostly) RESTful

• It allows us to create, read, update, delete resources

• A few common resource types are:

  • node (a machine — physical or virtual — in our cluster)

  • pod (group of containers running together on a node)

  • service (stable network endpoint to connect to one or multiple containers)

k8s/concepts-k8s.md

Scaling

• How would we scale the pod shown on the previous slide?

• Do create additional pods

  • each pod can be on a different node

  • each pod will have its own IP address

• Do not add more NGINX containers in the pod

  • all the NGINX containers would be on the same node

  • they would all have the same IP address
    (resulting in Address alreading in use errors)

k8s/concepts-k8s.md

Together or separate

• Should we put e.g. a web application server and a cache together?
  ("cache" being something like e.g. Memcached or Redis)

• Putting them in the same pod means:

  • they have to be scaled together

  • they can communicate very efficiently over localhost

• Putting them in different pods means:

  • they can be scaled separately

  • they must communicate over remote IP addresses
    (incurring more latency, lower performance)

• Both scenarios can make sense, depending on our goals

k8s/concepts-k8s.md

Credits

• The first diagram is courtesy of Lucas Käldström, in this presentation

  • it's one of the best Kubernetes architecture diagrams available!

• The second diagram is courtesy of Weave Works

  • a pod can have multiple containers working together

  • IP addresses are associated with pods, not with individual containers

Both diagrams used with permission.

First contact with kubectl

• kubectl is (almost) the only tool we'll need to talk to Kubernetes

• It is a rich CLI tool around the Kubernetes API

  (Everything you can do with kubectl, you can do directly with the API)

• On our machines, there is a ~/.kube/config file with:

  • the Kubernetes API address

  • the path to our TLS certificates used to authenticate

• You can also use the --kubeconfig flag to pass a config file

• Or directly --server, --user, etc.

• kubectl can be pronounced "Cube C T L", "Cube cuttle", "Cube cuddle"...

k8s/kubectlget.md

kubectl get

• Let's look at our Node resources with kubectl get!

k8s/kubectlget.md

Obtaining machine-readable output

• kubectl get can output JSON, YAML, or be directly formatted

k8s/kubectlget.md

(Ab)using kubectl and jq

• It's super easy to build custom reports

k8s/kubectlget.md

Type names

• The most common resource names have three forms:

  • singular (e.g. node, service, deployment)

  • plural (e.g. nodes, services, deployments)

  • short (e.g. no, svc, deploy)

• Some resources do not have a short name

• Endpoints only have a plural form

  (because even a single Endpoints resource is actually a list of endpoints)

k8s/kubectlget.md

Viewing details

• We can use kubectl get -o yaml to see all available details

• However, YAML output is often simultaneously too much and not enough

• For instance, kubectl get node node1 -o yaml is:

  • too much information (e.g.: list of images available on this node)

  • not enough information (e.g.: doesn't show pods running on this node)

  • difficult to read for a human operator

• For a comprehensive overview, we can use kubectl describe instead

k8s/kubectlget.md

kubectl describe

• kubectl describe needs a resource type and (optionally) a resource name

• It is possible to provide a resource name prefix

  (all matching objects will be displayed)

• kubectl describe will retrieve some extra information about the resource

(We should notice a bunch of control plane pods.)

k8s/kubectlget.md

Listing running containers

• Containers are manipulated through pods

• A pod is a group of containers:

  • running together (on the same node)

  • sharing resources (RAM, CPU; but also network, volumes)

Namespaces

• Namespaces allow us to segregate resources

Accessing namespaces

• By default, kubectl uses the default namespace

• We can see resources in all namespaces with --all-namespaces

Here are our system pods!

k8s/kubectlget.md

What are all these control plane pods?

• etcd is our etcd server

• kube-apiserver is the API server

• kube-controller-manager and kube-scheduler are other control plane components

• coredns provides DNS-based service discovery (replacing kube-dns as of 1.11)

• kube-proxy is the (per-node) component managing port mappings and such

• weave is the (per-node) component managing the network overlay

• the READY column indicates the number of containers in each pod

  (1 for most pods, but weave has 2, for instance)

k8s/kubectlget.md

Scoping another namespace

• We can also look at a different namespace (other than default)

k8s/kubectlget.md

Namespaces and other kubectl commands

• We can use -n/--namespace with almost every kubectl command

• Example:

  • kubectl create --namespace=X to create something in namespace X

• We can use -A/--all-namespaces with most commands that manipulate multiple objects

• Examples:

  • kubectl delete can delete resources across multiple namespaces

  • kubectl label can add/remove/update labels across multiple namespaces

k8s/kubectlget.md

Services

• A service is a stable endpoint to connect to "something"

  (In the initial proposal, they were called "portals")

ClusterIP services

• A ClusterIP service is internal, available from the cluster only

• This is useful for introspection from within containers

The command above should either time out, or show an authentication error. Why?

k8s/kubectlget.md

Time out

• Connections to ClusterIP services only work from within the cluster

• If we are outside the cluster, the curl command will probably time out

  (Because the IP address, e.g. 10.96.0.1, isn't routed properly outside the cluster)

• This is the case with most "real" Kubernetes clusters

• To try the connection from within the cluster, we can use shpod

k8s/kubectlget.md

Authentication error

This is what we should see when connecting from within the cluster:

$ curl -k https://10.96.0.1
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
  },
  "code": 403
}

k8s/kubectlget.md

Explanations

• We can see kind, apiVersion, metadata

• These are typical of a Kubernetes API reply

• Because we are talking to the Kubernetes API

• The Kubernetes API tells us "Forbidden"

  (because it requires authentication)

• The Kubernetes API is reachable from within the cluster

  (many apps integrating with Kubernetes will use this)

k8s/kubectlget.md

DNS integration

• Each service also gets a DNS record

• The Kubernetes DNS resolver is available from within pods

  (and sometimes, from within nodes, depending on configuration)

• Code running in pods can connect to services using their name

  (e.g. https://kubernetes/...)

Running our first containers on Kubernetes

• First things first: we cannot run a container

Starting a simple pod with kubectl run

• kubectl run is convenient to start a single pod

• We need to specify at least a name and the image we want to use

• Optionally, we can specify the command to run in the pod

The output tells us that a Pod was created:

pod/pingpong created

k8s/kubectl-run.md

Viewing container output

• Let's use the kubectl logs command

• It takes a Pod name as argument

• Unless specified otherwise, it will only show logs of the first container in the pod

  (Good thing there's only one in ours!)

k8s/kubectl-run.md

Streaming logs in real time

• Just like docker logs, kubectl logs supports convenient options:

  • -f/--follow to stream logs in real time (à la tail -f)

  • --tail to indicate how many lines you want to see (from the end)

  • --since to get logs only after a given timestamp

k8s/kubectl-run.md

Scaling our application

• kubectl gives us a simple command to scale a workload:

  kubectl scale TYPE NAME --replicas=HOWMANY

• Let's try it on our Pod, so that we have more Pods!

🤔 We get the following error, what does that mean?

Error from server (NotFound): the server could not find the requested resource

k8s/kubectl-run.md

Scaling a Pod

• We cannot "scale a Pod"

  (that's not completely true; we could give it more CPU/RAM)

• If we want more Pods, we need to create more Pods

  (i.e. execute kubectl run multiple times)

• There must be a better way!

  (spoiler alert: yes, there is a better way!)

k8s/kubectl-run.md

Creating more pods

• We are going to create a ReplicaSet

  (= set of replicas = set of identical pods)

• In fact, we will create a Deployment, which itself will create a ReplicaSet

• Why so many layers? We'll explain that shortly, don't worry!

k8s/kubectl-run.md

Creating a Deployment running ping

• Let's create a Deployment instead of a single Pod

• The -- is used to separate:

  • "options/flags of kubectl create

  • command to run in the container

k8s/kubectl-run.md

What has been created?

Note: kubectl get all is a lie. It doesn't show everything.

(But it shows a lot of "usual suspects", i.e. commonly used resources.)

k8s/kubectl-run.md

There's a lot going on here!

NAME                            READY   STATUS        RESTARTS   AGE
pod/pingpong                    1/1     Running       0          4m17s
pod/pingpong-6ccbc77f68-kmgfn   1/1     Running       0          11s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3h45
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pingpong   1/1     1            1           11s
NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/pingpong-6ccbc77f68   1         1         1       11s

Our new Pod is not named pingpong, but pingpong-xxxxxxxxxxx-yyyyy.

We have a Deployment named pingpong, and an extra ReplicaSet, too. What's going on?

k8s/kubectl-run.md

From Deployment to Pod

We have the following resources:

• deployment.apps/pingpong

  This is the Deployment that we just created.

• replicaset.apps/pingpong-xxxxxxxxxx

  This is a Replica Set created by this Deployment.

• pod/pingpong-xxxxxxxxxx-yyyyy

  This is a pod created by the Replica Set.

Let's explain what these things are.

k8s/kubectl-run.md

Pod

• Can have one or multiple containers

• Runs on a single node

  (Pod cannot "straddle" multiple nodes)

• Pods cannot be moved

  (e.g. in case of node outage)

• Pods cannot be scaled horizontally

  (except by manually creating more Pods)

k8s/kubectl-run.md

Replica Set

• Set of identical (replicated) Pods

• Defined by a pod template + number of desired replicas

• If there are not enough Pods, the Replica Set creates more

  (e.g. in case of node outage; or simply when scaling up)

• If there are too many Pods, the Replica Set deletes some

  (e.g. if a node was disconnected and comes back; or when scaling down)

• We can scale up/down a Replica Set

  • we update the manifest of the Replica Set

  • as a consequence, the Replica Set controller creates/deletes Pods

k8s/kubectl-run.md

Deployment

• Replica Sets control identical Pods

• Deployments are used to roll out different Pods

  (different image, command, environment variables, ...)

• When we update a Deployment with a new Pod definition:

  • a new Replica Set is created with the new Pod definition

  • that new Replica Set is progressively scaled up

  • meanwhile, the old Replica Set(s) is(are) scaled down

• This is a rolling update, minimizing application downtime

• When we scale up/down a Deployment, it scales up/down its Replica Set

k8s/kubectl-run.md

Can we scale now?

• Let's try kubectl scale again, but on the Deployment!

k8s/kubectl-run.md

Checking Deployment logs

• kubectl logs needs a Pod name

• But it can also work with a type/name

  (e.g. deployment/pingpong)

• It shows us the logs of the first Pod of the Deployment

• We'll see later how to get the logs of all the Pods!

k8s/kubectl-run.md

Resilience

• The deployment pingpong watches its replica set

• The replica set ensures that the right number of pods are running

• What happens if pods disappear?

k8s/kubectl-run.md

What happened?

• kubectl delete pod terminates the pod gracefully

  (sending it the TERM signal and waiting for it to shutdown)

• As soon as the pod is in "Terminating" state, the Replica Set replaces it

• But we can still see the output of the "Terminating" pod in kubectl logs

• Until 30 seconds later, when the grace period expires

• The pod is then killed, and kubectl logs exits

k8s/kubectl-run.md

Deleting a standalone Pod

• What happens if we delete a standalone Pod?

  (like the first pingpong Pod that we created)

• No replacement Pod gets created because there is no controller watching it

• That's why we will rarely use standalone Pods in practice

  (except for e.g. punctual debugging or executing a short supervised task)

Kubernetes network model

• TL,DR:

  Our cluster (nodes and pods) is one big flat IP network.

Kubernetes network model: the good

• Everything can reach everything

• No address translation

• No port translation

• No new protocol

• The network implementation can decide how to allocate addresses

• IP addresses don't have to be "portable" from a node to another

  (We can use e.g. a subnet per node and use a simple routed topology)

• The specification is simple enough to allow many various implementations

k8s/kubenet.md

Kubernetes network model: the less good

• Everything can reach everything

  • if you want security, you need to add network policies

  • the network implementation that you use needs to support them

• There are literally dozens of implementations out there

  (https://github.com/containernetworking/cni/ lists more than 25 plugins)

• Pods have level 3 (IP) connectivity, but services are level 4 (TCP or UDP)

  (Services map to a single UDP or TCP port; no port ranges or arbitrary IP packets)

• kube-proxy is on the data path when connecting to a pod or container,
  and it's not particularly fast (relies on userland proxying or iptables)

k8s/kubenet.md

Kubernetes network model: in practice

• The nodes that we are using have been set up to use Weave

• We don't endorse Weave in a particular way, it just Works For Us

• Don't worry about the warning about kube-proxy performance

• Unless you:

  • routinely saturate 10G network interfaces
  • count packet rates in millions per second
  • run high-traffic VOIP or gaming platforms
  • do weird things that involve millions of simultaneous connections
    (in which case you're already familiar with kernel tuning)

• If necessary, there are alternatives to kube-proxy; e.g. kube-router

k8s/kubenet.md

Exposing containers

• We can connect to our pods using their IP address

• Then we need to figure out a lot of things:

  • how do we look up the IP address of the pod(s)?

  • how do we connect from outside the cluster?

  • how do we load balance traffic?

  • what if a pod fails?

• Kubernetes has a resource type named Service

• Services address all these questions!

k8s/kubectlexpose.md

Services in a nutshell

• Services give us a stable endpoint to connect to a pod or a group of pods

• An easy way to create a service is to use kubectl expose

• If we have a deployment named my-little-deploy, we can run:

  kubectl expose deployment my-little-deploy --port=80

  ... and this will create a service with the same name (my-little-deploy)

• Services are automatically added to an internal DNS zone

  (in the example above, our code can now connect to http://my-little-deploy/)

k8s/kubectlexpose.md

Advantages of services

• We don't need to look up the IP address of the pod(s)

  (we resolve the IP address of the service using DNS)

• There are multiple service types; some of them allow external traffic

  (e.g. LoadBalancer and NodePort)

• Services provide load balancing

  (for both internal and external traffic)

• Service addresses are independent from pods' addresses

  (when a pod fails, the service seamlessly sends traffic to its replacement)

k8s/kubectlexpose.md

Many kinds and flavors of service

• There are different types of services:

  ClusterIP, NodePort, LoadBalancer, ExternalName

• There are also headless services

• Services can also have optional external IPs

• There is also another resource type called Ingress

  (specifically for HTTP services)

• Wow, that's a lot! Let's start with the basics ...

k8s/kubectlexpose.md

ClusterIP

• It's the default service type

• A virtual IP address is allocated for the service

  (in an internal, private range; e.g. 10.96.0.0/12)

• This IP address is reachable only from within the cluster (nodes and pods)

• Our code can connect to the service using the original port number

• Perfect for internal communication, within the cluster

k8s/kubectlexpose.md

LoadBalancer

• An external load balancer is allocated for the service

  (typically a cloud load balancer, e.g. ELB on AWS, GLB on GCE ...)

• This is available only when the underlying infrastructure provides some kind of "load balancer as a service"

• Each service of that type will typically cost a little bit of money

  (e.g. a few cents per hour on AWS or GCE)

• Ideally, traffic would flow directly from the load balancer to the pods

• In practice, it will often flow through a NodePort first

k8s/kubectlexpose.md

NodePort

• A port number is allocated for the service

  (by default, in the 30000-32767 range)

• That port is made available on all our nodes and anybody can connect to it

  (we can connect to any node on that port to reach the service)

• Our code needs to be changed to connect to that new port number

• Under the hood: kube-proxy sets up a bunch of iptables rules on our nodes

• Sometimes, it's the only available option for external traffic

  (e.g. most clusters deployed with kubeadm or on-premises)

k8s/kubectlexpose.md

Running containers with open ports

• Since ping doesn't have anything to connect to, we'll have to run something else

• We could use the nginx official image, but ...

  ... we wouldn't be able to tell the backends from each other!

• We are going to use jpetazzo/color, a tiny HTTP server written in Go

• jpetazzo/color listens on port 80

• It serves a page showing the pod's name

  (this will be useful when checking load balancing behavior)

k8s/kubectlexpose.md

Creating a deployment for our HTTP server

• We will create a deployment with kubectl create deployment

• Then we will scale it with kubectl scale

k8s/kubectlexpose.md

Exposing our deployment

• We'll create a default ClusterIP service

k8s/kubectlexpose.md

Services are layer 4 constructs

• You can assign IP addresses to services, but they are still layer 4

  (i.e. a service is not an IP address; it's an IP address + protocol + port)

• This is caused by the current implementation of kube-proxy

  (it relies on mechanisms that don't support layer 3)

• As a result: you have to indicate the port number for your service

  (with some exceptions, like ExternalName or headless services, covered later)

k8s/kubectlexpose.md

Testing our service

• We will now send a few HTTP requests to our pods

Shipping images with a registry

• Initially, our app was running on a single node

• We could build and run in the same place

• Therefore, we did not need to ship anything

• Now that we want to run on a cluster, things are different

• The easiest way to ship container images is to use a registry

k8s/shippingimages.md

How Docker registries work (a reminder)

• What happens when we execute docker run alpine ?

• If the Engine needs to pull the alpine image, it expands it into library/alpine

• library/alpine is expanded into index.docker.io/library/alpine

• The Engine communicates with index.docker.io to retrieve library/alpine:latest

• To use something else than index.docker.io, we specify it in the image name

• Examples:

  docker pull gcr.io/google-containers/alpine-with-bash:1.0
  docker build -t registry.mycompany.io:5000/myimage:awesome .
  docker push registry.mycompany.io:5000/myimage:awesome

k8s/shippingimages.md

Running DockerCoins on Kubernetes

• Create one deployment for each component

  (hasher, redis, rng, webui, worker)

• Expose deployments that need to accept connections

  (hasher, redis, rng, webui)

• For redis, we can use the official redis image

• For the 4 others, we need to build images and push them to some registry

k8s/shippingimages.md

Building and shipping images

• There are many options!

• Manually:

  • build locally (with docker build or otherwise)

  • push to the registry

• Automatically:

  • build and test locally

  • when ready, commit and push a code repository

  • the code repository notifies an automated build system

  • that system gets the code, builds it, pushes the image to the registry

k8s/shippingimages.md

Which registry do we want to use?

• There are SAAS products like Docker Hub, Quay ...

• Each major cloud provider has an option as well

  (ACR on Azure, ECR on AWS, GCR on Google Cloud...)

• There are also commercial products to run our own registry

  (Docker EE, Quay...)

• And open source options, too!

• When picking a registry, pay attention to its build system

  (when it has one)

k8s/shippingimages.md

Building on the fly

• Conceptually, it is possible to build images on the fly from a repository

• Example: ctr.run

  (deprecated in August 2020, after being aquired by Datadog)

• It did allow something like this:

  docker run ctr.run/github.com/jpetazzo/container.training/dockercoins/hasher

• No alternative yet

  (free startup idea, anyone?)

Using images from the Docker Hub

• For everyone's convenience, we took care of building DockerCoins images

• We pushed these images to the DockerHub, under the dockercoins user

• These images are tagged with a version number, v0.1

• The full image names are therefore:

  • dockercoins/hasher:v0.1

  • dockercoins/rng:v0.1

  • dockercoins/webui:v0.1

  • dockercoins/worker:v0.1

k8s/buildshiprun-dockerhub.md

Exercise — Deploy Dockercoins

• We want to deploy the dockercoins app

• There are 5 components in the app:

  hasher, redis, rng, webui, worker

• We'll use one Deployment for each component

  (created with kubectl create deployment)

• We'll connect them with Services

  (create with kubectl expose)

exercises/k8sfundamentals-details.md

Images

• We'll use the following images:

  • hasher → dockercoins/hasher:v0.1

  • redis → redis

  • rng → dockercoins/rng:v0.1

  • webui → dockercoins/webui:v0.1

  • worker → dockercoins/worker:v0.1

• All services should be internal services, except the web UI

  (since we want to be able to connect to the web UI from outside)

exercises/k8sfundamentals-details.md

Goal

• We should be able to see the web UI in our browser

  (with the graph showing approximately 3-4 hashes/second)

exercises/k8sfundamentals-details.md

Hints

• Make sure to expose services with the right ports

  (check the logs of the worker; they indicate the port numbers)

• The web UI can be exposed with a NodePort or LoadBalancer Service

exercises/k8sfundamentals-details.md

Running our application on Kubernetes

• We can now deploy our code (as well as a redis instance)

k8s/ourapponkube.md

Is this working?

• After waiting for the deployment to complete, let's look at the logs!

  (Hint: use kubectl get deploy -w to watch deployment events)

Connecting containers together

• Three deployments need to be reachable by others: hasher, redis, rng

• worker doesn't need to be exposed

• webui will be dealt with later

k8s/ourapponkube.md

Is this working yet?

• The worker has an infinite loop, that retries 10 seconds after an error

Exposing services for external access

• Now we would like to access the Web UI

• We will expose it with a NodePort

  (just like we did for the registry)

k8s/ourapponkube.md

Accessing the web UI

• We can now connect to any node, on the allocated node port, to view the web UI

Labels and annotations

• Most Kubernetes resources can have labels and annotations

• Both labels and annotations are arbitrary strings

  (with some limitations that we'll explain in a minute)

• Both labels and annotations can be added, removed, changed, dynamically

• This can be done with:

  • the kubectl edit command

  • the kubectl label and kubectl annotate

  • ... many other ways! (kubectl apply -f, kubectl patch, ...)

k8s/labels-annotations.md

Viewing labels and annotations

• Let's see what we get when we create a Deployment

So, what do we get?

k8s/labels-annotations.md

Labels and annotations for our Deployment

• We see one label:

  Labels: app=clock

• This is added by kubectl create deployment

• And one annotation:

  Annotations: deployment.kubernetes.io/revision: 1

• This is to keep track of successive versions when doing rolling updates

k8s/labels-annotations.md

And for the related Pod?

• Let's look up the Pod that was created and check it too

So, what do we get?

k8s/labels-annotations.md

Labels and annotations for our Pod

• We see two labels:

  Labels: app=clock
          pod-template-hash=xxxxxxxxxx

• app=clock comes from kubectl create deployment too

• pod-template-hash was assigned by the Replica Set

  (when we will do rolling updates, each set of Pods will have a different hash)

• There are no annotations:

  Annotations: <none>

k8s/labels-annotations.md

Selectors

• A selector is an expression matching labels

• It will restrict a command to the objects matching at least all these labels

k8s/labels-annotations.md

Settings labels and annotations

• The easiest method is to use kubectl label and kubectl annotate

k8s/labels-annotations.md

Other ways to view labels

• kubectl get gives us a couple of useful flags to check labels

• kubectl get --show-labels shows all labels

• kubectl get -L xyz shows the value of label xyz

k8s/labels-annotations.md

Other ways to view labels

• We can use the --show-labels flag with kubectl get

k8s/labels-annotations.md

Differences between labels and annotations

• The key for both labels and annotations:

  • must start and end with a letter or digit

  • can also have . - _ (but not in first or last position)

  • can be up to 63 characters, or 253 + / + 63

• Label values are up to 63 characters, with the same restrictions

• Annotations values can have arbitrary characters (yes, even binary)

• Maximum length isn't defined

  (dozens of kilobytes is fine, hundreds maybe not so much)

Revisiting kubectl logs

• In this section, we assume that we have a Deployment with multiple Pods

  (e.g. pingpong that we scaled to at least 3 pods)

• We will highlights some of the limitations of kubectl logs

k8s/kubectl-logs.md

Streaming logs of multiple pods

• By default, kubectl logs shows us the output of a single Pod

kubectl logs only shows us the logs of one of the Pods.

k8s/kubectl-logs.md

Viewing logs of multiple pods

• When we specify a deployment name, only one single pod's logs are shown

• We can view the logs of multiple pods by specifying a selector

• If we check the pods created by the deployment, they all have the label app=pingpong

  (this is just a default label that gets added when using kubectl create deployment)

k8s/kubectl-logs.md

Streaming logs of multiple pods

• Can we stream the logs of all our pingpong pods?

Note: combining -l and -f is only possible since Kubernetes 1.14!

Let's try to understand why ...

k8s/kubectl-logs.md

Shortcomings of kubectl logs

• We don't see which pod sent which log line

• If pods are restarted / replaced, the log stream stops

• If new pods are added, we don't see their logs

• To stream the logs of multiple pods, we need to write a selector

• There are external tools to address these shortcomings

  (e.g.: Stern)

k8s/kubectl-logs.md

Accessing logs from the CLI

• The kubectl logs command has limitations:

  • it cannot stream logs from multiple pods at a time

  • when showing logs from multiple pods, it mixes them all together

• We are going to see how to do it better

k8s/logs-cli.md

Doing it manually

• We could (if we were so inclined) write a program or script that would:

  • take a selector as an argument

  • enumerate all pods matching that selector (with kubectl get -l ...)

  • fork one kubectl logs --follow ... command per container

  • annotate the logs (the output of each kubectl logs ... process) with their origin

  • preserve ordering by using kubectl logs --timestamps ... and merge the output

Stern

Stern is an open source project originally by Wercker.

From the README:

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.

The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed.

Exactly what we need!

k8s/logs-cli.md

Checking if Stern is installed

• Run stern (without arguments) to check if it's installed:

  $ stern
  Tail multiple pods and containers from Kubernetes
  Usage:
  stern pod-query [flags]

• If it's missing, let's see how to install it

k8s/logs-cli.md

Installing Stern

• Stern is written in Go

• Go programs are usually very easy to install

  (no dependencies, extra libraries to install, etc)

• Binary releases are available here on GitHub

• Stern is also available through most package managers

  (e.g. on macOS, we can brew install stern or sudo port install stern)

k8s/logs-cli.md

Using Stern

• There are two ways to specify the pods whose logs we want to see:

  • -l followed by a selector expression (like with many kubectl commands)

  • with a "pod query," i.e. a regex used to match pod names

• These two ways can be combined if necessary

k8s/logs-cli.md

Stern convenient options

• The --tail N flag shows the last N lines for each container

  (Instead of showing the logs since the creation of the container)

• The -t / --timestamps flag shows timestamps

• The --all-namespaces flag is self-explanatory

k8s/logs-cli.md

Using Stern with a selector

• When specifying a selector, we can omit the value for a label

• This will match all objects having that label (regardless of the value)

• Everything created with kubectl run has a label run

• Everything created with kubectl create deployment has a label app

• We can use that property to view the logs of all the pods created with kubectl create deployment

Namespaces

• We would like to deploy another copy of DockerCoins on our cluster

• We could rename all our deployments and services:

  hasher → hasher2, redis → redis2, rng → rng2, etc.

• That would require updating the code

• There has to be a better way!

Identifying a resource

• We cannot have two resources with the same name

  (or can we...?)

Uniquely identifying a resource

• For namespaced resources:

  the tuple (kind, name, namespace) needs to be unique

• For resources at the cluster scope:

  the tuple (kind, name) needs to be unique

k8s/namespaces.md

Pre-existing namespaces

• If we deploy a cluster with kubeadm, we have three or four namespaces:

  • default (for our applications)

  • kube-system (for the control plane)

  • kube-public (contains one ConfigMap for cluster discovery)

  • kube-node-lease (in Kubernetes 1.14 and later; contains Lease objects)

• If we deploy differently, we may have different namespaces

k8s/namespaces.md

Creating namespaces

• Let's see two identical methods to create a namespace

k8s/namespaces.md

Using namespaces

• We can pass a -n or --namespace flag to most kubectl commands:

  kubectl -n blue get svc

• We can also change our current context

• A context is a (user, cluster, namespace) tuple

• We can manipulate contexts with the kubectl config command

k8s/namespaces.md

Viewing existing contexts

• On our training environments, at this point, there should be only one context

• The current context (the only one!) is tagged with a *

• What are NAME, CLUSTER, AUTHINFO, and NAMESPACE?

k8s/namespaces.md

What's in a context

• NAME is an arbitrary string to identify the context

• CLUSTER is a reference to a cluster

  (i.e. API endpoint URL, and optional certificate)

• AUTHINFO is a reference to the authentication information to use

  (i.e. a TLS client certificate, token, or otherwise)

• NAMESPACE is the namespace

  (empty string = default)

k8s/namespaces.md

Switching contexts

• We want to use a different namespace

• Solution 1: update the current context

  This is appropriate if we need to change just one thing (e.g. namespace or authentication).

• Solution 2: create a new context and switch to it

  This is appropriate if we need to change multiple things and switch back and forth.

• Let's go with solution 1!

k8s/namespaces.md

Updating a context

• This is done through kubectl config set-context

• We can update a context by passing its name, or the current context with --current

k8s/namespaces.md

Using our new namespace

• Let's check that we are in our new namespace, then deploy a new copy of Dockercoins

k8s/namespaces.md

Deploying DockerCoins with YAML files

• The GitHub repository jpetazzo/kubercoins contains everything we need!

If the argument behind -f is a directory, all the files in that directory are processed.

The subdirectories are not processed, unless we also add the -R flag.

k8s/namespaces.md

Viewing the deployed app

• Let's see if this worked correctly!

If the graph shows up but stays at zero, give it a minute or two!

k8s/namespaces.md

Namespaces and isolation

• Namespaces do not provide isolation

• A pod in the green namespace can communicate with a pod in the blue namespace

• A pod in the default namespace can communicate with a pod in the kube-system namespace

• CoreDNS uses a different subdomain for each namespace

• Example: from any pod in the cluster, you can connect to the Kubernetes API with:

  https://kubernetes.default.svc.cluster.local:443/

k8s/namespaces.md

Isolating pods

• Actual isolation is implemented with network policies

• Network policies are resources (like deployments, services, namespaces...)

• Network policies specify which flows are allowed:

  • between pods

  • from pods to the outside world

  • and vice-versa

k8s/namespaces.md

Switch back to the default namespace

• Let's make sure that we don't run future exercises and labs in the blue namespace

Note: we could have used --namespace=default for the same result.

k8s/namespaces.md

Switching namespaces more easily

• We can also use a little helper tool called kubens:

  # Switch to namespace foo
  kubens foo
  # Switch back to the previous namespace
  kubens -

• On our clusters, kubens is called kns instead

  (so that it's even fewer keystrokes to switch namespaces)

k8s/namespaces.md

kubens and kubectx

• With kubens, we can switch quickly between namespaces

• With kubectx, we can switch quickly between contexts

• Both tools are simple shell scripts available from https://github.com/ahmetb/kubectx

• On our clusters, they are installed as kns and kctx

  (for brevity and to avoid completion clashes between kubectx and kubectl)

k8s/namespaces.md

kube-ps1

• It's easy to lose track of our current cluster / context / namespace

• kube-ps1 makes it easy to track these, by showing them in our shell prompt

• It is installed on our training clusters, and when using shpod

• It gives us a prompt looking like this one:

  [123.45.67.89] (kubernetes-admin@kubernetes:default) docker@node1 ~

  (The highlighted part is context:namespace, managed by kube-ps1)

• Highly recommended if you work across multiple contexts or namespaces!

k8s/namespaces.md

Installing kube-ps1

• It's a simple shell script available from https://github.com/jonmosco/kube-ps1

• It needs to be installed in our profile/rc files

  (instructions differ depending on platform, shell, etc.)

• Once installed, it defines aliases called kube_ps1, kubeon, kubeoff

  (to selectively enable/disable it when needed)

• Pro-tip: install it on your machine during the next break!

Deploying with YAML

• So far, we created resources with the following commands:

  • kubectl run

  • kubectl create deployment

  • kubectl expose

• We can also create resources directly with YAML manifests

k8s/yamldeploy.md

kubectl apply vs create

• kubectl create -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, don't alter them
    (and display error message)

• kubectl apply -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, update them
    (to match the definition provided by the YAML file)

  • stores the manifest as an annotation in the resource

k8s/yamldeploy.md

Creating multiple resources

• The manifest can contain multiple resources separated by ---

 kind: ...
 apiVersion: ...
 metadata: ...
   name: ...
 ...
 ---
 kind: ...
 apiVersion: ...
 metadata: ...
   name: ...
 ...

k8s/yamldeploy.md

Creating multiple resources

• The manifest can also contain a list of resources

 apiVersion: v1
 kind: List
 items:
 - kind: ...
   apiVersion: ...
   ...
 - kind: ...
   apiVersion: ...
   ...

k8s/yamldeploy.md

Deploying dockercoins with YAML

• We provide a YAML manifest with all the resources for Dockercoins

  (Deployments and Services)

• We can use it if we need to deploy or redeploy Dockercoins

(If we deployed Dockercoins earlier, we will see warning messages, because the resources that we created lack the necessary annotation. We can safely ignore them.)

k8s/yamldeploy.md

Deleting resources

• We can also use a YAML file to delete resources

• kubectl delete -f ... will delete all the resources mentioned in a YAML file

  (useful to clean up everything that was created by kubectl apply -f ...)

• The definitions of the resources don't matter

  (just their kind, apiVersion, and name)

k8s/yamldeploy.md

Pruning¹ resources

• We can also tell kubectl to remove old resources

• This is done with kubectl apply -f ... --prune

• It will remove resources that don't exist in the YAML file(s)

• But only if they were created with kubectl apply in the first place

  (technically, if they have an annotation kubectl.kubernetes.io/last-applied-configuration)

¹If English is not your first language: to prune means to remove dead or overgrown branches in a tree, to help it to grow.

k8s/yamldeploy.md

YAML as source of truth

• Imagine the following workflow:

  • do not use kubectl run, kubectl create deployment, kubectl expose ...

  • define everything with YAML

  • kubectl apply -f ... --prune --all that YAML

  • keep that YAML under version control

  • enforce all changes to go through that YAML (e.g. with pull requests)

• Our version control system now has a full history of what we deploy

• Compares to "Infrastructure-as-Code", but for app deployments

k8s/yamldeploy.md

Declarative vs imperative

• Our container orchestrator puts a very strong emphasis on being declarative

• Declarative:

  I would like a cup of tea.

• Imperative:

  Boil some water. Pour it in a teapot. Add tea leaves. Steep for a while. Serve in a cup.

Declarative vs imperative

• What declarative would really be:

  I want a cup of tea, obtained by pouring an infusion¹ of tea leaves in a cup.

Declarative vs imperative

• Imperative systems:

  • simpler

  • if a task is interrupted, we have to restart from scratch

• Declarative systems:

  • if a task is interrupted (or if we show up to the party half-way through), we can figure out what's missing and do only what's necessary

  • we need to be able to observe the system

  • ... and compute a "diff" between what we have and what we want

shared/declarative.md

Declarative vs imperative in Kubernetes

• With Kubernetes, we cannot say: "run this container"

• All we can do is write a spec and push it to the API server

  (by creating a resource like e.g. a Pod or a Deployment)

• The API server will validate that spec (and reject it if it's invalid)

• Then it will store it in etcd

• A controller will "notice" that spec and act upon it

k8s/declarative.md

Reconciling state

• Watch for the spec fields in the YAML files later!

• The spec describes how we want the thing to be

• Kubernetes will reconcile the current state with the spec
  (technically, this is done by a number of controllers)

• When we want to change some resource, we update the spec

• Kubernetes will then converge that resource

19,000 words

They say, "a picture is worth one thousand words."

The following 19 slides show what really happens when we run:

kubectl create deployment web --image=nginx

k8s/deploymentslideshow.md

Authoring YAML

• We have already generated YAML implicitly, with e.g.:

  • kubectl run

  • kubectl create deployment (and a few other kubectl create variants)

  • kubectl expose

• When and why do we need to write our own YAML?

• How do we write YAML from scratch?

k8s/authoring-yaml.md

The limits of generated YAML

• Many advanced (and even not-so-advanced) features require to write YAML:

  • pods with multiple containers

  • resource limits

  • healthchecks

  • DaemonSets, StatefulSets

  • and more!

• How do we access these features?

k8s/authoring-yaml.md

Various ways to write YAML

• Completely from scratch with our favorite editor

  (yeah, right)

• Dump an existing resource with kubectl get -o yaml ...

  (it is recommended to clean up the result)

• Ask kubectl to generate the YAML

  (with a kubectl create --dry-run=client -o yaml)

• Use The Docs, Luke

  (the documentation almost always has YAML examples)

k8s/authoring-yaml.md