Kubernetes on Ubuntu 22.04 with CRI-Docker
Installation of Kubernetes Server
Introduction
We will install a Kubernetes on Ubuntu 22.04 with Docker and CRI-Docker. We are using an r5.2xlarge on Amazon EC2 using Amazon’s provided Ubuntu 22.04. You can get by with a much smaller instance depending on what you plan to run.
Installation
Update Ubuntu
Install the updates on your fresh server from Amazon.
## Update Ubunutu
sudo bash
sudo apt update -y
sudo apt upgrade -y
sudo apt autoremove -y
# if you see kenerl updates you should reboot
# init 6
Hostname
Your hostname should be in /etc/hosts. If its not there, put it there. If this is a fresh Amazon instance fix the hostname first.
sudo hostnamectl set-hostname k8
echo "172.31.74.59 k8.nathanobert.com k8" | sudo tee -a /etc/hosts
hostname
hostname --fqdn
# reboot
init 6
Install Docker
Documentation at https://docs.docker.com/engine/install/ubuntu/
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
sudo docker run hello-world
Verify Docker works with regular user account
sudo usermod -aG docker $USER
newgrp docker
docker run hello-world
CRI-Docker
Container Runtime Interface - Docker allows for you to use a CRI compliant interface to Kubernetes that back-ends to Docker. This allows you to share the images between Kubernetes and Docker. You can build an image with Docker, and then immediately its available to kubernetes without the need to push the image to a repository with Docker, and pull the image back with Kubernetes. Very useful for development.
Documentation at https://github.com/Mirantis/cri-dockerd/releases
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd_0.3.1.3-0.ubuntu-jammy_amd64.deb
sudo apt install ./cri-dockerd_0.3.1.3-0.ubuntu-jammy_amd64.deb -y
sudo systemctl status docker.service
sudo systmctl status cri-docker.service
Kubernetes
Install Kubernetes, based on https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
# This will prevent updates from breaking Kubernetes unexpectidly
sudo apt-mark hold kubelet kubeadm kubectl
Set selinux to permissive
Amazon’s image did not have selinux enabled, but you want it off.
sudo sestatus
# how to disable if needed
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
sudo sestatus
Disable swap
Amazon’s image did not have swap enabled, but you dont want swap!
sudo swapoff -a
# amazon workspace (ubuntu)
sudo rm -f /etc/udev/rules.d/10-zram0.rules
# regular ubuntu
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
free -m
Enable the kernel modules: overlay & br_netfilter
These kernel modules are necessary
sudo modprobe overlay
sudo modprobe br_netfilter
Create script to load modules on every reboot
These modules are necessary
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
Remove Resolved
If you have 127.0.0.1 in your /etc/resolv.conf file most likely you will have coredns problems when you start Kubernetes. Essentially there will be a loop inside of Kubernetes and it won’t know how to get to a DNS Server. These are the steps to bypass resolved and go directly to a real DNS server. Note you can choose your own DNS servers. 8.8.x.x. is google :)
ls -ltr /etc/resolv.conf
# lrwxrwxrwx 1 root root 39 Mar 25 03:04 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
sudo rm /etc/resolv.conf
echo "nameserver 8.8.8.8" | sudo tee -a /etc/resolv.conf
echo "nameserver 8.8.4.4" | sudo tee -a /etc/resolv.conf
echo "nameserver 1.1.1.1" | sudo tee -a /etc/resolv.conf
echo "nameserver 1.0.0.1" | sudo tee -a /etc/resolv.conf
echo "options edns0 trust-ad" | sudo tee -a /etc/resolv.conf
# this should be your domain name
echo "search nathanobert.com" | sudo tee -a /etc/resolv.conf
sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
At this point you need to decide which ip address you want to choose to use, as well as which Container Network Interface. We are choosing Calico with the default IP Range of 192.168.10.0/16. We are also assuming the control plan endpoint is this local machine’s hostname that we put in /etc/hosts. The two commans in the hostname force it to be in all lowercase to follow RFC-1123 DNS subdomain standards.
Start Kubernetes
sudo systemctl enable --now kubelet
sudo kubeadm config images pull --cri-socket /var/run/cri-dockerd.sock
sudo kubeadm init --pod-network-cidr=192.168.10.0/16 --control-plane-endpoint {`hostname --fqdn`,,} --cri-socket /var/run/cri-dockerd.sock
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo export KUBECONFIG=/etc/kubernetes/admin.conf | sudo tee -a /root/.bashrc
Taints
Remote taints from Node so it can schedule containers on this Kubernetes node. (Note: the master taint is usually not necessary)
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
Calico
Install Container Network Interface (CNI) - Calico. Note if you choose a different IP Range you will need to modify this Yaml file. If you used the default of 192.168.10/16, you will be happy. Also verify you are using the newest/stable calico.
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/calico.yaml
# commands to troubleshoot
ip addr | grep cni
kubectl get pods -n kube-system
HELM
Most of the time people want HELM if they are installing Kubernetes.
Documentation: https://helm.sh/docs/intro/install/
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
HA Proxy Ingress Controller
There are different ingress controllers, here is how you add HA Proxy if you would like it. Documentation: https://haproxy-ingress.github.io/docs/getting-started/
helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts
kubectl cluster-info
echo "controller:" > haproxy-ingress-values.yaml
echo " hostNetwork: true" >> haproxy-ingress-values.yaml
helm install haproxy-ingress haproxy-ingress/haproxy-ingress --create-namespace --namespace ingress-controller --version 0.14.2 -f haproxy-ingress-values.yaml
kubectl --namespace ingress-controller get services haproxy-ingress -o wide
# Note mine always says <pending> under EXTERNAL-IP