Kubernetes on Ubuntu 20.04

Install K8s

 1# remove docker if installed via snap 
 2sudo snap remove docker
 3
 4sudo apt install -y docker.io
 5sudo apt install -y apt-transport-https curl
 6
 7# switch to systemd
 8sudo tee /etc/docker/daemon.json <<EOF
 9{
10  "exec-opts": ["native.cgroupdriver=systemd"],
11  "log-driver": "json-file",
12  "log-opts": {
13    "max-size": "100m"
14  },
15  "storage-driver": "overlay2"
16}
17EOF
18
19## Enable Docker Service
20sudo systemctl enable docker.service
21
22sudo systemctl stop docker
23sudo systemctl start docker
24
25## Disable Swap
26sudo swapoff -a
27sudo sed -i '$ d' /etc/fstab
28
29
30## Add Kubernetes signing key, as of this writing xenial is latest
31curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
32sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
33
34## install kubernetes
35sudo apt install -y kubeadm=1.19.5-00 kubelet=1.19.5-00 kubectl=1.19.5-00 kubernetes-cni=0.8.7-00
36sudo apt-mark hold kubeadm=1.19.5-00 kubelet=1.19.5-00 kubectl=1.19.5-00 kubernetes-cni=0.8.7-00
37sudo apt-mark showhold
38## only on master
39sudo kubeadm init --pod-network-cidr=10.244.0.0/16
40
41   ## -- or on worker -- ##
42
43## use join command as printed by init
44sudo kubeadm join 192.168.1.44:6443 --token 8n8r48.7n4wdkt42nw4j436 \
45    --discovery-token-ca-cert-hash sha256:89b472970d7a3332559b06a01ddbd1f341bc8e4261ad98aa07878dda3ba0e411
46
47###### stop here if this is a worker node
48
49## setup local kube config
50
51mkdir -p $HOME/.kube
52sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
53sudo chown $(id -u):$(id -g) $HOME/.kube/config
54
55## deploy pod networking
56kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
57kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml    
58
59## install metric server
60kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

POST Setup

install MetalLB

1kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
2kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
3# On first install only
4kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Create MetalLB config

 1apiVersion: v1
 2kind: ConfigMap
 3metadata:
 4  namespace: metallb-system
 5  name: config
 6data:
 7  config: |
 8    address-pools:
 9    - name: default
10      protocol: layer2
11      addresses:
12      - 192.168.1.200-192.168.1.220    

nfs provisioner

install nfs client on all nodes with

1sudo apt-get install -y nfs-common

Install Provisioner

1sudo snap install helm --classic
2helm repo add stable https://charts.helm.sh/stable
3
4helm repo update
5
6helm install nfs stable/nfs-client-provisioner --set nfs.server=192.168.1.87 --set nfs.path=/mnt/nfs_share --set storageClass.defaultClass=true

concourse

1helm repo add concourse https://concourse-charts.storage.googleapis.com/
2
3helm install concourse  concourse/concourse
4
5kubectl expose deployment concourse-web --target-port=8080 --port=80 --type=LoadBalancer --name lb-concourse
6
7helm install concourse concourse/concourse --set web.service.api.type=LoadBalancer  --set concourse.web.externalUrl=http://concourse.ellin.net --set concourse.web.bindPort=80 --set worker.persistence.enabled=false   --set postgresql.persistence.enabled=false 

ArgoCD

1kubectl create namespace argocd
2
3kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
4
5kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
6
7# get the password
8kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

Misc Linux options

resize partition

You can’t resize a mounted filesystem with parted and resize2fs won’t resize the underlying partition. The workaround is a bit tricky and you have to be careful to keep the start cylinder the same when doing this. What you do is, in fdisk you delete the partition and recreate it with a larger size making sure you keep the start location (cylinder) the same. The example below illustrates this:

1[root@temeria ~] fdisk /dev/sda

expand the partition in fdisk by deleting the partition, create a new one using the same starting cylinder,

Pertinent information marked with <------

 1WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to switch off the mode (command ‘c’) and change display units to sectors (command ‘u’).
 2Command (m for help): p
 3Disk /dev/sda: 25.8 GB, 25769803776 bytes
 4255 heads, 63 sectors/track, 3133 cylinders
 5Units = cylinders of 16065 * 512 = 8225280 bytes
 6Sector size (logical/physical): 512 bytes / 512 bytes
 7I/O size (minimum/optimal): 512 bytes / 512 bytes
 8Disk identifier: 0x00073409
 9Device Boot Start End Blocks Id System
10/dev/sda1 * 1 39 307200 83 Linux <------ Starts at 39
11Partition 1 does not end on cylinder boundary.
12/dev/sda2 39 2097 16534528 83 Linux
13Command (m for help): d <------  delete the original partition
14Partition number (1-4): 2
15Command (m for help): n <------ new partition
16Command action
17e extended
18p primary partition (1-4)
19p <------ primary
20Partition number (1-4): 2 <------ usually 2
21First cylinder (39-3133, default 39):
22Using default value 39  <------ Starting positiono of or
23Last cylinder, +cylinders or +size{K,M,G} (39-3133, default 3133):
24Using default value 3133  <------ default is full disk
25Command (m for help): p
26Disk /dev/sda: 25.8 GB, 25769803776 bytes
27255 heads, 63 sectors/track, 3133 cylinders
28Units = cylinders of 16065 * 512 = 8225280 bytes
29Sector size (logical/physical): 512 bytes / 512 bytes
30I/O size (minimum/optimal): 512 bytes / 512 bytes
31Disk identifier: 0x00073409
32Device Boot Start End Blocks Id System
33/dev/sda1 * 1 39 307200 83 Linux
34Partition 1 does not end on cylinder boundary.
35/dev/sda2 39 3133 24857598+ 83 Linux
36Command (m for help): w <------ write changes
37The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8)
38Syncing disks.

Reboot the system to ensure the partition table is reread.

Resize the filesystem

This is perhaps the simplest step. Simply execute the resize2fs command with your partition as an argument.

1[root@temeria ~] resize2fs /dev/sda2

Reset IpTables after kubeadm reset

If you run kubeadm reset on the master you may need to reset iptables before running kubeadm again.

1iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

Enable system trust on Ubuntu

1mkdir /usr/local/share/ca-certificates/k8s
2
3sudo cp file.crt /usr/local/share/ca-certificates/k8s
4
5sudo update-ca-certificates

file copeied to /etc/ssl/certs

Recreating the Join Command

To rebuild join command

1jeff@kube-0:~$ kubeadm token generate
2
3jeff@kube-0:~$ kubeadm token create <token> --print-join-command --ttl=0

retrieve cert

1openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout |
2openssl pkey -pubin -outform DER |
3openssl dgst -sha256

nfs client

1sudo apt-get install nfs-common

upgrade K8s

Do on master, then repeat on each worker

  1. Upgrade Kubeadm
1apt-mark unhold kubeadm && \
2apt-get update && apt-get install -y kubeadm=1.19.3-00 && \
3apt-mark hold kubeadm
  1. Drain the Node
1kubectl drain kube-2 --ignore-daemonsets
  1. Upgrade the Node.
1sudo kubeadm upgrade node
  1. Upgrade the Kubelet
1apt-mark unhold kubelet kubectl && \
2apt-get update && apt-get install -y kubelet=1.19.3-00 kubectl=1.19.3-00 && \
3apt-mark hold kubelet kubectl
4
5sudo systemctl daemon-reload
6sudo systemctl restart kubelet
  1. Uncorden the node
1  kubectl uncordon kube-2
comments powered by Disqus