• Install basic ubuntu image using Win32DiskImager
    1. ubuntu-20.04.1-preinstalled-server-arm64+raspi.img
  • Plug in ethernet, get IP and SSH in.  Takes a few seconds to generate SSL.
sudo apt update
sudo apt upgrade
  • Install net-tools
sudo apt install net-tools
sudo apt install network-manager
  • Use nmcli to connect to WiFi if desired
nmcli r wifi on
nmcli d wifi list
sudo nmcli d wifi connect <ssid> password <password>
  • Change host name to something unique if creating a cluster
sudo vi /etc/hostname
  • Add enable cgroups
sudo vi /boot/firmware/cmdline.txt
Add -> cgroup_enable=memory cgroup_memory=1
  • Reboot
  • Install Docker
sudo snap install docker
  • Install microK8S
sudo snap install microk8s --classic
If you don’t want to type SUDO all the time do this:
sudo usermod -a -G microk8s $USER
mick
  • Reboot
  • Start microk8s
microk8s start
microk8s status
microk8s inspect
  • Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool.
sudo iptables -P FORWARD ACCEPT
  • Edit /etc/hosts to add each node hostname and IP  (note these are the VIRTUAL NIC IPs) use ifconfig to get them.  This way they can all find each other.
127.0.0.1 localhost
10.1.18.0 kube1
10.1.181.64 kube4
10.1.190.64 kube5
10.1.9.128 kube3
  • Use ping to check health
  • Run this to make sure everything is healthy
microk8s kubectl get all --all-namespaces
sudo microk8s inspect

Do all steps above on all Pi’s you want in the cluster.  BE PATIENT!  Don’t jump to conclusions and make rapid changes because a nodes does not show up or the status does not return quickly.  You will jam things up if you go to fast enabling things like dashboard.  It takes time for them to start running.

Always use these watched commands to monitor progress so you do not go too fast.

watch -n 5 microk8s kubectl get nodes
watch -n 5 microk8s kubectl get all --all-namespaces

Pick a Pi to be the master

microk8s add-node

Go to Pi that is being added to master and run the command that add-node output

microk8s join 10.11.0.163:25000/c06c125e577e113d8c7a31b55b520808

Verify 

microk8s status --wait-ready //(this is the be patient part.  It waits until the service is ready to get status.

microk8s.kubectl get nodes

After adding three Pi they will automatically start using HA (High Availability)

If things get unstable you can reset everything.  For nuclear you can run snap remove then install

sudo snap remove microk8s

For a less nuclear you can reset your microk8s node to default.

microk8s reset

You can then do microk8s add-node to rejoin the cluster

Great resource on debugging DNS

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

My pods can’t reach the internet or each other (but my MicroK8s host machine can)…

Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool. Such changes can be made persistent by installing the iptables-persistent package:

sudo iptables -P FORWARD ACCEPT

apt-get install iptables-persistent

su or sudo -i

iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

If you want to push the Dashboard to be external facing if your Unbutu does not have a GUI. Change the yaml “type” from ClusterIP to NodePort.

microk8s enable dashboard 

WAIT FOR IT TO INSTALL AND RUN

microk8s kubectl -n kube-system edit service kubernetes-dashboard

Change ClusterIP to NodePort

Get the mapped port of the dash-board service

ubuntu@kube3:~$ microk8s kubectl -n kube-system get services

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE

kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   126m

metrics-server              ClusterIP   10.152.183.189   <none>        443/TCP                  13m

dashboard-metrics-scraper   ClusterIP   10.152.183.11    <none>        8000/TCP                 12m

kubernetes-dashboard        NodePort    10.152.183.139   <none>        443:32746/TCP            12m

ubuntu@kube3:~$ sudo lsof -i tcp:32746

COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

kube-prox 75772 root    7u  IPv4 570050      0t0  TCP *:32746 (LISTEN)

Now you can hit one of the Pi’s IPs on port 32414 on https to load the Dashboard.

https://10.11.0.163:30804/

To login get a token (if you have not already set a password for su;  sudo passwd):

su

token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d ” ” -f1)

microk8s kubectl -n kube-system describe secret $token

Useful commands

sudo microk8s kubectl get deployment -n kube-system kubernetes-dashboard -o yaml

sudo microk8s.kubectl get all –all-namespaces

sudo microk8s kubectl get nodes

sudo microk8s.kubectl exec -i -t dnsutils — nslookup kubernetes.default

sudo microk8s kubectl exec -i -t dnsutils — ifconfig

sudo microk8s inspect

sudo microk8s status

sudo microk8s start

sudo microk8s add-node

microk8s remove-node 10.22.254.79

microk8s leave

sudo microk8s kubectl logs -n kube-system sukubernetes-dashboard-7ffd448895-klqnb