Setting Up a Raspberry Pi Kubernetes Cluster

Christian Højsager included in Kubernetes Raspberry Pi

2022-01-15 1652 words 8 minutes

/images/setting-up-a-raspberry-pi-kubernetes-cluster/featured_image.jpeg

Contents

In the last couple of months, I have been focused a lot on learning new DevOps technologies such as Kubernetes, Ansible, Terraform, and Jenkins Pipelines. And as I was learning about Kubernetes I decided to build my own Raspberry Pi Kubernetes cluster for a homelab. In this post, I will walk through how I build my Kubernetes Cluster, with high available master nodes running k3s lightweight Kubernetes. Just a few things to be aware of, if you decide to follow this guide to build your own Raspberry Pi Cluster. In this guide, I am using Docker instead of the default containerd as the Kubernetes container technology. If you want to use containerd instead, I will explain later down how to achieve this.

What you will need for the build

Minimum requirements:

2x Raspberry Pi 3/4 Model B — Link
Power supply for the Raspberry Pi’s

Part list of my build:

6x Raspberry Pi 3/4 Model B — Link
C4Labs Zebra Bramble Case — Link
6 port 60W USB charger — Link
NetGear 8-port Gigabit Switch — Link

Installing an Operating System on the Pi’s

The Operating System. I am installing on the Pi’s is Ubuntu 64 bit arm. To image the sd card I will be using the software Raspberry Pi Imager, which can be downloaded here. The Raspberry Pi Imager already contains the ubuntu OS, so there is no need to manually download it.

To install the OS on an SD card, place the SD card into your computer and open the Imager software.

Then click on “Choose OS”, then “Other general purpose OS”, then “Ubuntu” and then choose the OS “Ubuntu Server 20.04.3 LTS (RPI 3/4/400) – 64-bit OS”

Be sure not to choose version 21.10 I had a lot of trouble getting Kubernetes to work on it.

Then on the main screen of the Raspberry Pi Imager, choose Storage and choose the SD card you placed in your computer.

Then press Write to install the OS on the SD card.

Initial Setup of the Raspberry Pi

Set a static IP Addres
Change the hostname of the Pi
enable cgroup_memory
Update and Upgrade the OS
Install Docker

the first time you log in to the Pi you will need to use the username “ubuntu” and the password “ubuntu”. You will immediately be forced to change the default password. Now I always recommend creating a new user and locking down the default user, but for this guide, I will keep using the default user Ubuntu, if you want to change the user you can find a lot of great guides online by doing a quick google search.

Once you have changed the default password, you can ssh back into the Pi with your new password, and you will be ready to continue.

Setting a static ip address

we need to configure all the nodes in the cluster with a static IP address since otherwise none of the nodes will be able to communicate with each other.

you can set a static IP address by editing the file /etc/netplan/50-cloud-init.yaml

By default, the netplan will look similar to below

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eth0:
            dhcp4: true
            optional: true

Now, to set a static IP address you will need to provide the IP, the default gateway, and the DNS servers. You can do this by using the yaml below, just change the IP’s for your own.

1
2
3
4
5
6
7
8
9


network:
    ethernets:
        eth0:
            dhcp4: false
            optional: true
                addresses: [192.168.1.20/24]
        gateway4: 192.168.1.1
        nameservers:
            addresses: [1.1.1.1,1.0.0.1]

Once you have changed the file you can set it into effect by running the command

1

sudo netplan apply

If you don’t see any warning or error it should have been applied successfully. You can run the command ip a to see if the IP address is correct.

Setting a hostname on the Pi

To change the hostname you will need to edit the file /etc/hostname. You can name the nodes whatever you want, but it might be a good idea to name them according to which is a master and which is a worker.

1

sudo vim /etc/hostname

Enabling cgroup_memory

Now, this part you will need to do in order for the Kubernetes service to be able to run on your Pi. I won’t go into details on what it is, but it is basically some Linux kernel feature control group which you will need to set correctly to get k3s to work.

to enable cgroup_memory you will need to edit the file /boot/firmware/cmdling.txt. In the file, you will need to add the following to the beginning of the line

1

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

And you final file should look similar to below

1

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc

Installing Docker

To install docker you will need to start updating your apt packages. Run the command below.

1

sudo apt update && sudo apt upgrade -y

Then to install docker you can run:

1

sudo apt install docker.io -y

You can run the command below to check that the docker installation worked.

1
2


$ docker --version
Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.2

Now restart the Raspberry Pi

1

sudo reboot now

Setting up the Raspberry Pi Cluster nodes

All the previous steps in this guide should be repeated for all the Raspberry Pi’s in your cluster. Now once you have done this you are ready to continue with the next part.

Here we will install the k3s cluster with high available master nodes, and we will add the worker nodes to the cluster.

Configuring the first master node

Setting up a Raspberry Pi k3s node is as simple as running a single command. The command consists of a couple of parts.

curl -sfL https://get.k3s.io Is used for downloading the k3s package

K3S_KUBECONFIG_MODE=’644′ Is used if you ever want to add the cluster to a Rancher UI for management. If you don’t want to use Rancher you can still leave the config in.

*K3S_TOKEN=* Is used to create a token that you can use to add the Kubernetes nodes to the cluster. This is just a random string with characters and symbols you define.

sh -s – server –cluster-init –disable servicelb –docker Is used for installing the cluster package and configuring the master node for high availability with Docker as the container runtime.

Now run the complete command on your first Master node

1

sudo curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE='644' K3S_TOKEN=JKLklncasal123elmasd!@pokasdOASD sh -s - server --cluster-init --disable servicelb --docker

you should now the see k3s service being downloaded and installed.

Once the process is done, you can check if the k3s service is running by running the command:

1

And here you should see that the service is active and running.

Configuring the second master node

To configure the second master node to the cluster is as simple as running a simple command.

1

curl -sfL https://get.k3s.io | K3S_TOKEN=JKLklncasal123elmasd!@pokasdOASD sh -s - server --server https://192.168.1.20:6443 --no-deploy servicelb --docker

Here you will need to use the same K3S_TOKEN as when you configured the first master node. You will also need to change the IP address of the server to the master nodes IP address.

Once the command has been completed you can ssh into the first master node and run the command below, to check that it worked:

1

sudo kubectl get nodes

And you should see a similar output to below

NAME       STATUS   ROLES                       AGE   VERSION
master01   Ready    control-plane,etcd,master   15d   v1.22.5+k3s1
master02   Ready    control-plane,etcd,master   15d   v1.22.5+k3s1

Adding the worker nodes to the Kubernetes cluster

Again as with the master nodes, you can add worker nodes to the cluster with a single command.

1

curl -sfL https://get.k3s.io | K3S_URL=https://192.168.1.20:6443 K3S_TOKEN=JKLklncasal123elmasd!@pokasdOASD sh - --docker

Now, this command should be run on every worker node you want to join to the Kubernetes cluster.

Again, to check that it worked you can run the command below on the master node.

1

sudo kubectl get nodes

And you should see an output listing all nodes and their status in the cluster.

NAME       STATUS   ROLES                       AGE   VERSION
master01   Ready    control-plane,etcd,master   15d   v1.22.5+k3s1
master02   Ready    control-plane,etcd,master   15d   v1.22.5+k3s1
worker01   Ready                          15d   v1.22.5+k3s1
worker02   Ready                          15d   v1.22.5+k3s1
worker03   Ready                          15d   v1.22.5+k3s1
worker04   Ready                          15d   v1.22.5+k3s1

If you want to check that the cluster nodes are using docker as their container runtime, you can run the command below

1

sudo kubectl get nodes -o wide

and in the output, you should see that the container runtime is set to docker.

NAME       STATUS   ROLES                       AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
master01   Ready    control-plane,etcd,master   15d   v1.22.5+k3s1   192.168.1.20           Ubuntu 20.04.3 LTS   5.4.0-1048-raspi   docker://20.10.7

Now if something has gone wrong and it has not been configured with docker the output should look similar to below.

NAME       STATUS   ROLES                       AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
master01   Ready    control-plane,etcd,master   15d   v1.22.5+k3s1   192.168.1.20           Ubuntu 20.04.3 LTS   5.4.0-1048-raspi   containerd://1.5.8-k3s1

Conclusion

In my process of learning how to build this cluster, I learned a lot regarding working in Linux and about the Kubernetes technology. I have also created an Ansible playbook for automating almost the entire process in this guide. I will soon create a blog article explaining how to do this with Ansible.

I think running a Kubernetes cluster on Raspberry Pi’s is a great way of getting started with Kubernetes in a cheap way. There are endless possibilities in how you can utilize the Raspberry Pi cluster for training on Kubernetes technologies. You can even use a service as Teleport to gain access to the cluster from anywhere in the world, using the cluster as your own private cloud.