I recently got frustrated with my home router, and wanted to replace it with my much more powerful home-hosting server that I use with KubeSail. However, most firewall software is not exactly container friendly. I've been using FireHOL for years and years, but there are no guides anywhere to let it play nicely on the same host as a Kubernetes cluster. I wanted to document a nice FireHOL configuration for anyone else looking to turn a spare PC into a router-and-cluster-in-one, so here we go!

FireHOL is a firewall and QOS program that we'll use to secure our router. You can install both with:

apt install firehol fireqos

Next up, let's edit /etc/firehol/firehol.conf and setup a few basic facts:

FireHOL uses interfaces and routers to group traffic. In our case, we'll create two groups of interfaces: local and remote. Our "local" group needs to include all of the interfaces that will be created by tools like Docker, containerd, etc:

And our other group will be just our internet-facing interface. We'll also go ahead and set a few sensible rules - obviously feel free to customize to your needs!

Next up we're going to need to define a bunch of router sections - most importantly, we need to ensure LAN <-> WAN Routing and NATing works properly. We also need to ensure that traffic between various container interfaces is allowed to flow freely.

And finally, let's allow all traffic between various container interfaces. This list isn't exhaustive - so you may want to add any additional networks here depending on your needs! It contains docker+ and vxlan+, interfaces used by Docker, and flannel+ cni+ cali+, interfaces uses by Kubernetes networking systems like Flannel and Calico. Remember that we're not overriding those systems, they will still be able to drop traffic as they desire, but the default rule should be 'allow' for these interfaces (which would be the default behavior before we installed FireHOL).

You can then refresh the FireHOL system with firehol start.

Next up, let's setup FireQOS, which will help us provide fair service for different kinds of traffic. For example, one persons bittorrent downloads shouldn't slow down someone elses game of StarCraft. As before, let's start with some basic facts:

Next up, let's define an interface, which tells fireQOS how much bandwidth we have available (and how fast we want to allow connections to be) for each network. We'll also add an outline of our classes, which are categories of traffic. Classes are limited to a particular speed, priority, or reserved bandwidth. They use match statements to select various source or destination ports in order to classify traffic.

This will differ depending on the operating system you used, but for those using modern Ubuntu, here is a Netplan configuration you may want to use:

Note that you may want to remove the mtu lines, as it should be correctly auto-detected. I tend to set it explicitly to avoid any packet fragmentation that may accidentally rear its head.

Assuming we're using this server as a DNS and DHCP server as well, we'll want to install dnsmasq.

apt install dnsmasq

A few things you'll want to set include:

We recommend k3s, a lightweight Kubernetes distribution that runs great on almost any sort of hardware. You can install it with:

curl -sfL https://get.k3s.io | sh -

After a few moments, you can check that things are healthy:

k3s kubectl get nodes

At which point, install KubeSail!

k3s kubectl apply -f https://byoc.kubesail.com/YOUR_USERNAME.yaml

In just a few seconds, you'll see a notification on the KubeSail.com dashboard about a new cluster. Give it a name and accept the cluster, and you're off to the races! If for some reason the cluster doesn't appear on your dashboard, you can check the logs of the agent with a command like: k3s kubectl -n kubesail-agent logs -f -l app=kubesail-agent.

Dynamic DNS will automatically be setup and should 'just work', assuming we setup our firewall based on the FireHOL settings above. You can access your cluster at:

http://cluster-name.username.ksdns.io/

There are a couple things I'd like to get to next - for example:

I'm sure there are other clever things I can do with my home-cluster-turned-router, but so far that's all there is to it!

Feel free to join us in chat if you have any questions or comments!