Kubernetes (almost) in your Pocket

There are tons of options to learn and play with Kubernetes. Everything from managed services like Amazon EKS down to minikube on your local machine. When I started learning, I quickly realized I wanted a cluster of physcial machines to play with.

Hardware

Research and false starts

When starting on my plan to build a cluster, I set out some goals for myself:

  • At least 5 nodes
  • Two cables out the back - network and power
  • x86 based
  • Portable

With those thoughts in mind, I started my research the way I always do: Googling stuff. There were quite a few examples of smaller x86 clusters out there, including one that caught my attention made of “stick” pcs. It hit all my goals, someone had blazed the trail so I knew it would work…slam dunk.

And then I ran into hurdle number one: Supply on some of these compute sticks/single board computers is very limited. When I tried to purchase the same pc sticks as the writer, they were unavailable and the only similar ones on the market were almost 3 times the price he’d paid - putting this project well over what I wanted to spend.

I spent several months exploring other options, but didn’t come up with anything satisfactory. They were either too large, completely unavailable to consumers, or too expensive. I decided to shelve the project and built a boring 3 node cluster in Hyper-V on my local machine.

Why not Arm?

Put away the pitchforks, ARM is plenty cool. I have ARM nodes in my kubernetes cluster at home (which I’ll write more about in a later post).

I wanted to go x86 for two main reasons: 1. It was more difficult 2. More container availability

Honestly, it just felt cooler somehow to do it with x86 chips. I could have built a Raspberry Pi cluster in a weekend (with parts on hand, lets not talk about the box of pi 2s and 3s gathering dust) but it wouldn’t have been as interesting

Atomic powered

Enter the Atomic Pi.

I came across the Atomic Pi thanks to a post on Linux Gizmos and it re-ignited the spark for this project.

Initial testing proved positive so I bought 5 nodes and started planning the cluster. I grabbed an old itx case I had from another project and a small ATX psu to power everything. A gigabit switch from dlink rounded out v1s parts.

(Side note: I know I’m being rather short on details in this post. Check this link for a spreadsheet detailing parts and sources. Future posts will have more detailed build logs. Think of this as a very long tl;dr)

Here came mistake #1: I can do it better/cheaper than they can. The people behind the Atomic Pis offer an afforable power board (under $10 usd) that would have worked fine. I decided to build my own. That’s how I fried the first Pi (only hardware killed in the entire project!).

After a few revisions, I ended up with custom power boards that could power all 5 Pis from the PSU, as well as the switch. With the use of some brass standoffs, some creative drilling, and plenty of zipties, the cluster went into the case and the hardware was “done”. I’ve made a few minor adjustments since then, but the main hardware is all still the same

ToDo: Glamour shots of the current cluster

Software

Do it Myself

If reading about some of my hardware decisions hasn’t made it clear: I can be stubborn. I figured if I was going to learn kubernetes, I should LEARN it and write my own config management to deploy it. So I did.

I wrote a bunch of Ansible that could deploy kubernetes on top of Alpine nodes and it would mostly work, sometimes, depending on the phases of the moon.

Don’t do this. Go use other people’s work. I use for my main home cluster. It’s fantastic

Enter k3s/k3os

After getting a cluster “working” using ansible, I wasn’t happy with the results. I hadn’t really tuned or scaled the cluster to the hardware so it wasn’t usable - just the basic kube-system services were consuming most of the nodes resources.

Enter k3s from Rancher labs. k3os

ToDo: Write more about software, do basic demo

ToDo: Write a conclusion

- Andrew Herrington