Today we will have beginner friendly look on how to house Kubernetes in Proxmox. But not in the old-world SSH/Ansible way we will do it in the AWS/Azure/GCP way. So we are talking about scaling from 10th to 100 Kubernetes in Minutes with automated Cluster creation and upgrades in a reproductive way.

Why do you need this?

I don't need this at all - its just funny to have the same functionality like a big cloud provider have in my tiny home-lab.
But if you don't have a scaleble Kubernetes platform in your company it could be go hard in today's service-oriented world which is dominated by major cloud providers, managing a private cloud efficiently is more crucial than ever and one of the most loved tools in the Cloud are Kuberenetes. So how can you keep up? The answer lies in one of my favorite open source Projects Cluster-API and Proxmox.

Proxmox
widly known in the opensource and home-lab community made in Vienna and got a HUGE boost in small to medium scale private clouds after VMware fucked up there customers with there new pricing model. It's a simple but yet powerful Opensource Hypervisor based on KVM. So it's also the core of my home-lab for nearly a decade.

Cluster-API
less known in the home-lab community but loved and honored in enterprises and by Operations and Kubernetes enthusteds. So what is cluster-API actually. Cluster-API is a project under the CNCF and backed by the Kubernetes-Community with large support from variours vendor like VMware, Apple and NVIDIA which creates a unified way to create and manage Kubernetes Clusters on different "Providers" e.g. Proxmox or VMware. For example also VMWare uses a big part for commercial product Tanzu is nearly entiily based on the Open-Source Kubernetes. Cluster-API currently counts over 30 Infrastrucutre-Providers and Proxmox is just one of them. So long story short Cluster-API provides one unfilied API/Way to create production ready Kubernetes Clusters on ten of providers and is at least for me also the Defacto-Standard for Multi-Cloud Kubernetes and Kubernetes On-Prem.

The Sketch

Cluster-API uses the concept which is more known from the movie Inception. What do i Mean? Cluster-API itself uses the extensible API from Kubernetes to create Kubernetes-Clusters with Kubernetes. Thanks for the magic words wizard but How does it work?

We have this thing in Cluster-API which is called a management Cluster. The management clusters is responsible for or so called workload clusters. The management Cluster then takes care of the workload clusters. So for example the management cluster controls and update of a existing cluster or creates a new clusters. If we think a step forward you maybe notice we have a kind of a chicken-egg problem. We need a managment cluster to create workload-clusters but how do we create a managment cluster if we already
don't have a mangament cluster and how and so... You are right - because of this we will use use k3sup to bootstrap our management clusteron Proxmox

The managment cluster and the workload clusters will be as described on my Proxmox Datacenter.

Create the management Cluster with k3sup

[Disclaimer all the steps shown should not be used for a production or even development environment - its just a point you can start from - if you look for a more production level enviroment free to reach out to me.

Create a new VM for our managment Cluster

To Setup our VM for our management Cluster we will use the Debian-12 Proxmox VE Helper Script. Which makes it easy to create a new VM. Feel free to use other mechanism like Terraform, cloud-init or templates for that.

Log in into one node and execute:

bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/vm/debian-vm.sh)"

You will get a TUI which leads you through the VM please don't use the default setting - we need a bigger Disk (30GiB) and more RAM (4GiB):

  • Virtual Mchine ID = do not change
  • Machine Type = do not change
  • Disk Size = 30GiB
  • Disk Cache = do not change
  • Hostname = mgmt-cluster
  • CPU Model = do not change
  • CPU Cores = do not change
  • RAM = Change to 4096
  • Bridge = do not change
  • MAC = do not change
  • VLAN = do not change
  • MTU = do not change
  • Start VM when completed = yes

After that you can just open the Console and login as root (use xtrem.js for copy&paste working). After you loged in we can proceed with our next step

Install Kubernetes (K3s) on our newly created VM

First we need to install our kubernetes Cluster as mentioned we will use k3sup - its a really simple cli tool which allows you to create a k3s-kubernetes cluster within seconds. K3s is a Kubernetes-Distrubtion from Rancher which is now fully community driven by the CNCF. Its a popolur Kubernetes-Distro for small to mid large clusters. This commands downloads k3sup and then creates a simple cluster with the second command.

# install k3s-Kubernetes with k3sup
curl -sLS https://get.k3sup.dev | sh
k3sup install --local

You can test if the cluster is working with k3s kubectl get nodes it should show something like this.

root@localhost:~# k3s kubectl get nodes
NAME        STATUS   ROLES                  AGE     VERSION
localhost   Ready    control-plane,master   6m30s   v1.32.4+k3s1

Concrates you have setup up a simple single node kubernetes Cluster.

install cluster-api cli

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.10.1/clusterctl-linux-amd64 -o clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

initialize cluster-api components