Over some time I was really happy with my podman + ansible setup. It was great, but do you know what wasn’t such great? Deployment rollbacks. It all started with linkwarden. On my miniflux, I received a notification - that a new minor release is ready.

You can use GitHub repos as RSS links and received
notification about new releases. If you're using miniflux
just past https://github.com/ansible/ansible/releases.atom

Let's say version 2.8.0. When the time came, I just changed my variable and executed ansible-playbook command. Then after 3 minutes, my uptime-kuma started screaming, and saying that my app was dead. Initially, I ignored that, due to my experience with podman. During the service restart it makes an app unavailable, then downloads the new image and tries to run it. If the package is big, it could take a while. It wasn't great, but that is what could be acceptable for self-hosting service. Unfortunately, the main problem was with something different. Maintainers of Linkwarden sometimes inform users about a new release, however, it's only delivered as an rpm/deb package. The container image could be still in the building, or the building process could fail. It doesn't matter, sometimes my typo could occur as well. Then podman can't handle that, nor Ansible as it just restarts the service. Podman on another hand just stops service and does not roll back to the old version automatically. So sometimes it becomes annoying as I need to log into the server with ssh and manually fix it. That is why I decided to switch to some Kubernetes distribution.

k3s

So why is the based version of small Kubernetes supported by Suse?
Hmm mostly due to popularity, stability, and small resource consumption. It was good enough to run on a single host, and it's shipped as just a binary. This fits my needs, a small, easy-to-use package with build-in ingress, local-path storage provider, and flannel that implements Kubernetes CNI. Here is also a really nice
project for installation and configuration base cluster, which I used for my setup.

The migration process

I did not have much time for migration, as I really enjoy spending time with my family, also with a full-time job it is just hard to put in the calendar. That is why I decided to join the "5 AM Club", and start re-learning Kubernetes again. I have a full log of my activities with dates, but probably that is not what you could be interested in. Let's say my regular everyday process look like this:

  1. Apply terraform code

    cd hetzner
terrafrom apply

  2. Run k3s-ansible project, (check IP in inventory.yaml).

    cd k3s-ansible
ansible-playbook playbooks/site.yml -i inventory.yml

  3. With ready cluster update KUBECONFIG.

    export KUBECONFIG=~/.kube/config.new
kubectl config use-context k3s-ansible

  4. Patch k3s setup with traefik config.

    cd cluster-config
k apply -f traefik-ext-conf.yaml

  5. Create External Secret Operator main token for doppler

    kubectl create namespace external-secrets
kubectl create secret generic \
    -n external-secrets \
    doppler-token-argocd \
    --from-literal dopplerToken="dp.st.xx"

  6. Install EOS operator

    helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

  7. Check if the webhook is up and running (sometimes is not)

    k logs -l app.kubernetes.io/name=external-secrets-webhook -n external-secrets

  8. If yes, apply the overlay and create ClusterSecretStore

    cd ESO/
kubectl apply -k overlay/

  9. Install tailscale

    cd tailscale
kubectl apply -k .

  10. Install argo

    cd argocd
kubectl apply -k base/
kubectl apply -k overlay/

  11. Get Argo init admin password

    kubectl --namespace argocd get \
    secret argocd-initial-admin-secret \
    -o json \
    | jq -r '.data.password' \
    | base64 -d

    So far so good. Now let's break down used services.

Operators

I started by extending my traefik configuration to be able to handle regular requests from the internet. If you decide to do it,
as well please be aware that k3s for today (07-Feb-2025) is using version 2+, not 3+, which is why not everything straight from documentation will work. For example, my code is:

kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
  # we're still on with k3s https://github.com/traefik/traefik-helm-chart/blob/v27.0.2/traefik/values.yaml
spec:
  valuesContent: |-
    additionalArguments:
      - "[email protected]"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
      # - "--certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory"

With that, and duckdns domain (Hetzner,
very often gave you the same IP address. Thanks folks!). I was
able to expose my service directly to the internet with the usage HTTPS.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd
  namespace: argocd
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: "letsencrypt"
    traefik.ingress.kubernetes.io/service.serversscheme: "h2c"
spec:
  ingressClassName: "traefik"
  tls:
    - hosts:
        - nginx997.duckdns.org
  rules:
  - host: nginx997.duckdns.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

Then we have a challenge called:

How to store secrets with the usage of external sources?

I decided to use two products: External Secret Operator, and Doppler. In the beginning, I thought about Bitwarden's "not-so-new" Secret Manager, however, after a short investigation, the product seems to be not so well-supported by the ESO, which IMO is useful as it allows me to have one cluster-wide secret for getting secrets from external sources, which is great.

Doppler and ESO combo requires another post, so check my website from time to time.

Then I wanted to add Tailscle which besides being a "best in class VPN" for the homelabbers, allows you to add k8s services directly into your tailnet. What does it mean? The Tailscale operator allows you to access your k8s applications only when you are logged into your private
network (tailnet), with the usage of your domain for ended with ts.net. You can configure it in two ways on the resource side, with ingress or with service annotation.

  1. Ingress

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello
  namespace: hello
spec:
  ingressClassName: tailscale
  tls:
    - hosts:
        - hello.john-kira.ts.net
  rules:
    - host: hello.john-kira.ts.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80

  2. Service

    ---
apiVersion: v1
kind: Service
metadata:
  name: hello
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/tailnet-fqdn: "hello.john-kira.ts.net"
    tailscale.com/hostname: "hello"
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: nginx

Also please be aware that service by default exposes your service into tailnet over HTTP, where ingress provides a TLS certification as well.

Nice, at this point I was able to access my public service, as well as internal. What was the next step? ArgoCD.

Deploying ArgoCD was simple. During the first iteration, I decided to split my repo into base and overlay folders. The first directory contains the file for deploying an instance of Argo, exposing it to the tailnet. To archive it I just created a simple kustomization.yaml file as below:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: argocd
resources:
- namespace.yaml
- https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.2/manifests/install.yaml

# add --insecure flag to deployment, to avoid 307 redirection loop
patches:
- target:
    kind: ConfigMap
    name: argocd-cmd-params-cm
  path: configmap-patch.yaml
- target:
    kind: Service
    name: argocd-server
  path: patch-argocd-server-annotations.yaml

Then two patches were:

  • patch-argocd-server-annotations.yaml

    apiVersion: apps/v1
kind: Service
metadata:
  name: argocd-server
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/tailnet-fqdn: "argo.john-kira.ts.net"
    tailscale.com/hostname: "argo"

  • configmap-patch.yaml

    apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  server.insecure: "true"

Then my overlay directory contains only Argo's objects definitions:

overlay
├── applications.yaml
├── kustomization.yaml
├── projects.yaml
└── repositories.yaml

Summary

That was my first iteration of the "5 AM Club" Kubernetes migration. It takes me a bit longer than one or two mornings. Based on my notes it was around 7 days of work, ~1h per day. Not bad at all, however, there were a few things that are missing and which I should improve to make my setup much more flexible, stable, and easier to test. Where "to test" I understand deploying a new version of the operator and watching how the cluster is burning. Or not, I hope rather not.

With that in mind, thanks for your time, hope you enjoy reading it. If you would like to reach me or you have questions please use about page infos.