From Proxmox to Kubernetes - Evolving My Homelab (part 2)

This is part 2 of my series detailing a transition of my Homelab architecture to using Kubernetes with Proxmox. You can check out part one here: From Proxmox to Kubernetes - Evolving My Homelab (part 1)

In this post, I continue my journey evolving my homelab from a simple Proxmox setup to a more robust Kubernetes-based architecture. Building on part one, I’ll share what worked, what didn’t, and how my approach to self-hosting and automation has changed over time.

My Proxmox Foundation: What Worked (And What Didn't)

Starting out, I had a single device with Proxmox installed and configured. This was as an experiment but also with the intent to get a proper firewall configured to protect my home network. For this, I ran pfSense as a VM within Proxmox, and then configured networking to use pfSense completely. My device had two network interfaces, so it was rather simple to configure. One interface was for my incoming network traffic, and the second interface for my home network. For anyone wondering, I only allowed the management of Proxmox to occur on my home network interface.

Eventually I added a PCI card with four additional network interfaces. This was mostly because I needed them for other home network connections and I wanted to experiment configuring pfSense for this rather than just buying a simple unmanaged switch.

What worked with my Proxmox was having an amazing learning lab at home to get my hands dirty with hypervisor configuration, VMs, and all the glory that is resource usage. Using pfSense allowed me to also dig deep into firewall rules, VLANs, and segmenting network traffic in more ways than just subnetting.

What didn't work was obvious after a few short months. Everything was running on a single device. If I had to update the hypervisor or pfSense, the whole network had to lose connectivity. If there was a hardware failure, the whole network had to lose connectivity. You get the idea. What concerned me most was that I was running all this on just old desktop hardware. It may have been vastly overboard in terms of compute (CPU and Memory) but wasn't built specifically for these tasks.

Eventually I purchased a Protectli Vault, installed OPNsense, and configured that as the firewall for my home network. My Proxmox instead became my homelab to experiment with self-hosted apps. This became my playground for a few years.

Self-Hosted Haven on Proxmox

I first started with running Docker containers in Debian-based linux distributions. This was itself a great experience. Learning containers, how to configure them for persistent storage and network access, and problem-solving all the little things that happened along the way - that was when the real fun in my homelab began. And everything being controlled with code through a docker-compose YAML file.

My little self-hosted haven eventually grew to deploying containers directly on Proxmox, known as LXCs. Most of these were again running on Debian-based linux. However, I had started to deploy Fedora VMs (RedHat/RPM-based) and experimenting with Podman. My interest in Podman came about when reading about container privileges, and wanting to run my containers as non-root (unprivileged). That's outside the scope of my story here (maybe a future post). I now had LXC containers for Docmost, PiHole, Caddy, Paperless-ngx, Postiz, and n8n. And on my Fedora VM a Podman container for my Omada SDN, which manages my switch, and access point for my home network.

While I was starting to utilize Ansible for configuration management, it was not quite everything I hoped it would be. I began with the idea of "how can I use Ansible to build all this up from scratch" and quickly learned the concept of idempotency. So I developed Ansible playbooks to validate initial system/container configurations and continuously ensure configuration integrity over time.

This was great, but if a container went down unexpectedly, I had to manually troubleshoot and restore it. I had no way of orchestrating this process with Ansible to the degree I wanted. In my mind, Ansible was not well-suited for managing containerized workloads. It might be excellent at provisioning and enforcing, but lacks the dynamic scheduling, self-healing, and service discovery required for container orchestration.

This was a homelab environment after all, where I constantly experiment and tinker. To address these newfound needs, I decided to adopt Kubernetes, which provides native support for many of the limitations I found with Ansible. I'm still using Ansible, and paired with Kubernetes it's been a game changer.

Wrap-Up

Moving from a single-device Proxmox setup to a more distributed and orchestrated environment will fundamentally improved both the reliability and flexibility of my homelab. By layering Ansible’s configuration management with Kubernetes’ orchestration capabilities, I’ll be able to automate, scale, and recover my self-hosted services far more efficiently than ever before. This evolution should not only solve the limitations I encountered with manual intervention and single points of failure, but also open up new possibilities for experimentation and growth. Stay tuned for part 3, where I’ll dive into the details of my implementation strategy for Kubernetes and how it will further transform my homelab.