Skip to content
BranchesTags

Repository files navigation

🚀 Home Operations Repository 🚧

Kubernetes on Flatcar Container Linux + kubeadm · TrueNAS NVMe-oF storage (scale-csi) · GitOps managed 🤖


Stack Versions
Cluster Health
Resource Usage
GitOps & Reliability
Network Status

💡 Overview

This repository contains the configuration for my homelab Kubernetes cluster built for learning, experimentation, and running self-hosted applications. The setup emphasizes Infrastructure as Code (IaC) and GitOps practices using Flatcar Container Linux + kubeadm, Kubernetes, Flux, Renovate, and GitHub Actions.

Architecture: The cluster runs on Proxmox VE 9.2 with scale-csi providing primary storage — NVMe-oF/TCP (plus iSCSI and NFS) volumes served by a TrueNAS SCALE box over its WebSocket API — and node-local scratch volumes via the OpenEBS hostpath provisioner backed by a dedicated ZFS RAID10 SSD pool on the hypervisor.

🛠 homeops-cli — Custom Go CLI

Built from scratch — ~25,000 lines of Go powering the entire cluster lifecycle.

homeops-cli is a purpose-built command-line tool that automates every operational aspect of this infrastructure. It is not a wrapper around shell scripts — it's a full Go application with 19 internal packages, native API clients, and embedded template rendering.

The built binary and CLI command name are homeops-cli.

Command What it does
homeops-cli bootstrap --provider flatcar End-to-end Flatcar/kubeadm cluster init: kubeadm init/join over SSH, then Cilium + CRDs + Flux, with preflight checks and 1Password secret injection
homeops-cli flatcar deploy-vm Provisions Flatcar Container Linux VMs on Proxmox (Ignition via fw_cfg) with batch/concurrent deployment
homeops-cli flatcar render-ignition Renders the Butane→Ignition config for a node (debug/inspection)
homeops-cli k8s view-secret Decodes secret data with interactive secret and namespace selection
homeops-cli volsync snapshot Triggers Kopia-backed PVC snapshots via VolSync
homeops-cli volsync restore Point-in-time PVC recovery from Kopia repository
homeops-cli volsync migrate Safe storage-class migration of an app's PVC (fresh backup → guarded cutover → restore-on-create)
homeops-cli workstation Developer workstation setup and validation

Key internals:

  • Native API clients for Proxmox VE and TrueNAS Scale, plus Flatcar release-image resolution — no shelling out
  • 1Password CLI integration for zero-plaintext secret management
  • Embedded Butane→Ignition transpilation (CoreOS Butane library) + kubeadm v1beta4 config rendering for Flatcar nodes
  • Interactive TUI with rich prompts, spinners, and progress indicators
  • Full test suite with unit and integration tests

See the dedicated CLI guide at cmd/homeops-cli/README.md for current operator workflows, and the supporting docs for testing, code review findings, and coverage notes.

cmd/homeops-cli/
├── cmd/           # CLI commands (bootstrap, flatcar, volsync, kubernetes, workstation)
├── internal/      # packages: proxmox, truenas, flatcar, ssh, iso, config, security, ui, ...
├── main.go        # Cobra root command with signal handling
└── Makefile       # Build, test, lint, coverage

🌱 Kubernetes

The Kubernetes cluster is deployed using Flatcar Container Linux with kubeadm on Proxmox VE 9.2 VMs. Primary storage is provided by scale-csi, a purpose-built CSI driver that provisions ZFS-backed NVMe-oF/TCP, iSCSI, and NFS volumes on a TrueNAS SCALE appliance (NVMe SLOG-backed, sub-millisecond commit latency), with VolSync/Kopia handling per-app backup and restore.

Infrastructure Details

  • Hypervisor: Proxmox VE 9.2 with KVM/QEMU virtualization
  • Primary Storage: scale-csi (NVMe-oF/TCP, iSCSI, NFS) on TrueNAS SCALE — scale-nvmeof is the default StorageClass; secure-by-default NVMe host-NQN allowlisting; driver-side orphan GC
  • Local Storage: OpenEBS hostpath provisioner on a dedicated 4-drive ZFS RAID10 SSD pool (openebs-ssd) on the Proxmox host — high-throughput scratch for download landing zones and VolSync mover caches
  • Network Infrastructure:
    • 4x 10GbE Intel X540 NICs bonded via IEEE 802.3ad LACP (40Gbps) on the Proxmox host
    • Cisco switch providing high-speed interconnect
    • Jumbo frames (MTU 9000) enabled end-to-end
  • OS / Distribution: Flatcar Container Linux (immutable, auto-updating) bootstrapped with kubeadm (v1beta4); Kubernetes v1.36.1. The kubelet/kubeadm/kubectl/CNI binaries are delivered as a systemd-sysext image and version-managed by systemd-sysupdate.
  • Control-plane endpoint: kube-vip (ARP/L2) VIP 192.168.123.253:6443 — CNI-independent, so it is usable as the kubeadm control-plane endpoint during init/join.
  • VM Configuration: 3 control plane nodes, each with 16 vCPUs, 96GB RAM, and NUMA-pinned CPU affinity
  • Storage Strategy: Multiple storage tiers per VM:
    • Boot Disk: 100GB VirtIO SCSI disk on the nvme-mirror ZFS RAID1 for the Flatcar OS (/dev/sda)
    • Local Scratch: 700GB VirtIO SCSI zvol from the openebs-ssd ZFS RAID10 pool, mounted at /var/mnt/local-hostpath for OpenEBS hostPath workloads (/dev/sdc)
    • App Volumes: attached on demand by scale-csi as NVMe-oF/TCP (or iSCSI) block devices from TrueNAS — no static data disks
  • Networking:
    • Cilium CNI with eBPF datapath
    • kgateway (Gateway API) for ingress with L2/BGP announcements
    • VirtIO network adapters: 16 queues on the primary NIC, 8 on VLAN 90 and the storage fabric, and no queues key on VLAN 20
    • Network interface: eth0 (Flatcar names the Proxmox VirtIO NIC eth0, not ens18)
  • Guest Integration: QEMU Guest Agent for enhanced VM management
  • Ingress: kgateway (Gateway API) with Cilium L2/BGP LoadBalancer services
  • DNS: external-dns for Cloudflare (public) and PowerDNS via RFC2136 (internal) DNS management

Core Components

  • actions-runner-controller: Self-hosted GitHub runners for CI/CD workflows.
  • cert-manager: Automated TLS certificate management with Google Trust Services.
  • cilium: eBPF-based networking, security, and L2/BGP announcements for LoadBalancer IP allocation.
  • cloudflared: Secure tunnels to Cloudflare for external access via Cloudflare Tunnel.
  • kgateway: Gateway API controller using Envoy proxy for ingress routing and traffic management.
  • external-dns: Automated DNS record management with Cloudflare and PowerDNS (RFC2136) integration.
  • external-secrets: Kubernetes External Secrets Operator with 1Password Connect integration.
  • flux: GitOps continuous delivery for Kubernetes with SOPS decryption support.
  • openebs: Local persistent volume provisioner for hostPath scratch storage on the dedicated openebs-ssd ZFS RAID10 pool.
  • scale-csi: Primary storage — a purpose-built TrueNAS SCALE CSI driver (WebSocket API, zero SSH) providing NVMe-oF/TCP, iSCSI, and NFS StorageClasses with snapshots, detached clones, expansion, and a driver-side orphan reconciler.
  • sops: Managed secrets for Kubernetes using age encryption, committed to Git.
  • spegel: Stateless cluster local OCI registry mirror for improved image pull performance.
  • kured: Coordinates safe, one-at-a-time node reboots (GitOps-managed) when the Kubernetes systemd-sysext patch updates or a Flatcar OS update have staged a reboot — Flatcar's locksmithd is masked, so kured is the reboot orchestrator. Minor Kubernetes upgrades are driven via system-upgrade-controller (see apps/system-upgrade).
  • volsync: Backup and recovery of persistent volume claims with Kopia.

GitOps

Flux provides GitOps continuous delivery, watching the kubernetes folder and applying changes based on Git repository state. The setup includes:

  • SOPS Integration: Automatic decryption of secrets using age encryption
  • Dependency Management: HelmReleases and Kustomizations with explicit dependencies
  • Multi-tenancy: Namespace isolation with proper RBAC
  • Webhook Integration: GitHub webhook receiver for immediate sync on push

The workflow recursively searches the kubernetes/apps folder for kustomization.yaml files, which typically contain namespace resources and Flux Kustomizations (ks.yaml). Each Kustomization manages HelmReleases or other Kubernetes resources for applications.

Renovate provides automated dependency management across the entire repository, creating pull requests for updates to:

  • Container images with digest pinning
  • Helm chart versions
  • Kubernetes manifests
  • GitHub Actions workflows

Repository Structure

This Git repository is organized for GitOps workflows and infrastructure management:

📁 home-ops
├── 📁 bootstrap          # Initial cluster bootstrap resources
├── 📁 kubernetes
│   ├── 📁 apps          # Application deployments by namespace
│   │   ├── 📁 actions-runner-system # Self-hosted GitHub runners
│   │   ├── 📁 automation     # Workflow automation (n8n)
│   │   ├── 📁 cert-manager   # Certificate management
│   │   ├── 📁 downloads      # Media acquisition stack
│   │   ├── 📁 external-secrets # Secret management
│   │   ├── 📁 flux-system    # Flux controllers
│   │   ├── 📁 kube-system    # Core Kubernetes components
│   │   ├── 📁 media          # Media serving applications
│   │   ├── 📁 network        # Networking applications
│   │   ├── 📁 observability  # Monitoring and logging
│   │   ├── 📁 openebs-system # Local storage provisioner
│   │   ├── 📁 scale-csi      # TrueNAS CSI driver (NVMe-oF/iSCSI/NFS)
│   │   ├── 📁 self-hosted    # Productivity and tools
│   │   ├── 📁 system-upgrade # Automated upgrades
│   │   └── 📁 volsync-system # Volume backup and recovery
│   ├── 📁 components    # Reusable Kustomize components
│   │   ├── 📁 alerts         # AlertManager configurations
│   │   ├── 📁 cluster-secret # Cluster-wide secrets
│   │   ├── 📁 nfs-scaler     # NFS availability scaling
│   │   └── 📁 volsync-direct # Direct volume backup/restore
│   └── 📁 flux          # Flux system configuration
├── 📁 cmd               # HomeOps CLI source code
│   └── 📁 homeops-cli   # Go-based automation tool
└── 📁 scripts           # Automation and utility scripts

Node OS + kubeadm configuration (Butane/Ignition, kubeadm v1beta4, kube-vip) lives as embedded templates inside cmd/homeops-cli, not a top-level directory.

Flux Workflow

This is a high-level look how Flux deploys my applications with dependencies. In most cases a HelmRelease will depend on other HelmRelease's, in other cases a Kustomization will depend on other Kustomization's, and in rare situations an app can depend on a HelmRelease and a Kustomization. The example below shows an app whose persistent volume is provisioned by scale-csi and backed up by VolSync — the app's PVC is created with a dataSourceRef to a VolSync ReplicationDestination, so data restores automatically on (re)creation.

graph TD
    A>Kustomization: scale-csi] -->|Creates| B[HelmRelease: scale-csi]
    C>Kustomization: volsync] -->|Creates| D[HelmRelease: volsync]
    E>Kustomization: atuin] -->|Creates| F(HelmRelease: atuin)
    E>Kustomization: atuin] -->|Creates| G(PVC: atuin on scale-nvmeof)
    G>PVC: atuin] -->|Provisioned by| B>HelmRelease: scale-csi]
    F>HelmRelease: atuin] -->|Backed up by| D>HelmRelease: volsync]
Loading

Automation & Tooling

The repository includes comprehensive automation for cluster management through a custom Go-based CLI:

HomeOps CLI (cmd/homeops-cli)

A purpose-built Go application that provides complete infrastructure automation:

Core Capabilities:

  • Bootstrap: Complete cluster initialization (--provider flatcar: kubeadm init/join over SSH, then Cilium/CRDs/Flux) with preflight checks and 1Password integration
  • Flatcar Provisioning: Butane→Ignition rendering, kubeadm config generation, and kubeadm init/join orchestration over SSH
  • VM Management: Proxmox VE 9.2 VM creation booting Flatcar images with Ignition injected via qemu fw_cfg
  • Volume Operations: VolSync-based backup and restore with Kopia integration
  • Kubernetes Management: Deployment restarts, PVC browsing, and maintenance operations

Key Commands:

# Bootstrap entire cluster
homeops-cli bootstrap

# Flatcar node provisioning + cluster bootstrap (kubeadm)
homeops-cli flatcar deploy-vm --nodes k8s-0,k8s-1,k8s-2 --image-path /var/lib/vz/template/flatcar.img --concurrent 3
homeops-cli flatcar render-ignition --node k8s-0      # inspect rendered Ignition
homeops-cli bootstrap --provider flatcar              # kubeadm init/join + Cilium/CRDs/Flux

# Kubernetes and Flux operations
homeops-cli k8s view-secret
homeops-cli k8s apply-ks ./kubernetes/apps/observability/grafana/ks.yaml --name grafana-instance

# Volume backup/restore
homeops-cli volsync snapshot --pvc data-pvc --namespace default
homeops-cli volsync restore --pvc data-pvc --namespace default

Supporting Tools:

  • Template Rendering: Embedded Butane→Ignition transpilation (CoreOS Butane) + kubeadm v1beta4 configs for Flatcar
  • Secret Injection: 1Password CLI integration for secure secret management
  • Environment Management: mise for tool and environment variable management
  • Configuration Validation: Pre-commit hooks with kubeconform and YAML linting
  • CI/CD: GitHub Actions for automated testing, schema validation, and deployment

😶 Cloud Dependencies

While most infrastructure and workloads are self-hosted, I rely on cloud services for critical functions to avoid chicken/egg scenarios and ensure availability of essential services regardless of cluster state. This approach balances self-hosting benefits with operational reliability.

Alternative solutions would involve running a separate cloud-hosted Kubernetes cluster for critical services like Vault, Vaultwarden, or ntfy, but the operational overhead and costs would likely exceed the current cloud service expenses.

Service Use Cost
1Password Secrets with External Secrets ~$65/yr
Cloudflare Domain, DNS, and tunnel services ~$30/yr
Google Workspace Email hosting and productivity suite ~$72/yr
GitHub Repository hosting and CI/CD with Actions Free
iLert Incident management and alerting Free (tier)
Pushover Mobile notifications for alerts $5 OTP
Total: ~$14/mo

🌎 DNS & Networking

The cluster implements a sophisticated networking architecture using Cilium and kgateway (Gateway API):

External Access

  • Cloudflare Tunnel: Secure external access via cloudflared without port forwarding
  • External DNS (Cloudflare): Automatic DNS record management in Cloudflare for public services
  • Gateway API: kgateway-based ingress with dedicated LoadBalancer IPs per application

Internal Resolution

  • External DNS (PowerDNS): external-dns deployment using RFC2136 provider for internal DNS record management via PowerDNS
  • CoreDNS: Kubernetes cluster DNS with custom configurations
  • Cilium Announcements: Cilium L2/BGP announcements for LoadBalancer IP allocation

Network Architecture

  • Physical Layer:
    • 40Gbps LACP bond aggregating 4x 10GbE Intel X540 NICs
    • IEEE 802.3ad Link Aggregation Control Protocol (LACP)
    • Jumbo frames (MTU 9000) enabled end-to-end
    • Bidirectional bandwidth between Proxmox host and TrueNAS Scale
  • CNI: Cilium with eBPF datapath for high-performance networking
  • Load Balancing: Maglev algorithm with DSR (Direct Server Return) mode
  • IP Management: Kubernetes IPAM with native routing (Pod CIDR: 10.42.0.0/16)
  • BGP Peering:
    • Cilium ASN: 64550
    • Cisco C9300 ASN: 64541
    • LoadBalancer IP pool: 192.168.255.0/24
  • Gateway IPs: LoadBalancer services advertised via L2 and BGP
  • Kernel Optimizations:
    • TCP Congestion Control: BBR
    • TCP Buffer Sizes: 64MB max
    • Socket Buffers: 128MB (rcvbuf/sndbuf)
    • NFS nconnect: 16 parallel connections

📱 Applications

The cluster hosts a variety of self-hosted applications organized by namespace and function:

Productivity & Tools (self-hosted namespace)

Application Purpose Access
Atuin Shell history sync sh.${SECRET_DOMAIN}
OCIS Personal file sync & sharing ocis.${SECRET_DOMAIN}
The Lounge Persistent IRC/web chat client thelounge.${SECRET_DOMAIN}

Content & Finance (self-hosted namespace)

Application Purpose Access
Actual Personal budgeting actual.${SECRET_DOMAIN}
FreshRSS RSS feed aggregator feeds.${SECRET_DOMAIN}

All self-hosted apps now share the self-hosted namespace so VolSync movers and Kopia ownership stay aligned (snapshots live under identities like app@self-hosted:/data).

Media & Requests (media namespace)

Application Purpose Access
Jellyseerr Media discovery & request management requests.${SECRET_DOMAIN}

Media workloads live in the media namespace so VolSync and Kopia identities follow app@media:/data for consistent restores.

Downloads & Indexers (downloads namespace)

Application Purpose Access
Autobrr Real-time announce filtering & actions autobrr.${SECRET_DOMAIN}
Bazarr Subtitle management for Radarr/Sonarr libraries bazarr.${SECRET_DOMAIN}
Cross-Seed Torrent cross-seeding suggestions Internal only
NZBGet Usenet downloader nzbget.${SECRET_DOMAIN}
Pinchflat Long-form video & podcast archiving pinchflat.${SECRET_DOMAIN}
Prowlarr Indexer proxy & search aggregator prowlarr.${SECRET_DOMAIN}
qBittorrent VPN-protected torrent client with VueTorrent UI qbittorrent.${SECRET_DOMAIN}
Qui Autobrr queue monitor & dashboard qui.${SECRET_DOMAIN}
Radarr Movie library automation radarr.${SECRET_DOMAIN}
Recyclarr Radarr/Sonarr config synchronisation Internal only
TQM Automated qBittorrent retag/cleanup jobs Internal only
Sonarr TV library automation sonarr.${SECRET_DOMAIN}

The entire download stack now lives in the downloads namespace so VolSync movers and Kopia ownership stay aligned (app@downloads:/data) and restores remain consistent.

Automation & Workflows (automation namespace)

Application Purpose Access
n8n Workflow automation & integrations n8n.${SECRET_DOMAIN}

Automation workloads run in the automation namespace so VolSync restores and Kopia ownership continue to match app@automation:/data.

Observability Stack (observability namespace)

Fully native VictoriaMetrics stack — single vendor, native protocols, no translation layers.

Application Purpose Access
VMSingle Metrics storage (300Gi, 14d retention) metrics.${SECRET_DOMAIN}
VMAgent Metrics scraping with streaming aggregation Internal only
VMAuth Unified auth proxy (host + path routing) Routes all external/Grafana traffic
VMAlert Metrics + log alerting (2 instances) Internal only
VMAlertManager Alert routing to Pushover alertmanager.${SECRET_DOMAIN}
VictoriaLogs Log storage with native syslog ingestion (500Gi, 14d) logs.${SECRET_DOMAIN}
vlagent K8s pod log collection (DaemonSet) Internal only
vmbackup Daily incremental metrics backup to NFS Internal only
Grafana Dashboards via grafana-operator grafana.${SECRET_DOMAIN}
Blackbox Exporter ICMP/HTTP/TCP probing Internal only
Gatus Uptime monitoring status.${SECRET_DOMAIN}
KEDA Event-driven autoscaling Internal only

Log pipeline: Pod logs → vlagent (native protocol) → VictoriaLogs. Network syslog (Cisco/VyOS/UniFi) → VictoriaLogs native syslog listener (UDP 514/5514/5515 on LB 192.168.255.254). No fluent-bit, no vector.

Storage & Infrastructure

Application Purpose Access
scale-csi Primary storage — TrueNAS NVMe-oF/iSCSI/NFS CSI driver Internal only
OpenEBS Local persistent volume provisioner (ZFS RAID10 scratch) Internal only

All applications use kgateway (Gateway API) for ingress with automatic TLS certificates from Google Trust Services via cert-manager.

💾 Storage Architecture

The cluster uses a multi-tier storage architecture with scale-csi (TrueNAS SCALE) as the primary storage layer:

Storage Tiers

Tier Provider StorageClass Use Case
NVMe-oF Block scale-csi scale-nvmeof (default) Application PVCs — highest IOPS, sub-ms commit latency
iSCSI Block scale-csi scale-iscsi Block volumes over iSCSI
NFS Filesystem scale-csi scale-nfs Shared (RWX-capable) storage
Local Scratch OpenEBS openebs-hostpath Download landing zones + VolSync caches on node-local ZFS RAID10
Backup VolSync + Kopia — Automated PVC backup and restore

scale-csi Configuration

scale-csi is a purpose-built CSI driver for TrueNAS SCALE, talking exclusively to its WebSocket JSON-RPC API (zero SSH):

  • Backend: TrueNAS SCALE all-flash ZFS pool with mirrored NVMe SLOG (~1ms fsync commit latency) and NVMe L2ARC
  • Protocols: NVMe-oF/TCP (primary), iSCSI, and NFS from a single driver
  • Security: NVMe subsystem host-NQN allowlisting (secure-by-default, no allowAnyHost), non-root controller, path-traversal-hardened volume IDs
  • Data safety: detached (fully independent) volumes from snapshots, foreign-snapshot deletion guards, and a driver-side orphan reconciler (read-only detection with Prometheus gauges + a gated, capped nightly GC)
  • Features: snapshots/clones/expansion, VolumeSnapshotClass scale-snapshot, online NVMe-oF expansion, integrated metrics + alerts

Backup Strategy

VolSync with Kopia provides automated backup:

  • ReplicationSource: Scheduled backups of PVCs to S3-compatible storage
  • ReplicationDestination: Point-in-time recovery with dataSource references
  • Identity Alignment: Namespace-based identity (app@namespace:/data) for consistent restores

⚙ Hardware

Physical Infrastructure

Component Specifications Function
Proxmox Host Proxmox VE 9.2 (KVM/QEMU) VM compute & management
├─ CPU 2x Intel Xeon E5-2697A v4 @ 2.60GHz (32 cores / 64 threads) VM compute resources
├─ Memory 512GB DDR4-2400 ECC (16x 32GB) VM memory allocation
├─ Network 4x 10GbE Intel X540 NICs (40Gbps LACP to Cisco switch) High-speed VM networking
└─ Storage 2x NVMe SSDs (ZFS RAID1 mirror — VM boot, 30GB/drive over-provisioned) + 4x 1TB SSD (ZFS RAID10 openebs-ssd, 50GB/drive over-provisioned) VM boot mirror + local scratch zvols
Storage Server TrueNAS Scale iSCSI, NVMe-oF & NFS storage
├─ CPU 2x Intel Xeon E5-2690 v4 @ 2.60GHz (28 cores / 56 threads) Storage processing
├─ Memory 120GB DDR4-2400 ECC (8x 16GB, reduced for VM allocation) ZFS ARC cache and services
├─ L2ARC 2x 1TB NVMe (1.8TB read cache) Extended read cache
├─ SLOG 2x 60GB NVMe (mirrored) Synchronous write log
├─ Network 4x 10GbE Intel X540 NICs (40Gbps LACP to Cisco switch) Storage network
└─ Protocols iSCSI (block) + NVMe-oF (block) + NFS 4.2 (file) Primary k8s storage via scale-csi
Network Switch Cisco C9300 Infrastructure interconnect
├─ LACP Configuration IEEE 802.3ad Link Aggregation (40Gbps total) High-bandwidth storage path
├─ BGP Peering AS 64541 (peering with Cilium AS 64550) LoadBalancer IP advertisement
└─ MTU Jumbo frames (9000 bytes) enabled Optimized large frame throughput

Storage Architecture

Storage Tier Hardware Purpose
TrueNAS Primary Pool 3x RAIDZ1 vdevs (3 disks each = 28.5TB usable) scale-csi volumes (NVMe-oF/iSCSI zvols + NFS) + media exports
SLOG (Intent Log) 2x 60GB NVMe (mirrored) Synchronous write acceleration
L2ARC (Read Cache) 2x 1TB NVMe (1.8TB total) Extended ARC read cache

Virtual Machine Configuration

VM Role Count vCPU Memory Storage Layout OS
Kubernetes Control Plane 3 16 96GB 100GB boot (nvme-mirror RAID1) + 700GB local scratch (openebs-ssd RAID10 zvol) Flatcar Container Linux + kubeadm (k8s v1.36.1)

Storage Details:

  • Boot Disk (scsi0, /dev/sda): 100GB VirtIO SCSI disk on the nvme-mirror ZFS RAID1 for the Flatcar OS
  • Local Scratch (scsi3, /dev/sdc): 700GB VirtIO SCSI zvol from the openebs-ssd ZFS RAID10 pool (ext4, LABEL=openebs-local), mounted at /var/mnt/local-hostpath for OpenEBS hostPath workloads
  • App Volumes: attached dynamically by scale-csi as NVMe-oF/TCP (or iSCSI) devices from TrueNAS — no static data disks

VM Configuration:

  • Provisioning: Ignition via qemu fw_cfg (opt/org.flatcar-linux/config) — the rendered Ignition is attached at VM create; the disk imports a Flatcar image (no install ISO)
  • BIOS: OVMF (UEFI)
  • CPU Type: host,flags=+pdpe1gb;-spec-ctrl
  • Network: VirtIO on vmbr0; primary net0 uses MTU 9000, VLAN 999, and 16 queues; VLAN 20 net1 uses MTU 1500 with no queues key; VLAN 90 net2 uses MTU 1500 and 8 queues; storage net3-net6 use MTU 9000 and 8 queues
  • Guest Agent: QEMU Guest Agent for enhanced management
  • Machine Type: Q35 chipset

CPU & Memory Optimization:

  • NUMA Pinning: Each VM is pinned to specific CPU cores with HT siblings for optimal memory locality
    • k8s-0 (VMID 200): Cores 0-7,32-39 (Socket 0, NUMA 0)
    • k8s-1 (VMID 201): Cores 16-23,48-55 (Socket 1, NUMA 1)
    • k8s-2 (VMID 202): Cores 8-15,40-47 (Socket 0, NUMA 0)
  • Memory: 96GB (98304 MB) per VM, no overcommit

Total VM Resources: 48 vCPUs, 288GB RAM allocated from the 64-thread, 512GB host system.

📚 Learning & Credits

This homelab serves as a continuous learning platform for cloud-native technologies, GitOps practices, and infrastructure automation. The setup provides hands-on experience with production-grade tools and practices in a controlled environment.

Special thanks to onedr0p and the k8s-at-home community. This repository was heavily inspired by onedr0p's home-ops repository, which served as an excellent learning resource and foundation for understanding GitOps workflows and Kubernetes cluster management.

🙏 Gratitude and Thanks

Thanks to all the people who donate their time to the Home Operations Discord community. Be sure to check out kubesearch.dev for ideas on how to deploy applications or get ideas on what you could deploy.

About

Kubernetes cluster on Flatcar Linux managed with Flux & GitOps — includes homeops-cli, a custom Go CLI for cluster lifecycle, VM provisioning, and backup automation

Topics

cliflatcarflatcar-linuxfluxgitopsgolanghome-opshomelabhomeops-cliinfrastructurekuberneteskubesearchtruenas-scale

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository

Used by

Contributors

Languages