What 450+ Days of Operating k3s Taught Me

My k3s cluster has been running for more than 450 days. It now spans nine Linux nodes, 40 CPU cores, roughly 288 GB of memory, and more than 25 applications reconciled through Argo CD.

The biggest lesson is not that Kubernetes can run in a homelab. It is that operating Kubernetes turns abstract concepts into connected decisions.

Git Has to Be the Control Plane

Manual cluster changes are fast exactly once. After that, they become undocumented state.

Argo CD gives the platform a durable answer to three questions:

1. What should be running?
2. What changed?
3. Can the environment be reconstructed?

GitOps does not eliminate incidents, but it sharply reduces ambiguity during them.

Storage Is a Workload Decision

The cluster uses Longhorn, NFS, and local-path storage because no single storage mode is correct for every workload.

Replicated block storage is useful for state that must move with workloads. NFS is useful for shared files and capacity backed by TrueNAS. Local storage is useful when simplicity and node affinity are acceptable.

The important skill is not installing a provisioner. It is knowing which failure domain a workload can tolerate.

Networking Becomes Concrete

Cilium and Hubble make service traffic visible. MetalLB gives bare-metal services addresses. Traefik and Kong handle different ingress and gateway concerns. cert-manager automates certificates.

When these systems are diagrams in a course, they appear independent. In operation, a failed request can cross all of them. Troubleshooting becomes the practice of following evidence across layers instead of guessing at the layer you know best.

Observability Must Observe the Platform

Prometheus, Grafana, Loki, Tempo, OpenTelemetry, and network-flow visibility are not portfolio decorations. They are how the platform explains itself.

The useful question is never “Do I have a dashboard?” It is “Can I move from a user-visible symptom to the responsible component with the evidence available?”

A Healthy Lab Includes Failure

Nodes go unavailable. Images fail to pull. volumes need attention. Old workloads leave debris. A lab that never fails is either not doing enough or not being observed closely enough.

Operating the cluster changed how I think about production engineering. Reliability is not the absence of failure. It is the quality of the system's response: detection, containment, recovery, and the change that prevents repetition.

That is why I keep the cluster running. Certification material teaches the objects. Operations teaches the system.