EDGE
TLS valid · Caddy reachable
The request reached the edge and received a deliberate HTTP response.
← Failure LabThe proxy was healthy.
002 / Kubernetes / Stateful workload / Incident
The proxy was healthy.
The service had disappeared.
Gitea returned “503 — no available server.” The failure was not DNS, TLS, or Caddy. A lost Kubernetes node had stranded a Longhorn RWO volume, blocked the replacement pod, and emptied the Service endpoints behind the proxy.
Impact: Web, API, HTTPS clone, automation, and dependent integrations lost their source-control upstream. No repository data was lost.
USER SIGNAL
HTTP 503
DEPLOYMENT
0 / 1 available
ENDPOINTS
0 ready
RECOVERY
HTTP 200 · volume healthy
The diagnostic
Follow the request inward.
A 503 from a proxy is evidence that the proxy is alive. The next useful question is whether it has a healthy backend.
SERVICE
0 ready endpoints
The reverse proxy was reporting an upstream failure, not causing one.
WORKLOAD
Deployment 0/1
The old pod was terminating and its replacement could not initialize.
STORAGE
RWO attached to failed node
The new pod could not safely mount the same filesystem elsewhere.
Failure propagation
One node crossed five control planes.
[01]
Node stops reporting
[02]
Pod cannot terminate
[03]
RWO stays attached
[04]
Replacement cannot mount
[05]
Service loses endpoints
[06]
Caddy returns 503
Incident timeline
State stopped converging.
10:36 UTC
Node heartbeat lost
The kubelet stopped posting status. Memory, disk, PID, and readiness conditions became Unknown.
10:44 UTC
Gitea pod began terminating
The original pod could not complete termination because its node was unavailable.
+ replacement
A new pod landed elsewhere
Kubernetes scheduled a replacement, but it remained before initialization with no pod IP.
+ storage
RWO attachment stayed behind
Longhorn still recorded the 10 GiB volume as attached to the failed node.
+ endpoint
The Service became empty
Neither HTTP nor SSH had a ready endpoint, leaving the edge proxy with no usable upstream.
recovery
Node and storage state converged
The volume detached cleanly, attached to the replacement node, became healthy, and Gitea returned HTTP 200.
Recovery decision
Safety before speed.
The tempting response was to force-delete the pod and detach the volume. That would have been unsafe while the old node might still have been capable of writing.
The preferred path was to restore the node, let Kubernetes complete termination, and allow Longhorn to detach and reattach cleanly. If restoration had failed, the prerequisite for forced recovery would have been fencing: proof that the old node could no longer access storage.
Decision gate
Can the old node be restored?
YES → restore → clean termination → clean detach
NO → prove fencing before forced detach
No fencing evidence → stop. Concurrent writers are a data-integrity risk.
Platform response
Recovery is an event. Reliability is the follow-through.
Detect the user outcome
Alert when Gitea has no available replica or Service endpoint—not only when a container restarts.
Detect stuck lifecycle state
Alert on pods terminating beyond a bounded interval and volumes stuck attaching or detaching.
Document fencing
Force-detach is permitted only after proving the previous node cannot continue writing.
Exercise recovery
Repeat the sequence with a sacrificial RWO workload and capture detection and recovery time.
Protect the source of truth
Test Gitea repository, database, and volume restoration outside the failure domain it manages.
Measure the platform
Track availability, endpoint readiness, volume attachment latency, and successful recovery—not just pod count.
The lesson
A healthy edge can only report the absence of a service. Stateful recovery begins where scheduling, storage, and node truth stop agreeing.