# FM-KED-001 — VM Disk Space Exhaustion

**Severity:**<span style="white-space: pre-wrap;"> S2 — High</span>  
**Recovery Class:**<span style="white-space: pre-wrap;"> A — Quick Fix</span>  
**Covered by Monthly Support:**<span style="white-space: pre-wrap;"> Yes</span>

---

### Description

Disk space on a virtual machine reaches critical levels, leading to degraded system behavior, application instability, or failed background operations.

This issue is operational, recurrent, and typically caused by uncontrolled growth of logs, containers, temporary files, or Kubernetes artifacts.

---

### Typical Symptoms

- Services failing to write logs or temporary files
- Background jobs failing without explicit errors
- <span style="white-space: pre-wrap;">Kubernetes pods entering </span>`<span class="editor-theme-code">Evicted</span>`<span style="white-space: pre-wrap;"> or </span>`<span class="editor-theme-code">Terminating</span>`<span style="white-space: pre-wrap;"> state</span>
- System warnings related to low disk space

---

### Diagnostic Checklist

#### Identify Top Disk Consumers

```bash
sudo du -ahx / | sort -rh | head -n 20
```

---

### Recovery Procedure

<span style="white-space: pre-wrap;">Follow the steps below </span>**as needed**, not necessarily all of them.

---

#### 1. Clean Package Manager Artifacts

```bash
sudo apt-get autoremove
sudo du -sh /var/cache/apt
sudo apt-get autoclean
sudo apt-get clean
```

---

#### 2. Clean System Journals

```bash
sudo journalctl --vacuum-time=3d
```

---

#### 3. Truncate Docker Logs

```bash
sudo truncate -s 0 /var/lib/docker/containers/**/*-json.log
```

---

#### 4. Prune Docker Resources

```bash
sudo docker system prune
```

---

#### 5. Remove Obsolete Kubernetes ReplicaSets

```bash
kubectl get rs -A -o wide | tail -n +2 | \
awk '{if ($3 + $4 + $5 == 0) print "kubectl delete rs -n "$1, $2 }' | sh
```

---

#### 6. Clear Evicted Kubernetes Pods

```bash
kubectl get pods | grep Evicted | awk '{print $1}' | xargs kubectl delete pod
```

With explicit kubeconfig:

```bash
kubectl --kubeconfig bank.yaml get pods | grep Evicted | \
awk '{print $1}' | xargs kubectl --kubeconfig bank.yaml delete pod
```

---

#### 7. Force Remove Stuck Terminating Pods

```bash
for p in $(kubectl --kubeconfig bank.yaml get pods | grep Terminating | awk '{print $1}');
do
  kubectl --kubeconfig bank.yaml delete pod $p --grace-period=0 --force
done
```

---

### Optional Diagnostics

#### Inspect Memory Usage (for runaway processes)

```bash
ps -eo size,pid,user,command --sort -size | \
awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } \
{ for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }'
```

---

---

### Preventive Notes

- Disk usage monitoring is strongly recommended
- Log rotation must be verified after updates
- Kubernetes cleanup should be part of routine maintenance

---

###