Recently while developing Cozystack, we encountered a serious issue that required an urgent fix: if /var/lib/kubelet
and /var/lib/containerd
are located on different partitions, Kubernetes garbage collection doesn't work properly.
Here's what happens: /var/lib/containerd
gets filled up, while kubelet keeps reporting plenty of free space in /var/lib/kubelet
, saying something like "Still a 95% of free space available." As a result, pods can't launch because they can't pull new images. We had to use bind mounts to move them to a shared partition.
Previously, we mounted two different disks to /var/lib/kubelet
and /var/lib/containerd
. Now, we mount a single disk to /ephemeral
and then create bind-mounts for both /var/lib/kubelet and /var/lib/containerd
.
Since the directories don't exist on a fresh, empty disk when the virtual machine starts, they need to be created and mounted: https://github.com/aenix-io/cozystack/pull/239/files#diff-abed09760d63acdd1c484818f4118fe9c3cd13cd2c1cdc25711d8e91d773a048R139-R141.
Additionally, we preserve the configuration in the fstab so that the directories are automatically mounted upon restarting the virtual machine: https://github.com/aenix-io/cozystack/pull/239/files#diff-abed09760d63acdd1c484818f4118fe9c3cd13cd2c1cdc25711d8e91d773a048R133-R134.