Some issues with Kubernetes garbage collection

Andrei Kvapil - Aug 1 - - Dev Community

Recently while developing Cozystack, we encountered a serious issue that required an urgent fix: if /var/lib/kubelet and /var/lib/containerd are located on different partitions, Kubernetes garbage collection doesn't work properly.

Here's what happens: /var/lib/containerd gets filled up, while kubelet keeps reporting plenty of free space in /var/lib/kubelet, saying something like "Still a 95% of free space available." As a result, pods can't launch because they can't pull new images. We had to use bind mounts to move them to a shared partition.

Previously, we mounted two different disks to /var/lib/kubelet and /var/lib/containerd. Now, we mount a single disk to /ephemeral and then create bind-mounts for both /var/lib/kubelet and /var/lib/containerd.

Since the directories don't exist on a fresh, empty disk when the virtual machine starts, they need to be created and mounted: https://github.com/aenix-io/cozystack/pull/239/files#diff-abed09760d63acdd1c484818f4118fe9c3cd13cd2c1cdc25711d8e91d773a048R139-R141.

Additionally, we preserve the configuration in the fstab so that the directories are automatically mounted upon restarting the virtual machine: https://github.com/aenix-io/cozystack/pull/239/files#diff-abed09760d63acdd1c484818f4118fe9c3cd13cd2c1cdc25711d8e91d773a048R133-R134.

.
Terabox Video Player