We have seen issues where agent nodes run out of disk space despite having a conservative free space threshold. The nodes have a 100GB disk with a 40GB free space threshold, they are created dynamically by the Google Compute Engine plugin.

Looking on a node that had reached 100% disk usage, I believe the node was running below the threshold for an hour without being marked offline.

While monitoring the node status I saw the ‘Data obtained’ reach 45 minutes at one point, so I’m wondering if it’s possible the master simply didn’t gather the node status for so long that the node was able to run out of disk space. Is there a way to set the maximum time between checking the nodes?

Is there anyway to debug what is happening to cause this issue?