This is a somewhat vague bug report, so I apologise in advance for this, but maybe it will help solve this problem for others (assuming it's not our problem alone).

We are running Jenkins on top of Kubernetes, and most of the configuration of Jenkins is ephemeral, meaning, whenever we restart the pod running Jenkins master, all configuration reverts to our default configuration, provided by our Git repository.

The only directory that is persistent, is the jobs directory.

We've noticed that whenever Jenkins restarts, all the GitHub commit status checkmarks (the ones you see in a PR on github.com for example, next to the commit lines) go from green to red, yet when you click on the checkmark, and go to the job on Jenkins, the job is actually (still) green, and in fact, hasn't run after the restart, but is still showing the status since the last run.

This is weird, because we expect the GitHub status to only be updated by Jenkins when a job runs, but it seems that somehow Jenkins is letting GitHub know about failed builds after a restart, when in fact those builds didn't fail, and they didn't run after the restart.

Another explanation could be that GitHub is polling Jenkins for status of a commit, and somehow hitting a Jenkins instance that is rebooting, and thus marking the checkmark as failed, but with my knowledge about the GitHub API, that is not how things work, you need to push the status update to change it from green to red, or vise-versa, so the problem seems to lie with Jenkins, not GitHub.

Any thoughts on this?