Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44908

GitHub commit status resets on Jenkins reboot

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      This is a somewhat vague bug report, so I apologise in advance for this, but maybe it will help solve this problem for others (assuming it's not our problem alone).

      We are running Jenkins on top of Kubernetes, and most of the configuration of Jenkins is ephemeral, meaning, whenever we restart the pod running Jenkins master, all configuration reverts to our default configuration, provided by our Git repository.

      The only directory that is persistent, is the jobs directory.

      We've noticed that whenever Jenkins restarts, all the GitHub commit status checkmarks (the ones you see in a PR on github.com for example, next to the commit lines) go from green to red, yet when you click on the checkmark, and go to the job on Jenkins, the job is actually (still) green, and in fact, hasn't run after the restart, but is still showing the status since the last run.

      This is weird, because we expect the GitHub status to only be updated by Jenkins when a job runs, but it seems that somehow Jenkins is letting GitHub know about failed builds after a restart, when in fact those builds didn't fail, and they didn't run after the restart.

      Another explanation could be that GitHub is polling Jenkins for status of a commit, and somehow hitting a Jenkins instance that is rebooting, and thus marking the checkmark as failed, but with my knowledge about the GitHub API, that is not how things work, you need to push the status update to change it from green to red, or vise-versa, so the problem seems to lie with Jenkins, not GitHub.

      Any thoughts on this?

        Attachments

          Activity

          jeanmertz Jean Mertz created issue -
          jeanmertz Jean Mertz made changes -
          Field Original Value New Value
          Description This is a somewhat vague bug report, so I apologise in advance for this, but maybe it will help solve this problem for others (assuming it's not our problem alone).

          We are running Jenkins on top of Kubernetes, and most of the configuration of Jenkins is ephemeral, meaning, whenever we restart the pod running Jenkins master, all configuration reverts to our default configuration, provided by our Git repository.

          The only directory that is persistent, is the jobs directory.

          We've noticed that whenever Jenkins restarts, all(?) the GitHub commit status checkmarks go from green to red, yet when you click on the checkmark, and go to the job on Jenkins, the job is actually (still) green, and in fact, hasn't run after the restart, but is still showing the status since the last run.

          This is weird, because we expect the GitHub status to only be updated by Jenkins when a job runs, but it seems that somehow Jenkins is letting GitHub know about failed builds after a restart, when in fact those builds didn't fail, and they didn't run after the restart.

          Another explanation could be that GitHub is polling Jenkins for status of a commit, but with my knowledge about the GitHub API, that is not how things work, you need to push the status update to change it from green to red, or vise-versa, so the problem seems to lie with Jenkins, not GitHub.

          Any thoughts on this?
          This is a somewhat vague bug report, so I apologise in advance for this, but maybe it will help solve this problem for others (assuming it's not our problem alone).

          We are running Jenkins on top of Kubernetes, and most of the configuration of Jenkins is ephemeral, meaning, whenever we restart the pod running Jenkins master, all configuration reverts to our default configuration, provided by our Git repository.

          The only directory that is persistent, is the jobs directory.

          We've noticed that whenever Jenkins restarts, all(?) the GitHub commit status checkmarks (the ones you see in a PR on github.com for example, next to the commit lines) go from green to red, yet when you click on the checkmark, and go to the job on Jenkins, the job is actually (still) green, and in fact, hasn't run after the restart, but is still showing the status since the last run.

          This is weird, because we expect the GitHub status to only be updated by Jenkins when a job runs, but it seems that somehow Jenkins is letting GitHub know about failed builds after a restart, when in fact those builds didn't fail, and they didn't run after the restart.

          Another explanation could be that GitHub is polling Jenkins for status of a commit, but with my knowledge about the GitHub API, that is not how things work, you need to push the status update to change it from green to red, or vise-versa, so the problem seems to lie with Jenkins, not GitHub.

          Any thoughts on this?
          jeanmertz Jean Mertz made changes -
          Description This is a somewhat vague bug report, so I apologise in advance for this, but maybe it will help solve this problem for others (assuming it's not our problem alone).

          We are running Jenkins on top of Kubernetes, and most of the configuration of Jenkins is ephemeral, meaning, whenever we restart the pod running Jenkins master, all configuration reverts to our default configuration, provided by our Git repository.

          The only directory that is persistent, is the jobs directory.

          We've noticed that whenever Jenkins restarts, all(?) the GitHub commit status checkmarks (the ones you see in a PR on github.com for example, next to the commit lines) go from green to red, yet when you click on the checkmark, and go to the job on Jenkins, the job is actually (still) green, and in fact, hasn't run after the restart, but is still showing the status since the last run.

          This is weird, because we expect the GitHub status to only be updated by Jenkins when a job runs, but it seems that somehow Jenkins is letting GitHub know about failed builds after a restart, when in fact those builds didn't fail, and they didn't run after the restart.

          Another explanation could be that GitHub is polling Jenkins for status of a commit, but with my knowledge about the GitHub API, that is not how things work, you need to push the status update to change it from green to red, or vise-versa, so the problem seems to lie with Jenkins, not GitHub.

          Any thoughts on this?
          This is a somewhat vague bug report, so I apologise in advance for this, but maybe it will help solve this problem for others (assuming it's not our problem alone).

          We are running Jenkins on top of Kubernetes, and most of the configuration of Jenkins is ephemeral, meaning, whenever we restart the pod running Jenkins master, all configuration reverts to our default configuration, provided by our Git repository.

          The only directory that is persistent, is the jobs directory.

          We've noticed that whenever Jenkins restarts, all(?) the GitHub commit status checkmarks (the ones you see in a PR on github.com for example, next to the commit lines) go from green to red, yet when you click on the checkmark, and go to the job on Jenkins, the job is actually (still) green, and in fact, hasn't run after the restart, but is still showing the status since the last run.

          This is weird, because we expect the GitHub status to only be updated by Jenkins when a job runs, but it seems that somehow Jenkins is letting GitHub know about failed builds after a restart, when in fact those builds didn't fail, and they didn't run after the restart.

          Another explanation could be that GitHub is polling Jenkins for status of a commit, and somehow hitting a Jenkins instance that is rebooting, and thus marking the checkmark as failed, but with my knowledge about the GitHub API, that is not how things work, you need to push the status update to change it from green to red, or vise-versa, so the problem seems to lie with Jenkins, not GitHub.

          Any thoughts on this?
          Hide
          integer Kanstantsin Shautsou added a comment -

          I have no even idea, maybe try tcpdump/wireshark connection between jenkins and github to check whether jenkins hits gh

          Show
          integer Kanstantsin Shautsou added a comment - I have no even idea, maybe try tcpdump/wireshark connection between jenkins and github to check whether jenkins hits gh

            People

            Assignee:
            lanwen Kirill Merkushev
            Reporter:
            jeanmertz Jean Mertz
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: