Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-3094

ci.jenkins.io: ghost build (unable to resume after a controller restart)

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have an "erratical" behavior on ci.jenkins.io: When a restart had been triggered outside Jenkins (through a command such as

      service restart docker-jenkins
      

      or

      docker restart jenkins

      ), we tend to see "ghosts builds": these builds are individually marked as "failed", with their logs indicating a JNLP connectivity error, or an "unable to "resume" the pipeline".

      But from the home page, these builds are still waiting on the queue.

      Examples: https://ci.jenkins.io/job/Core/job/jenkins/job/PR-5802/2/ (see screenshots)

        Attachments

          Activity

          Hide
          dduportal Damien Duportal added a comment - - edited

          After discussing with others, it seems that the durability mode of the pipelines having a low steps to work ratio, such as jenkinsci/jenkins could be set to maximum persistence (e.g. the default mode). In other means: reverting https://github.com/jenkinsci/jenkins/pull/3297.

          The impact is a bit more I/O load on the controller, but we have SSDs, it's fine \o/

          Resulting PR: https://github.com/jenkinsci/jenkins/pull/5809

          Show
          dduportal Damien Duportal added a comment - - edited After discussing with others, it seems that the durability mode of the pipelines having a low steps to work ratio, such as jenkinsci/jenkins could be set to maximum persistence (e.g. the default mode). In other means: reverting https://github.com/jenkinsci/jenkins/pull/3297 . The impact is a bit more I/O load on the controller, but we have SSDs, it's fine \o/ Resulting PR: https://github.com/jenkinsci/jenkins/pull/5809
          Hide
          dduportal Damien Duportal added a comment -

          Same for jenkins-infra/pipeline-library: https://github.com/jenkins-infra/pipeline-library/pull/233

          Show
          dduportal Damien Duportal added a comment - Same for jenkins-infra/pipeline-library: https://github.com/jenkins-infra/pipeline-library/pull/233
          Hide
          dduportal Damien Duportal added a comment -

          Merged and applied: we did not see ghosts builds after 3 restarts: considering this is solved unless it re-appears.

          Show
          dduportal Damien Duportal added a comment - Merged and applied: we did not see ghosts builds after 3 restarts: considering this is solved unless it re-appears.

            People

            Assignee:
            dduportal Damien Duportal
            Reporter:
            dduportal Damien Duportal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: