Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34712

"master is offline" preventing Pipeline from executing

      Our own Jenkins Pipeline projects seem to be getting stuck in this state of "master is offline" when attempting to run on our clusters which have zero executors assigned to the master node.

      It's unclear what, past a service restart, will clear this up

      Steps to reproduce:

      1. Start a pipeline job
      2. forcing the master to run out of storage
      3. shutdown master, clear up storage
      4. restart master, confirm it's up
      5. observe that it is still marked as offline for a long time. 30+ minutes

          [JENKINS-34712] "master is offline" preventing Pipeline from executing

          Jesse Glick added a comment -

          I have never heard of this problem before, and have no idea offhand how it could occur, since as danielbeck notes, there is always a MasterComputer even if you have configured zero heavyweight executors—WorkflowJob uses flyweights.

          As far as I know I lack administrative access to the server in question to do any live debugging.

          Jesse Glick added a comment - I have never heard of this problem before, and have no idea offhand how it could occur, since as danielbeck notes, there is always a MasterComputer even if you have configured zero heavyweight executors— WorkflowJob uses flyweights. As far as I know I lack administrative access to the server in question to do any live debugging.

          Jesse Glick added a comment -

          Jenkins.instance.selfLabel.offline, which should never be possible.

          Jesse Glick added a comment - Jenkins.instance.selfLabel.offline , which should never be possible.

          Daniel Beck added a comment -

          jglick We learned a few hours ago that master was marked offline due to disk space, and since it has zero executors, it wasn't apparent from the UI (as an executor-less master isn't shown on the executors pane).

          For some reason that offline state was preserved across restarts, and apparently longer then disk space cleanup + 30 minutes for the next monitor run, so maybe something was wrong there, but that was the offline cause.

          Daniel Beck added a comment - jglick We learned a few hours ago that master was marked offline due to disk space, and since it has zero executors, it wasn't apparent from the UI (as an executor-less master isn't shown on the executors pane). For some reason that offline state was preserved across restarts, and apparently longer then disk space cleanup + 30 minutes for the next monitor run, so maybe something was wrong there, but that was the offline cause.

          Daniel Beck added a comment -

          Looks a lot like Not A Defect to me. If the master is offline (especially for disk space reasons), no need to run any builds anywhere. The only RFE I could think of would be to not hide the executor-less master node in the executors sidepanel if it's marked offline.

          Daniel Beck added a comment - Looks a lot like Not A Defect to me. If the master is offline (especially for disk space reasons), no need to run any builds anywhere. The only RFE I could think of would be to not hide the executor-less master node in the executors sidepanel if it's marked offline.

          Jesse Glick added a comment -

          Sounds like a core bug.

          Jesse Glick added a comment - Sounds like a core bug.

          Daniel Beck added a comment -

          jglick What's the bug? That the node monitors work? That flyweight tasks don't run on marked-offline nodes?

          Daniel Beck added a comment - jglick What's the bug? That the node monitors work? That flyweight tasks don't run on marked-offline nodes?

          Jesse Glick added a comment -

          I guess that the master node should be displayed when it is offline.

          Jesse Glick added a comment - I guess that the master node should be displayed when it is offline.

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/resources/lib/hudson/executors.jelly
          http://jenkins-ci.org/commit/jenkins/b67a30f8daff936c91fd54b90bef6c366707a8f1
          Log:
          Merge pull request #3294 from dwnusbaum/JENKINS-34712

          JENKINS-34712 Always show the master node when it is offline

          Compare: https://github.com/jenkinsci/jenkins/compare/5c8cc45900bf...b67a30f8daff

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/resources/lib/hudson/executors.jelly http://jenkins-ci.org/commit/jenkins/b67a30f8daff936c91fd54b90bef6c366707a8f1 Log: Merge pull request #3294 from dwnusbaum/ JENKINS-34712 JENKINS-34712 Always show the master node when it is offline Compare: https://github.com/jenkinsci/jenkins/compare/5c8cc45900bf...b67a30f8daff

          Daniel Beck added a comment -

          Released in 2.108.

          Daniel Beck added a comment - Released in 2.108.

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/resources/lib/hudson/executors.jelly
          http://jenkins-ci.org/commit/jenkins/20d44c5aa750f6fece96f83f0f7ed519e9df2e54
          Log:
          Merge pull request #3294 from dwnusbaum/JENKINS-34712

          JENKINS-34712 Always show the master node when it is offline

          (cherry picked from commit b67a30f8daff936c91fd54b90bef6c366707a8f1)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/resources/lib/hudson/executors.jelly http://jenkins-ci.org/commit/jenkins/20d44c5aa750f6fece96f83f0f7ed519e9df2e54 Log: Merge pull request #3294 from dwnusbaum/ JENKINS-34712 JENKINS-34712 Always show the master node when it is offline (cherry picked from commit b67a30f8daff936c91fd54b90bef6c366707a8f1)

            dnusbaum Devin Nusbaum
            rtyler R. Tyler Croy
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: