Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9688

Build runs on master without executor and get stuck

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      In the ASF Jenkins, we do not allow builds to run on master. Thus, we have set the number of executors on master to zero. Even so, one build recently (https://builds.apache.org/hudson/job/Axis2/774/) got assigned to run on master. The build is configured with a label to run on one of the slaves, but somehow Jenkins assigned it incorrectly.
      Running on master also meant it got stuck, presumably due to the lack of executors, without any way of stopping it.

        Attachments

          Activity

          Hide
          guptapranav01 Pranav Gupta added a comment -

          Hi,

          whats the best way to run couple of jenkins jobs using groovy script. I don't find any good script for this. Any kind of help is greatly appreciated

          Show
          guptapranav01 Pranav Gupta added a comment - Hi, whats the best way to run couple of jenkins jobs using groovy script. I don't find any good script for this. Any kind of help is greatly appreciated
          Hide
          danielbeck Daniel Beck added a comment -

          Too old and too little detail in the issue report to help further investigation, so resolving as incomplete.

          Please file a new issue if this happens with a recent version of Jenkins (no older than ten weeks or so), and provide relevant information: Installed plugins, job configuration, how this can be reproduced, etc. See also the wiki for what other information may be relevant:
          https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Show
          danielbeck Daniel Beck added a comment - Too old and too little detail in the issue report to help further investigation, so resolving as incomplete. Please file a new issue if this happens with a recent version of Jenkins (no older than ten weeks or so), and provide relevant information: Installed plugins, job configuration, how this can be reproduced, etc. See also the wiki for what other information may be relevant: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue
          Hide
          dimitkoto Dimitar Sakarov added a comment -

          I still have the same issue, using Jenkins ver. 1.552 on 64bit Linux RHEL, and only Restart of Jenkins is able to resolve these zombie jobs staying there forever and preventing new builds from starting.

          Show
          dimitkoto Dimitar Sakarov added a comment - I still have the same issue, using Jenkins ver. 1.552 on 64bit Linux RHEL, and only Restart of Jenkins is able to resolve these zombie jobs staying there forever and preventing new builds from starting.
          Hide
          josesa Jose Sa added a comment -

          After several similar situations in the past weeks like this one where concurrent jobs using the same lock I think the problem may be mostly between chair and keyboard, but also due to lack of informative messages.

          Lets assume 3 jobs A, B and C using the same lock, the following timeline occurs:

          1. A is already running (so it has the lock) and we start B and C.
          2. B and C show in queue stating they are being blocked by A
          3. A finishes
          4. B and C already waited the 'delay' so they are ready to run, so they do run at the same time
          5. B gets the lock so it states it in the log is activelly running
          6. C on the other hand didn't get the lock, but is running anyway and showing no output that it is actively waiting for the lock
          7. Users panic when they see a job running for 2 days without any output in the log and cancel the C
          8. C get's inconsistent state and starts showing "null" instead of real dates
          9. Jenkins needs to be restarted to get rid of C because there is no way to kill it

          Also we observed that if we don't Cancel C eventually it starts executing normally when B finishes and showing it has acquired the lock in the log file.

          Bottom line the problem is lack of informative messages from the plugin to provide the proper feedback to users so they don't panic.

          Workaround: Revoke users permissions to Cancel builds that are using locks.

          Show
          josesa Jose Sa added a comment - After several similar situations in the past weeks like this one where concurrent jobs using the same lock I think the problem may be mostly between chair and keyboard, but also due to lack of informative messages. Lets assume 3 jobs A, B and C using the same lock, the following timeline occurs: A is already running (so it has the lock) and we start B and C. B and C show in queue stating they are being blocked by A A finishes B and C already waited the 'delay' so they are ready to run, so they do run at the same time B gets the lock so it states it in the log is activelly running C on the other hand didn't get the lock, but is running anyway and showing no output that it is actively waiting for the lock Users panic when they see a job running for 2 days without any output in the log and cancel the C C get's inconsistent state and starts showing "null" instead of real dates Jenkins needs to be restarted to get rid of C because there is no way to kill it Also we observed that if we don't Cancel C eventually it starts executing normally when B finishes and showing it has acquired the lock in the log file. Bottom line the problem is lack of informative messages from the plugin to provide the proper feedback to users so they don't panic. Workaround: Revoke users permissions to Cancel builds that are using locks.
          Hide
          josesa Jose Sa added a comment -

          Even with delays it got stuck (maybe because I used exact same quiet period). Now I've configured different quiet periods to avoid this.

          Show
          josesa Jose Sa added a comment - Even with delays it got stuck (maybe because I used exact same quiet period). Now I've configured different quiet periods to avoid this.

            People

            Assignee:
            stephenconnolly Stephen Connolly
            Reporter:
            protocol7b protocol7b
            Votes:
            9 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: