Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9913

Not obvious why some post-build tasks enforce serial behavior even when builds are concurrent

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • RedHat Enterprise Linux 4.8, Jenkins 1.414

      We're experiencing an issue with concurrent builds where Jenkins appears to be associating separate builds (run on different machines) such that they won't be marked as completed until all jobs are completed. For example, if we kick off 5 concurrent builds on 5 different nodes, builds 1-4 won't be marked as completed if build #5 is still running, even though builds 1-4 are finished. I've seen a report of someone experiencing this issue elsewhere:

      http://groups.google.com/group/jenkinsci-users/browse_thread/thread/e477e25910266d2a?fwc=1

      but a solution wasn't posted. We do not have the batch plugin or the locks and latches plugin installed. We've disabled all post-build processing and switched between different containers (Glassfish/Tomcat), but the problem persists. I couldn't find an issue logged for this other than the aforementioned posting.

          [JENKINS-9913] Not obvious why some post-build tasks enforce serial behavior even when builds are concurrent

          Jesse Glick added a comment -

          Around half of those have more than one commit, as I said.

          Jesse Glick added a comment - Around half of those have more than one commit, as I said.

          I give up, you win man. thanks for your help.

          surya gaddipati added a comment - I give up, you win man. thanks for your help.

          Jesse Glick added a comment -

          I am trying to get a patch through in JENKINS-16376; please push for changes in other affected plugins.

          Jesse Glick added a comment - I am trying to get a patch through in JENKINS-16376 ; please push for changes in other affected plugins.

          Byron Brummer added a comment -

          I have to wholeheartedly agree with surya et al: There needs to be a (non-plugin) switch for this. Perhaps not a global switch (although, why not?), but at an absolute minimum a per-job switch. This is a Jenkins-wide issue and giving plugins enough rope to hang the users does not strike me as the best process.

          The only "workaround" really is to throw out the entire Post-Build functionality and recreated it from whole cloth as Build Steps. That is really what this bug means: The entire Post-Build feature must be considered suspect and unreliable. At the very minimum there should be a huge user warning around the Concurrent Build check box telling users that Post-Build actions may likely serialize their jobs anyway and should be avoided.


          It should also be noted that Jenkins jobs are leveraged for MUCH more than simply "pulling code from an SCM and compiling it" type jobs. There are a huge number of utility tasks around a typical development lifecycle (deployments, environment setups and refreshes, data jobs, etc) that fit very well into Jenkins. In the real world I see such utility jobs far, far outnumber "real" build jobs. I mention this because of all the debate above about Git vs Subversion workflows and the like: It doesn't matter...in real world Jenkins usage it's common to find that MOST jobs aren't calling SCM at all because they aren't "buliding" anything, rendering all such workflow debates mute.

          Byron Brummer added a comment - I have to wholeheartedly agree with surya et al: There needs to be a (non-plugin) switch for this. Perhaps not a global switch (although, why not?), but at an absolute minimum a per-job switch. This is a Jenkins-wide issue and giving plugins enough rope to hang the users does not strike me as the best process. The only "workaround" really is to throw out the entire Post-Build functionality and recreated it from whole cloth as Build Steps. That is really what this bug means: The entire Post-Build feature must be considered suspect and unreliable. At the very minimum there should be a huge user warning around the Concurrent Build check box telling users that Post-Build actions may likely serialize their jobs anyway and should be avoided. It should also be noted that Jenkins jobs are leveraged for MUCH more than simply "pulling code from an SCM and compiling it" type jobs. There are a huge number of utility tasks around a typical development lifecycle (deployments, environment setups and refreshes, data jobs, etc) that fit very well into Jenkins. In the real world I see such utility jobs far, far outnumber "real" build jobs. I mention this because of all the debate above about Git vs Subversion workflows and the like: It doesn't matter...in real world Jenkins usage it's common to find that MOST jobs aren't calling SCM at all because they aren't "buliding" anything, rendering all such workflow debates mute.

          Jesse Glick added a comment -

          Build step monitors are an aspect of any build step, including both “builders” and “post-build steps” (recorders and notifiers), so that is an artificial distinction. Prior to the introduction of concurrent builds, plugins were free to assume that a build with a lower number than the current one was in fact completed—and some did in fact make that assumption. Monitors were introduced to ensure full binary compatibility. All plugins with build steps need to be updated to specify the monitor style they actually require, which is usually none. This has to be done on a case-by-case basis, while reviewing the plugin code for calls such as getPreviousBuild and ensuring that the logic continues to make sense when this build might be finishing well before earlier builds.

          Jesse Glick added a comment - Build step monitors are an aspect of any build step, including both “builders” and “post-build steps” (recorders and notifiers), so that is an artificial distinction. Prior to the introduction of concurrent builds, plugins were free to assume that a build with a lower number than the current one was in fact completed—and some did in fact make that assumption. Monitors were introduced to ensure full binary compatibility. All plugins with build steps need to be updated to specify the monitor style they actually require, which is usually none. This has to be done on a case-by-case basis, while reviewing the plugin code for calls such as getPreviousBuild and ensuring that the logic continues to make sense when this build might be finishing well before earlier builds.

          Not Again added a comment -

          Got bitten by the same thing. Now planning to move the post build stuff into an additional last step of the build job.
          Only problem is that I need to get data from the slave to master before that. And that is also a Post build action (I use Copy-To-Slave plugin). Any way I can do it as a part of build job only?

          Not Again added a comment - Got bitten by the same thing. Now planning to move the post build stuff into an additional last step of the build job. Only problem is that I need to get data from the slave to master before that. And that is also a Post build action (I use Copy-To-Slave plugin). Any way I can do it as a part of build job only?

          M S added a comment -

          To resolve this problem use xplugin as an intermediary to nunit/junit.

          M S added a comment - To resolve this problem use xplugin as an intermediary to nunit/junit.

          Code changed in jenkins
          User: Tim-Clifford
          Path:
          src/main/java/jenkins/plugins/slack/ActiveNotifier.java
          http://jenkins-ci.org/commit/slack-plugin/832e7b50028ca086dd49311ce454e7a19c9a3af9
          Log:
          Change .getPreviousBuild() to allow for higher concurrency

          Jenkins changed getPreviousBuild() to halt jobs if a previous
          build hasn't finished, meaning jobs of variable runtime end up
          being serialized:

          https://issues.jenkins-ci.org/browse/JENKINS-9913?focusedCommentId=184188&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-184188

          This change switches to the latest completed build, allowing
          this plugin to work with jobs of variable length.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Tim-Clifford Path: src/main/java/jenkins/plugins/slack/ActiveNotifier.java http://jenkins-ci.org/commit/slack-plugin/832e7b50028ca086dd49311ce454e7a19c9a3af9 Log: Change .getPreviousBuild() to allow for higher concurrency Jenkins changed getPreviousBuild() to halt jobs if a previous build hasn't finished, meaning jobs of variable runtime end up being serialized: https://issues.jenkins-ci.org/browse/JENKINS-9913?focusedCommentId=184188&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-184188 This change switches to the latest completed build, allowing this plugin to work with jobs of variable length.

          Eric Cooper added a comment -

          My apologies if this is the wrong please to ask this but I really did look for the right place and couldn't find it - is there a schedule for when this fix will be released? I am running into this issue and would benefit from the fix.

          Eric Cooper added a comment - My apologies if this is the wrong please to ask this but I really did look for the right place and couldn't find it - is there a schedule for when this fix will be released? I am running into this issue and would benefit from the fix.

          Jesse Glick added a comment -

          ericc there are distinct issues for various plugins.

          Jesse Glick added a comment - ericc there are distinct issues for various plugins.

            jglick Jesse Glick
            pomvr Philip Metting van Rijn
            Votes:
            19 Vote for this issue
            Watchers:
            34 Start watching this issue

              Created:
              Updated:
              Resolved: