Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13810

Maven modules marked to wrong build when running concurrent job

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core, maven-plugin
    • None

      I have a Maven project build with one module, named jenkins-test. This module contains one test, which fails everytime (assertTrue(false) ). There is a problem when this project is executed with "Execute concurrent builds if necessary" turned on and when multiple jobs are started at same time. Multiple builds start and finish, but some of these jobs are completed succesfully and some are marked as failed. What I see in the build status page is something like this:

      (Success) Build #180 (May 12, 2012 8:17:09 PM)
      ...
      Module Builds
      jenkins-test (didn't run)

      (Success) Build #181 (May 12, 2012 8:17:12 PM)
      ...
      Module Builds
      (disabled) jenkins-test (didn't run)

      (Unstable) Build #182 (May 12, 2012 8:17:19 PM)
      ...
      Module Builds
      jenkins-test (Unstable) #182 (Unstable) #183 (Unstable) #184

      Logs show that the builds were executed properly (commit ids are right, and "[ERROR] There are test failures." is there). The next build then gets build number #187.

      In my setup I'm using Gerrit and Gerrit Trigger to launch jobs, so when I push multiple commits into Gerrit in one push, Jenkins starts multiple jobs at same time. I don't know if you can duplicate this behavior without Gerrit, though.

      Currently MavenModule and MavenModuleSet have their own build numbers, which are increment when a new module is built. These might get mixed up when multiple builds are executed concurrently. Multiple MavenModuletSets are built starting at the same time, and some how the numbering of MavenModules gets out of sync.

      The commit in https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4 fixes this, but I haven't tested it other cases than mine and I don't think that I'm qualified to change something that is this deeply on the jenkins core The commit probably has some unnecessary bits in it too at the moment. So I don't think that it should be merged

          [JENKINS-13810] Maven modules marked to wrong build when running concurrent job

          evernat added a comment -

          Is it reproduced with a recent Jenkins version?

          evernat added a comment - Is it reproduced with a recent Jenkins version?

          I've changed to freestyle project with Maven buildsteps at the end of February, and this was valid then. Jenkins version was 1.502.

          Jyrki Puttonen added a comment - I've changed to freestyle project with Maven buildsteps at the end of February, and this was valid then. Jenkins version was 1.502.

          evernat added a comment -

          Neither you or me are able to test this issue with recent Jenkins/Maven/Gerrit.
          So resolving this issue as can not reproduce. Please reopen if reproduced.

          evernat added a comment - Neither you or me are able to test this issue with recent Jenkins/Maven/Gerrit. So resolving this issue as can not reproduce. Please reopen if reproduced.

          I could replicate it on Jenkins 1.609, 1.625 and 1.642, with a simple project with 4 submodules.
          I launch 10 build with the command

          for i in {0..9}; 
          do 
              echo $i
              curl http://127.0.0.1:8080/job/mv/build?delay=0sec
          done
          

          Ivan Fernandez Calvo added a comment - I could replicate it on Jenkins 1.609, 1.625 and 1.642, with a simple project with 4 submodules. I launch 10 build with the command for i in {0..9}; do echo $i curl http: //127.0.0.1:8080/job/mv/build?delay=0sec done

          Still valid with 2.46.x and maven-plugin 2.14

          Just create a maven job to build maven itself (https://github.com/apache/maven) and configure it to "Execute concurrent builds if necessary"

          Configure your instance (several executors on master) and launch several builds in a short period of time.

          You'll quickly see several modules attached to same parent build by error

           

           

           

          Arnaud Héritier added a comment - Still valid with 2.46.x and maven-plugin 2.14 Just create a maven job to build maven itself ( https://github.com/apache/maven)  and configure it to "Execute concurrent builds if necessary" Configure your instance (several executors on master) and launch several builds in a short period of time. You'll quickly see several modules attached to same parent build by error      

          olamy do you have in mind how are linked modules builds with the project build?

          Arnaud Héritier added a comment - olamy do you have in mind how are linked modules builds with the project build?

          olamy cc oleg_nenashev

          The problem from my POV is that we don't/cannot pass an identifier to link a MavenModuleSetBuild and its MavenBuild(s) (We just trigger a build of each module from the Maven project)

          https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/MavenModuleSetBuild.java#L712

          And thus we hack to link them 

          https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/MavenModuleSetBuild.java#L518

          WDYT ?

           

           

          Arnaud Héritier added a comment - olamy  cc oleg_nenashev The problem from my POV is that we don't/cannot pass an identifier to link a MavenModuleSetBuild and its MavenBuild(s) (We just trigger a build of each module from the Maven project) https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/MavenModuleSetBuild.java#L712 And thus we hack to link them  https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/MavenModuleSetBuild.java#L518 WDYT ?    

          Olivier Lamy added a comment -

          aheritier good catch! I reckon I never use concurrent build option

          The fix by jyrkip seems to be ok https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4 

          Olivier Lamy added a comment - aheritier good catch! I reckon I never use concurrent build option The fix by  jyrkip seems to be ok https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4  

          good finding olamy you're the boss !!

          but https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4 will requires an update of Jenkins core API and the maven plugin. I will discuss about it with oleg_nenashev but I'm not sure to have the ability to convince the community to update the Jenkins core for a need from the maven plugin  (which is really specific due to the use of a real Jenkins job for each Maven module - I don't see others cases where you could need such kind of coherency across jobs). And I'm not sure that I can implement a similar fix without the core part

           

          Arnaud Héritier added a comment - good finding olamy you're the boss !! but https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4  will requires an update of Jenkins core API and the maven plugin. I will discuss about it with oleg_nenashev but I'm not sure to have the ability to convince the community to update the Jenkins core for a need from the maven plugin  (which is really specific due to the use of a real Jenkins job for each Maven module - I don't see others cases where you could need such kind of coherency across jobs). And I'm not sure that I can implement a similar fix without the core part  

          Oleg Nenashev added a comment -

          > I will discuss about it with Oleg Nenashev but I'm not sure to have the ability to convince the community to update the Jenkins core for a need from the maven plugin

          I see no problem with that if it's implemented in a graceful way. After a quick review I do not see anything insane in https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4

          Oleg Nenashev added a comment - > I will discuss about it with Oleg Nenashev but I'm not sure to have the ability to convince the community to update the Jenkins core for a need from the maven plugin I see no problem with that if it's implemented in a graceful way. After a quick review I do not see anything insane in https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4

          oleg_nenashev no the code itself seems ok (but it has no tests - TBD). But if we go in that direction it means that we have to update the Jenkins core and to do a new major release of the maven plugin which is targeting this core release at minimum (For now we are still compatible with 1.625.3)

          Arnaud Héritier added a comment - oleg_nenashev no the code itself seems ok (but it has no tests - TBD). But if we go in that direction it means that we have to update the Jenkins core and to do a new major release of the maven plugin which is targeting this core release at minimum (For now we are still compatible with 1.625.3)

          Olivier Lamy added a comment - - edited

          aheritier OMG good luck!!   well it's only a new field with a buildNumber nothing really big change and should not break anything (IMHO)

          Olivier Lamy added a comment - - edited aheritier OMG good luck!!    well it's only a new field with a buildNumber nothing really big change and should not break anything (IMHO)

          Owen Mehegan added a comment -

          Is there any chance this is going to be resolved?

          Owen Mehegan added a comment - Is there any chance this is going to be resolved?

          Olivier Lamy added a comment -

          pr here https://github.com/jenkinsci/maven-plugin/pull/118 

          I tried to avoid core changes but need more testing especially with large projects....

          Olivier Lamy added a comment - pr here https://github.com/jenkinsci/maven-plugin/pull/118   I tried to avoid core changes but need more testing especially with large projects....

          Olivier Lamy added a comment - - edited

          uhm I have an issue with core  

          java.lang.IllegalStateException: JENKINS-27530: cannot create a build with number 47 since that (or higher) is already in use among [5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30, 32, 33, 34, 35, 36, 38, 39, 41, 42, 43, 44, 48, 49]

          Well I guess 47 is possible as not used
          oleg_nenashev jglick any idea?

          Olivier Lamy added a comment - - edited uhm I have an issue with core   java.lang.IllegalStateException: JENKINS-27530: cannot create a build with number 47 since that (or higher) is already in use among [5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30, 32, 33, 34, 35, 36, 38, 39, 41, 42, 43, 44, 48, 49] Well I guess 47 is possible as not used oleg_nenashev jglick any idea?

          Oleg Nenashev added a comment -

          Yes, I would expect core to handle such cases when parallel runs are enabled for AbstractProject. Unfortunately many APIs actually do not support it

          Oleg Nenashev added a comment - Yes, I would expect core to handle such cases when parallel runs are enabled for AbstractProject. Unfortunately many APIs actually do not support it

          Jesse Glick added a comment -

          No oleg_nenashev, parallel builds are normally fine, since build number allocation is done by a synchronized method on the project.

          When this ISE occurs, something is badly broken. We just do not know what it is.

          Jesse Glick added a comment - No oleg_nenashev , parallel builds are normally fine, since build number allocation is done by a synchronized method on the project. When this ISE occurs, something is badly broken. We just do not know what it is.

          Any chance of this being resolved at some point? We had a test job that had failures reported in another concurrent run, so the job was green, even though it had failures. Thanks

          Jacques-Etienne Beaudet added a comment - Any chance of this being resolved at some point? We had a test job that had failures reported in another concurrent run, so the job was green, even though it had failures. Thanks

          We also got this issue, a build was reported as red because of another job's failing test.

          Frederic Boutin added a comment - We also got this issue, a build was reported as red because of another job's failing test.

          Karan Dhingra added a comment -

          We've also been experiencing the same issue. Hope it gets fixed soon.

          Karan Dhingra added a comment - We've also been experiencing the same issue. Hope it gets fixed soon.

          Jesse Glick added a comment -

          Use of maven-plugin has been strongly discouraged for years, so I would not expect any Jenkins developers to spend time investigating this. Switch to freestyle or Pipeline builds.

          Jesse Glick added a comment - Use of maven-plugin has been strongly discouraged for years, so I would not expect any Jenkins developers to spend time investigating this. Switch to freestyle or Pipeline builds.

            olamy Olivier Lamy
            jyrkip Jyrki Puttonen
            Votes:
            6 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: