Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34389

starting parallel matrix jobs with dynamic axes causes in wrong build configuration

      We create a matrix project which gets its dynamic axes from build parameter. The job is doeing nothing instead printing some messages and wait an amount of time. Now we create an upstream project. This project calls the matrix project with different axis as a build parameter. This means we call the project and this starts parallel the matrix project with diferent axis.

      The projects are run with the correct matrix, but sometimes the job configuration contains the matrix values from another parallel build. This is visible if you compare the console output with the generated links in the build status UI.

      We attached the job configurations and one build.xml where you can see that the builded matrix differs from the given build parameter.

          [JENKINS-34389] starting parallel matrix jobs with dynamic axes causes in wrong build configuration

          Jan Bottek added a comment -

          addendum: i thin this issue is similar to JENKINS-30437

          Jan Bottek added a comment - addendum: i thin this issue is similar to JENKINS-30437

          I have a similar issue and tried to fix it.

          Bug origin:
          I found that the method "MatrixBuild.MatrixBuildExecution.doRun" is called in multiple threads without locks. If there are parallel builds of the same projects they sometimes call the "MatrixProject.rebuildConfigurations" at the same time before they can get the axis -> the build that is a bit slower overwrites the axes of the first one.

          Fix:
          I created a mutex for the MatrixProject instance and locked it before the "MatrixProject.rebuildConfigurations" is triggered and unlocked it after the axis is copied. Additional I used the fix from JENKINS-30437 for a deep copy of the axes.

          The full fix can be found here (based on matrix-project 1.6).

          Benjamin Böhmke added a comment - I have a similar issue and tried to fix it. Bug origin: I found that the method "MatrixBuild.MatrixBuildExecution.doRun" is called in multiple threads without locks. If there are parallel builds of the same projects they sometimes call the "MatrixProject.rebuildConfigurations" at the same time before they can get the axis -> the build that is a bit slower overwrites the axes of the first one. Fix: I created a mutex for the MatrixProject instance and locked it before the "MatrixProject.rebuildConfigurations" is triggered and unlocked it after the axis is copied. Additional I used the fix from JENKINS-30437 for a deep copy of the axes. The full fix can be found here (based on matrix-project 1.6).

          Jan Bottek added a comment -

          Hi all,

          something new about this issue?

          Can i help by giving more analyze information?

          Thanks in advance

          Kind regards

          Jan

          Jan Bottek added a comment - Hi all, something new about this issue? Can i help by giving more analyze information? Thanks in advance Kind regards Jan

          bboehmke I couldn't find an PR for your mutex patch. Was it never submitted or is there a reason why it isn't mergable?

          Joshua Hoblitt added a comment - bboehmke I couldn't find an PR for your mutex patch. Was it never submitted or is there a reason why it isn't mergable?

          The patch I made was only a simple fix to get it working.

          I think the part with the mutex is OK but the way the deep copy is done is not very well.
          This should be done with correct copy constructors of the "AxisList" I think.

          If you think the fix is enough I can create a pull request.

          Benjamin Böhmke added a comment - The patch I made was only a simple fix to get it working. I think the part with the mutex is OK but the way the deep copy is done is not very well. This should be done with correct copy constructors of the "AxisList" I think. If you think the fix is enough I can create a pull request.

          Joshua Hoblitt added a comment - - edited

          bboehmke I am completely unqualified to judge thread synchronization in the jenkins core. I would recommend opening a PR as some people (at least me) often check open PRs for unmerged fixes. I haven't tried the patch yet but I am contemplating trying to rebase it on the current 1.18 release.

          Joshua Hoblitt added a comment - - edited bboehmke I am completely unqualified to judge thread synchronization in the jenkins core. I would recommend opening a PR as some people (at least me) often check open PRs for unmerged fixes. I haven't tried the patch yet but I am contemplating trying to rebase it on the current 1.18 release.

          I created a pull request here.

          We are currently using this patch in our Jenkins instance at work.
          There is only one visible down side: The mutex object is dumped to the job config.
          This has no effect but looks strange.

          Benjamin Böhmke added a comment - I created a pull request here . We are currently using this patch in our Jenkins instance at work. There is only one visible down side: The mutex object is dumped to the job config. This has no effect but looks strange.

          Code changed in jenkins
          User: Benjamin Böhmke
          Path:
          src/test/java/hudson/matrix/MatrixTest.java
          src/test/java/hudson/matrix/helper/DynamicTestAxis.java
          http://jenkins-ci.org/commit/matrix-project-plugin/a0c12239e9f294c3317a0ffa03571474ef726539
          Log:
          added unit test for JENKINS-34389

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Benjamin Böhmke Path: src/test/java/hudson/matrix/MatrixTest.java src/test/java/hudson/matrix/helper/DynamicTestAxis.java http://jenkins-ci.org/commit/matrix-project-plugin/a0c12239e9f294c3317a0ffa03571474ef726539 Log: added unit test for JENKINS-34389

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          src/main/java/hudson/matrix/MatrixBuild.java
          src/main/java/hudson/matrix/MatrixProject.java
          src/test/java/hudson/matrix/MatrixTest.java
          src/test/java/hudson/matrix/helper/DynamicTestAxis.java
          http://jenkins-ci.org/commit/matrix-project-plugin/4cc27615094beb4540685c2b00efde93ce6e9835
          Log:
          Merge pull request #41 from bboehmke/dynamicAxis-fix

          JENKINS-34389 Fixed handling of dynamic axis

          Compare: https://github.com/jenkinsci/matrix-project-plugin/compare/ca0aa4607a2b...4cc27615094b

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/matrix/MatrixBuild.java src/main/java/hudson/matrix/MatrixProject.java src/test/java/hudson/matrix/MatrixTest.java src/test/java/hudson/matrix/helper/DynamicTestAxis.java http://jenkins-ci.org/commit/matrix-project-plugin/4cc27615094beb4540685c2b00efde93ce6e9835 Log: Merge pull request #41 from bboehmke/dynamicAxis-fix JENKINS-34389 Fixed handling of dynamic axis Compare: https://github.com/jenkinsci/matrix-project-plugin/compare/ca0aa4607a2b...4cc27615094b

            kohsuke Kohsuke Kawaguchi
            wreggyl Jan Bottek
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: