Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11993

matrix flyweight and configuration builds have "overlapping" workspace directories

      I have a (few) matrix jobs on a distributed build environment, I'm using SCM(SVN) polling using "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'" strategy.

      I just noticed that sometimes a random configuration job from my matrix job lose it's workspace, and has to do a clean checkout:
      "Checking out a fresh workspace because there's no workspace at /home/jenkins/workspace/php-src-5.4-matrix-build"

      After experiencing it a few times I also noticed, that it is correlated to the other issue, that I was seeing, namely that sometimes the flyweight build of my matrix job gets executed on a different node that I was pinning to using the matrix tie parent plugin.
      Of course executing that on a different node means that it has to do a clean checkout from SCM, and this is when I got enlightened:

      • the flyweight build will use $JENKINS_HOME/$JOB_NAME as the workspace directory
      • the configuration jobs will use the $JENKINS_HOME/$JOB_NAME/$AXES
        This means that if both the flyweight and one of the configuration jobs will be executed on the same node, they will interfere with each others workspaces.
        What I see to happen, is that the flyweight job will delete everything from $JENKINS_HOME/$JOB_NAME, and create a clean checkout there.
        Which in turn will force my configuration build to also do a clean checkout, as it won't find it's workspace.

      I created a bug report for the immediate problem (which made this issue visible for me): that the matrix tie parent plugin won't schedule the flyweight job to the tied label in some cases:
      https://issues.jenkins-ci.org/browse/JENKINS-11987

      The author of that plugin confirmed, that this could indeed happen, and marked the bug as "Not A Defect".

      Now I don't really see a way to prevent this issue from happening, as the matrix tie parent can only reduce this situation, but not prevent it, and I can't tell jenkins to use a different workspace path for the flyweight and the configuration job(different == one of them not containing the other one).

      Do you think that this could be fixed somehow, or do you know any workaround for this problem?

          [JENKINS-11993] matrix flyweight and configuration builds have "overlapping" workspace directories

          Ferenc Kovacs added a comment -

          I just created a much more simplified test scenarion:

          • create a matrix project using an svn repo and "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'"
          • create a custom axis named test, with dummy values like "1 2 3"
          • execute the build, make sure you have changes (update to a previous revision manually or commit something)

          first time, you will get a checkout for all of your builds (flyweight + 3 configuration job)

          on the second(and every other) run, you will get the following for the flyweight build:
          Cleaning up /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/.
          Deleting /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/./test

          then the configuration builds will have their workspace removed (they were under /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/test) so a full checkout will happen.

          for now, I will change my "Check-out Strategy", as this is the easier fix, but here are some possible solutions:

          • if the matrixtieparent would work every time, that would prevent this from happening.
          • if jenkins would use different workspace directory for the flyweight and the configuration builds, that would also fix this.
          • the svn plugin could be aware of this edge case, and not removing the configuration build workspace from the flyweight build workspace.

          Ferenc Kovacs added a comment - I just created a much more simplified test scenarion: create a matrix project using an svn repo and "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'" create a custom axis named test, with dummy values like "1 2 3" execute the build, make sure you have changes (update to a previous revision manually or commit something) first time, you will get a checkout for all of your builds (flyweight + 3 configuration job) on the second(and every other) run, you will get the following for the flyweight build: Cleaning up /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/. Deleting /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/./test then the configuration builds will have their workspace removed (they were under /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/test) so a full checkout will happen. for now, I will change my "Check-out Strategy", as this is the easier fix, but here are some possible solutions: if the matrixtieparent would work every time, that would prevent this from happening. if jenkins would use different workspace directory for the flyweight and the configuration builds, that would also fix this. the svn plugin could be aware of this edge case, and not removing the configuration build workspace from the flyweight build workspace.

          This is problem for me as well. Basically makes the emulate clean checkout strategy useless with matrix projects.

          David Humeniuk added a comment - This is problem for me as well. Basically makes the emulate clean checkout strategy useless with matrix projects.

          Ferenc Kovacs added a comment -

          for the record: I've just realized that the Exclude flyweight tasks plugin can be used to workaround this problem.

          Ferenc Kovacs added a comment - for the record: I've just realized that the Exclude flyweight tasks plugin can be used to workaround this problem.

          Kevin Moody added a comment -

          Any updates on this issue? I believe that I might be seeing something similar. However, my matrix builds are using the git plugin, as the scm, and execute the "clean after checkout" option.

          Kevin Moody added a comment - Any updates on this issue? I believe that I might be seeing something similar. However, my matrix builds are using the git plugin, as the scm, and execute the "clean after checkout" option.

          Oleg Nenashev added a comment -

          Use Throttle Concurrent Builds plugin (or its equivalents).
          The described issue is not a defect IMO

          Oleg Nenashev added a comment - Use Throttle Concurrent Builds plugin (or its equivalents). The described issue is not a defect IMO

          Ferenc Kovacs added a comment -

          I don't have this problem anymore, as the project using this setup was eventually moved to git, but from a quick look into Throttle Concurrent Builds plugin, I'm not sure that this would solve the issue.
          AFAIR you don't have to enable concurrent execution for the matrix job to be able to bump into this issue, and I don't see any mention about flyweight builds at https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin and even if it would affect the execution of the flyweight jobs, this issue isn't about executing the jobs in concurrent, but executing the flyweight and the matrix configuration jobs on the same node (even in a serialized matter).
          could you ellaborate how would your suggestion work in the practice?
          I can see why could somebody argue that the svn plugin is doing what it was instructed to do (eg. wiping out any files/dirs from the checkout) but I think that it is rather unfortunate, that by default the checkout dir for the flyweight job to fetch the remote changes will use the same directory which the matrix configuration builds are residing in.

          Ferenc Kovacs added a comment - I don't have this problem anymore, as the project using this setup was eventually moved to git, but from a quick look into Throttle Concurrent Builds plugin, I'm not sure that this would solve the issue. AFAIR you don't have to enable concurrent execution for the matrix job to be able to bump into this issue, and I don't see any mention about flyweight builds at https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin and even if it would affect the execution of the flyweight jobs, this issue isn't about executing the jobs in concurrent, but executing the flyweight and the matrix configuration jobs on the same node (even in a serialized matter). could you ellaborate how would your suggestion work in the practice? I can see why could somebody argue that the svn plugin is doing what it was instructed to do (eg. wiping out any files/dirs from the checkout) but I think that it is rather unfortunate, that by default the checkout dir for the flyweight job to fetch the remote changes will use the same directory which the matrix configuration builds are residing in.

          Daniel Beck added a comment -

          Well, this is kind of messed up to be honest.

          Simplest solution would be to have the parent build use a subdirectory as well, so cleaning that up would not interfere with child build workspaces.

          Daniel Beck added a comment - Well, this is kind of messed up to be honest. Simplest solution would be to have the parent build use a subdirectory as well, so cleaning that up would not interfere with child build workspaces.

          Todd Lipcon added a comment -

          Still no update on this? It would be great if there were a way to specify the "custom workspace" separately for the parent from the child builds. "Throttle concurrent builds" is a hacky workaround at best. Another hack that might be helpful is a variable like "$IS_PARENT_BUILD" or some way to programatically generate the workspace patch via a template language or groovy snippet or somesuch.

          Todd Lipcon added a comment - Still no update on this? It would be great if there were a way to specify the "custom workspace" separately for the parent from the child builds. "Throttle concurrent builds" is a hacky workaround at best. Another hack that might be helpful is a variable like "$IS_PARENT_BUILD" or some way to programatically generate the workspace patch via a template language or groovy snippet or somesuch.

            kbertelson kbertelson
            tyrael Ferenc Kovacs
            Votes:
            3 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: