Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30231

Build creates second workspace@2 for non-concurrent build configuration

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Minor Minor
    • core, matrix-project-plugin
    • None
    • OS: Ubuntu 12.04 64-bit
      Jenkins: 1.609.1 (upgraded from 1.580.3)
      disk-usage plugin: 0.25 (upgraded from 0.24)
      Matrix Project Plugin: 1.6 (upgraded from 1.4.1)
    • Jenkins Core 2.136

      I didn't have this issue when I was using ver 1.580.3.
      After upgrading Jenkins ver 1.609.1 (together with other Jenkins plugins update),
      I started to see this issue.

      My jenkins job is using multi-config project type (aka Matrix project),
      and has 3 variants.
      I didn't set to be run concurrently,
      so the 3 variants will start the build one by one.

      In the previous version 1.580.3,
      it will reuse the same workspace folder for all of 3 variant builds.

      In the new version 1.609.1,
      when multiple jobs are triggerd,
      I can see sometimes the first variant build of the second job
      will start to create and use workspace@2
      instead of re-using original worksapce.

      Note:
      I'm also using disk-usage plugin.
      At the end of build,
      it will trigger that plugin to calculate workspace disk space.
      My work space tends to be large and sometimes it take several minutes to finish.

      I suspect it's that disk-usage plugin still occupying the original worksapce,
      but the second job in queue starts earlier,
      so it creates workspace@2 for the first variable build.

      From workspace@2,
      I always see only one variant only under AXIS folder.

          [JENKINS-30231] Build creates second workspace@2 for non-concurrent build configuration

          Rick Liu added a comment -

          Per comment from Jesse Glick,
          I create this separate issue
          which has similar symptom.

          Rick Liu added a comment - Per comment from Jesse Glick, I create this separate issue which has similar symptom.

          Will Berriss added a comment -

          I am seeing this issue today on one of my pipelines. Is there a quick fix workaround I can use to get rid of it and reset the pipeline? I don't mind starting the pipeline from scratch again - just wondering how to get rid of the @2 workspace. Anyone know?

          Will Berriss added a comment - I am seeing this issue today on one of my pipelines. Is there a quick fix workaround I can use to get rid of it and reset the pipeline? I don't mind starting the pipeline from scratch again - just wondering how to get rid of the @2 workspace. Anyone know?

          Matthew Smith added a comment -

          will177 It looks like this might be a solution: https://issues.jenkins-ci.org/browse/JENKINS-14354

          It looks like you can replace the "@" separator with a separator of your choice.

          Matthew Smith added a comment - will177 It looks like this might be a solution: https://issues.jenkins-ci.org/browse/JENKINS-14354 It looks like you can replace the "@" separator with a separator of your choice.

          Rick Liu added a comment -

          I use customize worksapce configuration to hardcoded the workspace name to work-around this problem

          Rick Liu added a comment - I use customize worksapce configuration to hardcoded the workspace name to work-around this problem

          This is still a problem. We restrict our jobs to one per node and still see a number of @2 workspaces. Replacing @ with _ (like JENKINS-14354) helps builds which get tripped up by @ but we still have duplication. As we have a number of large workspaces this wastes a lot of diskspace.

          Russell Gallop added a comment - This is still a problem. We restrict our jobs to one per node and still see a number of @2 workspaces. Replacing @ with _ (like JENKINS-14354 ) helps builds which get tripped up by @ but we still have duplication. As we have a number of large workspaces this wastes a lot of diskspace.

          We've recently started seeing a duplicate @2 workspace as well, when we moved from 2.60.3 to 2.73.2 with some new plug in.

           

          nolan de souza added a comment - We've recently started seeing a duplicate @2 workspace as well, when we moved from 2.60.3 to 2.73.2 with some new plug in.  

          Devin Nusbaum added a comment -

          JENKINS-41127, which may have been the root cause of this issue, was just fixed in Jenkins 2.136. Does anyone have a test environment in which they can consistently reproduce this issue? If so, it would be great if you could test whether this problem still occurs in Jenkins 2.136 or newer.

          Devin Nusbaum added a comment - JENKINS-41127 , which may have been the root cause of this issue, was just fixed in Jenkins 2.136 . Does anyone have a test environment in which they can consistently reproduce this issue? If so, it would be great if you could test whether this problem still occurs in Jenkins 2.136 or newer.

          Devin Nusbaum added a comment - - edited

          Since I have not heard back either way, I am going to go ahead and close this issue because at least one way it could happen was fixed in Jenkins 2.136. Please feel free to reopen the issue if you are still seeing it in Jenkins 2.136 or newer.

          Devin Nusbaum added a comment - - edited Since I have not heard back either way, I am going to go ahead and close this issue because at least one way it could happen was fixed in Jenkins 2.136. Please feel free to reopen the issue if you are still seeing it in Jenkins 2.136 or newer.

          J S added a comment -

          I have the same problem at Jenkins Version : 2.121.3

          J S added a comment - I have the same problem at Jenkins Version : 2.121.3

          J S added a comment -

          I still have the same problem on Jenkins Version 2.121.3 LTS .

          On my Slave i get the Folder @2 that are very big.

          J S added a comment - I still have the same problem on Jenkins Version 2.121.3 LTS . On my Slave i get the Folder @2 that are very big.

          Devin Nusbaum added a comment - - edited

          openjenkins The change I made to fix JENKINS-41127 is in 2.136, and it was not backported to the 2.121.x LTS line (It is in the 2.138.x LTS line). Please only reopen the issue if you are seeing it in 2.136 or newer (unless you have additional information that leads you to believe it is distinct from JENKINS-41127, such as seeing the issue for something other than a pipeline or matrix job, and please include any relevant info if that is the case).

          Devin Nusbaum added a comment - - edited openjenkins The change I made to fix JENKINS-41127 is in 2.136, and it was not backported to the 2.121.x LTS line (It is in the 2.138.x LTS line). Please only reopen the issue if you are seeing it in 2.136 or newer (unless you have additional information that leads you to believe it is distinct from JENKINS-41127 , such as seeing the issue for something other than a pipeline or matrix job, and please include any relevant info if that is the case).

          Kamil Szuster added a comment -

          Hi! 

          Yesterday one of my jobs started to show the same behavior, but we are not using the matrix-project-plugin. At some point (maybe after we deleted the original workspace-folder of a certain job) jenkins started to build the project in a folder suffixed by "@2" - but only on one specific node. Recreating the Pipelinejob with the same name didn't help. Today i did a restart of the server and that seems to have soled the issue for now.

          We are using Jenkins 2.138.2.

          Kamil Szuster added a comment - Hi!  Yesterday one of my jobs started to show the same behavior, but we are not using the matrix-project-plugin. At some point (maybe after we deleted the original workspace-folder of a certain job) jenkins started to build the project in a folder suffixed by "@2" - but only on one specific node. Recreating the Pipelinejob with the same name didn't help. Today i did a restart of the server and that seems to have soled the issue for now. We are using Jenkins 2.138.2.

          Jatin Mehta added a comment -

          This issue is also seen in Jenkins ver 2.176.1. Jenkins is appending "@2" at the end of the workspace name for one of my nodes. This is causing the tests being run by the pipeline to fail since they look for files in a configured workspace name without the suffix @2. Jenkins seems to do that to prevent mishaps during concurrent builds but this happens for me even when I check Do not allow concurrent builds in the pipeline configuration. Is this new behavior due to a Jenkins version upgrade? Do we have any workarounds? I changed the project name that prevented it from happening but it came up again today. Can't keep changing project name to new ones. Please help!

          Jatin Mehta added a comment - This issue is also seen in Jenkins ver 2.176.1. Jenkins is appending "@2" at the end of the workspace name for one of my nodes. This is causing the tests being run by the pipeline to fail since they look for files in a configured workspace name without the suffix  @2 . Jenkins seems to do that to prevent mishaps during concurrent builds but this happens for me even when I check  Do not allow concurrent builds  in the pipeline configuration. Is this new behavior due to a Jenkins version upgrade? Do we have any workarounds? I changed the project name that prevented it from happening but it came up again today. Can't keep changing project name to new ones. Please help!

          Devin Nusbaum added a comment -

          j1mehta if you have a consistent way to reproduce the problem from scratch, please file a new issue with full reproduction steps (i.e. download Jenkins 2.176.1, start it up, install plugins X version Y and Z version W, create a Pipeline with the following (ideally minimal) script, etc.).

          Devin Nusbaum added a comment - j1mehta if you have a consistent way to reproduce the problem from scratch, please file a new issue with full reproduction steps (i.e. download Jenkins 2.176.1, start it up, install plugins X version Y and Z version W, create a Pipeline with the following (ideally minimal) script, etc.).

          Jatin Mehta added a comment - - edited

          dnusbaum, thank you for the reply. Makes sense, will do that. However, it will take some time since this problem occurs in a Jenkins environment of which I am not an admin and would need to replicate it locally. Meanwhile, if I could get any pointers, that would be great. 

          EDIT: It doesn't get reproduced right away. It starts occurring only after several builds and hence is kinda hard to reproduce.

          Jatin Mehta added a comment - - edited dnusbaum , thank you for the reply. Makes sense, will do that. However, it will take some time since this problem occurs in a Jenkins environment of which I am not an admin and would need to replicate it locally. Meanwhile, if I could get any pointers, that would be great.  EDIT: It doesn't get reproduced right away. It starts occurring only after several builds and hence is kinda hard to reproduce.

          Kamil Szuster added a comment -

          j1mehta dnusbaum I think we triggered this issue by removing the working directory of the job within the workspace of a node by hand. 

          Something like 

          1. Run job_a on node b
          2. let it finish
          3. login via ssh onto node b
          4. rm -R {JENKINS_WORKSPACE}/job_a
          5. rerun job a
          6. new folder with @2 suffix should be created

          Unfortunately i cant evaluate this right now.

          Kamil Szuster added a comment - j1mehta dnusbaum I think we triggered this issue by removing the working directory of the job within the workspace of a node by hand.  Something like  Run job_a on node b let it finish login via ssh onto node b rm -R {JENKINS_WORKSPACE}/job_a rerun job a new folder with @2 suffix should be created Unfortunately i cant evaluate this right now.

          Jatin Mehta added a comment -

          Hi kamil02. I tried that too. With every new build, my Jenkinsfile removes the previous build's workspace.  Now, even if I remove the original workspace by hand, in the next build, it still goes on to create the workspace with an @2 suffix. Not sure if Jenkins caches the data (that a workspace xyz was present once upon a time on this node) using which it creates the new workspace (xyz@2)

          Jatin Mehta added a comment - Hi kamil02 . I tried that too. With every new build, my Jenkinsfile removes the previous build's workspace.  Now, even if I remove the original workspace by hand, in the next build, it still goes on to create the workspace with an @2 suffix. Not sure if Jenkins caches the data (that a workspace xyz was present once upon a time on this node) using which it creates the new workspace (xyz@2)

          Kamil Szuster added a comment -

          j1mehta did you also try to restart the jenkins master after the deletion of both (original and @2) workspaces? If i remember correctly, i also had the feeling, that jenkins uses some kind of cache to remember where the workspace is supposed to be. As mentioned before, restarting the jenkins master instance after deleting the original and @2 workspace solved our problem.

          Kamil Szuster added a comment - j1mehta did you also try to restart the jenkins master after the deletion of both (original and @2) workspaces? If i remember correctly, i also had the feeling, that jenkins uses some kind of cache to remember where the workspace is supposed to be. As mentioned before, restarting the jenkins master instance after deleting the original and @2 workspace solved our problem.

          Jatin Mehta added a comment - - edited

          kamil02, I don't have admin access to restart Jenkins master. Besides, thats only a temporary workaround. If this was a permanent solution, it shouldn't have cropped up for a new project (or same project with its name changed for that matter). It is still a bug. Will try to come up with a more consistent way to produce it. 

          Jatin Mehta added a comment - - edited kamil02 , I don't have admin access to restart Jenkins master. Besides, thats only a temporary workaround. If this was a permanent solution, it shouldn't have cropped up for a new project (or same project with its name changed for that matter). It is still a bug. Will try to come up with a more consistent way to produce it. 

          Adam Kolarik added a comment -

          I'm seeing this issue on version 2.176.1. We also don't have concurrent or Matrix builds setup, just one executor per node. On one specific node, we got the workspace@2 duplicated workspace. I tried deleting both workspaces on the node manually and restart Jenkins server to see if that helps and it did.

          Adam Kolarik added a comment - I'm seeing this issue on version 2.176.1. We also don't have concurrent or Matrix builds setup, just one executor per node. On one specific node, we got the workspace@2 duplicated workspace. I tried deleting both workspaces on the node manually and restart Jenkins server to see if that helps and it did.

          Kamil Szuster added a comment -

          Today the same error occurred again: Jenkins created a ..@2 and ...@3 folder in the .../workspace directory. I had to delete these via SSH and restart the master => back to normal job workspace again.  

          Jenkins ver. 2.176.3

          Kamil Szuster added a comment - Today the same error occurred again: Jenkins created a ..@2 and ...@3 folder in the .../workspace directory. I had to delete these via SSH and restart the master => back to normal job workspace again.   Jenkins ver. 2.176.3

          Peter Shen added a comment - - edited

          I have the same issue on Jenkins ver. 2.176.3.

          Peter Shen added a comment - - edited I have the same issue on Jenkins ver. 2.176.3.

          I have the same issue with Jenkins version 2.222.3

          I have admin access, so I rebooted Jenkins master after deleting ..@2 folder. This did not help either.

          Naveen Sathyanarayana added a comment - I have the same issue with Jenkins version 2.222.3 I have admin access, so I rebooted Jenkins master after deleting ..@2 folder. This did not help either.

          Sten Olsson added a comment -

          I also have this problem.  I checkout two repositories into my workspace and often times Jenkins creates a second workspace for the parallel-running job which checks out the second repo.  The tools assume they are in the same workspace and promptly fail.  Possible workaround might be to save the location of WORKSPACE in the beginning and then with each parallel task, always change directory (cd) back to that location before doing anything. 

          For example,

          Sten Olsson added a comment - I also have this problem.  I checkout two repositories into my workspace and often times Jenkins creates a second workspace for the parallel-running job which checks out the second repo.  The tools assume they are in the same workspace and promptly fail.  Possible workaround might be to save the location of WORKSPACE in the beginning and then with each parallel task, always change directory (cd) back to that location before doing anything.  For example,

          Sten Olsson added a comment -

          dnusbaum - I see that the issue is marked as Resolved.  In this case, should a new issue be created or the preference is to Reopen?  I didn't want to reopen if it really needs to be a new issue instead.

          Sten Olsson added a comment - dnusbaum - I see that the issue is marked as Resolved.  In this case, should a new issue be created or the preference is to Reopen?  I didn't want to reopen if it really needs to be a new issue instead.

          Devin Nusbaum added a comment -

          olssons My guess is that there are probably many distinct issues being reported here (e.g. some scenarios are likely specific to (multibranch) pipelines, but there were also bugs fixed that affected freestyle and other project types). I would go ahead and open a new issue with full details on your specific case, ideally with steps on how to reproduce the problem from scratch.

          Devin Nusbaum added a comment - olssons My guess is that there are probably many distinct issues being reported here (e.g. some scenarios are likely specific to (multibranch) pipelines, but there were also bugs fixed that affected freestyle and other project types). I would go ahead and open a new issue with full details on your specific case, ideally with steps on how to reproduce the problem from scratch.

          Sten Olsson added a comment -

          dnusbaum - With my issue, the double workspace never occurs during the "Build Software" parallel tasks.  However, when I have 3 parallel tasks in a row, like under "Analysis and Adaptation", this is where the problem sometimes happens.  I say sometimes because it happens maybe half the time or 3 out of 4 builds in a row. 

          I first started my workarounds by removing the second checkout like I mentioned before (see "Checkout tools" above).  That did not fix the problem.  I then made all the tasks in Analysis and Adaptation serial.  That (obviously) fixed the problem.  If I were to guess, it didn't like 3 tasks running in parallel, all using the same workspace.  My builds take about 10-15 minutes longer now, but no more failures at least.

          I may open an issue in the future. 

          PS - I could never recreate the problem from the Jenkins Pipeline script, only from a checked in Jenkinsfile itself

          Sten Olsson added a comment - dnusbaum - With my issue, the double workspace never occurs during the "Build Software" parallel tasks.  However, when I have 3 parallel tasks in a row, like under "Analysis and Adaptation", this is where the problem sometimes happens.  I say sometimes because it happens maybe half the time or 3 out of 4 builds in a row.  I first started my workarounds by removing the second checkout like I mentioned before (see "Checkout tools" above).  That did not fix the problem.  I then made all the tasks in Analysis and Adaptation serial.  That (obviously) fixed the problem.  If I were to guess, it didn't like 3 tasks running in parallel, all using the same workspace.  My builds take about 10-15 minutes longer now, but no more failures at least. I may open an issue in the future.  PS - I could never recreate the problem from the Jenkins Pipeline script, only from a checked in Jenkinsfile itself

          Ori Wiesel added a comment -

          Is there anyway to disable this option?

          so no new workspace will be created and only the original workspace will be used alone? (I'm aware of the risks)

          Ori Wiesel added a comment - Is there anyway to disable this option? so no new workspace will be created and only the original workspace will be used alone? (I'm aware of the risks)

            dnusbaum Devin Nusbaum
            totoroliu Rick Liu
            Votes:
            4 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated:
              Resolved: