Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30231

Build creates second workspace@2 for non-concurrent build configuration

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Duplicate
    • core, matrix-project-plugin
    • None
    • OS: Ubuntu 12.04 64-bit
      Jenkins: 1.609.1 (upgraded from 1.580.3)
      disk-usage plugin: 0.25 (upgraded from 0.24)
      Matrix Project Plugin: 1.6 (upgraded from 1.4.1)
    • Jenkins Core 2.136

    Description

      I didn't have this issue when I was using ver 1.580.3.
      After upgrading Jenkins ver 1.609.1 (together with other Jenkins plugins update),
      I started to see this issue.

      My jenkins job is using multi-config project type (aka Matrix project),
      and has 3 variants.
      I didn't set to be run concurrently,
      so the 3 variants will start the build one by one.

      In the previous version 1.580.3,
      it will reuse the same workspace folder for all of 3 variant builds.

      In the new version 1.609.1,
      when multiple jobs are triggerd,
      I can see sometimes the first variant build of the second job
      will start to create and use workspace@2
      instead of re-using original worksapce.

      Note:
      I'm also using disk-usage plugin.
      At the end of build,
      it will trigger that plugin to calculate workspace disk space.
      My work space tends to be large and sometimes it take several minutes to finish.

      I suspect it's that disk-usage plugin still occupying the original worksapce,
      but the second job in queue starts earlier,
      so it creates workspace@2 for the first variable build.

      From workspace@2,
      I always see only one variant only under AXIS folder.

      Attachments

        Issue Links

          Activity

            totoroliu Rick Liu created issue -
            totoroliu Rick Liu added a comment -

            Per comment from Jesse Glick,
            I create this separate issue
            which has similar symptom.

            totoroliu Rick Liu added a comment - Per comment from Jesse Glick, I create this separate issue which has similar symptom.
            totoroliu Rick Liu made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-21622 [ JENKINS-21622 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 165324 ] JNJira + In-Review [ 181925 ]
            will177 Will Berriss added a comment -

            I am seeing this issue today on one of my pipelines. Is there a quick fix workaround I can use to get rid of it and reset the pipeline? I don't mind starting the pipeline from scratch again - just wondering how to get rid of the @2 workspace. Anyone know?

            will177 Will Berriss added a comment - I am seeing this issue today on one of my pipelines. Is there a quick fix workaround I can use to get rid of it and reset the pipeline? I don't mind starting the pipeline from scratch again - just wondering how to get rid of the @2 workspace. Anyone know?
            mksmith Matthew Smith added a comment -

            will177 It looks like this might be a solution: https://issues.jenkins-ci.org/browse/JENKINS-14354

            It looks like you can replace the "@" separator with a separator of your choice.

            mksmith Matthew Smith added a comment - will177 It looks like this might be a solution: https://issues.jenkins-ci.org/browse/JENKINS-14354 It looks like you can replace the "@" separator with a separator of your choice.
            totoroliu Rick Liu added a comment -

            I use customize worksapce configuration to hardcoded the workspace name to work-around this problem

            totoroliu Rick Liu added a comment - I use customize worksapce configuration to hardcoded the workspace name to work-around this problem

            This is still a problem. We restrict our jobs to one per node and still see a number of @2 workspaces. Replacing @ with _ (like JENKINS-14354) helps builds which get tripped up by @ but we still have duplication. As we have a number of large workspaces this wastes a lot of diskspace.

            rg Russell Gallop added a comment - This is still a problem. We restrict our jobs to one per node and still see a number of @2 workspaces. Replacing @ with _ (like JENKINS-14354 ) helps builds which get tripped up by @ but we still have duplication. As we have a number of large workspaces this wastes a lot of diskspace.

            We've recently started seeing a duplicate @2 workspace as well, when we moved from 2.60.3 to 2.73.2 with some new plug in.

             

            ndza2017 nolan de souza added a comment - We've recently started seeing a duplicate @2 workspace as well, when we moved from 2.60.3 to 2.73.2 with some new plug in.  
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal CD-480 (Web Link)" [ 20531 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal CD-473 (Web Link)" [ 20532 ]
            recampbell Ryan Campbell made changes -
            Link This issue relates to JENKINS-41127 [ JENKINS-41127 ]
            dnusbaum Devin Nusbaum added a comment -

            JENKINS-41127, which may have been the root cause of this issue, was just fixed in Jenkins 2.136. Does anyone have a test environment in which they can consistently reproduce this issue? If so, it would be great if you could test whether this problem still occurs in Jenkins 2.136 or newer.

            dnusbaum Devin Nusbaum added a comment - JENKINS-41127 , which may have been the root cause of this issue, was just fixed in Jenkins 2.136 . Does anyone have a test environment in which they can consistently reproduce this issue? If so, it would be great if you could test whether this problem still occurs in Jenkins 2.136 or newer.
            dnusbaum Devin Nusbaum added a comment - - edited

            Since I have not heard back either way, I am going to go ahead and close this issue because at least one way it could happen was fixed in Jenkins 2.136. Please feel free to reopen the issue if you are still seeing it in Jenkins 2.136 or newer.

            dnusbaum Devin Nusbaum added a comment - - edited Since I have not heard back either way, I am going to go ahead and close this issue because at least one way it could happen was fixed in Jenkins 2.136. Please feel free to reopen the issue if you are still seeing it in Jenkins 2.136 or newer.
            dnusbaum Devin Nusbaum made changes -
            Released As Jenkins Core 2.136
            Assignee Kohsuke Kawaguchi [ kohsuke ] Devin Nusbaum [ dnusbaum ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            openjenkins J S added a comment -

            I have the same problem at Jenkins Version : 2.121.3

            openjenkins J S added a comment - I have the same problem at Jenkins Version : 2.121.3
            openjenkins J S added a comment -

            I still have the same problem on Jenkins Version 2.121.3 LTS .

            On my Slave i get the Folder @2 that are very big.

            openjenkins J S added a comment - I still have the same problem on Jenkins Version 2.121.3 LTS . On my Slave i get the Folder @2 that are very big.
            openjenkins J S made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            dnusbaum Devin Nusbaum added a comment - - edited

            openjenkins The change I made to fix JENKINS-41127 is in 2.136, and it was not backported to the 2.121.x LTS line (It is in the 2.138.x LTS line). Please only reopen the issue if you are seeing it in 2.136 or newer (unless you have additional information that leads you to believe it is distinct from JENKINS-41127, such as seeing the issue for something other than a pipeline or matrix job, and please include any relevant info if that is the case).

            dnusbaum Devin Nusbaum added a comment - - edited openjenkins The change I made to fix JENKINS-41127 is in 2.136, and it was not backported to the 2.121.x LTS line (It is in the 2.138.x LTS line). Please only reopen the issue if you are seeing it in 2.136 or newer (unless you have additional information that leads you to believe it is distinct from JENKINS-41127 , such as seeing the issue for something other than a pipeline or matrix job, and please include any relevant info if that is the case).
            dnusbaum Devin Nusbaum made changes -
            Resolution Duplicate [ 3 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            kamil02 Kamil Szuster added a comment -

            Hi! 

            Yesterday one of my jobs started to show the same behavior, but we are not using the matrix-project-plugin. At some point (maybe after we deleted the original workspace-folder of a certain job) jenkins started to build the project in a folder suffixed by "@2" - but only on one specific node. Recreating the Pipelinejob with the same name didn't help. Today i did a restart of the server and that seems to have soled the issue for now.

            We are using Jenkins 2.138.2.

            kamil02 Kamil Szuster added a comment - Hi!  Yesterday one of my jobs started to show the same behavior, but we are not using the matrix-project-plugin. At some point (maybe after we deleted the original workspace-folder of a certain job) jenkins started to build the project in a folder suffixed by "@2" - but only on one specific node. Recreating the Pipelinejob with the same name didn't help. Today i did a restart of the server and that seems to have soled the issue for now. We are using Jenkins 2.138.2.
            j1mehta Jatin Mehta added a comment -

            This issue is also seen in Jenkins ver 2.176.1. Jenkins is appending "@2" at the end of the workspace name for one of my nodes. This is causing the tests being run by the pipeline to fail since they look for files in a configured workspace name without the suffix @2. Jenkins seems to do that to prevent mishaps during concurrent builds but this happens for me even when I check Do not allow concurrent builds in the pipeline configuration. Is this new behavior due to a Jenkins version upgrade? Do we have any workarounds? I changed the project name that prevented it from happening but it came up again today. Can't keep changing project name to new ones. Please help!

            j1mehta Jatin Mehta added a comment - This issue is also seen in Jenkins ver 2.176.1. Jenkins is appending "@2" at the end of the workspace name for one of my nodes. This is causing the tests being run by the pipeline to fail since they look for files in a configured workspace name without the suffix  @2 . Jenkins seems to do that to prevent mishaps during concurrent builds but this happens for me even when I check  Do not allow concurrent builds  in the pipeline configuration. Is this new behavior due to a Jenkins version upgrade? Do we have any workarounds? I changed the project name that prevented it from happening but it came up again today. Can't keep changing project name to new ones. Please help!
            dnusbaum Devin Nusbaum added a comment -

            j1mehta if you have a consistent way to reproduce the problem from scratch, please file a new issue with full reproduction steps (i.e. download Jenkins 2.176.1, start it up, install plugins X version Y and Z version W, create a Pipeline with the following (ideally minimal) script, etc.).

            dnusbaum Devin Nusbaum added a comment - j1mehta if you have a consistent way to reproduce the problem from scratch, please file a new issue with full reproduction steps (i.e. download Jenkins 2.176.1, start it up, install plugins X version Y and Z version W, create a Pipeline with the following (ideally minimal) script, etc.).
            j1mehta Jatin Mehta added a comment - - edited

            dnusbaum, thank you for the reply. Makes sense, will do that. However, it will take some time since this problem occurs in a Jenkins environment of which I am not an admin and would need to replicate it locally. Meanwhile, if I could get any pointers, that would be great. 

            EDIT: It doesn't get reproduced right away. It starts occurring only after several builds and hence is kinda hard to reproduce.

            j1mehta Jatin Mehta added a comment - - edited dnusbaum , thank you for the reply. Makes sense, will do that. However, it will take some time since this problem occurs in a Jenkins environment of which I am not an admin and would need to replicate it locally. Meanwhile, if I could get any pointers, that would be great.  EDIT: It doesn't get reproduced right away. It starts occurring only after several builds and hence is kinda hard to reproduce.
            kamil02 Kamil Szuster added a comment -

            j1mehta dnusbaum I think we triggered this issue by removing the working directory of the job within the workspace of a node by hand. 

            Something like 

            1. Run job_a on node b
            2. let it finish
            3. login via ssh onto node b
            4. rm -R {JENKINS_WORKSPACE}/job_a
            5. rerun job a
            6. new folder with @2 suffix should be created

            Unfortunately i cant evaluate this right now.

            kamil02 Kamil Szuster added a comment - j1mehta dnusbaum I think we triggered this issue by removing the working directory of the job within the workspace of a node by hand.  Something like  Run job_a on node b let it finish login via ssh onto node b rm -R {JENKINS_WORKSPACE}/job_a rerun job a new folder with @2 suffix should be created Unfortunately i cant evaluate this right now.
            j1mehta Jatin Mehta added a comment -

            Hi kamil02. I tried that too. With every new build, my Jenkinsfile removes the previous build's workspace.  Now, even if I remove the original workspace by hand, in the next build, it still goes on to create the workspace with an @2 suffix. Not sure if Jenkins caches the data (that a workspace xyz was present once upon a time on this node) using which it creates the new workspace (xyz@2)

            j1mehta Jatin Mehta added a comment - Hi kamil02 . I tried that too. With every new build, my Jenkinsfile removes the previous build's workspace.  Now, even if I remove the original workspace by hand, in the next build, it still goes on to create the workspace with an @2 suffix. Not sure if Jenkins caches the data (that a workspace xyz was present once upon a time on this node) using which it creates the new workspace (xyz@2)
            kamil02 Kamil Szuster added a comment -

            j1mehta did you also try to restart the jenkins master after the deletion of both (original and @2) workspaces? If i remember correctly, i also had the feeling, that jenkins uses some kind of cache to remember where the workspace is supposed to be. As mentioned before, restarting the jenkins master instance after deleting the original and @2 workspace solved our problem.

            kamil02 Kamil Szuster added a comment - j1mehta did you also try to restart the jenkins master after the deletion of both (original and @2) workspaces? If i remember correctly, i also had the feeling, that jenkins uses some kind of cache to remember where the workspace is supposed to be. As mentioned before, restarting the jenkins master instance after deleting the original and @2 workspace solved our problem.
            j1mehta Jatin Mehta added a comment - - edited

            kamil02, I don't have admin access to restart Jenkins master. Besides, thats only a temporary workaround. If this was a permanent solution, it shouldn't have cropped up for a new project (or same project with its name changed for that matter). It is still a bug. Will try to come up with a more consistent way to produce it. 

            j1mehta Jatin Mehta added a comment - - edited kamil02 , I don't have admin access to restart Jenkins master. Besides, thats only a temporary workaround. If this was a permanent solution, it shouldn't have cropped up for a new project (or same project with its name changed for that matter). It is still a bug. Will try to come up with a more consistent way to produce it. 
            adamkolarik Adam Kolarik added a comment -

            I'm seeing this issue on version 2.176.1. We also don't have concurrent or Matrix builds setup, just one executor per node. On one specific node, we got the workspace@2 duplicated workspace. I tried deleting both workspaces on the node manually and restart Jenkins server to see if that helps and it did.

            adamkolarik Adam Kolarik added a comment - I'm seeing this issue on version 2.176.1. We also don't have concurrent or Matrix builds setup, just one executor per node. On one specific node, we got the workspace@2 duplicated workspace. I tried deleting both workspaces on the node manually and restart Jenkins server to see if that helps and it did.
            kamil02 Kamil Szuster added a comment -

            Today the same error occurred again: Jenkins created a ..@2 and ...@3 folder in the .../workspace directory. I had to delete these via SSH and restart the master => back to normal job workspace again.  

            Jenkins ver. 2.176.3

            kamil02 Kamil Szuster added a comment - Today the same error occurred again: Jenkins created a ..@2 and ...@3 folder in the .../workspace directory. I had to delete these via SSH and restart the master => back to normal job workspace again.   Jenkins ver. 2.176.3
            xianpeng Peter Shen added a comment - - edited

            I have the same issue on Jenkins ver. 2.176.3.

            xianpeng Peter Shen added a comment - - edited I have the same issue on Jenkins ver. 2.176.3.

            I have the same issue with Jenkins version 2.222.3

            I have admin access, so I rebooted Jenkins master after deleting ..@2 folder. This did not help either.

            naveen6s Naveen Sathyanarayana added a comment - I have the same issue with Jenkins version 2.222.3 I have admin access, so I rebooted Jenkins master after deleting ..@2 folder. This did not help either.
            olssons Sten Olsson made changes -
            Attachment image-2020-08-16-13-41-37-491.png [ 52222 ]
            olssons Sten Olsson made changes -
            Attachment image-2020-08-16-13-42-26-759.png [ 52223 ]
            olssons Sten Olsson made changes -
            Attachment image-2020-08-16-13-43-23-877.png [ 52224 ]
            olssons Sten Olsson added a comment -

            I also have this problem.  I checkout two repositories into my workspace and often times Jenkins creates a second workspace for the parallel-running job which checks out the second repo.  The tools assume they are in the same workspace and promptly fail.  Possible workaround might be to save the location of WORKSPACE in the beginning and then with each parallel task, always change directory (cd) back to that location before doing anything. 

            For example,

            olssons Sten Olsson added a comment - I also have this problem.  I checkout two repositories into my workspace and often times Jenkins creates a second workspace for the parallel-running job which checks out the second repo.  The tools assume they are in the same workspace and promptly fail.  Possible workaround might be to save the location of WORKSPACE in the beginning and then with each parallel task, always change directory (cd) back to that location before doing anything.  For example,
            olssons Sten Olsson added a comment -

            dnusbaum - I see that the issue is marked as Resolved.  In this case, should a new issue be created or the preference is to Reopen?  I didn't want to reopen if it really needs to be a new issue instead.

            olssons Sten Olsson added a comment - dnusbaum - I see that the issue is marked as Resolved.  In this case, should a new issue be created or the preference is to Reopen?  I didn't want to reopen if it really needs to be a new issue instead.
            dnusbaum Devin Nusbaum added a comment -

            olssons My guess is that there are probably many distinct issues being reported here (e.g. some scenarios are likely specific to (multibranch) pipelines, but there were also bugs fixed that affected freestyle and other project types). I would go ahead and open a new issue with full details on your specific case, ideally with steps on how to reproduce the problem from scratch.

            dnusbaum Devin Nusbaum added a comment - olssons My guess is that there are probably many distinct issues being reported here (e.g. some scenarios are likely specific to (multibranch) pipelines, but there were also bugs fixed that affected freestyle and other project types). I would go ahead and open a new issue with full details on your specific case, ideally with steps on how to reproduce the problem from scratch.
            olssons Sten Olsson added a comment -

            dnusbaum - With my issue, the double workspace never occurs during the "Build Software" parallel tasks.  However, when I have 3 parallel tasks in a row, like under "Analysis and Adaptation", this is where the problem sometimes happens.  I say sometimes because it happens maybe half the time or 3 out of 4 builds in a row. 

            I first started my workarounds by removing the second checkout like I mentioned before (see "Checkout tools" above).  That did not fix the problem.  I then made all the tasks in Analysis and Adaptation serial.  That (obviously) fixed the problem.  If I were to guess, it didn't like 3 tasks running in parallel, all using the same workspace.  My builds take about 10-15 minutes longer now, but no more failures at least.

            I may open an issue in the future. 

            PS - I could never recreate the problem from the Jenkins Pipeline script, only from a checked in Jenkinsfile itself

            olssons Sten Olsson added a comment - dnusbaum - With my issue, the double workspace never occurs during the "Build Software" parallel tasks.  However, when I have 3 parallel tasks in a row, like under "Analysis and Adaptation", this is where the problem sometimes happens.  I say sometimes because it happens maybe half the time or 3 out of 4 builds in a row.  I first started my workarounds by removing the second checkout like I mentioned before (see "Checkout tools" above).  That did not fix the problem.  I then made all the tasks in Analysis and Adaptation serial.  That (obviously) fixed the problem.  If I were to guess, it didn't like 3 tasks running in parallel, all using the same workspace.  My builds take about 10-15 minutes longer now, but no more failures at least. I may open an issue in the future.  PS - I could never recreate the problem from the Jenkins Pipeline script, only from a checked in Jenkinsfile itself
            wizzzozzz Ori Wiesel added a comment -

            Is there anyway to disable this option?

            so no new workspace will be created and only the original workspace will be used alone? (I'm aware of the risks)

            wizzzozzz Ori Wiesel added a comment - Is there anyway to disable this option? so no new workspace will be created and only the original workspace will be used alone? (I'm aware of the risks)

            People

              dnusbaum Devin Nusbaum
              totoroliu Rick Liu
              Votes:
              4 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: