Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23693

Cannot delete workspace

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: ws-cleanup-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.570, workspace clean up plugin 0.2.1, Debian wheezy (master+slave)
    • Similar Issues:

      Description

      After updating Jenkins from 1.567 to 1.570 and Workspace Cleanup Plugin from 0.20 to 0.21, all jobs using workspace cleanup are failing:

      ...
      00:00:00.508 Baue auf Slave comitdev1 (karma git linux fast 64bit) in Arbeitsbereich /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893
      00:00:00.508 
      00:00:00.508 Deleting project workspace... Using command: /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts 
      00:00:03.932 Using command: /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts 
      00:00:06.941 Using command: /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts 
      00:00:06.943 Cannot delete workspace: java.io.IOException: Cannot run program "/home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts": java.io.IOException: error=13, Permission denied
      00:00:06.945 ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "/home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts": java.io.IOException: error=13, Permission denied
      00:00:06.945 [FINDBUGS] Skipping publisher since build result is FAILURE
      00:00:06.946 [PMD] Skipping publisher since build result is FAILURE
      ...
      

      I have checked the workspace, everything looks ok, file permissions are all correct. Downgrade the plugin back to 0.20 fixed the problem.

        Attachments

          Activity

          zhh Hua Zhang created issue -
          Hide
          vjuranek vjuranek added a comment -

          Hi,could you please share your configuration of the plugin? Do you use external program for cleanup? Looks like it's trying to use external program for cleanup and for some reason it wants to use workspace/SD-2893/APPLICATION/artifacts which obviously has to fail.

          Show
          vjuranek vjuranek added a comment - Hi,could you please share your configuration of the plugin? Do you use external program for cleanup? Looks like it's trying to use external program for cleanup and for some reason it wants to use workspace/SD-2893/APPLICATION/artifacts which obviously has to fail.
          zhh Hua Zhang made changes -
          Field Original Value New Value
          Description After updating Jenkins from 1.567 to 1.570 and Workspace Cleanup Plugin from 0.20 to 0.21, all jobs using workspace cleanup are failing:

          {code}
          00:00:06.943 Cannot delete workspace: java.io.IOException: Cannot run program "/home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts": java.io.IOException: error=13, Permission denied
          00:00:06.945 ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "/home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts": java.io.IOException: error=13, Permission denied
          {code}

          I have checked the workspace, everything looks ok, file permissions are all correct. Downgrade the plugin back to 0.20 fixed the problem.
          After updating Jenkins from 1.567 to 1.570 and Workspace Cleanup Plugin from 0.20 to 0.21, all jobs using workspace cleanup are failing:

          {code}
          ...
          00:00:00.508 Baue auf Slave comitdev1 (karma git linux fast 64bit) in Arbeitsbereich /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893
          00:00:00.508
          00:00:00.508 Deleting project workspace... Using command: /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts
          00:00:03.932 Using command: /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts
          00:00:06.941 Using command: /home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts
          00:00:06.943 Cannot delete workspace: java.io.IOException: Cannot run program "/home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts": java.io.IOException: error=13, Permission denied
          00:00:06.945 ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "/home/jenkins-slave/jenkins-slave-fs-root/workspace/SD-2893/APPLICATION/artifacts": java.io.IOException: error=13, Permission denied
          00:00:06.945 [FINDBUGS] Skipping publisher since build result is FAILURE
          00:00:06.946 [PMD] Skipping publisher since build result is FAILURE
          ...
          {code}

          I have checked the workspace, everything looks ok, file permissions are all correct. Downgrade the plugin back to 0.20 fixed the problem.
          Hide
          zhh Hua Zhang added a comment -

          Please find the configuration in attachment.
          No, I am not using any other external programs for cleanup. FWIW: the slaves are all Debian boxes (not VM), the master is a Debian VM, and JENKINS_HOME is on a NFS partition

          Show
          zhh Hua Zhang added a comment - Please find the configuration in attachment. No, I am not using any other external programs for cleanup. FWIW: the slaves are all Debian boxes (not VM), the master is a Debian VM, and JENKINS_HOME is on a NFS partition
          zhh Hua Zhang made changes -
          Attachment cannot-delete-workspace-config.zip [ 26346 ]
          Hide
          vjuranek vjuranek added a comment -

          Thanks for details. Unfortunately, I'm not able to reproduce it: I used your config with ws-cleanup 0.20, it worked, then upgraded to 0.21 and it still works. Anyway, from log record Using command: ... is obvious, that method handling externalDelete is called. Could you please check once again, that you job configs don't contain any externalDelete (e.g. there is only <externalDelete/> element without any content)? Right now I have no idea how <externalDelete/> can be interpreted as not to be null
          Thanks

          Show
          vjuranek vjuranek added a comment - Thanks for details. Unfortunately, I'm not able to reproduce it: I used your config with ws-cleanup 0.20, it worked, then upgraded to 0.21 and it still works. Anyway, from log record Using command: ... is obvious, that method handling externalDelete is called. Could you please check once again, that you job configs don't contain any externalDelete (e.g. there is only <externalDelete/> element without any content)? Right now I have no idea how <externalDelete/> can be interpreted as not to be null Thanks
          zhh Hua Zhang made changes -
          Attachment ws-cleanup-config.png [ 26349 ]
          Hide
          zhh Hua Zhang added a comment -

          I added a screenshot of the job configuration. The input field "External Deletion Command" is empty, no whitespaces in there.
          And in config.xml:

              <hudson.plugins.ws__cleanup.PreBuildCleanup plugin="ws-cleanup@0.20">
                <patterns>
                  <hudson.plugins.ws__cleanup.Pattern>
                    <pattern>APPLICATION/artifacts</pattern>
                    <type>INCLUDE</type>
                  </hudson.plugins.ws__cleanup.Pattern>
                </patterns>
                <deleteDirs>true</deleteDirs>
                <cleanupParameter/>
                <externalDelete/>
              </hudson.plugins.ws__cleanup.PreBuildCleanup>
          

          Nothing looks suspicious to me.

          I will try upgrade again to see if this is only an one-time issue.

          Show
          zhh Hua Zhang added a comment - I added a screenshot of the job configuration. The input field "External Deletion Command" is empty, no whitespaces in there. And in config.xml: <hudson.plugins.ws__cleanup.PreBuildCleanup plugin= "ws-cleanup@0.20" > <patterns> <hudson.plugins.ws__cleanup.Pattern> <pattern>APPLICATION/artifacts</pattern> <type>INCLUDE</type> </hudson.plugins.ws__cleanup.Pattern> </patterns> <deleteDirs> true </deleteDirs> <cleanupParameter/> <externalDelete/> </hudson.plugins.ws__cleanup.PreBuildCleanup> Nothing looks suspicious to me. I will try upgrade again to see if this is only an one-time issue.
          Hide
          vjuranek vjuranek added a comment -

          thanks for double check. I'll go through the code once again and try to figure out the reason how this can happen

          Show
          vjuranek vjuranek added a comment - thanks for double check. I'll go through the code once again and try to figure out the reason how this can happen
          Hide
          danielbeck Daniel Beck added a comment -

          What happens when this line returns an empty string instead of null? Could that happen with empty node properties (i.e. empty string as key and value)?
          https://github.com/jenkinsci/ws-cleanup-plugin/blob/master/src/main/java/hudson/plugins/ws_cleanup/Cleanup.java#L35

          Show
          danielbeck Daniel Beck added a comment - What happens when this line returns an empty string instead of null? Could that happen with empty node properties (i.e. empty string as key and value)? https://github.com/jenkinsci/ws-cleanup-plugin/blob/master/src/main/java/hudson/plugins/ws_cleanup/Cleanup.java#L35
          Hide
          vjuranek vjuranek added a comment -

          quickly looking into the code, it would probably result into behavior described in this issue, but IMHO this should never happen, as env. var. expansion should return null for null param: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/Util.java#L143

          Anyway, will check it more carefully in the eventing. Thanks for good pointer!

          Show
          vjuranek vjuranek added a comment - quickly looking into the code, it would probably result into behavior described in this issue, but IMHO this should never happen, as env. var. expansion should return null for null param: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/Util.java#L143 Anyway, will check it more carefully in the eventing. Thanks for good pointer!
          Hide
          danielbeck Daniel Beck added a comment -

          I'm probably wrong about this:

          An empty entry of EnvironmentVariablesNodeProperty maps "" to "", and I got an NPE trying to pass null to get(String)...

          Show
          danielbeck Daniel Beck added a comment - I'm probably wrong about this: An empty entry of EnvironmentVariablesNodeProperty maps "" to "", and I got an NPE trying to pass null to get(String)...
          Hide
          drmonte Mantas Balnys added a comment - - edited

          Same here:

          ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "<first found file in inclusion list>": java.io.IOException: error=13, Permission denied
          

          It gets even more interesting on Windows:

          ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "<first found file in inclusion list>": CreateProcess error=193, %1 is not a valid Win32 application
          

          Note that by default xml is saved as:

                <cleanupParameter></cleanupParameter>
                <externalDelete></externalDelete>
          

          Though that does not change the behavior of the plugin.
          I would recommend to 1st check for null, 2nd trim leading trailing spaces and check for empty string.

          Show
          drmonte Mantas Balnys added a comment - - edited Same here: ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "<first found file in inclusion list>": java.io.IOException: error=13, Permission denied It gets even more interesting on Windows: ERROR: Cannot delete workspace: java.io.IOException: Cannot run program "<first found file in inclusion list>": CreateProcess error=193, %1 is not a valid Win32 application Note that by default xml is saved as: <cleanupParameter></cleanupParameter> <externalDelete></externalDelete> Though that does not change the behavior of the plugin. I would recommend to 1st check for null, 2nd trim leading trailing spaces and check for empty string.
          Hide
          lanfeust Jean-Jacques Lafay added a comment -

          Also hit by this one. But with an interesting precision : we have several very similar jobs, and only one of these had the problem. Checking the differences (and the previous comments), I'm pretty sure it's related to the use of EnvInject plugin.

          Show
          lanfeust Jean-Jacques Lafay added a comment - Also hit by this one. But with an interesting precision : we have several very similar jobs, and only one of these had the problem. Checking the differences (and the previous comments), I'm pretty sure it's related to the use of EnvInject plugin.
          Hide
          vjuranek vjuranek added a comment -

          There's one small fix related to how env. variables are expanded. Could someone how is able to reproduce this issue try to check, if this change fixes the problem?

          Any further details how to reproduce the issue will be more than welcome.
          Thanks.

          Show
          vjuranek vjuranek added a comment - There's one small fix related to how env. variables are expanded. Could someone how is able to reproduce this issue try to check, if this change fixes the problem? Any further details how to reproduce the issue will be more than welcome. Thanks.
          Hide
          christian_galsterer_gd Christian Galsterer added a comment -

          We have the same problem on RHEL/CentOS and there is no external delete command defined. The config.xml has an empty <externalDelete/> element.

          Show
          christian_galsterer_gd Christian Galsterer added a comment - We have the same problem on RHEL/CentOS and there is no external delete command defined. The config.xml has an empty <externalDelete/> element.
          Hide
          vjuranek vjuranek added a comment -

          I've just released ws-clenaup 0.22 which contains small fix for expanding env. var. on nodes and may be related to this issue. Could you please try it and let me know if it fixes the issue? If not, could anyone provide a reproducer for this issue?
          Thanks

          Show
          vjuranek vjuranek added a comment - I've just released ws-clenaup 0.22 which contains small fix for expanding env. var. on nodes and may be related to this issue. Could you please try it and let me know if it fixes the issue? If not, could anyone provide a reproducer for this issue? Thanks
          Hide
          yoann_dubreuil Yoann Dubreuil (Inactive) added a comment -

          We also hit the issue in version 0.22. I don't have a case to reproduce, but referring to the following stacktrace, I think it's required to check for null before calling get(command) on EnvVars

          Aug 19, 2014 6:23:15 PM hudson.plugins.ws_cleanup.WsCleanup perform
          SEVERE: null
          java.lang.NullPointerException
          	at hudson.util.CaseInsensitiveComparator.compare(CaseInsensitiveComparator.java:40)
          	at hudson.util.CaseInsensitiveComparator.compare(CaseInsensitiveComparator.java:34)
          	at java.util.TreeMap.getEntryUsingComparator(TreeMap.java:369)
          	at java.util.TreeMap.getEntry(TreeMap.java:340)
          	at java.util.TreeMap.get(TreeMap.java:273)
          	at hudson.plugins.ws_cleanup.Cleanup.<init>(Cleanup.java:35)
          	at hudson.plugins.ws_cleanup.WsCleanup.perform(WsCleanup.java:165)
          	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          

          I will propose a PR with the null check if it works.

          Show
          yoann_dubreuil Yoann Dubreuil (Inactive) added a comment - We also hit the issue in version 0.22. I don't have a case to reproduce, but referring to the following stacktrace, I think it's required to check for null before calling get(command) on EnvVars Aug 19, 2014 6:23:15 PM hudson.plugins.ws_cleanup.WsCleanup perform SEVERE: null java.lang.NullPointerException at hudson.util.CaseInsensitiveComparator.compare(CaseInsensitiveComparator.java:40) at hudson.util.CaseInsensitiveComparator.compare(CaseInsensitiveComparator.java:34) at java.util.TreeMap.getEntryUsingComparator(TreeMap.java:369) at java.util.TreeMap.getEntry(TreeMap.java:340) at java.util.TreeMap.get(TreeMap.java:273) at hudson.plugins.ws_cleanup.Cleanup.<init>(Cleanup.java:35) at hudson.plugins.ws_cleanup.WsCleanup.perform(WsCleanup.java:165) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) I will propose a PR with the null check if it works.
          Hide
          yoann_dubreuil Yoann Dubreuil (Inactive) added a comment -

          I pushed a patch and a pull request here: https://github.com/jenkinsci/ws-cleanup-plugin/pull/18

          It fixes the NPE on our environment.

          Show
          yoann_dubreuil Yoann Dubreuil (Inactive) added a comment - I pushed a patch and a pull request here: https://github.com/jenkinsci/ws-cleanup-plugin/pull/18 It fixes the NPE on our environment.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Yoann Dubreuil
          Path:
          src/main/java/hudson/plugins/ws_cleanup/Cleanup.java
          http://jenkins-ci.org/commit/ws-cleanup-plugin/a8067fbb2ca46ccfbefc65714730d62267cfb7e9
          Log:
          [FIX JENKINS-23693]: Checks if delete command is null

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Yoann Dubreuil Path: src/main/java/hudson/plugins/ws_cleanup/Cleanup.java http://jenkins-ci.org/commit/ws-cleanup-plugin/a8067fbb2ca46ccfbefc65714730d62267cfb7e9 Log: [FIX JENKINS-23693] : Checks if delete command is null
          Hide
          vjuranek vjuranek added a comment -

          Hopefully fixed by PR #18, fixed is contained in 0.23 released right now. If the issue is still present, please reopen.

          Show
          vjuranek vjuranek added a comment - Hopefully fixed by PR #18 , fixed is contained in 0.23 released right now. If the issue is still present, please reopen.
          vjuranek vjuranek made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 156478 ] JNJira + In-Review [ 195433 ]

            People

            Assignee:
            vjuranek vjuranek
            Reporter:
            zhh Hua Zhang
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: