Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65829

Workspace Cleanup does not delete @libs directories created by libraries

    • 2.454

      In Jenkins Pipelines one can use shared libraries. With them a additional directory is created:

      Usual-Workspace: /var/lib/jenkins/workspace/job-id/
      Additional SharedLibrary-Folder: /var/lib/jenkins/workspace/job-id@tmp

      Seems as if there the shared library sources are stored in case one want s to rerun the build.

      In order to completely wipe out old build data, a option to delete this @tmp folder could be introduced.

          [JENKINS-65829] Workspace Cleanup does not delete @libs directories created by libraries

          same issue here, i have just cleanup 30Go of remaining @libs directories ...

          Cyril Pottiers added a comment - same issue here, i have just cleanup 30Go of remaining @libs directories ...

          Same here. Are there any workarounds or other plugins to use here instead? 

          Vitaliy Schreibmann added a comment - Same here. Are there any workarounds or other plugins to use here instead? 

          Similar issue is still observed that's creating unnecessary space issue.

          But a couple of differences in my observation:

          • Shared Library is being cloned to a folder with @lib suffix, NOT @tmp suffix... e.g. /var/lib/jenkins/workspace/job-id@libs
          • Even though the Jenkins controller (master) is never used for running a Build, these folders are getting created on Jenkins Controller's job workspace.

          Is someone taking a look at this issue?

          Arnab Banerjee added a comment - Similar issue is still observed that's creating unnecessary space issue. But a couple of differences in my observation: Shared Library is being cloned to a folder with @lib suffix, NOT @tmp suffix... e.g. /var/lib/jenkins/workspace/job-id@libs Even though the Jenkins controller (master) is never used for running a Build, these folders are getting created on Jenkins Controller's job workspace. Is someone taking a look at this issue?

          Jesse Glick added a comment -

          JENKINS-70869 allows you to never create the …@libs directory to begin with.

          Jesse Glick added a comment - JENKINS-70869 allows you to never create the …@libs directory to begin with.

          Arnab Banerjee added a comment - - edited

          hi jglick, thanks for sharing the info but the issue you mentioned (https://issues.jenkins.io/browse/JENKINS-70869) talks about an option is marked as resolved..
          Is this the default behavior of pipeline-groovy-lib plugin now that it doesn't create the @libs folders?
          If that's the case can you please share the Jenkins version that has the fix (We are using Jenkins 2.332.4 with "Pipeline: Groovy LibrariesVersion 629.vb_5627b_ee2104" and it still creates these on controller nodes)?
          If not, then could you please add some guidance on how to achieve this?

          Arnab Banerjee added a comment - - edited hi jglick , thanks for sharing the info but the issue you mentioned ( https://issues.jenkins.io/browse/JENKINS-70869 ) talks about an option is marked as resolved.. Is this the default behavior of pipeline-groovy-lib plugin now that it doesn't create the @libs folders? If that's the case can you please share the Jenkins version that has the fix (We are using Jenkins 2.332.4 with "Pipeline: Groovy LibrariesVersion 629.vb_5627b_ee2104" and it still creates these on controller nodes)? If not, then could you please add some guidance on how to achieve this?

          Okay.. I see the new version released 3 days back has some relevant information..
          https://plugins.jenkins.io/pipeline-groovy-lib/releases/

          Adding this to the other ticket as well for easy access

          Arnab Banerjee added a comment - Okay.. I see the new version released 3 days back has some relevant information.. https://plugins.jenkins.io/pipeline-groovy-lib/releases/ Adding this to the other ticket as well for easy access

          Allan BURDAJEWICZ added a comment - - edited

          Actually the WorkspaceClleanerThread is supposed to cleanup the suffixed directories since the fix for https://issues.jenkins.io/browse/JENKINS-41805 but this is not happening for @libs suffixed workspaces on the controller for builds that are executed on non-controller executors. cc timja

          In those cases, there is only a @libs suffixed workspace but not the original and we don'tpass the condition https://github.com/jenkinsci/jenkins/blob/jenkins-2.447/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L107.

          In such case, the WorkspaceLocator finds out that there is a workspace on the controller FS, but it looks for existence of $WORKSPACE and not $WORKSPACE@libs.

          Here is an example of a job built by test agent test-agent that use a shared library. The library is cloned under /var/jenkins_home/workspace/pipelines/pipelineFromScriptWithLib@libs but there is no /var/jenkins_home/workspace/pipelines/pipelineFromScriptWithLib:

          Mar. 05, 2024 1:12:03 PM FINE jenkins.branch.WorkspaceLocatorImpl locate
          ignoring non-branch project org.jenkinsci.plugins.workflow.job.WorkflowJob@1744e0b[pipelines/pipelineFromScriptWithLib] on hudson.model.Hudson@5787278d
          Mar. 05, 2024 1:12:03 PM FINE hudson.model.WorkspaceCleanupThread shouldBeDeleted
          Directory /var/jenkins_home/workspace/pipelines/pipelineFromScriptWithLib does not exist
          

          Here is another example with a branch jobs, the library is cloned under /var/jenkins_home/workspace/_catalog-poc-pipelines_withLib_2@libs but there is no /var/jenkins_home/workspace/_catalog-poc-pipelines_withLib_2:

          Mar. 05, 2024 1:12:03 PM FINER jenkins.branch.WorkspaceLocatorImpl locate
          index already lists /var/jenkins_home/workspace/_catalog-poc-pipelines_withLib_2 for org.jenkinsci.plugins.workflow.job.WorkflowJob@21ed6f70[organizations/organizationGithubScm/ci-catalog-poc-pipelines/withLib] on hudson.model.Hudson@5787278d
          Mar. 05, 2024 1:12:03 PM FINE hudson.model.WorkspaceCleanupThread shouldBeDeleted
          Directory /var/jenkins_home/workspace/_catalog-poc-pipelines_withLib_2 does not exist
          

          To reproduce:

          • start Jenkins with -Dhudson.model.WorkspaceCleanupThread.retainForDays=0
          • Set up a remote agent
          • Create a pipeline that clones a library:
          @Library("testLibs") _
          node ('test') {
              echo "test"
          }
          
          • Build the job
          • Run hudson.model.AsyncPeriodicWork.all().get(WorkspaceCleanupThread.class).run() in the script console to execute the thread run

          Allan BURDAJEWICZ added a comment - - edited Actually the WorkspaceClleanerThread is supposed to cleanup the suffixed directories since the fix for https://issues.jenkins.io/browse/JENKINS-41805 but this is not happening for @libs suffixed workspaces on the controller for builds that are executed on non-controller executors. cc timja In those cases, there is only a @libs suffixed workspace but not the original and we don'tpass the condition https://github.com/jenkinsci/jenkins/blob/jenkins-2.447/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L107 . In such case, the WorkspaceLocator finds out that there is a workspace on the controller FS, but it looks for existence of $WORKSPACE and not $WORKSPACE@libs . Here is an example of a job built by test agent test-agent that use a shared library. The library is cloned under /var/jenkins_home/workspace/pipelines/pipelineFromScriptWithLib@libs but there is no /var/jenkins_home/workspace/pipelines/pipelineFromScriptWithLib : Mar. 05, 2024 1:12:03 PM FINE jenkins.branch.WorkspaceLocatorImpl locate ignoring non-branch project org.jenkinsci.plugins.workflow.job.WorkflowJob@1744e0b[pipelines/pipelineFromScriptWithLib] on hudson.model.Hudson@5787278d Mar. 05, 2024 1:12:03 PM FINE hudson.model.WorkspaceCleanupThread shouldBeDeleted Directory / var /jenkins_home/workspace/pipelines/pipelineFromScriptWithLib does not exist Here is another example with a branch jobs, the library is cloned under /var/jenkins_home/workspace/_catalog-poc-pipelines_withLib_2@libs but there is no /var/jenkins_home/workspace/_catalog-poc-pipelines_withLib_2 : Mar. 05, 2024 1:12:03 PM FINER jenkins.branch.WorkspaceLocatorImpl locate index already lists / var /jenkins_home/workspace/_catalog-poc-pipelines_withLib_2 for org.jenkinsci.plugins.workflow.job.WorkflowJob@21ed6f70[organizations/organizationGithubScm/ci-catalog-poc-pipelines/withLib] on hudson.model.Hudson@5787278d Mar. 05, 2024 1:12:03 PM FINE hudson.model.WorkspaceCleanupThread shouldBeDeleted Directory / var /jenkins_home/workspace/_catalog-poc-pipelines_withLib_2 does not exist To reproduce: start Jenkins with -Dhudson.model.WorkspaceCleanupThread.retainForDays=0 Set up a remote agent Create a pipeline that clones a library: @Library( "testLibs" ) _ node ( 'test' ) { echo "test" } Build the job Run hudson.model.AsyncPeriodicWork.all().get(WorkspaceCleanupThread.class).run() in the script console to execute the thread run

            allan_burdajewicz Allan BURDAJEWICZ
            xbw1220 Bowei
            Votes:
            7 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: