Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53579

Asynchronous cleanup not removing renamed workspace directories on slaves

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: ws-cleanup-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.121.3
      ws-cleanup 0.34
    • Similar Issues:

      Description

      We noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}ws-cleanup${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures.

        Attachments

          Issue Links

            Activity

            jbochenski Jakub Bochenski created issue -
            jbochenski Jakub Bochenski made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-27648 [ JENKINS-27648 ]
            jbochenski Jakub Bochenski made changes -
            Environment Jenkins 1.579
            ws-cleanup 0.24
            ws-cleanup 0.34
            jbochenski Jakub Bochenski made changes -
            Environment ws-cleanup 0.34 Jenkins ver. 2.121.3
            ws-cleanup 0.34
            jbochenski Jakub Bochenski made changes -
            Description After upgrading to ws-cleanup 0.24 in order to get the asynchronous cleanup function we noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}_ws-cleanup_${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior under ws-cleanup 0.24 is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures. We noticed the workspaces on our slaves getting renamed to the form of ${WORKSPACE}_ws-cleanup_${TIMESTAMP}. (ie, job1 would become job1_ws-cleanup_1411197183394). The expected behavior is that these were temporary to support asynchronous processing and would be deleted. However, these directories never get removed from the slave. Over time, the slave hard drives filled up resulting in build failures.
            jbochenski Jakub Bochenski made changes -
            Link This issue depends on JENKINS-24824 [ JENKINS-24824 ]
            Hide
            oleksandrp Oleksandr Perepelytsya added a comment -

            We're experiencing same issue on Jenkins - v. 2.138.2 / WS-Cleanup plugin - v. 0.34

            Show
            oleksandrp Oleksandr Perepelytsya added a comment - We're experiencing same issue on Jenkins - v. 2.138.2 / WS-Cleanup plugin - v. 0.34
            Hide
            spiderman Collin R added a comment - - edited

            Same here on Jenkins v. 2.140 / WS-Cleanup plugin v. 0.34. Seems that the ${WORKSPACE}_ws-cleanup_${TIMESTAMP} directory is somehow owned by root and therefore causes the async resource disposer to fail due to lack of permissions. Workaround is to add a cron job to chown ${WORKSPACE}_ws-cleanup_${TIMESTAMP} which allows async cleanup to occur.

            Show
            spiderman Collin R added a comment - - edited Same here on Jenkins v. 2.140 / WS-Cleanup plugin v. 0.34. Seems that the ${WORKSPACE}_ws-cleanup_${TIMESTAMP} directory is somehow owned by root and therefore causes the async resource disposer to fail due to lack of permissions. Workaround is to add a cron job to chown ${WORKSPACE}_ws-cleanup_${TIMESTAMP} which allows async cleanup to occur.
            Hide
            jbochenski Jakub Bochenski added a comment -

            On second thoughts I think this is an improvement, not a defect. Nobody promised async cleanup will handle ownership errors etc.
            It would be nice to have support for it, but any solution will be hacky and

            Show
            jbochenski Jakub Bochenski added a comment - On second thoughts I think this is an improvement, not a defect. Nobody promised async cleanup will handle ownership errors etc. It would be nice to have support for it, but any solution will be hacky and
            jbochenski Jakub Bochenski made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            Hide
            olivergondza Oliver Gondža added a comment -

            Agreed, in general, Jenkins does not promis it will delete anything not owned by the user running the agent. The correct course of action is to see, why are those directories owned by other users.

            Show
            olivergondza Oliver Gondža added a comment - Agreed, in general, Jenkins does not promis it will delete anything not owned by the user running the agent. The correct course of action is to see, why are those directories owned by other users.
            Hide
            mookayla Michaela Ervin added a comment - - edited

            Why are the ${WORKSPACE}ws-cleanup${TIMESTAMP} directories owned by someone else?  AFAIK jenkins creates them so it should delete them.

             

            Edit: I see, our scripts run some commands with sudo and thus stuff is owned by root.

            Maybe I just need to run `chown -R jenkins:jenkins` on the directory at the end of my script.

             

            What about an option to "try" to run the cleanup as root?

            Show
            mookayla Michaela Ervin added a comment - - edited Why are the ${WORKSPACE} ws-cleanup ${TIMESTAMP} directories owned by someone else?  AFAIK jenkins creates them so it should delete them.   Edit: I see, our scripts run some commands with sudo and thus stuff is owned by root. Maybe I just need to run `chown -R jenkins:jenkins` on the directory at the end of my script.   What about an option to "try" to run the cleanup as root?
            Hide
            markewaite Mark Waite added a comment -

            I hope for security purposes that the Jenkins agent process cannot become the root user as a general pattern. I don't think the plugin has any way to ask the operating system to promote the cleanup operation to root permission. Even if it did, I don't think it would be wise to make such a request, since agents that can easily become the root user can then also be used to harm the operating system that is hosting the agent

            Show
            markewaite Mark Waite added a comment - I hope for security purposes that the Jenkins agent process cannot become the root user as a general pattern. I don't think the plugin has any way to ask the operating system to promote the cleanup operation to root permission. Even if it did, I don't think it would be wise to make such a request, since agents that can easily become the root user can then also be used to harm the operating system that is hosting the agent
            Hide
            mookayla Michaela Ervin added a comment - - edited

            Understandable!

             

            I added the following to the end of my build script to test changing the perms.

             

            workspace = getcwd()
            LOGGER.info('Workspace directory is: {}'.format(workspace)) 
            system('cd "{}" && sudo chown -R jenkins:jenkins *'.format(workspace))
            

            It seems to have worked except a file still existed that I didn't realize was getting created.  Running again to see if it works.

            Show
            mookayla Michaela Ervin added a comment - - edited Understandable!   I added the following to the end of my build script to test changing the perms.   workspace = getcwd() LOGGER.info( 'Workspace directory is: {}' .format(workspace)) system( 'cd "{}" && sudo chown -R jenkins:jenkins *' .format(workspace)) It seems to have worked except a file still existed that I didn't realize was getting created.  Running again to see if it works.
            Hide
            gdartiguelongue Gilles Dartiguelongue added a comment -

            For some reason, here the problems appears on jobs that are using Docker images running root user inside. I would expect Jenkins to establish a permission mapping between its euid and container's root. What is even stranger is that not all files created in the relevant steps are surviving.

            Show
            gdartiguelongue Gilles Dartiguelongue added a comment - For some reason, here the problems appears on jobs that are using Docker images running root user inside. I would expect Jenkins to establish a permission mapping between its euid and container's root. What is even stranger is that not all files created in the relevant steps are surviving.
            Hide
            jbochenski Jakub Bochenski added a comment - - edited

            > I would expect Jenkins to establish a permission mapping between its euid and container's root.

            Can you suggest a technical solution to this. It would be useful but I see no robust and easy way to do it

            EDIT: my argument is same as https://issues.jenkins-ci.org/browse/JENKINS-53579?focusedCommentId=374472&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-374472

            Show
            jbochenski Jakub Bochenski added a comment - - edited > I would expect Jenkins to establish a permission mapping between its euid and container's root. Can you suggest a technical solution to this. It would be useful but I see no robust and easy way to do it EDIT: my argument is same as https://issues.jenkins-ci.org/browse/JENKINS-53579?focusedCommentId=374472&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-374472

              People

              Assignee:
              olivergondza Oliver Gondža
              Reporter:
              jbochenski Jakub Bochenski
              Votes:
              15 Vote for this issue
              Watchers:
              26 Start watching this issue

                Dates

                Created:
                Updated: