Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24440

Jenkins cannot delete root-owned files/folders in jenkins-owned directories

    XMLWordPrintable

Details

    • Improvement
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • core

    Description

      We want to be able to wipe workspaces containing files created via Docker containers (and thus owned by root). In a shell, as jenkins, we can delete root-owned files/folders as long as they are within jenkins-owned directories. But this function ( https://github.com/jenkinsci/jenkins/blob/6c2fffb/core/src/main/java/hudson/Util.java#L299 ) fails in that case.

      Perhaps, it's due to a windowsOS java limitation which cannot delete a non-empty directory?

      Could it be possible to create a special implementation for unix, that allows deletion of non-empty directories and thus supports the use-case mentioned above?

      Attachments

        Issue Links

          Activity

            pdupont Philippe Dupont created issue -
            pdupont Philippe Dupont made changes -
            Field Original Value New Value
            Summary Jenkins cannot delete files owned by root in folders he own Jenkins cannot delete root-owned files/folders in jenkins-owned directories
            danielbeck Daniel Beck made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            akropp anton kropp added a comment -

            +1

            akropp anton kropp added a comment - +1
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 157421 ] JNJira + In-Review [ 179575 ]
            metametadata Yuri Govorushchenko added a comment - - edited

            Is there any workaround? My shell scripts wrapped by insideImage create several root-owned folders (e.g. node-modules/) in the workspace and jenkins user is not able to clean them on the next build via git clean.

            metametadata Yuri Govorushchenko added a comment - - edited Is there any workaround? My shell scripts wrapped by insideImage create several root -owned folders (e.g. node-modules/ ) in the workspace and jenkins user is not able to clean them on the next build via git clean .
            markewaite Mark Waite added a comment -

            One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root.

            Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root.

            markewaite Mark Waite added a comment - One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root. Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root.
            metametadata Yuri Govorushchenko added a comment - - edited

            Thank you for the suggestions!

            One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root.

            Unfortunately I have to run docker container with `-u root` explicitly because I had some problems due to JENKINS-38438.

            Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root.

            But I suppose the script will be still run by jenkins user so it cannot act as root?

            I ended up adding jenkins user to sudoers and calling my command with sudo explicitly: sudo git clean. For future reference these are the steps to make jenkins a sudoer:

            • sudo -u root bash (login as root)
            • EDITOR=nano visudo (edit sudoers using nano editor instead of default vi)
            • Add these lines and exit (Ctrl+x, y):
            ## CUSTOM: allow jenkins user to sudo without password
            jenkins	ALL=(ALL) 	NOPASSWD: ALL
            
            metametadata Yuri Govorushchenko added a comment - - edited Thank you for the suggestions! One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root. Unfortunately I have to run docker container with `-u root` explicitly because I had some problems due to JENKINS-38438 . Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root. But I suppose the script will be still run by jenkins user so it cannot act as root ? I ended up adding jenkins user to sudoers and calling my command with sudo explicitly: sudo git clean . For future reference these are the steps to make jenkins a sudoer: sudo -u root bash (login as root) EDITOR=nano visudo (edit sudoers using nano editor instead of default vi) Add these lines and exit (Ctrl+x, y): ## CUSTOM: allow jenkins user to sudo without password jenkins ALL=(ALL) NOPASSWD: ALL

            Another workaround (that doesn't require making jenkins a sudoer) is to clean the workspace inside the image:

            // Given arbitrary string returns a strongly escaped shell string literal.
            // I.e. it will be in single quotes which turns off interpolation of $(...), etc.
            // E.g.: 1'2\3\'4 5"6 (groovy string) -> '1'\''2\3\'\''4 5"6' (groovy string which can be safely pasted into shell command).
            // Pretty much a HACK: track https://issues.jenkins-ci.org/browse/JENKINS-44231.
            def shellString(s) {
                // Replace ' with '\'' (https://unix.stackexchange.com/a/187654/260156). Then enclose with '...'.
                // 1) Why not replace \ with \\? Because '...' does not treat backslashes in a special way.
                // 2) And why not use ANSI-C quoting? I.e. we could replace ' with \'
                // and enclose using $'...' (https://stackoverflow.com/a/8254156/4839573).
                // Because ANSI-C quoting is not yet supported by Dash (default shell in Ubuntu & Debian) (https://unix.stackexchange.com/a/371873).
                '\'' + s.replace('\'', '\'\\\'\'') + '\''
            }
            
            // ...
            
            node {
                stage('Checkout') {
                    checkout(scm)
                }
            
                def pathsToClean = sh(returnStdout: true,
                                      script: 'git clean -fdx --dry-run --exclude node_modules | sed "s/Would remove //"').split('\n')
            
                // ...
            
                myImage.inside("...") {
                	stage('Git Clean') {
                        // We don't call `git clean ...` here directly
                        // because it requires injecting SCM credentials into the image which is cumbersome.
                        pathsToClean.each {
                            sh('rm -rf ' + shellString(it))
                        }
                    }
            
                    // ...
                }
            }
            
            metametadata Yuri Govorushchenko added a comment - Another workaround (that doesn't require making jenkins a sudoer) is to clean the workspace inside the image: // Given arbitrary string returns a strongly escaped shell string literal. // I.e. it will be in single quotes which turns off interpolation of $(...), etc. // E.g.: 1 '2\3\' 4 5 "6 (groovy string) -> '1' \ ''2\3\' \ ''4 5" 6' (groovy string which can be safely pasted into shell command). // Pretty much a HACK: track https://issues.jenkins-ci.org/browse/JENKINS-44231. def shellString(s) { // Replace ' with ' \ '' (https://unix.stackexchange.com/a/187654/260156). Then enclose with ' ...'. // 1) Why not replace \ with \\? Because '...' does not treat backslashes in a special way. // 2) And why not use ANSI-C quoting? I.e. we could replace ' with \' // and enclose using $ '...' (https://stackoverflow.com/a/8254156/4839573). // Because ANSI-C quoting is not yet supported by Dash ( default shell in Ubuntu & Debian) (https://unix.stackexchange.com/a/371873). '\' ' + s.replace(' \ '', ' \ '\\\' \ '') + ' \'' } // ... node { stage( 'Checkout' ) { checkout(scm) } def pathsToClean = sh(returnStdout: true , script: 'git clean -fdx --dry-run --exclude node_modules | sed "s/Would remove //" ' ).split( '\n' ) // ... myImage.inside( "..." ) { stage( 'Git Clean' ) { // We don't call `git clean ...` here directly // because it requires injecting SCM credentials into the image which is cumbersome. pathsToClean.each { sh( 'rm -rf ' + shellString(it)) } } // ... } }
            patanouk pata nouk added a comment - - edited

            Same problem here.

            A quick and dirty fix I found was to do this:

             

            pipeline {
                ...
                post {
                    always {
                        sh "chmod -R 777 ."
                        cleanWs()
                    }
                }
            }
            

             

            It allows anyone to modify workspace files so that Jenkins can clean it.

            patanouk pata nouk added a comment - - edited Same problem here. A quick and dirty fix I found was to do this:   pipeline {     ...     post {        always {         sh "chmod -R 777 ."            cleanWs() } } }   It allows anyone to modify workspace files so that Jenkins can clean it.
            starrett67 Josh Starrett added a comment -

            +1

            starrett67 Josh Starrett added a comment - +1
            kivagant Eugene G added a comment - - edited

            This helps in my case:

            stages {
            // ... image.inside('-u root) { some code creates files and folders}
            
            post {
                cleanup {
                    script {
                        image.inside('-u root') {
                          sh 'find . -user root -name \'*\' | xargs chmod ugo+rw'
                        }
                    }
                    deleteDir()
                }
            } 
            kivagant Eugene G added a comment - - edited This helps in my case: stages { // ... image.inside('-u root) { some code creates files and folders} post { cleanup { script { image.inside( '-u root' ) { sh 'find . -user root -name \' *\ ' | xargs chmod ugo+rw' } } deleteDir() } }
            xianpeng Peter Shen added a comment -

            +1

            xianpeng Peter Shen added a comment - +1
            jdupont Jules Dupont made changes -
            Comment [ +1 – I'm not sure if our setup is atypical, but we're running the jenkins user as non-root (to avoid potential security issues and because we're starting agents from the master node via SSH) and we're running docker as root to avoid to a bunch of problems that happen otherwise. As a result, this bug leads to the accumulation of hundreds of GBs of files on all of our Linux agents. This is definitely not minor for us. ]
            artalus Artalus S. added a comment - - edited

            A partial workaround for the generic "Jenkins cannot delete root files" - setup Linux ACL permissions for the  workspace directory:

            $ rm -rf workspace # to avoid any inconsistencies in existing files
            $ mkdir workspace
            $ sudo setfacl -dm 'u:<your_jenkins_user_name>:rwx' workspace

            See internet (e.g. arch wiki) for explanations on ACL, but this basically will allow jenkins access to all NEWLY CREATED files. Note the emphasis on newly created. If, for example, you `sudo tar -xf archive_with_files_owned_by_root.tar.xz`, these files will not be assigned with this ACL expansion =/

             

            artalus Artalus S. added a comment - - edited A partial workaround for the generic "Jenkins cannot delete root files" - setup Linux ACL permissions for the  workspace directory: $ rm -rf workspace # to avoid any inconsistencies in existing files $ mkdir workspace $ sudo setfacl -dm 'u:<your_jenkins_user_name>:rwx' workspace See internet (e.g. arch wiki ) for explanations on ACL, but this basically will allow jenkins access to all NEWLY CREATED files. Note the emphasis on newly created. If, for example, you `sudo tar -xf archive_with_files_owned_by_root.tar.xz`, these files will not be assigned with this ACL expansion =/  

            setfacl didn't work for me. And post chmod/chown is too flaky. If the agent crashes for some reason, that workspace will be left undeletable without manual intervention. 

            instead i tried setting `umask` to the container by overriding the docker run entrypoint and it worked. This basically sets the container's default permission mask so any files written by the docker container to the jenkins workspaces mounted dir will be accessible by Jenkins or any other. 

            For example this is how it worked for our Cypress execution. Once `umask` is set, it continues with the originally intended entrypoint command

            docker run -v $PWD:/e2e -w /e2e -e CYPRESS_baseUrl=http://127.0.0.1:8888 --ipc host --network host --entrypoint /bin/bash cypress/included:4.12.1 -c \"umask 0000; cypress run --browser chrome --headless\""
            

            Moreover, you can also experiment with user mapping

            cjayawickrema Chandima Jayawickrema added a comment - setfacl didn't work for me. And post chmod/chown is too flaky. If the agent crashes for some reason, that workspace will be left undeletable without manual intervention.  instead i tried setting `umask` to the container by overriding the docker run entrypoint and it worked. This basically sets the container's default permission mask so any files written by the docker container to the jenkins workspaces mounted dir will be accessible by Jenkins or any other.  For example this is how it worked for our Cypress execution. Once `umask` is set, it continues with the originally intended entrypoint command docker run -v $PWD:/e2e -w /e2e -e CYPRESS_baseUrl=http: //127.0.0.1:8888 --ipc host --network host --entrypoint /bin/bash cypress/included:4.12.1 -c \ "umask 0000; cypress run --browser chrome --headless\" " Moreover, you can also experiment with user mapping
            nickbrown Nicholas Brown made changes -
            Link This issue relates to JENKINS-64874 [ JENKINS-64874 ]

            People

              Unassigned Unassigned
              pdupont Philippe Dupont
              Votes:
              11 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated: