Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24440

Jenkins cannot delete root-owned files/folders in jenkins-owned directories

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core

      We want to be able to wipe workspaces containing files created via Docker containers (and thus owned by root). In a shell, as jenkins, we can delete root-owned files/folders as long as they are within jenkins-owned directories. But this function ( https://github.com/jenkinsci/jenkins/blob/6c2fffb/core/src/main/java/hudson/Util.java#L299 ) fails in that case.

      Perhaps, it's due to a windowsOS java limitation which cannot delete a non-empty directory?

      Could it be possible to create a special implementation for unix, that allows deletion of non-empty directories and thus supports the use-case mentioned above?

          [JENKINS-24440] Jenkins cannot delete root-owned files/folders in jenkins-owned directories

          Yuri Govorushchenko added a comment - - edited

          Is there any workaround? My shell scripts wrapped by insideImage create several root-owned folders (e.g. node-modules/) in the workspace and jenkins user is not able to clean them on the next build via git clean.

          Yuri Govorushchenko added a comment - - edited Is there any workaround? My shell scripts wrapped by insideImage create several root -owned folders (e.g. node-modules/ ) in the workspace and jenkins user is not able to clean them on the next build via git clean .

          Mark Waite added a comment -

          One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root.

          Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root.

          Mark Waite added a comment - One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root. Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root.

          Yuri Govorushchenko added a comment - - edited

          Thank you for the suggestions!

          One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root.

          Unfortunately I have to run docker container with `-u root` explicitly because I had some problems due to JENKINS-38438.

          Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root.

          But I suppose the script will be still run by jenkins user so it cannot act as root?

          I ended up adding jenkins user to sudoers and calling my command with sudo explicitly: sudo git clean. For future reference these are the steps to make jenkins a sudoer:

          • sudo -u root bash (login as root)
          • EDITOR=nano visudo (edit sudoers using nano editor instead of default vi)
          • Add these lines and exit (Ctrl+x, y):
          ## CUSTOM: allow jenkins user to sudo without password
          jenkins	ALL=(ALL) 	NOPASSWD: ALL
          

          Yuri Govorushchenko added a comment - - edited Thank you for the suggestions! One alternative might be to configure docker so that it runs as the same user that is running the agent. It should then create the files as that user, rather than creating them as root. Unfortunately I have to run docker container with `-u root` explicitly because I had some problems due to JENKINS-38438 . Otherwise, you could create some form of separate program (script, compiled executable, etc.) which is called before the SCM checkout and cleans the directory as root. But I suppose the script will be still run by jenkins user so it cannot act as root ? I ended up adding jenkins user to sudoers and calling my command with sudo explicitly: sudo git clean . For future reference these are the steps to make jenkins a sudoer: sudo -u root bash (login as root) EDITOR=nano visudo (edit sudoers using nano editor instead of default vi) Add these lines and exit (Ctrl+x, y): ## CUSTOM: allow jenkins user to sudo without password jenkins ALL=(ALL) NOPASSWD: ALL

          Another workaround (that doesn't require making jenkins a sudoer) is to clean the workspace inside the image:

          // Given arbitrary string returns a strongly escaped shell string literal.
          // I.e. it will be in single quotes which turns off interpolation of $(...), etc.
          // E.g.: 1'2\3\'4 5"6 (groovy string) -> '1'\''2\3\'\''4 5"6' (groovy string which can be safely pasted into shell command).
          // Pretty much a HACK: track https://issues.jenkins-ci.org/browse/JENKINS-44231.
          def shellString(s) {
              // Replace ' with '\'' (https://unix.stackexchange.com/a/187654/260156). Then enclose with '...'.
              // 1) Why not replace \ with \\? Because '...' does not treat backslashes in a special way.
              // 2) And why not use ANSI-C quoting? I.e. we could replace ' with \'
              // and enclose using $'...' (https://stackoverflow.com/a/8254156/4839573).
              // Because ANSI-C quoting is not yet supported by Dash (default shell in Ubuntu & Debian) (https://unix.stackexchange.com/a/371873).
              '\'' + s.replace('\'', '\'\\\'\'') + '\''
          }
          
          // ...
          
          node {
              stage('Checkout') {
                  checkout(scm)
              }
          
              def pathsToClean = sh(returnStdout: true,
                                    script: 'git clean -fdx --dry-run --exclude node_modules | sed "s/Would remove //"').split('\n')
          
              // ...
          
              myImage.inside("...") {
              	stage('Git Clean') {
                      // We don't call `git clean ...` here directly
                      // because it requires injecting SCM credentials into the image which is cumbersome.
                      pathsToClean.each {
                          sh('rm -rf ' + shellString(it))
                      }
                  }
          
                  // ...
              }
          }
          

          Yuri Govorushchenko added a comment - Another workaround (that doesn't require making jenkins a sudoer) is to clean the workspace inside the image: // Given arbitrary string returns a strongly escaped shell string literal. // I.e. it will be in single quotes which turns off interpolation of $(...), etc. // E.g.: 1 '2\3\' 4 5 "6 (groovy string) -> '1' \ ''2\3\' \ ''4 5" 6' (groovy string which can be safely pasted into shell command). // Pretty much a HACK: track https://issues.jenkins-ci.org/browse/JENKINS-44231. def shellString(s) { // Replace ' with ' \ '' (https://unix.stackexchange.com/a/187654/260156). Then enclose with ' ...'. // 1) Why not replace \ with \\? Because '...' does not treat backslashes in a special way. // 2) And why not use ANSI-C quoting? I.e. we could replace ' with \' // and enclose using $ '...' (https://stackoverflow.com/a/8254156/4839573). // Because ANSI-C quoting is not yet supported by Dash ( default shell in Ubuntu & Debian) (https://unix.stackexchange.com/a/371873). '\' ' + s.replace(' \ '', ' \ '\\\' \ '') + ' \'' } // ... node { stage( 'Checkout' ) { checkout(scm) } def pathsToClean = sh(returnStdout: true , script: 'git clean -fdx --dry-run --exclude node_modules | sed "s/Would remove //" ' ).split( '\n' ) // ... myImage.inside( "..." ) { stage( 'Git Clean' ) { // We don't call `git clean ...` here directly // because it requires injecting SCM credentials into the image which is cumbersome. pathsToClean.each { sh( 'rm -rf ' + shellString(it)) } } // ... } }

          pata nouk added a comment - - edited

          Same problem here.

          A quick and dirty fix I found was to do this:

           

          pipeline {
              ...
              post {
                  always {
                      sh "chmod -R 777 ."
                      cleanWs()
                  }
              }
          }
          

           

          It allows anyone to modify workspace files so that Jenkins can clean it.

          pata nouk added a comment - - edited Same problem here. A quick and dirty fix I found was to do this:   pipeline {     ...     post {        always {         sh "chmod -R 777 ."            cleanWs() } } }   It allows anyone to modify workspace files so that Jenkins can clean it.

          Josh Starrett added a comment -

          +1

          Josh Starrett added a comment - +1

          Eugene G added a comment - - edited

          This helps in my case:

          stages {
          // ... image.inside('-u root) { some code creates files and folders}
          
          post {
              cleanup {
                  script {
                      image.inside('-u root') {
                        sh 'find . -user root -name \'*\' | xargs chmod ugo+rw'
                      }
                  }
                  deleteDir()
              }
          } 

          Eugene G added a comment - - edited This helps in my case: stages { // ... image.inside('-u root) { some code creates files and folders} post { cleanup { script { image.inside( '-u root' ) { sh 'find . -user root -name \' *\ ' | xargs chmod ugo+rw' } } deleteDir() } }

          Peter Shen added a comment -

          +1

          Peter Shen added a comment - +1

          Artalus S. added a comment - - edited

          A partial workaround for the generic "Jenkins cannot delete root files" - setup Linux ACL permissions for the  workspace directory:

          $ rm -rf workspace # to avoid any inconsistencies in existing files
          $ mkdir workspace
          $ sudo setfacl -dm 'u:<your_jenkins_user_name>:rwx' workspace

          See internet (e.g. arch wiki) for explanations on ACL, but this basically will allow jenkins access to all NEWLY CREATED files. Note the emphasis on newly created. If, for example, you `sudo tar -xf archive_with_files_owned_by_root.tar.xz`, these files will not be assigned with this ACL expansion =/

           

          Artalus S. added a comment - - edited A partial workaround for the generic "Jenkins cannot delete root files" - setup Linux ACL permissions for the  workspace directory: $ rm -rf workspace # to avoid any inconsistencies in existing files $ mkdir workspace $ sudo setfacl -dm 'u:<your_jenkins_user_name>:rwx' workspace See internet (e.g. arch wiki ) for explanations on ACL, but this basically will allow jenkins access to all NEWLY CREATED files. Note the emphasis on newly created. If, for example, you `sudo tar -xf archive_with_files_owned_by_root.tar.xz`, these files will not be assigned with this ACL expansion =/  

          setfacl didn't work for me. And post chmod/chown is too flaky. If the agent crashes for some reason, that workspace will be left undeletable without manual intervention. 

          instead i tried setting `umask` to the container by overriding the docker run entrypoint and it worked. This basically sets the container's default permission mask so any files written by the docker container to the jenkins workspaces mounted dir will be accessible by Jenkins or any other. 

          For example this is how it worked for our Cypress execution. Once `umask` is set, it continues with the originally intended entrypoint command

          docker run -v $PWD:/e2e -w /e2e -e CYPRESS_baseUrl=http://127.0.0.1:8888 --ipc host --network host --entrypoint /bin/bash cypress/included:4.12.1 -c \"umask 0000; cypress run --browser chrome --headless\""
          

          Moreover, you can also experiment with user mapping

          Chandima Jayawickrema added a comment - setfacl didn't work for me. And post chmod/chown is too flaky. If the agent crashes for some reason, that workspace will be left undeletable without manual intervention.  instead i tried setting `umask` to the container by overriding the docker run entrypoint and it worked. This basically sets the container's default permission mask so any files written by the docker container to the jenkins workspaces mounted dir will be accessible by Jenkins or any other.  For example this is how it worked for our Cypress execution. Once `umask` is set, it continues with the originally intended entrypoint command docker run -v $PWD:/e2e -w /e2e -e CYPRESS_baseUrl=http: //127.0.0.1:8888 --ipc host --network host --entrypoint /bin/bash cypress/included:4.12.1 -c \ "umask 0000; cypress run --browser chrome --headless\" " Moreover, you can also experiment with user mapping

            Unassigned Unassigned
            pdupont Philippe Dupont
            Votes:
            11 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated: