Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50471

Linux /tmp directory mounted noexec breaks git clone

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • None
    • Jenkins ver. 2.107.1
      git-plugin 2.7.1
      kubernetes-plugin 1.4

      When I try to build something I always got this error:

       > git fetch --no-tags --progress git@bitbucket.org:xxxxxx/xxxxx.git +refs/heads/*:refs/remotes/origin/*
      ERROR: Error cloning remote repo 'origin'
      hudson.plugins.git.GitException: Command "git fetch --no-tags --progress git@bitbucket.org:xxxxxx/xxxxx.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: fatal: cannot exec '/home/jenkins/workspace/service-skeleton_master-7JMZPV5GKPYJ7FXGSQK6YYEOA6VIRJOEPUFA5ATSGLP5FPZSOMBA@tmp/ssh6122672630332030715.sh': Permission denied
      fatal: unable to fork
      
      

      It seems to look like what's described in Git plugin wiki: https://plugins.jenkins.io/git-client

       

      I just connected to my slave to check folder rights and everything belongs to the "jenkins" user... so I don't understand what is the problem.

       

      May it be a problem from Kubernetes?

       

      I'm looking for a solution for 2 days... But still blocked with this issue. Could you help me please? Thanks.

       

      NB: I've already ask help on IRC but no concluent solution

      PS: Additionnal info in comments

          [JENKINS-50471] Linux /tmp directory mounted noexec breaks git clone

          Thomas Ramé added a comment - - edited

          When executing the command "mount" I can see this line:

          /dev/sda1 on /home/jenkins type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30,data=ordered)

           

          It could be the origin of the issue because of the "noexec". How can I fix this simply since I'm not managing slave creation (due to Kubernetes plugin) please?

           

          And even if I could fix that by overriding this mount or whatever... it seems strange this is not working from basic configuration of Kubernetes plugin, no?

          Thomas Ramé added a comment - - edited When executing the command "mount" I can see this line: /dev/sda1 on /home/jenkins type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30,data=ordered)   It could be the origin of the issue because of the "noexec". How can I fix this simply since I'm not managing slave creation (due to Kubernetes plugin) please?   And even if I could fix that by overriding this mount or whatever... it seems strange this is not working from basic configuration of Kubernetes plugin, no?

          Mark Waite added a comment - - edited

          The kubernetes plugin does not control the mount options of the temp folder. I think you'll need to work with your administrators to change the cluster configuration.

          Another alternative is to use the JGit provider from the git plugin instead of command line git.

          See the CloudBees article for more details.

          Mark Waite added a comment - - edited The kubernetes plugin does not control the mount options of the temp folder. I think you'll need to work with your administrators to change the cluster configuration. Another alternative is to use the JGit provider from the git plugin instead of command line git. See the CloudBees article for more details.

          Thomas Ramé added a comment - - edited

          markewaite I saw this article few hours ago but they are some problems:

          • They are already with a 755 chmod. It doesn't work due to the mount having the parameter "noexec"
          • And even if I remount partition be myself, my jenkins-slave are temporary due kubernetes plugin so I'm looking for a automatic way to do that. I guess I need to modify a setting in pod deployment YAML but I don't see at this moment where should I do that

           

          If you have some ideas/indications, don't hesitate to give them

           

          Thanks

          Thomas Ramé added a comment - - edited markewaite  I saw this article few hours ago but they are some problems: They are already with a 755 chmod. It doesn't work due to the mount having the parameter "noexec" And even if I remount partition be myself, my jenkins-slave are temporary due kubernetes plugin so I'm looking for a automatic way to do that. I guess I need to modify a setting in pod deployment YAML but I don't see at this moment where should I do that   If you have some ideas/indications, don't hesitate to give them   Thanks

          Mark Waite added a comment -

          sneko use JGit. The Jenkins git plugin includes support for command line git (default) and for JGit (needs to be enabled in the Jenkins global configuration by adding JGit as an option). Once you've enabled JGit globally in Jenkins, a pick list will appear in the job definition to allow you to choose the git implementation.

          Mark Waite added a comment - sneko use JGit. The Jenkins git plugin includes support for command line git (default) and for JGit (needs to be enabled in the Jenkins global configuration by adding JGit as an option). Once you've enabled JGit globally in Jenkins, a pick list will appear in the job definition to allow you to choose the git implementation.

          Thomas Ramé added a comment -

          I'm gonna try tomorrow, it's 4am here. I will keep you updated and will mark as resolved if I succeed

           

          Thanks for your help

          Thomas Ramé added a comment - I'm gonna try tomorrow, it's 4am here. I will keep you updated and will mark as resolved if I succeed   Thanks for your help

          Thomas Ramé added a comment - - edited

          Hahaha... that's was so simple to solve this issue by using JGit... Big thank you markewaite

           

          By the way I will try to figure it out with docker image maintainer since /home/jenkins is the only volume I provide from Kubernetes and that by executing "mount" I can get this list:

          /dev/sda1 on /home/jenkins type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30,data=ordered)
          /dev/sda1 on /home/jenkins/agent type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
          /dev/sda1 on /home/jenkins/.jenkins type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
          

          So I just looked into the slave Dockerfile (https://hub.docker.com/r/jenkins/slave/~/dockerfile/) to see why some subfolders don't have noexec options... It appears that they are created during the docker image build. I'm not totally confident that I'm understanding the whole thing but we could imagine that by creating /home/jenkins/workspace during image build we would avoid the "noexec" option.

          After that I don't have enough knowledge about Jenkins processes to know if it's secure and if it makes sense...

           

          Thanks again you were very helpful 

          Thomas Ramé added a comment - - edited Hahaha... that's was so simple to solve this issue by using JGit... Big thank you markewaite   By the way I will try to figure it out with docker image maintainer since /home/jenkins is the only volume I provide from Kubernetes and that by executing "mount" I can get this list: /dev/sda1 on /home/jenkins type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30,data=ordered) /dev/sda1 on /home/jenkins/agent type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered) /dev/sda1 on /home/jenkins/.jenkins type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered) So I just looked into the slave Dockerfile ( https://hub.docker.com/r/jenkins/slave/~/dockerfile/ ) to see why some subfolders don't have noexec options... It appears that they are created during the docker image build. I'm not totally confident that I'm understanding the whole thing but we could imagine that by creating /home/jenkins/workspace during image build we would avoid the "noexec" option. After that I don't have enough knowledge about Jenkins processes to know if it's secure and if it makes sense...   Thanks again you were very helpful 

          Thomas Ramé added a comment -

          See in comments one of the solution provided by markewaite

          Thomas Ramé added a comment - See in comments one of the solution provided by  markewaite

          Thomas Ramé added a comment -

          I just want to update the solution found since it was not scoped to Git, but also to all scripted parts written in my Jenkinsfiles.

          By following these instructions:
          https://cloud.google.com/solutions/configuring-jenkins-kubernetes-engine
           
          You will use volumes from host, that apply for using docker and to locate the workspace in the /tmp of the host.
           
          By doing so, it solves all the problems related to the "noexec" mount option on slave partitions.
           
          I hope it will help!

          Thomas Ramé added a comment - I just want to update the solution found since it was not scoped to Git, but also to all scripted parts written in my Jenkinsfiles. By following these instructions: https://cloud.google.com/solutions/configuring-jenkins-kubernetes-engine   You will use volumes from host, that apply for using docker and to locate the workspace in the /tmp of the host.   By doing so, it solves all the problems related to the "noexec" mount option on slave partitions.   I hope it will help!

            Unassigned Unassigned
            sneko Thomas Ramé
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: