Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46230

git-lfs not working in kubernetes container

    XMLWordPrintable

Details

    Description

      Trying to make use of the git-lfs addition to the checkout step, but it's failing.

      The error I'm getting is:

      Fetching without tags
      Fetching upstream changes from https://github.com/---
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress https://github.com/--- +refs/heads/master:refs/remotes/origin/master --depth=2
      Checking out Revision 33d759e (master)
      Commit message: "---"
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 33d759e
      
      hudson.plugins.git.GitException: Command "git checkout -f 33d759e" returned status code 128:
      stdout: 
      stderr: git-lfs filter-process: line 1: git-lfs: not found
      fatal: The remote end hung up unexpectedly
      

      The example pipeline code that is failing: 

      podTemplate(label: "test", containers: [
          containerTemplate(name: "golang", image: "golang:1.8", ttyEnabled: true, command: "cat"
          )
      ]) {
      	node("test") {
      		container(name: "golang") {
      			stage("checkout") {
      				// install git lfs on debian based pod
      				sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
      				sh "apt-get install git-lfs"
      				sh "git lfs install"
      				checkout([
      					$class: "GitSCM",
      					branches: scm.branches,
      					extensions: scm.extensions + [
      						[$class: "CloneOption", depth: 2, shallow: true],
      						[$class: "GitLFSPull"]
      					],
      					userRemoteConfigs: scm.userRemoteConfigs
      				])
      				sh 'ls -l'
      			}
      		}
      	}
      }
       

      If I log into the pod after git-lfs is installed and run `git clone` directly, it works and pulls the large file. My guess is that this problem is related to this bug/workaround, but I don't know how to implement this in the k8s/jenkins env
      https://github.com/git-lfs/git-lfs/issues/1821#issuecomment-272889540

      Attachments

        Issue Links

          Activity

            markewaite Mark Waite added a comment -

            Closing as a duplicate of JENKINS-30600.

            markewaite Mark Waite added a comment - Closing as a duplicate of JENKINS-30600 .
            jknurek J Knurek added a comment -

            yes, I think you're right.

             

            I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by: https://github.com/jenkinsci/docker-slave/pull/39

            jknurek J Knurek added a comment - yes, I think you're right.   I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by:  https://github.com/jenkinsci/docker-slave/pull/39

            my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container

            csanchez Carlos Sanchez added a comment - my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container
            jknurek J Knurek added a comment -

            I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code:

            container(name: "golang") {
            	stage("checkout") {
            		sh "ls -al"
            		sh "whoami"
            		checkout([
            			$class: "GitSCM",
            			branches: scm.branches,
            			extensions: scm.extensions + [
            				[$class: "CloneOption", depth: 2, shallow: true]
            			],
            			userRemoteConfigs: scm.userRemoteConfigs
            		])
            		sh "ls -al"
            	}
            }
             
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 8
            drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 .
            drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
            -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + whoami
            root
            [Pipeline] checkout
            Cloning the remote Git repository
            Cloning with configured refspecs honoured and without tags
            Using shallow clone
            shallow clone depth 2
            Cloning repository https://github.com/repo.git
             > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10
            Fetching upstream changes from https://github.com/repo.git
             > git --version # timeout=10
            using GIT_ASKPASS to set credentials 
             > git fetch --no-tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
             > git config remote.origin.url https://github.com/repo.git # timeout=10
             > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
             > git config remote.origin.url https://github.com/repo.git # timeout=10
            Fetching without tags
            Fetching upstream changes from https://github.com/repo.git
            using GIT_ASKPASS to set credentials 
             > git fetch --tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
            Checking out Revision 33d759ea0 (master)
            Commit message: "...."
             > git config core.sparsecheckout # timeout=10
             > git checkout -f 33d759ea0
             > git rev-list 33d759ea0 # timeout=10
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 112
            drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 .
            drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
            -rw-r--r-- 1 10000 10000  131 Aug 21 07:28 .dockerignore
            drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git
            -rw-r--r-- 1 10000 10000   71 Aug 21 07:28 .gitattributes
            -rw-r--r-- 1 10000 10000  120 Aug 21 07:28 .gitignore
            -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml
            -rw-r--r-- 1 10000 10000  522 Aug 21 07:28 Dockerfile
            -rw-r--r-- 1 10000 10000  197 Aug 21 07:28 Jenkinsfile
            -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile
            -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md
            -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml
            -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh
            -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock
            -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml
            -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
            drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts
            [Pipeline] }
            

            so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 

            jknurek J Knurek added a comment - I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code: container(name: "golang" ) { stage( "checkout" ) { sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ] ], userRemoteConfigs: scm.userRemoteConfigs ]) sh "ls -al" } }   [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 8 drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw------- 1 root root 0 Aug 21 07:28 nohup.out [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Using shallow clone shallow clone depth 2 Cloning repository https: //github.com/repo.git > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10 Fetching upstream changes from https: //github.com/repo.git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 > git config remote.origin.url https: //github.com/repo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10 > git config remote.origin.url https: //github.com/repo.git # timeout=10 Fetching without tags Fetching upstream changes from https: //github.com/repo.git using GIT_ASKPASS to set credentials > git fetch --tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 Checking out Revision 33d759ea0 (master) Commit message: "...." > git config core.sparsecheckout # timeout=10 > git checkout -f 33d759ea0 > git rev-list 33d759ea0 # timeout=10 [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 112 drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw-r--r-- 1 10000 10000 131 Aug 21 07:28 .dockerignore drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git -rw-r--r-- 1 10000 10000 71 Aug 21 07:28 .gitattributes -rw-r--r-- 1 10000 10000 120 Aug 21 07:28 .gitignore -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml -rw-r--r-- 1 10000 10000 522 Aug 21 07:28 Dockerfile -rw-r--r-- 1 10000 10000 197 Aug 21 07:28 Jenkinsfile -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml -rw------- 1 root root 0 Aug 21 07:28 nohup.out drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts [Pipeline] } so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 
            markewaite Mark Waite added a comment -

            Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file.

            You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

            markewaite Mark Waite added a comment - Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file. You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

            People

              Unassigned Unassigned
              jknurek J Knurek
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: