Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46230

git-lfs not working in kubernetes container

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Trying to make use of the git-lfs addition to the checkout step, but it's failing.

      The error I'm getting is:

      Fetching without tags
      Fetching upstream changes from https://github.com/---
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress https://github.com/--- +refs/heads/master:refs/remotes/origin/master --depth=2
      Checking out Revision 33d759e (master)
      Commit message: "---"
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 33d759e
      
      hudson.plugins.git.GitException: Command "git checkout -f 33d759e" returned status code 128:
      stdout: 
      stderr: git-lfs filter-process: line 1: git-lfs: not found
      fatal: The remote end hung up unexpectedly
      

      The example pipeline code that is failing: 

      podTemplate(label: "test", containers: [
          containerTemplate(name: "golang", image: "golang:1.8", ttyEnabled: true, command: "cat"
          )
      ]) {
      	node("test") {
      		container(name: "golang") {
      			stage("checkout") {
      				// install git lfs on debian based pod
      				sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
      				sh "apt-get install git-lfs"
      				sh "git lfs install"
      				checkout([
      					$class: "GitSCM",
      					branches: scm.branches,
      					extensions: scm.extensions + [
      						[$class: "CloneOption", depth: 2, shallow: true],
      						[$class: "GitLFSPull"]
      					],
      					userRemoteConfigs: scm.userRemoteConfigs
      				])
      				sh 'ls -l'
      			}
      		}
      	}
      }
       

      If I log into the pod after git-lfs is installed and run `git clone` directly, it works and pulls the large file. My guess is that this problem is related to this bug/workaround, but I don't know how to implement this in the k8s/jenkins env
      https://github.com/git-lfs/git-lfs/issues/1821#issuecomment-272889540

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment -

            Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file.

            You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

            Show
            markewaite Mark Waite added a comment - Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file. You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.
            Hide
            jknurek J Knurek added a comment -

            I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code:

            container(name: "golang") {
            	stage("checkout") {
            		sh "ls -al"
            		sh "whoami"
            		checkout([
            			$class: "GitSCM",
            			branches: scm.branches,
            			extensions: scm.extensions + [
            				[$class: "CloneOption", depth: 2, shallow: true]
            			],
            			userRemoteConfigs: scm.userRemoteConfigs
            		])
            		sh "ls -al"
            	}
            }
             
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 8
            drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 .
            drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
            -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + whoami
            root
            [Pipeline] checkout
            Cloning the remote Git repository
            Cloning with configured refspecs honoured and without tags
            Using shallow clone
            shallow clone depth 2
            Cloning repository https://github.com/repo.git
             > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10
            Fetching upstream changes from https://github.com/repo.git
             > git --version # timeout=10
            using GIT_ASKPASS to set credentials 
             > git fetch --no-tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
             > git config remote.origin.url https://github.com/repo.git # timeout=10
             > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
             > git config remote.origin.url https://github.com/repo.git # timeout=10
            Fetching without tags
            Fetching upstream changes from https://github.com/repo.git
            using GIT_ASKPASS to set credentials 
             > git fetch --tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
            Checking out Revision 33d759ea0 (master)
            Commit message: "...."
             > git config core.sparsecheckout # timeout=10
             > git checkout -f 33d759ea0
             > git rev-list 33d759ea0 # timeout=10
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 112
            drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 .
            drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
            -rw-r--r-- 1 10000 10000  131 Aug 21 07:28 .dockerignore
            drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git
            -rw-r--r-- 1 10000 10000   71 Aug 21 07:28 .gitattributes
            -rw-r--r-- 1 10000 10000  120 Aug 21 07:28 .gitignore
            -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml
            -rw-r--r-- 1 10000 10000  522 Aug 21 07:28 Dockerfile
            -rw-r--r-- 1 10000 10000  197 Aug 21 07:28 Jenkinsfile
            -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile
            -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md
            -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml
            -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh
            -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock
            -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml
            -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
            drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts
            [Pipeline] }
            

            so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 

            Show
            jknurek J Knurek added a comment - I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code: container(name: "golang" ) { stage( "checkout" ) { sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ] ], userRemoteConfigs: scm.userRemoteConfigs ]) sh "ls -al" } }   [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 8 drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw------- 1 root root 0 Aug 21 07:28 nohup.out [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Using shallow clone shallow clone depth 2 Cloning repository https: //github.com/repo.git > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10 Fetching upstream changes from https: //github.com/repo.git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 > git config remote.origin.url https: //github.com/repo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10 > git config remote.origin.url https: //github.com/repo.git # timeout=10 Fetching without tags Fetching upstream changes from https: //github.com/repo.git using GIT_ASKPASS to set credentials > git fetch --tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 Checking out Revision 33d759ea0 (master) Commit message: "...." > git config core.sparsecheckout # timeout=10 > git checkout -f 33d759ea0 > git rev-list 33d759ea0 # timeout=10 [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 112 drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw-r--r-- 1 10000 10000 131 Aug 21 07:28 .dockerignore drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git -rw-r--r-- 1 10000 10000 71 Aug 21 07:28 .gitattributes -rw-r--r-- 1 10000 10000 120 Aug 21 07:28 .gitignore -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml -rw-r--r-- 1 10000 10000 522 Aug 21 07:28 Dockerfile -rw-r--r-- 1 10000 10000 197 Aug 21 07:28 Jenkinsfile -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml -rw------- 1 root root 0 Aug 21 07:28 nohup.out drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts [Pipeline] } so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 
            Hide
            csanchez Carlos Sanchez added a comment -

            my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container

            Show
            csanchez Carlos Sanchez added a comment - my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container
            Hide
            jknurek J Knurek added a comment -

            yes, I think you're right.

             

            I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by: https://github.com/jenkinsci/docker-slave/pull/39

            Show
            jknurek J Knurek added a comment - yes, I think you're right.   I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by:  https://github.com/jenkinsci/docker-slave/pull/39
            Hide
            markewaite Mark Waite added a comment -

            Closing as a duplicate of JENKINS-30600.

            Show
            markewaite Mark Waite added a comment - Closing as a duplicate of JENKINS-30600 .

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jknurek J Knurek
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: