Trying to make use of the git-lfs addition to the checkout step, but it's failing.

      The error I'm getting is:

      Fetching without tags
      Fetching upstream changes from https://github.com/---
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress https://github.com/--- +refs/heads/master:refs/remotes/origin/master --depth=2
      Checking out Revision 33d759e (master)
      Commit message: "---"
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 33d759e
      
      hudson.plugins.git.GitException: Command "git checkout -f 33d759e" returned status code 128:
      stdout: 
      stderr: git-lfs filter-process: line 1: git-lfs: not found
      fatal: The remote end hung up unexpectedly
      

      The example pipeline code that is failing: 

      podTemplate(label: "test", containers: [
          containerTemplate(name: "golang", image: "golang:1.8", ttyEnabled: true, command: "cat"
          )
      ]) {
      	node("test") {
      		container(name: "golang") {
      			stage("checkout") {
      				// install git lfs on debian based pod
      				sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
      				sh "apt-get install git-lfs"
      				sh "git lfs install"
      				checkout([
      					$class: "GitSCM",
      					branches: scm.branches,
      					extensions: scm.extensions + [
      						[$class: "CloneOption", depth: 2, shallow: true],
      						[$class: "GitLFSPull"]
      					],
      					userRemoteConfigs: scm.userRemoteConfigs
      				])
      				sh 'ls -l'
      			}
      		}
      	}
      }
       

      If I log into the pod after git-lfs is installed and run `git clone` directly, it works and pulls the large file. My guess is that this problem is related to this bug/workaround, but I don't know how to implement this in the k8s/jenkins env
      https://github.com/git-lfs/git-lfs/issues/1821#issuecomment-272889540

          [JENKINS-46230] git-lfs not working in kubernetes container

          Mark Waite added a comment - - edited

          I don't understand which work around you believe would resolve your issue in your kubernetes container.

          That issue on GitHub seems to mention credential helpers, even though Linux machines (and kubernetes pods) rarely use credentials helpers. Your pod does not seem to be defining a credentials helper, so I assume it is not a credential helper issue.

          That issue also mentions case insensitive file name extensions, yet you said that the clone works when you perform a git clone from inside the kubernetes pod. I would expect case insensitive file name extensions to cause that to break no matter what you did, since the typical Linux file system is case sensitive.

          The message from the checkout step hints (to me) that the lfs command was not installed in a portion of the PATH defined for the checkout step. That is quite odd, since I assume the preceding git lfs install is not reporting any errors. Can you confirm that git lfs install reports that git lfs is installed?

          Can you perform the git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master command from a shell step immediately prior to your checkout step?

          What is the output of git --version on your kubernetes instances? Is the git new enough to support LFS?

          Mark Waite added a comment - - edited I don't understand which work around you believe would resolve your issue in your kubernetes container. That issue on GitHub seems to mention credential helpers, even though Linux machines (and kubernetes pods) rarely use credentials helpers. Your pod does not seem to be defining a credentials helper, so I assume it is not a credential helper issue. That issue also mentions case insensitive file name extensions, yet you said that the clone works when you perform a git clone from inside the kubernetes pod. I would expect case insensitive file name extensions to cause that to break no matter what you did, since the typical Linux file system is case sensitive. The message from the checkout step hints (to me) that the lfs command was not installed in a portion of the PATH defined for the checkout step. That is quite odd, since I assume the preceding git lfs install is not reporting any errors. Can you confirm that git lfs install reports that git lfs is installed? Can you perform the git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master command from a shell step immediately prior to your checkout step? What is the output of git --version on your kubernetes instances? Is the git new enough to support LFS?

          J Knurek added a comment -

          I tried running git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master but the 

          git fetch origin was not responding. Because I have other problems in later steps trying to use git's ssh connection, and have a work-around for it, I wrapped the calls and below are the results for the following commands:

          	sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
          	sh "apt-get install git-lfs"
          	sh "git lfs install"
          	sh "git init"
          	sh "git remote add origin git@github.com:repo.git"
          	sh "chmod 400 /root/.ssh/ssh-privatekey"
          	sh "echo \"Host github.com\n\tStrictHostKeyChecking no\n\" > /root/.ssh/config"
          	sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin'"
          	sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master'"
           

          oh, and the git version in this container is: 2.1.4 

          + bash
          + curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh
          Detected operating system as debian/8.
          Checking for curl...
          Detected curl...
          Running apt-get update... done.
          Installing debian-archive-keyring which is needed for installing 
          apt-transport-https on many Debian systems.
          Installing apt-transport-https... done.
          Installing /etc/apt/sources.list.d/github_git-lfs.list...done.
          Importing packagecloud gpg key... done.
          Running apt-get update... done.
          
          The repository is setup! You can now install packages.
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + apt-get install git-lfs
          Reading package lists...
          Building dependency tree...
          Reading state information...
          The following NEW packages will be installed:
            git-lfs
          0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded.
          Need to get 2766 kB of archives.
          After this operation, 11.3 MB of additional disk space will be used.
          Get:1 https://packagecloud.io/github/git-lfs/debian/ jessie/main git-lfs amd64 2.2.1 [2766 kB]
          debconf: delaying package configuration, since apt-utils is not installed
          Fetched 2766 kB in 1s (2049 kB/s)
          Selecting previously unselected package git-lfs.
          (Reading database ... 
          (Reading database ... 100%
          (Reading database ... 14788 files and directories currently installed.)
          Preparing to unpack .../git-lfs_2.2.1_amd64.deb ...
          Unpacking git-lfs (2.2.1) ...
          Setting up git-lfs (2.2.1) ...
          Git LFS initialized.
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + git lfs install
          Git LFS initialized.
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + git init
          Initialized empty Git repository in /home/jenkins/workspace/repo_master-FZOPC6NFFVCNA/.git/
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + git remote add origin git@github.com:repo.git
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + chmod 400 /root/.ssh/ssh-privatekey
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + echo Host github.com
          	StrictHostKeyChecking no
          
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin
          Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment)
          Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
          From github.com:repo
           * [new branch]      feature/16415-docker-fix -> origin/feature/16415-docker-fix
           * [new branch]      master     -> origin/master
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master
          Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment)
          Fetching origin/master
          
          Git LFS: (0 of 1 files) 0 B / 116.62 MB                                        
          Git LFS: (1 of 1 files) 116.62 MB / 116.62 MB                                  
          [Pipeline] checkout
           > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
           > git config remote.origin.url https://github.com/repo.git # timeout=10
          ERROR: Error fetching remote repo 'origin'
          hudson.plugins.git.GitException: Failed to fetch from https://github.com/repo.git
          	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
          	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
          	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
          	at hudson.security.ACL.impersonate(ACL.java:260)
          	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/repo.git" returned status code 255:
          stdout: 
          stderr: error: could not lock config file .git/config: Permission denied
          
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179)
          	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895)
          	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
          	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          	at hudson.remoting.Request$2.run(Request.java:336)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at hudson.remoting.Engine$1$1.run(Engine.java:94)
          	at java.lang.Thread.run(Thread.java:745)
          	at ......remote call to JNLP4-connect connection from 10.132.5.130/10.132.5.130:42512(Native Method)
          	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
          	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
          	at hudson.remoting.Channel.call(Channel.java:830)
          	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
          	at com.sun.proxy.$Proxy104.setRemoteUrl(Unknown Source)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
          	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805)
          	... 13 more
          
          

          J Knurek added a comment - I tried running git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master but the  git fetch origin was not responding. Because I have other problems in later steps trying to use git's ssh connection, and have a work-around for it, I wrapped the calls and below are the results for the following commands: sh "curl -s https: //packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash" sh "apt-get install git-lfs" sh "git lfs install" sh "git init" sh "git remote add origin git@github.com:repo.git" sh "chmod 400 /root/.ssh/ssh-privatekey" sh "echo \" Host github.com\n\tStrictHostKeyChecking no\n\ " > /root/.ssh/config" sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin' " sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master' "   oh, and the git version in this container is: 2.1.4  + bash + curl -s https: //packagecloud.io/install/repositories/github/git-lfs/script.deb.sh Detected operating system as debian/8. Checking for curl... Detected curl... Running apt-get update... done. Installing debian-archive-keyring which is needed for installing apt-transport-https on many Debian systems. Installing apt-transport-https... done. Installing /etc/apt/sources.list.d/github_git-lfs.list...done. Importing packagecloud gpg key... done. Running apt-get update... done. The repository is setup! You can now install packages. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + apt-get install git-lfs Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: git-lfs 0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded. Need to get 2766 kB of archives. After this operation, 11.3 MB of additional disk space will be used. Get:1 https: //packagecloud.io/github/git-lfs/debian/ jessie/main git-lfs amd64 2.2.1 [2766 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 2766 kB in 1s (2049 kB/s) Selecting previously unselected package git-lfs. (Reading database ... (Reading database ... 100% (Reading database ... 14788 files and directories currently installed.) Preparing to unpack .../git-lfs_2.2.1_amd64.deb ... Unpacking git-lfs (2.2.1) ... Setting up git-lfs (2.2.1) ... Git LFS initialized. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git lfs install Git LFS initialized. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git init Initialized empty Git repository in /home/jenkins/workspace/repo_master-FZOPC6NFFVCNA/.git/ [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git remote add origin git@github.com:repo.git [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + chmod 400 /root/.ssh/ssh-privatekey [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + echo Host github.com StrictHostKeyChecking no [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment) Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts. From github.com:repo * [ new branch] feature/16415-docker-fix -> origin/feature/16415-docker-fix * [ new branch] master -> origin/master [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment) Fetching origin/master Git LFS: (0 of 1 files) 0 B / 116.62 MB Git LFS: (1 of 1 files) 116.62 MB / 116.62 MB [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https: //github.com/repo.git # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https: //github.com/repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https: //github.com/repo.git" returned status code 255: stdout: stderr: error: could not lock config file .git/config: Permission denied at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179) at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang. Thread .run( Thread .java:745) at ......remote call to JNLP4-connect connection from 10.132.5.130/10.132.5.130:42512(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:830) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257) at com.sun.proxy.$Proxy104.setRemoteUrl(Unknown Source) at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805) ... 13 more

          Mark Waite added a comment - - edited

          The message

          could not lock config file .git/config: Permission denied

          would usually indicate that the wrong user owns that file, or the wrong ownership or permissions have been assigned to that file or the directory which contains the file.

          Mark Waite added a comment - - edited The message could not lock config file .git/config: Permission denied would usually indicate that the wrong user owns that file, or the wrong ownership or permissions have been assigned to that file or the directory which contains the file.

          J Knurek added a comment - - edited
          + ls -l .git/config
          -rwxr--r-- 1 root root 210 Aug 18 08:59 .git/config

           

          + whoami
          root

          J Knurek added a comment - - edited + ls -l .git/config -rwxr--r-- 1 root root 210 Aug 18 08:59 .git/config   + whoami root

          J Knurek added a comment -

          more specifically, this is after installing git lfs, and running init/fetch:

          the pipeline steps:

          	sh "ls -l .git/"
          	sh "ls -al"
          	sh "whoami"
          	checkout([
          		$class: "GitSCM",
          		branches: scm.branches,
          		extensions: scm.extensions + [
          			[$class: "CloneOption", depth: 2, shallow: true],
          			[$class: "GitLFSPull"]
          		],
          		userRemoteConfigs: scm.userRemoteConfigs
          	])
           

          and the output:

          + ls -l .git/
          total 48
          -rw-r--r-- 1 root root  938 Aug 18 09:08 FETCH_HEAD
          -rw-r--r-- 1 root root   23 Aug 18 09:08 HEAD
          drwxr-xr-x 2 root root 4096 Aug 18 09:08 branches
          -rwxr--r-- 1 root root  210 Aug 18 09:08 config
          -rw-r--r-- 1 root root   73 Aug 18 09:08 description
          drwxr-xr-x 2 root root 4096 Aug 18 09:08 hooks
          drwxr-xr-x 2 root root 4096 Aug 18 09:08 info
          drwxr-xr-x 4 root root 4096 Aug 18 09:08 lfs
          drwxr-xr-x 3 root root 4096 Aug 18 09:08 logs
          drwxr-xr-x 4 root root 4096 Aug 18 09:08 objects
          drwxr-xr-x 5 root root 4096 Aug 18 09:08 refs
          -rw-r--r-- 1 root root  287 Aug 18 09:08 shallow
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + ls -al
          total 12
          drwxr-xr-x 3 10000 10000 4096 Aug 18 09:08 .
          drwxr-xr-x 4 10000 10000 4096 Aug 18 09:08 ..
          drwxr-xr-x 9 root  root  4096 Aug 18 09:08 .git
          -rw------- 1 root  root     0 Aug 18 09:08 nohup.out
          [Pipeline] sh
          [repo_master-FZOPC6NFFVCNA] Running shell script
          + whoami
          root
          [Pipeline] checkout
           > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
           > git config remote.origin.url https://github.com/repo.git # timeout=10
          ERROR: Error fetching remote repo 'origin'
          hudson.plugins.git.GitException: Failed to fetch from https://github.com/repo.git
          	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
          	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
          	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
          	at hudson.security.ACL.impersonate(ACL.java:260)
          	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/repo.git" returned status code 255:
          stdout: 
          stderr: error: could not lock config file .git/config: Permission denied
           

          J Knurek added a comment - more specifically, this is after installing git lfs, and running init/fetch: the pipeline steps: sh "ls -l .git/" sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ], [$class: "GitLFSPull" ] ], userRemoteConfigs: scm.userRemoteConfigs ])   and the output: + ls -l .git/ total 48 -rw-r--r-- 1 root root 938 Aug 18 09:08 FETCH_HEAD -rw-r--r-- 1 root root 23 Aug 18 09:08 HEAD drwxr-xr-x 2 root root 4096 Aug 18 09:08 branches -rwxr--r-- 1 root root 210 Aug 18 09:08 config -rw-r--r-- 1 root root 73 Aug 18 09:08 description drwxr-xr-x 2 root root 4096 Aug 18 09:08 hooks drwxr-xr-x 2 root root 4096 Aug 18 09:08 info drwxr-xr-x 4 root root 4096 Aug 18 09:08 lfs drwxr-xr-x 3 root root 4096 Aug 18 09:08 logs drwxr-xr-x 4 root root 4096 Aug 18 09:08 objects drwxr-xr-x 5 root root 4096 Aug 18 09:08 refs -rw-r--r-- 1 root root 287 Aug 18 09:08 shallow [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ls -al total 12 drwxr-xr-x 3 10000 10000 4096 Aug 18 09:08 . drwxr-xr-x 4 10000 10000 4096 Aug 18 09:08 .. drwxr-xr-x 9 root root 4096 Aug 18 09:08 .git -rw------- 1 root root 0 Aug 18 09:08 nohup.out [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https: //github.com/repo.git # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https: //github.com/repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https: //github.com/repo.git" returned status code 255: stdout: stderr: error: could not lock config file .git/config: Permission denied  

          Mark Waite added a comment -

          Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file.

          You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

          Mark Waite added a comment - Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file. You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

          J Knurek added a comment -

          I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code:

          container(name: "golang") {
          	stage("checkout") {
          		sh "ls -al"
          		sh "whoami"
          		checkout([
          			$class: "GitSCM",
          			branches: scm.branches,
          			extensions: scm.extensions + [
          				[$class: "CloneOption", depth: 2, shallow: true]
          			],
          			userRemoteConfigs: scm.userRemoteConfigs
          		])
          		sh "ls -al"
          	}
          }
           
          [Pipeline] sh
          [rep_master-FZOPC6NFFVCNA] Running shell script
          + ls -al
          total 8
          drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 .
          drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
          -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
          [Pipeline] sh
          [rep_master-FZOPC6NFFVCNA] Running shell script
          + whoami
          root
          [Pipeline] checkout
          Cloning the remote Git repository
          Cloning with configured refspecs honoured and without tags
          Using shallow clone
          shallow clone depth 2
          Cloning repository https://github.com/repo.git
           > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10
          Fetching upstream changes from https://github.com/repo.git
           > git --version # timeout=10
          using GIT_ASKPASS to set credentials 
           > git fetch --no-tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
           > git config remote.origin.url https://github.com/repo.git # timeout=10
           > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
           > git config remote.origin.url https://github.com/repo.git # timeout=10
          Fetching without tags
          Fetching upstream changes from https://github.com/repo.git
          using GIT_ASKPASS to set credentials 
           > git fetch --tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
          Checking out Revision 33d759ea0 (master)
          Commit message: "...."
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 33d759ea0
           > git rev-list 33d759ea0 # timeout=10
          [Pipeline] sh
          [rep_master-FZOPC6NFFVCNA] Running shell script
          + ls -al
          total 112
          drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 .
          drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
          -rw-r--r-- 1 10000 10000  131 Aug 21 07:28 .dockerignore
          drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git
          -rw-r--r-- 1 10000 10000   71 Aug 21 07:28 .gitattributes
          -rw-r--r-- 1 10000 10000  120 Aug 21 07:28 .gitignore
          -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml
          -rw-r--r-- 1 10000 10000  522 Aug 21 07:28 Dockerfile
          -rw-r--r-- 1 10000 10000  197 Aug 21 07:28 Jenkinsfile
          -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile
          -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md
          -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml
          -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh
          -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock
          -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml
          -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
          drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts
          [Pipeline] }
          

          so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 

          J Knurek added a comment - I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code: container(name: "golang" ) { stage( "checkout" ) { sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ] ], userRemoteConfigs: scm.userRemoteConfigs ]) sh "ls -al" } }   [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 8 drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw------- 1 root root 0 Aug 21 07:28 nohup.out [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Using shallow clone shallow clone depth 2 Cloning repository https: //github.com/repo.git > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10 Fetching upstream changes from https: //github.com/repo.git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 > git config remote.origin.url https: //github.com/repo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10 > git config remote.origin.url https: //github.com/repo.git # timeout=10 Fetching without tags Fetching upstream changes from https: //github.com/repo.git using GIT_ASKPASS to set credentials > git fetch --tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 Checking out Revision 33d759ea0 (master) Commit message: "...." > git config core.sparsecheckout # timeout=10 > git checkout -f 33d759ea0 > git rev-list 33d759ea0 # timeout=10 [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 112 drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw-r--r-- 1 10000 10000 131 Aug 21 07:28 .dockerignore drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git -rw-r--r-- 1 10000 10000 71 Aug 21 07:28 .gitattributes -rw-r--r-- 1 10000 10000 120 Aug 21 07:28 .gitignore -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml -rw-r--r-- 1 10000 10000 522 Aug 21 07:28 Dockerfile -rw-r--r-- 1 10000 10000 197 Aug 21 07:28 Jenkinsfile -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml -rw------- 1 root root 0 Aug 21 07:28 nohup.out drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts [Pipeline] } so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 

          my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container

          Carlos Sanchez added a comment - my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container

          J Knurek added a comment -

          yes, I think you're right.

           

          I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by: https://github.com/jenkinsci/docker-slave/pull/39

          J Knurek added a comment - yes, I think you're right.   I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by:  https://github.com/jenkinsci/docker-slave/pull/39

          Mark Waite added a comment -

          Closing as a duplicate of JENKINS-30600.

          Mark Waite added a comment - Closing as a duplicate of JENKINS-30600 .

            Unassigned Unassigned
            jknurek J Knurek
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: