Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46230

git-lfs not working in kubernetes container

    XMLWordPrintable

Details

    Description

      Trying to make use of the git-lfs addition to the checkout step, but it's failing.

      The error I'm getting is:

      Fetching without tags
      Fetching upstream changes from https://github.com/---
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress https://github.com/--- +refs/heads/master:refs/remotes/origin/master --depth=2
      Checking out Revision 33d759e (master)
      Commit message: "---"
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 33d759e
      
      hudson.plugins.git.GitException: Command "git checkout -f 33d759e" returned status code 128:
      stdout: 
      stderr: git-lfs filter-process: line 1: git-lfs: not found
      fatal: The remote end hung up unexpectedly
      

      The example pipeline code that is failing: 

      podTemplate(label: "test", containers: [
          containerTemplate(name: "golang", image: "golang:1.8", ttyEnabled: true, command: "cat"
          )
      ]) {
      	node("test") {
      		container(name: "golang") {
      			stage("checkout") {
      				// install git lfs on debian based pod
      				sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
      				sh "apt-get install git-lfs"
      				sh "git lfs install"
      				checkout([
      					$class: "GitSCM",
      					branches: scm.branches,
      					extensions: scm.extensions + [
      						[$class: "CloneOption", depth: 2, shallow: true],
      						[$class: "GitLFSPull"]
      					],
      					userRemoteConfigs: scm.userRemoteConfigs
      				])
      				sh 'ls -l'
      			}
      		}
      	}
      }
       

      If I log into the pod after git-lfs is installed and run `git clone` directly, it works and pulls the large file. My guess is that this problem is related to this bug/workaround, but I don't know how to implement this in the k8s/jenkins env
      https://github.com/git-lfs/git-lfs/issues/1821#issuecomment-272889540

      Attachments

        Issue Links

          Activity

            jknurek J Knurek created issue -
            markewaite Mark Waite made changes -
            Field Original Value New Value
            Assignee Mark Waite [ markewaite ]
            markewaite Mark Waite added a comment - - edited

            I don't understand which work around you believe would resolve your issue in your kubernetes container.

            That issue on GitHub seems to mention credential helpers, even though Linux machines (and kubernetes pods) rarely use credentials helpers. Your pod does not seem to be defining a credentials helper, so I assume it is not a credential helper issue.

            That issue also mentions case insensitive file name extensions, yet you said that the clone works when you perform a git clone from inside the kubernetes pod. I would expect case insensitive file name extensions to cause that to break no matter what you did, since the typical Linux file system is case sensitive.

            The message from the checkout step hints (to me) that the lfs command was not installed in a portion of the PATH defined for the checkout step. That is quite odd, since I assume the preceding git lfs install is not reporting any errors. Can you confirm that git lfs install reports that git lfs is installed?

            Can you perform the git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master command from a shell step immediately prior to your checkout step?

            What is the output of git --version on your kubernetes instances? Is the git new enough to support LFS?

            markewaite Mark Waite added a comment - - edited I don't understand which work around you believe would resolve your issue in your kubernetes container. That issue on GitHub seems to mention credential helpers, even though Linux machines (and kubernetes pods) rarely use credentials helpers. Your pod does not seem to be defining a credentials helper, so I assume it is not a credential helper issue. That issue also mentions case insensitive file name extensions, yet you said that the clone works when you perform a git clone from inside the kubernetes pod. I would expect case insensitive file name extensions to cause that to break no matter what you did, since the typical Linux file system is case sensitive. The message from the checkout step hints (to me) that the lfs command was not installed in a portion of the PATH defined for the checkout step. That is quite odd, since I assume the preceding git lfs install is not reporting any errors. Can you confirm that git lfs install reports that git lfs is installed? Can you perform the git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master command from a shell step immediately prior to your checkout step? What is the output of git --version on your kubernetes instances? Is the git new enough to support LFS?
            jknurek J Knurek added a comment -

            I tried running git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master but the 

            git fetch origin was not responding. Because I have other problems in later steps trying to use git's ssh connection, and have a work-around for it, I wrapped the calls and below are the results for the following commands:

            	sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
            	sh "apt-get install git-lfs"
            	sh "git lfs install"
            	sh "git init"
            	sh "git remote add origin git@github.com:repo.git"
            	sh "chmod 400 /root/.ssh/ssh-privatekey"
            	sh "echo \"Host github.com\n\tStrictHostKeyChecking no\n\" > /root/.ssh/config"
            	sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin'"
            	sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master'"
             

            oh, and the git version in this container is: 2.1.4 

            + bash
            + curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh
            Detected operating system as debian/8.
            Checking for curl...
            Detected curl...
            Running apt-get update... done.
            Installing debian-archive-keyring which is needed for installing 
            apt-transport-https on many Debian systems.
            Installing apt-transport-https... done.
            Installing /etc/apt/sources.list.d/github_git-lfs.list...done.
            Importing packagecloud gpg key... done.
            Running apt-get update... done.
            
            The repository is setup! You can now install packages.
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + apt-get install git-lfs
            Reading package lists...
            Building dependency tree...
            Reading state information...
            The following NEW packages will be installed:
              git-lfs
            0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded.
            Need to get 2766 kB of archives.
            After this operation, 11.3 MB of additional disk space will be used.
            Get:1 https://packagecloud.io/github/git-lfs/debian/ jessie/main git-lfs amd64 2.2.1 [2766 kB]
            debconf: delaying package configuration, since apt-utils is not installed
            Fetched 2766 kB in 1s (2049 kB/s)
            Selecting previously unselected package git-lfs.
            (Reading database ... 
            (Reading database ... 100%
            (Reading database ... 14788 files and directories currently installed.)
            Preparing to unpack .../git-lfs_2.2.1_amd64.deb ...
            Unpacking git-lfs (2.2.1) ...
            Setting up git-lfs (2.2.1) ...
            Git LFS initialized.
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + git lfs install
            Git LFS initialized.
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + git init
            Initialized empty Git repository in /home/jenkins/workspace/repo_master-FZOPC6NFFVCNA/.git/
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + git remote add origin git@github.com:repo.git
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + chmod 400 /root/.ssh/ssh-privatekey
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + echo Host github.com
            	StrictHostKeyChecking no
            
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin
            Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment)
            Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
            From github.com:repo
             * [new branch]      feature/16415-docker-fix -> origin/feature/16415-docker-fix
             * [new branch]      master     -> origin/master
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master
            Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment)
            Fetching origin/master
            
            Git LFS: (0 of 1 files) 0 B / 116.62 MB                                        
            Git LFS: (1 of 1 files) 116.62 MB / 116.62 MB                                  
            [Pipeline] checkout
             > git rev-parse --is-inside-work-tree # timeout=10
            Fetching changes from the remote Git repository
             > git config remote.origin.url https://github.com/repo.git # timeout=10
            ERROR: Error fetching remote repo 'origin'
            hudson.plugins.git.GitException: Failed to fetch from https://github.com/repo.git
            	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
            	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
            	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
            	at hudson.security.ACL.impersonate(ACL.java:260)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/repo.git" returned status code 255:
            stdout: 
            stderr: error: could not lock config file .git/config: Permission denied
            
            	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924)
            	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
            	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
            	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
            	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545)
            	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179)
            	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:498)
            	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895)
            	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
            	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
            	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
            	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
            	at hudson.remoting.Request$2.run(Request.java:336)
            	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at hudson.remoting.Engine$1$1.run(Engine.java:94)
            	at java.lang.Thread.run(Thread.java:745)
            	at ......remote call to JNLP4-connect connection from 10.132.5.130/10.132.5.130:42512(Native Method)
            	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
            	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
            	at hudson.remoting.Channel.call(Channel.java:830)
            	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
            	at com.sun.proxy.$Proxy104.setRemoteUrl(Unknown Source)
            	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
            	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805)
            	... 13 more
            
            
            jknurek J Knurek added a comment - I tried running git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master but the  git fetch origin was not responding. Because I have other problems in later steps trying to use git's ssh connection, and have a work-around for it, I wrapped the calls and below are the results for the following commands: sh "curl -s https: //packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash" sh "apt-get install git-lfs" sh "git lfs install" sh "git init" sh "git remote add origin git@github.com:repo.git" sh "chmod 400 /root/.ssh/ssh-privatekey" sh "echo \" Host github.com\n\tStrictHostKeyChecking no\n\ " > /root/.ssh/config" sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin' " sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master' "   oh, and the git version in this container is: 2.1.4  + bash + curl -s https: //packagecloud.io/install/repositories/github/git-lfs/script.deb.sh Detected operating system as debian/8. Checking for curl... Detected curl... Running apt-get update... done. Installing debian-archive-keyring which is needed for installing apt-transport-https on many Debian systems. Installing apt-transport-https... done. Installing /etc/apt/sources.list.d/github_git-lfs.list...done. Importing packagecloud gpg key... done. Running apt-get update... done. The repository is setup! You can now install packages. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + apt-get install git-lfs Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: git-lfs 0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded. Need to get 2766 kB of archives. After this operation, 11.3 MB of additional disk space will be used. Get:1 https: //packagecloud.io/github/git-lfs/debian/ jessie/main git-lfs amd64 2.2.1 [2766 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 2766 kB in 1s (2049 kB/s) Selecting previously unselected package git-lfs. (Reading database ... (Reading database ... 100% (Reading database ... 14788 files and directories currently installed.) Preparing to unpack .../git-lfs_2.2.1_amd64.deb ... Unpacking git-lfs (2.2.1) ... Setting up git-lfs (2.2.1) ... Git LFS initialized. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git lfs install Git LFS initialized. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git init Initialized empty Git repository in /home/jenkins/workspace/repo_master-FZOPC6NFFVCNA/.git/ [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git remote add origin git@github.com:repo.git [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + chmod 400 /root/.ssh/ssh-privatekey [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + echo Host github.com StrictHostKeyChecking no [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment) Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts. From github.com:repo * [ new branch] feature/16415-docker-fix -> origin/feature/16415-docker-fix * [ new branch] master -> origin/master [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment) Fetching origin/master Git LFS: (0 of 1 files) 0 B / 116.62 MB Git LFS: (1 of 1 files) 116.62 MB / 116.62 MB [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https: //github.com/repo.git # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https: //github.com/repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https: //github.com/repo.git" returned status code 255: stdout: stderr: error: could not lock config file .git/config: Permission denied at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179) at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang. Thread .run( Thread .java:745) at ......remote call to JNLP4-connect connection from 10.132.5.130/10.132.5.130:42512(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:830) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257) at com.sun.proxy.$Proxy104.setRemoteUrl(Unknown Source) at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805) ... 13 more
            markewaite Mark Waite added a comment - - edited

            The message

            could not lock config file .git/config: Permission denied

            would usually indicate that the wrong user owns that file, or the wrong ownership or permissions have been assigned to that file or the directory which contains the file.

            markewaite Mark Waite added a comment - - edited The message could not lock config file .git/config: Permission denied would usually indicate that the wrong user owns that file, or the wrong ownership or permissions have been assigned to that file or the directory which contains the file.
            jknurek J Knurek added a comment - - edited
            + ls -l .git/config
            -rwxr--r-- 1 root root 210 Aug 18 08:59 .git/config

             

            + whoami
            root
            jknurek J Knurek added a comment - - edited + ls -l .git/config -rwxr--r-- 1 root root 210 Aug 18 08:59 .git/config   + whoami root
            jknurek J Knurek added a comment -

            more specifically, this is after installing git lfs, and running init/fetch:

            the pipeline steps:

            	sh "ls -l .git/"
            	sh "ls -al"
            	sh "whoami"
            	checkout([
            		$class: "GitSCM",
            		branches: scm.branches,
            		extensions: scm.extensions + [
            			[$class: "CloneOption", depth: 2, shallow: true],
            			[$class: "GitLFSPull"]
            		],
            		userRemoteConfigs: scm.userRemoteConfigs
            	])
             

            and the output:

            + ls -l .git/
            total 48
            -rw-r--r-- 1 root root  938 Aug 18 09:08 FETCH_HEAD
            -rw-r--r-- 1 root root   23 Aug 18 09:08 HEAD
            drwxr-xr-x 2 root root 4096 Aug 18 09:08 branches
            -rwxr--r-- 1 root root  210 Aug 18 09:08 config
            -rw-r--r-- 1 root root   73 Aug 18 09:08 description
            drwxr-xr-x 2 root root 4096 Aug 18 09:08 hooks
            drwxr-xr-x 2 root root 4096 Aug 18 09:08 info
            drwxr-xr-x 4 root root 4096 Aug 18 09:08 lfs
            drwxr-xr-x 3 root root 4096 Aug 18 09:08 logs
            drwxr-xr-x 4 root root 4096 Aug 18 09:08 objects
            drwxr-xr-x 5 root root 4096 Aug 18 09:08 refs
            -rw-r--r-- 1 root root  287 Aug 18 09:08 shallow
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 12
            drwxr-xr-x 3 10000 10000 4096 Aug 18 09:08 .
            drwxr-xr-x 4 10000 10000 4096 Aug 18 09:08 ..
            drwxr-xr-x 9 root  root  4096 Aug 18 09:08 .git
            -rw------- 1 root  root     0 Aug 18 09:08 nohup.out
            [Pipeline] sh
            [repo_master-FZOPC6NFFVCNA] Running shell script
            + whoami
            root
            [Pipeline] checkout
             > git rev-parse --is-inside-work-tree # timeout=10
            Fetching changes from the remote Git repository
             > git config remote.origin.url https://github.com/repo.git # timeout=10
            ERROR: Error fetching remote repo 'origin'
            hudson.plugins.git.GitException: Failed to fetch from https://github.com/repo.git
            	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
            	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
            	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
            	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
            	at hudson.security.ACL.impersonate(ACL.java:260)
            	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:745)
            Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/repo.git" returned status code 255:
            stdout: 
            stderr: error: could not lock config file .git/config: Permission denied
             
            jknurek J Knurek added a comment - more specifically, this is after installing git lfs, and running init/fetch: the pipeline steps: sh "ls -l .git/" sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ], [$class: "GitLFSPull" ] ], userRemoteConfigs: scm.userRemoteConfigs ])   and the output: + ls -l .git/ total 48 -rw-r--r-- 1 root root 938 Aug 18 09:08 FETCH_HEAD -rw-r--r-- 1 root root 23 Aug 18 09:08 HEAD drwxr-xr-x 2 root root 4096 Aug 18 09:08 branches -rwxr--r-- 1 root root 210 Aug 18 09:08 config -rw-r--r-- 1 root root 73 Aug 18 09:08 description drwxr-xr-x 2 root root 4096 Aug 18 09:08 hooks drwxr-xr-x 2 root root 4096 Aug 18 09:08 info drwxr-xr-x 4 root root 4096 Aug 18 09:08 lfs drwxr-xr-x 3 root root 4096 Aug 18 09:08 logs drwxr-xr-x 4 root root 4096 Aug 18 09:08 objects drwxr-xr-x 5 root root 4096 Aug 18 09:08 refs -rw-r--r-- 1 root root 287 Aug 18 09:08 shallow [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ls -al total 12 drwxr-xr-x 3 10000 10000 4096 Aug 18 09:08 . drwxr-xr-x 4 10000 10000 4096 Aug 18 09:08 .. drwxr-xr-x 9 root root 4096 Aug 18 09:08 .git -rw------- 1 root root 0 Aug 18 09:08 nohup.out [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https: //github.com/repo.git # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https: //github.com/repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https: //github.com/repo.git" returned status code 255: stdout: stderr: error: could not lock config file .git/config: Permission denied  
            markewaite Mark Waite added a comment -

            Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file.

            You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

            markewaite Mark Waite added a comment - Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file. You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.
            markewaite Mark Waite made changes -
            Component/s git-plugin [ 15543 ]
            Component/s kubernetes-plugin [ 20639 ]
            jknurek J Knurek added a comment -

            I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code:

            container(name: "golang") {
            	stage("checkout") {
            		sh "ls -al"
            		sh "whoami"
            		checkout([
            			$class: "GitSCM",
            			branches: scm.branches,
            			extensions: scm.extensions + [
            				[$class: "CloneOption", depth: 2, shallow: true]
            			],
            			userRemoteConfigs: scm.userRemoteConfigs
            		])
            		sh "ls -al"
            	}
            }
             
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 8
            drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 .
            drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
            -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + whoami
            root
            [Pipeline] checkout
            Cloning the remote Git repository
            Cloning with configured refspecs honoured and without tags
            Using shallow clone
            shallow clone depth 2
            Cloning repository https://github.com/repo.git
             > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10
            Fetching upstream changes from https://github.com/repo.git
             > git --version # timeout=10
            using GIT_ASKPASS to set credentials 
             > git fetch --no-tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
             > git config remote.origin.url https://github.com/repo.git # timeout=10
             > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
             > git config remote.origin.url https://github.com/repo.git # timeout=10
            Fetching without tags
            Fetching upstream changes from https://github.com/repo.git
            using GIT_ASKPASS to set credentials 
             > git fetch --tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
            Checking out Revision 33d759ea0 (master)
            Commit message: "...."
             > git config core.sparsecheckout # timeout=10
             > git checkout -f 33d759ea0
             > git rev-list 33d759ea0 # timeout=10
            [Pipeline] sh
            [rep_master-FZOPC6NFFVCNA] Running shell script
            + ls -al
            total 112
            drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 .
            drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
            -rw-r--r-- 1 10000 10000  131 Aug 21 07:28 .dockerignore
            drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git
            -rw-r--r-- 1 10000 10000   71 Aug 21 07:28 .gitattributes
            -rw-r--r-- 1 10000 10000  120 Aug 21 07:28 .gitignore
            -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml
            -rw-r--r-- 1 10000 10000  522 Aug 21 07:28 Dockerfile
            -rw-r--r-- 1 10000 10000  197 Aug 21 07:28 Jenkinsfile
            -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile
            -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md
            -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml
            -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh
            -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock
            -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml
            -rw------- 1 root  root     0 Aug 21 07:28 nohup.out
            drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts
            [Pipeline] }
            

            so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 

            jknurek J Knurek added a comment - I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code: container(name: "golang" ) { stage( "checkout" ) { sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ] ], userRemoteConfigs: scm.userRemoteConfigs ]) sh "ls -al" } }   [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 8 drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw------- 1 root root 0 Aug 21 07:28 nohup.out [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Using shallow clone shallow clone depth 2 Cloning repository https: //github.com/repo.git > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10 Fetching upstream changes from https: //github.com/repo.git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 > git config remote.origin.url https: //github.com/repo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10 > git config remote.origin.url https: //github.com/repo.git # timeout=10 Fetching without tags Fetching upstream changes from https: //github.com/repo.git using GIT_ASKPASS to set credentials > git fetch --tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 Checking out Revision 33d759ea0 (master) Commit message: "...." > git config core.sparsecheckout # timeout=10 > git checkout -f 33d759ea0 > git rev-list 33d759ea0 # timeout=10 [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 112 drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw-r--r-- 1 10000 10000 131 Aug 21 07:28 .dockerignore drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git -rw-r--r-- 1 10000 10000 71 Aug 21 07:28 .gitattributes -rw-r--r-- 1 10000 10000 120 Aug 21 07:28 .gitignore -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml -rw-r--r-- 1 10000 10000 522 Aug 21 07:28 Dockerfile -rw-r--r-- 1 10000 10000 197 Aug 21 07:28 Jenkinsfile -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml -rw------- 1 root root 0 Aug 21 07:28 nohup.out drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts [Pipeline] } so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user 
            jknurek J Knurek made changes -
            Component/s kubernetes-plugin [ 20639 ]
            jknurek J Knurek made changes -
            Link This issue is related to JENKINS-47389 [ JENKINS-47389 ]
            jknurek J Knurek made changes -
            Link This issue is related to JENKINS-46055 [ JENKINS-46055 ]

            my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container

            csanchez Carlos Sanchez added a comment - my guess is that shell commands are executed inside the golang container as expected (inside container("golang")) but the checkout step is running in the default jnlp container
            jknurek J Knurek added a comment -

            yes, I think you're right.

             

            I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by: https://github.com/jenkinsci/docker-slave/pull/39

            jknurek J Knurek added a comment - yes, I think you're right.   I was able to recently workaround this by using a custom jnlp image that has git lfs installed. Which to me implies that this issue is merely just blocked by:  https://github.com/jenkinsci/docker-slave/pull/39
            markewaite Mark Waite added a comment -

            Closing as a duplicate of JENKINS-30600.

            markewaite Mark Waite added a comment - Closing as a duplicate of JENKINS-30600 .
            markewaite Mark Waite made changes -
            Link This issue is related to JENKINS-30600 [ JENKINS-30600 ]
            markewaite Mark Waite made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Closed [ 6 ]

            People

              Unassigned Unassigned
              jknurek J Knurek
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: