[JENKINS-46230] git-lfs not working in kubernetes container

Type: Bug
Resolution: Duplicate
Priority: Minor
Component/s: git-client-plugin, kubernetes-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

Trying to make use of the git-lfs addition to the checkout step, but it's failing.

The error I'm getting is:

Fetching without tags
Fetching upstream changes from https://github.com/---
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress https://github.com/--- +refs/heads/master:refs/remotes/origin/master --depth=2
Checking out Revision 33d759e (master)
Commit message: "---"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 33d759e

hudson.plugins.git.GitException: Command "git checkout -f 33d759e" returned status code 128:
stdout: 
stderr: git-lfs filter-process: line 1: git-lfs: not found
fatal: The remote end hung up unexpectedly

The example pipeline code that is failing:

podTemplate(label: "test", containers: [
    containerTemplate(name: "golang", image: "golang:1.8", ttyEnabled: true, command: "cat"
    )
]) {
	node("test") {
		container(name: "golang") {
			stage("checkout") {
				// install git lfs on debian based pod
				sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
				sh "apt-get install git-lfs"
				sh "git lfs install"
				checkout([
					$class: "GitSCM",
					branches: scm.branches,
					extensions: scm.extensions + [
						[$class: "CloneOption", depth: 2, shallow: true],
						[$class: "GitLFSPull"]
					],
					userRemoteConfigs: scm.userRemoteConfigs
				])
				sh 'ls -l'
			}
		}
	}
}

If I log into the pod after git-lfs is installed and run `git clone` directly, it works and pulls the large file. My guess is that this problem is related to this bug/workaround, but I don't know how to implement this in the k8s/jenkins env
https://github.com/git-lfs/git-lfs/issues/1821#issuecomment-272889540

is related to

JENKINS-47389 sh step within container step does not work when user is non-root

Resolved

JENKINS-30600 git isn't run inside build container

Open

JENKINS-46055 "dir" fails to use existing directory owned by root

Open

J Knurek created issue - 2017-08-16 09:05

Mark Waite made changes - 2017-08-16 10:44

Assignee

Original: Mark Waite [ markewaite ]

Mark Waite added a comment - 2017-08-16 10:58 - edited

I don't understand which work around you believe would resolve your issue in your kubernetes container.

That issue on GitHub seems to mention credential helpers, even though Linux machines (and kubernetes pods) rarely use credentials helpers. Your pod does not seem to be defining a credentials helper, so I assume it is not a credential helper issue.

That issue also mentions case insensitive file name extensions, yet you said that the clone works when you perform a git clone from inside the kubernetes pod. I would expect case insensitive file name extensions to cause that to break no matter what you did, since the typical Linux file system is case sensitive.

The message from the checkout step hints (to me) that the lfs command was not installed in a portion of the PATH defined for the checkout step. That is quite odd, since I assume the preceding git lfs install is not reporting any errors. Can you confirm that git lfs install reports that git lfs is installed?

Can you perform the git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master command from a shell step immediately prior to your checkout step?

What is the output of git --version on your kubernetes instances? Is the git new enough to support LFS?

Mark Waite added a comment - 2017-08-16 10:58 - edited I don't understand which work around you believe would resolve your issue in your kubernetes container. That issue on GitHub seems to mention credential helpers, even though Linux machines (and kubernetes pods) rarely use credentials helpers. Your pod does not seem to be defining a credentials helper, so I assume it is not a credential helper issue. That issue also mentions case insensitive file name extensions, yet you said that the clone works when you perform a git clone from inside the kubernetes pod. I would expect case insensitive file name extensions to cause that to break no matter what you did, since the typical Linux file system is case sensitive. The message from the checkout step hints (to me) that the lfs command was not installed in a portion of the PATH defined for the checkout step. That is quite odd, since I assume the preceding git lfs install is not reporting any errors. Can you confirm that git lfs install reports that git lfs is installed? Can you perform the git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master command from a shell step immediately prior to your checkout step? What is the output of git --version on your kubernetes instances? Is the git new enough to support LFS?

J Knurek added a comment - 2017-08-17 14:15

I tried running git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master but the

git fetch origin was not responding. Because I have other problems in later steps trying to use git's ssh connection, and have a work-around for it, I wrapped the calls and below are the results for the following commands:

	sh "curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash"
	sh "apt-get install git-lfs"
	sh "git lfs install"
	sh "git init"
	sh "git remote add origin git@github.com:repo.git"
	sh "chmod 400 /root/.ssh/ssh-privatekey"
	sh "echo \"Host github.com\n\tStrictHostKeyChecking no\n\" > /root/.ssh/config"
	sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin'"
	sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master'"

oh, and the git version in this container is: 2.1.4

+ bash
+ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh
Detected operating system as debian/8.
Checking for curl...
Detected curl...
Running apt-get update... done.
Installing debian-archive-keyring which is needed for installing 
apt-transport-https on many Debian systems.
Installing apt-transport-https... done.
Installing /etc/apt/sources.list.d/github_git-lfs.list...done.
Importing packagecloud gpg key... done.
Running apt-get update... done.

The repository is setup! You can now install packages.
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ apt-get install git-lfs
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  git-lfs
0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded.
Need to get 2766 kB of archives.
After this operation, 11.3 MB of additional disk space will be used.
Get:1 https://packagecloud.io/github/git-lfs/debian/ jessie/main git-lfs amd64 2.2.1 [2766 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2766 kB in 1s (2049 kB/s)
Selecting previously unselected package git-lfs.
(Reading database ... 
(Reading database ... 100%
(Reading database ... 14788 files and directories currently installed.)
Preparing to unpack .../git-lfs_2.2.1_amd64.deb ...
Unpacking git-lfs (2.2.1) ...
Setting up git-lfs (2.2.1) ...
Git LFS initialized.
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ git lfs install
Git LFS initialized.
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ git init
Initialized empty Git repository in /home/jenkins/workspace/repo_master-FZOPC6NFFVCNA/.git/
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ git remote add origin git@github.com:repo.git
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ chmod 400 /root/.ssh/ssh-privatekey
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ echo Host github.com
	StrictHostKeyChecking no

[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin
Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment)
Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
From github.com:repo
 * [new branch]      feature/16415-docker-fix -> origin/feature/16415-docker-fix
 * [new branch]      master     -> origin/master
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master
Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment)
Fetching origin/master

Git LFS: (0 of 1 files) 0 B / 116.62 MB                                        
Git LFS: (1 of 1 files) 116.62 MB / 116.62 MB                                  
[Pipeline] checkout
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/repo.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/repo.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:260)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/repo.git" returned status code 255:
stdout: 
stderr: error: could not lock config file .git/config: Permission denied

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to JNLP4-connect connection from 10.132.5.130/10.132.5.130:42512(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:830)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy104.setRemoteUrl(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805)
	... 13 more

J Knurek added a comment - 2017-08-17 14:15 I tried running git init; git remote add origin your_url; git fetch origin; git lfs fetch origin origin/master but the git fetch origin was not responding. Because I have other problems in later steps trying to use git's ssh connection, and have a work-around for it, I wrapped the calls and below are the results for the following commands: sh "curl -s https: //packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash" sh "apt-get install git-lfs" sh "git lfs install" sh "git init" sh "git remote add origin git@github.com:repo.git" sh "chmod 400 /root/.ssh/ssh-privatekey" sh "echo \" Host github.com\n\tStrictHostKeyChecking no\n\ " > /root/.ssh/config" sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin' " sh "ssh-agent bash -c 'ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master' " oh, and the git version in this container is: 2.1.4 + bash + curl -s https: //packagecloud.io/install/repositories/github/git-lfs/script.deb.sh Detected operating system as debian/8. Checking for curl... Detected curl... Running apt-get update... done. Installing debian-archive-keyring which is needed for installing apt-transport-https on many Debian systems. Installing apt-transport-https... done. Installing /etc/apt/sources.list.d/github_git-lfs.list...done. Importing packagecloud gpg key... done. Running apt-get update... done. The repository is setup! You can now install packages. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + apt-get install git-lfs Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: git-lfs 0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded. Need to get 2766 kB of archives. After this operation, 11.3 MB of additional disk space will be used. Get:1 https: //packagecloud.io/github/git-lfs/debian/ jessie/main git-lfs amd64 2.2.1 [2766 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 2766 kB in 1s (2049 kB/s) Selecting previously unselected package git-lfs. (Reading database ... (Reading database ... 100% (Reading database ... 14788 files and directories currently installed.) Preparing to unpack .../git-lfs_2.2.1_amd64.deb ... Unpacking git-lfs (2.2.1) ... Setting up git-lfs (2.2.1) ... Git LFS initialized. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git lfs install Git LFS initialized. [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git init Initialized empty Git repository in /home/jenkins/workspace/repo_master-FZOPC6NFFVCNA/.git/ [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + git remote add origin git@github.com:repo.git [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + chmod 400 /root/.ssh/ssh-privatekey [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + echo Host github.com StrictHostKeyChecking no [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git fetch -v --depth=2 origin Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment) Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts. From github.com:repo * [ new branch] feature/16415-docker-fix -> origin/feature/16415-docker-fix * [ new branch] master -> origin/master [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ssh-agent bash -c ssh-add /root/.ssh/ssh-privatekey; git lfs fetch origin origin/master Identity added: /root/.ssh/ssh-privatekey (rsa w/o comment) Fetching origin/master Git LFS: (0 of 1 files) 0 B / 116.62 MB Git LFS: (1 of 1 files) 116.62 MB / 116.62 MB [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https: //github.com/repo.git # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https: //github.com/repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https: //github.com/repo.git" returned status code 255: stdout: stderr: error: could not lock config file .git/config: Permission denied at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179) at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang. Thread .run( Thread .java:745) at ......remote call to JNLP4-connect connection from 10.132.5.130/10.132.5.130:42512(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:830) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257) at com.sun.proxy.$Proxy104.setRemoteUrl(Unknown Source) at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805) ... 13 more

Mark Waite added a comment - 2017-08-17 14:19 - edited

The message

could not lock config file .git/config: Permission denied

would usually indicate that the wrong user owns that file, or the wrong ownership or permissions have been assigned to that file or the directory which contains the file.

Mark Waite added a comment - 2017-08-17 14:19 - edited The message could not lock config file .git/config: Permission denied would usually indicate that the wrong user owns that file, or the wrong ownership or permissions have been assigned to that file or the directory which contains the file.

J Knurek added a comment - 2017-08-18 09:04 - edited

+ ls -l .git/config
-rwxr--r-- 1 root root 210 Aug 18 08:59 .git/config

+ whoami
root

J Knurek added a comment - 2017-08-18 09:04 - edited + ls -l .git/config -rwxr--r-- 1 root root 210 Aug 18 08:59 .git/config + whoami root

J Knurek added a comment - 2017-08-18 09:13

more specifically, this is after installing git lfs, and running init/fetch:

the pipeline steps:

	sh "ls -l .git/"
	sh "ls -al"
	sh "whoami"
	checkout([
		$class: "GitSCM",
		branches: scm.branches,
		extensions: scm.extensions + [
			[$class: "CloneOption", depth: 2, shallow: true],
			[$class: "GitLFSPull"]
		],
		userRemoteConfigs: scm.userRemoteConfigs
	])

and the output:

+ ls -l .git/
total 48
-rw-r--r-- 1 root root  938 Aug 18 09:08 FETCH_HEAD
-rw-r--r-- 1 root root   23 Aug 18 09:08 HEAD
drwxr-xr-x 2 root root 4096 Aug 18 09:08 branches
-rwxr--r-- 1 root root  210 Aug 18 09:08 config
-rw-r--r-- 1 root root   73 Aug 18 09:08 description
drwxr-xr-x 2 root root 4096 Aug 18 09:08 hooks
drwxr-xr-x 2 root root 4096 Aug 18 09:08 info
drwxr-xr-x 4 root root 4096 Aug 18 09:08 lfs
drwxr-xr-x 3 root root 4096 Aug 18 09:08 logs
drwxr-xr-x 4 root root 4096 Aug 18 09:08 objects
drwxr-xr-x 5 root root 4096 Aug 18 09:08 refs
-rw-r--r-- 1 root root  287 Aug 18 09:08 shallow
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ ls -al
total 12
drwxr-xr-x 3 10000 10000 4096 Aug 18 09:08 .
drwxr-xr-x 4 10000 10000 4096 Aug 18 09:08 ..
drwxr-xr-x 9 root  root  4096 Aug 18 09:08 .git
-rw------- 1 root  root     0 Aug 18 09:08 nohup.out
[Pipeline] sh
[repo_master-FZOPC6NFFVCNA] Running shell script
+ whoami
root
[Pipeline] checkout
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/repo.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/repo.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:260)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/repo.git" returned status code 255:
stdout: 
stderr: error: could not lock config file .git/config: Permission denied

J Knurek added a comment - 2017-08-18 09:13 more specifically, this is after installing git lfs, and running init/fetch: the pipeline steps: sh "ls -l .git/" sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ], [$class: "GitLFSPull" ] ], userRemoteConfigs: scm.userRemoteConfigs ]) and the output: + ls -l .git/ total 48 -rw-r--r-- 1 root root 938 Aug 18 09:08 FETCH_HEAD -rw-r--r-- 1 root root 23 Aug 18 09:08 HEAD drwxr-xr-x 2 root root 4096 Aug 18 09:08 branches -rwxr--r-- 1 root root 210 Aug 18 09:08 config -rw-r--r-- 1 root root 73 Aug 18 09:08 description drwxr-xr-x 2 root root 4096 Aug 18 09:08 hooks drwxr-xr-x 2 root root 4096 Aug 18 09:08 info drwxr-xr-x 4 root root 4096 Aug 18 09:08 lfs drwxr-xr-x 3 root root 4096 Aug 18 09:08 logs drwxr-xr-x 4 root root 4096 Aug 18 09:08 objects drwxr-xr-x 5 root root 4096 Aug 18 09:08 refs -rw-r--r-- 1 root root 287 Aug 18 09:08 shallow [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + ls -al total 12 drwxr-xr-x 3 10000 10000 4096 Aug 18 09:08 . drwxr-xr-x 4 10000 10000 4096 Aug 18 09:08 .. drwxr-xr-x 9 root root 4096 Aug 18 09:08 .git -rw------- 1 root root 0 Aug 18 09:08 nohup.out [Pipeline] sh [repo_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https: //github.com/repo.git # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https: //github.com/repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https: //github.com/repo.git" returned status code 255: stdout: stderr: error: could not lock config file .git/config: Permission denied

Mark Waite added a comment - 2017-08-18 15:29

Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file.

You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

Mark Waite added a comment - 2017-08-18 15:29 Sorry, but I don't know how to diagnose the permissions issue which is causing git to be unable to lock the .git/config file. It could be a problem from the underlying file system where the volume is mounted. It could be a permissions issue on the file. It could be an ownership issue on the file. You might create a temporary subdirectory in that repository, change to that subdirectory, perform a git init, and a git config command in that subdirectory (to create the .git/config file), then compare the ownership and permissions on the resulting file to the problem file and its parent directory.

Mark Waite made changes - 2017-08-19 10:55

Component/s	Original: git-plugin [ 15543 ]
Component/s	Original: kubernetes-plugin [ 20639 ]

J Knurek added a comment - 2017-08-21 07:39

I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code:

container(name: "golang") {
	stage("checkout") {
		sh "ls -al"
		sh "whoami"
		checkout([
			$class: "GitSCM",
			branches: scm.branches,
			extensions: scm.extensions + [
				[$class: "CloneOption", depth: 2, shallow: true]
			],
			userRemoteConfigs: scm.userRemoteConfigs
		])
		sh "ls -al"
	}
}

[Pipeline] sh
[rep_master-FZOPC6NFFVCNA] Running shell script
+ ls -al
total 8
drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 .
drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
-rw------- 1 root  root     0 Aug 21 07:28 nohup.out
[Pipeline] sh
[rep_master-FZOPC6NFFVCNA] Running shell script
+ whoami
root
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Using shallow clone
shallow clone depth 2
Cloning repository https://github.com/repo.git
 > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10
Fetching upstream changes from https://github.com/repo.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
 > git config remote.origin.url https://github.com/repo.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
 > git config remote.origin.url https://github.com/repo.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/repo.git
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress https://github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2
Checking out Revision 33d759ea0 (master)
Commit message: "...."
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 33d759ea0
 > git rev-list 33d759ea0 # timeout=10
[Pipeline] sh
[rep_master-FZOPC6NFFVCNA] Running shell script
+ ls -al
total 112
drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 .
drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 ..
-rw-r--r-- 1 10000 10000  131 Aug 21 07:28 .dockerignore
drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git
-rw-r--r-- 1 10000 10000   71 Aug 21 07:28 .gitattributes
-rw-r--r-- 1 10000 10000  120 Aug 21 07:28 .gitignore
-rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml
-rw-r--r-- 1 10000 10000  522 Aug 21 07:28 Dockerfile
-rw-r--r-- 1 10000 10000  197 Aug 21 07:28 Jenkinsfile
-rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile
-rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md
-rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml
-rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh
-rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock
-rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml
-rw------- 1 root  root     0 Aug 21 07:28 nohup.out
drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts
[Pipeline] }

so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user

J Knurek added a comment - 2017-08-21 07:39 I don't think this has anything to do with the container but with either the kubernetes plugin OR the checkout step (or both). Because when I run this simplified step you can see that Jenkins is using a different user to checkout the code: container(name: "golang" ) { stage( "checkout" ) { sh "ls -al" sh "whoami" checkout([ $class: "GitSCM" , branches: scm.branches, extensions: scm.extensions + [ [$class: "CloneOption" , depth: 2, shallow: true ] ], userRemoteConfigs: scm.userRemoteConfigs ]) sh "ls -al" } } [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 8 drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw------- 1 root root 0 Aug 21 07:28 nohup.out [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + whoami root [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Using shallow clone shallow clone depth 2 Cloning repository https: //github.com/repo.git > git init /home/jenkins/workspace/rep_master-FZOPC6NFFVCNA # timeout=10 Fetching upstream changes from https: //github.com/repo.git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 > git config remote.origin.url https: //github.com/repo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10 > git config remote.origin.url https: //github.com/repo.git # timeout=10 Fetching without tags Fetching upstream changes from https: //github.com/repo.git using GIT_ASKPASS to set credentials > git fetch --tags --progress https: //github.com/repo.git +refs/heads/master:refs/remotes/origin/master --depth=2 Checking out Revision 33d759ea0 (master) Commit message: "...." > git config core.sparsecheckout # timeout=10 > git checkout -f 33d759ea0 > git rev-list 33d759ea0 # timeout=10 [Pipeline] sh [rep_master-FZOPC6NFFVCNA] Running shell script + ls -al total 112 drwxr-xr-x 7 10000 10000 4096 Aug 21 07:28 . drwxr-xr-x 4 10000 10000 4096 Aug 21 07:28 .. -rw-r--r-- 1 10000 10000 131 Aug 21 07:28 .dockerignore drwxr-xr-x 8 10000 10000 4096 Aug 21 07:28 .git -rw-r--r-- 1 10000 10000 71 Aug 21 07:28 .gitattributes -rw-r--r-- 1 10000 10000 120 Aug 21 07:28 .gitignore -rw-r--r-- 1 10000 10000 1189 Aug 21 07:28 .gitlab-ci.yml -rw-r--r-- 1 10000 10000 522 Aug 21 07:28 Dockerfile -rw-r--r-- 1 10000 10000 197 Aug 21 07:28 Jenkinsfile -rw-r--r-- 1 10000 10000 6445 Aug 21 07:28 Makefile -rw-r--r-- 1 10000 10000 4186 Aug 21 07:28 README.md -rw-r--r-- 1 10000 10000 2772 Aug 21 07:28 docker-compose.yml -rwxr-xr-x 1 10000 10000 1948 Aug 21 07:28 docker-entrypoint.sh -rw-r--r-- 1 10000 10000 8577 Aug 21 07:28 glide.lock -rw-r--r-- 1 10000 10000 3481 Aug 21 07:28 glide.yaml -rw------- 1 root root 0 Aug 21 07:28 nohup.out drwxr-xr-x 2 10000 10000 4096 Aug 21 07:28 scripts [Pipeline] } so in our previous example, when we use the `sh` step to run git init/fetch, the config is created as the root user, but then the checkout step is using the `10000` user

Assignee:: Unassigned

Reporter:: J Knurek

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2017-08-16 09:05

Updated:: 2020-10-29 11:51

Resolved:: 2020-10-29 11:51

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Mark Waite added a comment - 2017-08-16 10:58, Edited by Mark Waite - 2017-08-16 11:05

Expand comment: Mark Waite added a comment - 2017-08-16 10:58, Edited by Mark Waite - 2017-08-16 11:05

Collapse comment: J Knurek added a comment - 2017-08-17 14:15

Expand comment: J Knurek added a comment - 2017-08-17 14:15

Collapse comment: Mark Waite added a comment - 2017-08-17 14:19, Edited by Mark Waite - 2017-08-17 14:20

Expand comment: Mark Waite added a comment - 2017-08-17 14:19, Edited by Mark Waite - 2017-08-17 14:20

Collapse comment: J Knurek added a comment - 2017-08-18 09:04, Edited by J Knurek - 2017-08-18 09:06

Expand comment: J Knurek added a comment - 2017-08-18 09:04, Edited by J Knurek - 2017-08-18 09:06

Collapse comment: J Knurek added a comment - 2017-08-18 09:13

Expand comment: J Knurek added a comment - 2017-08-18 09:13

Collapse comment: Mark Waite added a comment - 2017-08-18 15:29

Expand comment: Mark Waite added a comment - 2017-08-18 15:29

Collapse comment: J Knurek added a comment - 2017-08-21 07:39

Expand comment: J Knurek added a comment - 2017-08-21 07:39

People

Dates