[JENKINS-60650] Git LFS *sometimes* fails on when git merge is involved

Type: Bug
Resolution: Duplicate
Priority: Major
Component/s: git-client-plugin
Labels:
- git
- jenkins
- plugin
Environment:

Hide
Jenkins version: 2.190.1
Bitbucket Source Branch Plugin version: 2.5.0
Git Plugin: 3.12.1
Git client Plugin: 2.8.6
Bitbucket server version: 6.1.2
Git LFS version: git-lfs/2.8.0 (GitHub; linux amd64; go 1.12.6)
Git version: 2.11.0

Jenkins is running inside a docker container.

Show
Jenkins version: 2.190.1 Bitbucket Source Branch Plugin version: 2.5.0 Git Plugin: 3.12.1 Git client Plugin: 2.8.6 Bitbucket server version: 6.1.2 Git LFS version: git-lfs/2.8.0 (GitHub; linux amd64; go 1.12.6) Git version: 2.11.0 Jenkins is running inside a docker container.

Similar Issues:
Powered by SuggestiMate

Show

Sometimes when an LFS file change is merged to master, all PR CI jobs will start failing on the merge step with:

Started by user Jupiter CI
 Restarted from build #11, stage Build Terminal
 Obtained Jenkinsfile from 5033c6390fdbbfa53f57975da66b0d55282353a3+bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667
 Running in Durability level: MAX_SURVIVABILITY
 [Pipeline] Start of Pipeline
 [Pipeline] milestone
 Trying to pass milestone 11
 [Pipeline] milestone
 Trying to pass milestone 12
 [Pipeline] stage
 [Pipeline] { (Build Terminal)
 [Pipeline] node
 Running on Jenkins in /var/jenkins_home/workspace/jupiter_PR-99
 [Pipeline]
{ [Pipeline] checkout using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
> git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from 2 remote Git repositories
> git config remote.origin.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10 Resetting working tree
> git reset --hard # timeout=10 > git clean -fdx # timeout=10
Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
> git --version # timeout=10 using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/pull-requests/99/from:refs/remotes/origin/PR-99
> git config remote.upstream.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10 Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10 Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/heads/master:refs/remotes/upstream/master Merging remotes/upstream/master commit bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 into PR head commit 5033c6390fdbbfa53f57975da66b0d55282353a3 Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin
> git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 # timeout=10
ERROR: Timeout after 10 minutes
Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin [Pipeline] }
[Pipeline] // node
 [Pipeline] }
 [Pipeline] // stage
 [Pipeline] End of Pipeline
 [Bitbucket] Notifying pull request build result
 [Bitbucket] Build result notified
 hudson.plugins.git.GitException: Command "git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667" returned status code 128:
 stdout: 
 stderr: Downloading apps/dashboard.tar.gz (3.8 MB)
 Terminated
 error: external filter 'git-lfs filter-process' failed
 fatal: apps/dashboard.tar.gz: smudge filter lfs failed
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:841)
 at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
 at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1100)
 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1193)
 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 Finished: FAILURE

This only seems to happen on PR jobs where it tries to merge the target branch first before building. And even then, only sometimes. Normal branch jobs don't encounter this problem.

The problem is, I am having a hard time reproducing this behavior outside of Jenkins. It seems like Jenkins is lying about which git commands it is actually running, because if I attach to the docker container running Jenkins, cd to the PR workspace folder, and then run the git commands shown in the log one by one, they all run without issue until I get to the git merge command at which point it complains about user and email not being configured. If I configure a user/email, then it complains about there not being an editor. But this works most of the time, so I'm wondering if it is secretly running git merge --no-edit or something. Oh, and if I do git merge --no-edit (by the way) it runs fine without error.

This seems similar to

https://issues.jenkins-ci.org/browse/JENKINS-50321

However, I get no errors at all related to credentials. Just the 10 minute timeout error after git merge, and then a lfs smudge error + backtrace. When this happens it's really bad as it halts merging of all PRs since Jenkins is now foobared.

The only way to "fix" it is to create a new PR that modifies the LFS file in the backtrace and merge it to master. Then, if you rebase all PRs, they will no longer fail with the LFS/merge issue.

duplicates

JENKINS-45228 Git merge requires authentication in LFS merges, plugin does not authenticate the git merge command

Closed

Bryan Gillespie created issue - 2020-01-06 17:53

Bryan Gillespie made changes - 2020-01-06 17:56

Description

Original: Sometimes when an LFS file change is merged to master, _all_ PR CI jobs will start failing on the merge step with:

{{Started by user Jupiter CI
Restarted from build #11, stage Build Terminal
Obtained Jenkinsfile from 5033c6390fdbbfa53f57975da66b0d55282353a3+bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] milestone
Trying to pass milestone 11
[Pipeline] milestone
Trying to pass milestone 12
[Pipeline] stage
[Pipeline] \{ (Build Terminal)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/jupiter_PR-99
[Pipeline] {
[Pipeline] checkout
using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from 2 remote Git repositories
> git config remote.origin.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching without tags
Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/pull-requests/99/from:refs/remotes/origin/PR-99
> git config remote.upstream.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching without tags
Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/heads/master:refs/remotes/upstream/master
Merging remotes/upstream/master commit bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 into PR head commit 5033c6390fdbbfa53f57975da66b0d55282353a3
Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin
> git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 # timeout=10
ERROR: Timeout after 10 minutes
Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
[Bitbucket] Notifying pull request build result
[Bitbucket] Build result notified
hudson.plugins.git.GitException: Command "git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667" returned status code 128:
stdout:
stderr: Downloading apps/dashboard.tar.gz (3.8 MB)
Terminated
error: external filter 'git-lfs filter-process' failed
fatal: apps/dashboard.tar.gz: smudge filter lfs failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:841)
at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1100)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1193)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE}}

This only seems to happen on PR jobs where it tries to merge the target branch first before building. And even then, only sometimes. Normal branch jobs don't encounter this problem.

The problem is, I am having a hard time reproducing this behavior outside of Jenkins. It seems like Jenkins is lying about which git commands it is actually running, because if I attach to the docker container running Jenkins, cd to the PR workspace folder, and then run the git commands shown in the log one by one, they all run without issue until I get to the {{git merge}} command at which point it complains about user and email not being configured. If I configure a user/email, then it complains about there not being an editor. But this works _most_ of the time, so I'm wondering if it is secretly running {{git merge --no-edit}} or something. Oh, and if I do {{git merge --no-edit}} (by the way) it runs fine without error.

This seems similar to

https://issues.jenkins-ci.org/browse/JENKINS-50321

However, I get no errors at all related to credentials. Just the 10 minute timeout error after git merge, and then a lfs smudge error + backtrace. When this happens it's really bad as it halts merging of all PRs since Jenkins is now foobared.

The only way to "fix" it is to create a new PR that modifies the LFS file in the backtrace and merge it to master. Then, if you rebase all PRs, they will no longer fail with the LFS/merge issue.

New: Sometimes when an LFS file change is merged to master, _all_ PR CI jobs will start failing on the merge step with:

{noformat}
Started by user Jupiter CI
Restarted from build #11, stage Build Terminal
Obtained Jenkinsfile from 5033c6390fdbbfa53f57975da66b0d55282353a3+bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] milestone
Trying to pass milestone 11
[Pipeline] milestone
Trying to pass milestone 12
[Pipeline] stage
[Pipeline] { (Build Terminal)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/jupiter_PR-99
[Pipeline]
{ [Pipeline] checkout using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
> git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from 2 remote Git repositories
> git config remote.origin.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10 Resetting working tree
> git reset --hard # timeout=10 > git clean -fdx # timeout=10
Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
> git --version # timeout=10 using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/pull-requests/99/from:refs/remotes/origin/PR-99
> git config remote.upstream.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10 Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10 Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/heads/master:refs/remotes/upstream/master Merging remotes/upstream/master commit bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 into PR head commit 5033c6390fdbbfa53f57975da66b0d55282353a3 Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin
> git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 # timeout=10
ERROR: Timeout after 10 minutes
Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin [Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
[Bitbucket] Notifying pull request build result
[Bitbucket] Build result notified
hudson.plugins.git.GitException: Command "git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667" returned status code 128:
stdout:
stderr: Downloading apps/dashboard.tar.gz (3.8 MB)
Terminated
error: external filter 'git-lfs filter-process' failed
fatal: apps/dashboard.tar.gz: smudge filter lfs failed
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:841)
at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1100)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1193)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
{noformat}

This only seems to happen on PR jobs where it tries to merge the target branch first before building. And even then, only sometimes. Normal branch jobs don't encounter this problem.

The problem is, I am having a hard time reproducing this behavior outside of Jenkins. It seems like Jenkins is lying about which git commands it is actually running, because if I attach to the docker container running Jenkins, cd to the PR workspace folder, and then run the git commands shown in the log one by one, they all run without issue until I get to the {{git merge}} command at which point it complains about user and email not being configured. If I configure a user/email, then it complains about there not being an editor. But this works _most_ of the time, so I'm wondering if it is secretly running {{git merge --no-edit}} or something. Oh, and if I do {{git merge --no-edit}} (by the way) it runs fine without error.

This seems similar to

https://issues.jenkins-ci.org/browse/JENKINS-50321

However, I get no errors at all related to credentials. Just the 10 minute timeout error after git merge, and then a lfs smudge error + backtrace. When this happens it's really bad as it halts merging of all PRs since Jenkins is now foobared.

The only way to "fix" it is to create a new PR that modifies the LFS file in the backtrace and merge it to master. Then, if you rebase all PRs, they will no longer fail with the LFS/merge issue.

Bryan Gillespie made changes - 2020-01-06 18:00

Environment

Original: Jenkins version: 2.190.1
Plugin version: 2.5.0
Bitbucket server version: 6.1.2
Git LFS version: git-lfs/2.8.0 (GitHub; linux amd64; go 1.12.6)
Git version: 2.11.0

Jenkins is running inside a docker container.

New: Jenkins version: 2.190.1
Bitbucket Source Branch Plugin version: 2.5.0
Git Plugin: 3.12.1
Git client Plugin: 2.8.6
Bitbucket server version: 6.1.2
Git LFS version: git-lfs/2.8.0 (GitHub; linux amd64; go 1.12.6)
Git version: 2.11.0

Jenkins is running inside a docker container.

Mark Waite made changes - 2020-01-07 00:17

Link

New: This issue duplicates ~~JENKINS-45228~~ [ ~~JENKINS-45228~~ ]

Mark Waite made changes - 2020-01-07 00:17

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Fixed but Unreleased [ 10203 ]

Mark Waite made changes - 2020-01-07 00:17

Status

Original: Fixed but Unreleased [ 10203 ]

New: Closed [ 6 ]

Assignee:: Mark Waite

Reporter:: Bryan Gillespie

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2020-01-06 17:53

Updated:: 2021-02-04 15:56

Resolved:: 2020-01-07 00:17

Jenkins

Git LFS sometimes fails on when git merge is involved

[JENKINS-60650] Git LFS sometimes fails on when git merge is involved

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates

[JENKINS-60650] Git LFS sometimes fails on when git merge is involved