-
Bug
-
Resolution: Duplicate
-
Major
Sometimes when an LFS file change is merged to master, all PR CI jobs will start failing on the merge step with:
Started by user Jupiter CI
Restarted from build #11, stage Build Terminal
Obtained Jenkinsfile from 5033c6390fdbbfa53f57975da66b0d55282353a3+bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] milestone
Trying to pass milestone 11
[Pipeline] milestone
Trying to pass milestone 12
[Pipeline] stage
[Pipeline] { (Build Terminal)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/jupiter_PR-99
[Pipeline]
{ [Pipeline] checkout using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
> git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from 2 remote Git repositories
> git config remote.origin.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10 Resetting working tree
> git reset --hard # timeout=10 > git clean -fdx # timeout=10
Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
> git --version # timeout=10 using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/pull-requests/99/from:refs/remotes/origin/PR-99
> git config remote.upstream.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10 Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10 Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/heads/master:refs/remotes/upstream/master Merging remotes/upstream/master commit bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 into PR head commit 5033c6390fdbbfa53f57975da66b0d55282353a3 Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin
> git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 # timeout=10
ERROR: Timeout after 10 minutes
Enabling Git LFS pull
> git config core.sparsecheckout # timeout=10
> git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git lfs pull origin [Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
[Bitbucket] Notifying pull request build result
[Bitbucket] Build result notified
hudson.plugins.git.GitException: Command "git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667" returned status code 128:
stdout:
stderr: Downloading apps/dashboard.tar.gz (3.8 MB)
Terminated
error: external filter 'git-lfs filter-process' failed
fatal: apps/dashboard.tar.gz: smudge filter lfs failed
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:841)
at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1100)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1193)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
This only seems to happen on PR jobs where it tries to merge the target branch first before building. And even then, only sometimes. Normal branch jobs don't encounter this problem.
The problem is, I am having a hard time reproducing this behavior outside of Jenkins. It seems like Jenkins is lying about which git commands it is actually running, because if I attach to the docker container running Jenkins, cd to the PR workspace folder, and then run the git commands shown in the log one by one, they all run without issue until I get to the git merge command at which point it complains about user and email not being configured. If I configure a user/email, then it complains about there not being an editor. But this works most of the time, so I'm wondering if it is secretly running git merge --no-edit or something. Oh, and if I do git merge --no-edit (by the way) it runs fine without error.
This seems similar to
https://issues.jenkins-ci.org/browse/JENKINS-50321
However, I get no errors at all related to credentials. Just the 10 minute timeout error after git merge, and then a lfs smudge error + backtrace. When this happens it's really bad as it halts merging of all PRs since Jenkins is now foobared.
The only way to "fix" it is to create a new PR that modifies the LFS file in the backtrace and merge it to master. Then, if you rebase all PRs, they will no longer fail with the LFS/merge issue.
- duplicates
-
-
- Closed
-