Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60650

Git LFS *sometimes* fails on when git merge is involved

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • git-client-plugin

      Sometimes when an LFS file change is merged to master, all PR CI jobs will start failing on the merge step with:

       

       

      Started by user Jupiter CI
       Restarted from build #11, stage Build Terminal
       Obtained Jenkinsfile from 5033c6390fdbbfa53f57975da66b0d55282353a3+bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667
       Running in Durability level: MAX_SURVIVABILITY
       [Pipeline] Start of Pipeline
       [Pipeline] milestone
       Trying to pass milestone 11
       [Pipeline] milestone
       Trying to pass milestone 12
       [Pipeline] stage
       [Pipeline] { (Build Terminal)
       [Pipeline] node
       Running on Jenkins in /var/jenkins_home/workspace/jupiter_PR-99
       [Pipeline]
      { [Pipeline] checkout using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
      > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from 2 remote Git repositories
      > git config remote.origin.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
      Cleaning workspace
      > git rev-parse --verify HEAD # timeout=10 Resetting working tree
      > git reset --hard # timeout=10 > git clean -fdx # timeout=10
      Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
      > git --version # timeout=10 using GIT_ASKPASS to set credentials
      > git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/pull-requests/99/from:refs/remotes/origin/PR-99
      > git config remote.upstream.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
      Cleaning workspace
      > git rev-parse --verify HEAD # timeout=10 Resetting working tree
      > git reset --hard # timeout=10
      > git clean -fdx # timeout=10 Fetching without tags Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git using GIT_ASKPASS to set credentials
      > git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/heads/master:refs/remotes/upstream/master Merging remotes/upstream/master commit bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 into PR head commit 5033c6390fdbbfa53f57975da66b0d55282353a3 Enabling Git LFS pull
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
      > git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials
      > git lfs pull origin
      > git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 # timeout=10
      ERROR: Timeout after 10 minutes
      Enabling Git LFS pull
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
      > git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials
      > git lfs pull origin [Pipeline] }
      [Pipeline] // node
       [Pipeline] }
       [Pipeline] // stage
       [Pipeline] End of Pipeline
       [Bitbucket] Notifying pull request build result
       [Bitbucket] Build result notified
       hudson.plugins.git.GitException: Command "git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667" returned status code 128:
       stdout: 
       stderr: Downloading apps/dashboard.tar.gz (3.8 MB)
       Terminated
       error: external filter 'git-lfs filter-process' failed
       fatal: apps/dashboard.tar.gz: smudge filter lfs failed
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:841)
       at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
       at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1100)
       at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1193)
       at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
       at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
       at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
       at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Finished: FAILURE
      

       

      This only seems to happen on PR jobs where it tries to merge the target branch first before building. And even then, only sometimes. Normal branch jobs don't encounter this problem.

      The problem is, I am having a hard time reproducing this behavior outside of Jenkins. It seems like Jenkins is lying about which git commands it is actually running, because if I attach to the docker container running Jenkins, cd to the PR workspace folder, and then run the git commands shown in the log one by one, they all run without issue until I get to the git merge command at which point it complains about user and email not being configured. If I configure a user/email, then it complains about there not being an editor. But this works most of the time, so I'm wondering if it is secretly running git merge --no-edit or something. Oh, and if I do git merge --no-edit (by the way) it runs fine without error.

      This seems similar to

      https://issues.jenkins-ci.org/browse/JENKINS-50321

      However, I get no errors at all related to credentials. Just the 10 minute timeout error after git merge, and then a lfs smudge error + backtrace. When this happens it's really bad as it halts merging of all PRs since Jenkins is now foobared.

      The only way to "fix" it is to create a new PR that modifies the LFS file in the backtrace and merge it to master. Then, if you rebase all PRs, they will no longer fail with the LFS/merge issue.

          [JENKINS-60650] Git LFS *sometimes* fails on when git merge is involved

          Mark Waite added a comment - - edited

          Duplicate of JENKINS-45228.

          Resolved in git client plugin 3.1.0-beta that was released Jan 2, 2019. Please test it with git plugin 4.1.0-beta to confirm that it fixes your issue as well.

          Beta releases are available from the Jenkins experimental update center.

          The git lfs extension has extended the git merge command to silently require authentication when performing some of the merges. Same condition happens with git checkout when it is used in the typical command sequence for a user. The git lfs extension uses a "smudge filter" by default to pull the correct large files for the SHA-1 in the checkout. The git plugin implemented LFS support before the more polished versions of LFS were implemented, so it explicitly configures command line git to not pull large files, then performs a separate large file pull after the checkout has completed.

          Mark Waite added a comment - - edited Duplicate of JENKINS-45228 . Resolved in git client plugin 3.1.0-beta that was released Jan 2, 2019. Please test it with git plugin 4.1.0-beta to confirm that it fixes your issue as well. Beta releases are available from the Jenkins experimental update center . The git lfs extension has extended the git merge command to silently require authentication when performing some of the merges. Same condition happens with git checkout when it is used in the typical command sequence for a user. The git lfs extension uses a "smudge filter" by default to pull the correct large files for the SHA-1 in the checkout. The git plugin implemented LFS support before the more polished versions of LFS were implemented, so it explicitly configures command line git to not pull large files, then performs a separate large file pull after the checkout has completed.

          Bharath R added a comment - - edited

          markewaite I already have git client plugin version 3.6.0 installed, Git plugin 4.5.2 and still face the issue described as https://issues.jenkins.io/browse/JENKINS-45228
          Can you help me? Posting my comment from the earlier ticket here too:

          The only workaround is to rebase the PR every time, which is annoying. The problem also occurs on master node as As others point out, in pre-pipeline stage in a Bitbucket Multi branch pipeline.

          The problem started after we enabled LFS.
          I use GIT LFS Pull after checkout as an additional step in Jenkins configuration. Have tried 'Clean before/after checkout' and 'Wipeout repository and force clone' options too. None helps

          And I get the following error:
          stdout:
          stderr: error: Your local changes to the following files would be overwritten by merge: LFSfile.zip

          I have git client plugin version 3.6.0 installed, Git plugin 4.5.2.
          Is there something, I could be doing wrong? Any help would be appreciated.

          Bharath R added a comment - - edited markewaite I already have git client plugin version 3.6.0 installed, Git plugin 4.5.2 and still face the issue described as https://issues.jenkins.io/browse/JENKINS-45228 Can you help me? Posting my comment from the earlier ticket here too: The only workaround is to rebase the PR every time, which is annoying. The problem also occurs on master node as As others point out, in pre-pipeline stage in a Bitbucket Multi branch pipeline. The problem started after we enabled LFS. I use GIT LFS Pull after checkout as an additional step in Jenkins configuration. Have tried 'Clean before/after checkout' and 'Wipeout repository and force clone' options too. None helps And I get the following error: stdout: stderr: error: Your local changes to the following files would be overwritten by merge: LFSfile.zip I have git client plugin version 3.6.0 installed, Git plugin 4.5.2. Is there something, I could be doing wrong? Any help would be appreciated.

            markewaite Mark Waite
            bgillesp Bryan Gillespie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: