Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62382

Checkout scm hangs or fails on private LFS repos without GitLFSPull extension

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Jenkins 2.222.3
      Git plugin 4.2.1
      Git Client 3.0.0
      Git cli 2.26.2
      git-lfs/2.10.0 (GitHub; linux amd64; go 1.12.7; git a526ba6b)

      Checkout scm will fail or hang in repositories that

      • Contain LFS files
      • Are non-public

       

      The following repository will exhibit the issue when made private: https://github.com/res0nance/git-lfs-test

      With the error:

       > git config core.sparsecheckout # timeout=10
       > git checkout -f c146a3ca4fb0380e0e336a850a5a695b669cac3d # timeout=10
      
      GitHub has been notified of this commit’s build result
      
      hudson.plugins.git.GitException: Command "git checkout -f c146a3ca4fb0380e0e336a850a5a695b669cac3d" returned status code 128:
      stdout: 
      stderr: Downloading AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe (378 KB)
      Error downloading object: AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe (4e7d17e): Smudge error: Error downloading AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe (4e7d17e18251183ddb766620b7e1f04cf9a979184e88ddc0dba8d288c1dc3522): batch response: Bad credentials
      
      Errors logged to /mnt/data/jenkins/workspace/s0nance_git-lfs-test-priv_master/.git/lfs/logs/20200521T063801.838088036.log
      Use `git lfs logs last` to view the log.
      error: external filter 'git-lfs filter-process' failed
      fatal: AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe: smudge filter lfs failed
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2372)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1000(CliGitAPIImpl.java:80)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2683)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (common-env) - ForgeCI-Linux-Agent (i-0bb64dada62d9c488)
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      		at hudson.remoting.Channel.call(Channel.java:998)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
      		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      		at java.lang.reflect.Method.invoke(Method.java:498)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
      		at com.sun.proxy.$Proxy120.execute(Unknown Source)
      		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1217)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      Caused: hudson.plugins.git.GitException: Could not checkout c146a3ca4fb0380e0e336a850a5a695b669cac3d
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2707)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

       

      The checkout scm command seems to fail because of the lack of credentials when running the checkout with the smudge filter enabled. https://github.com/jenkinsci/git-client-plugin/blob/eeec334af0b6447f3db9fb88d55728911a092d73/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2735

       

      I confirmed it manually by recompiling the git-client plugin so that it disables the smudge filter unconditionally that this results in checkout completing but the LFS file being in its "pointer" state a 131 byte file.

       

      This is not an issue on a significantly older version git 2.14.

          [JENKINS-62382] Checkout scm hangs or fails on private LFS repos without GitLFSPull extension

          Raihaan Shouhell created issue -

          Mark Waite added a comment - - edited

          I have a private GitHub LFS repository (docker-private-lfs) that I use very often from a Jenkins job. I just confirmed the the git lfs works on my Debian testing machine with CLI git 2.26.2 and git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.8).

          Does the job have the GitLFSPull extension enabled? When the GitLFSPull extension is enabled, the git checkout should not need to authenticate because the necessary large files are already available locally. Early implementations of the smudge filter were slow, so the git plugin chooses to call git lfs pull directly. If there is not a call to git lfs pull in the job log, then that may indicate some other problem.

          Mark Waite added a comment - - edited I have a private GitHub LFS repository ( docker-private-lfs ) that I use very often from a Jenkins job. I just confirmed the the git lfs works on my Debian testing machine with CLI git 2.26.2 and git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.8). Does the job have the GitLFSPull extension enabled? When the GitLFSPull extension is enabled, the git checkout should not need to authenticate because the necessary large files are already available locally. Early implementations of the smudge filter were slow, so the git plugin chooses to call git lfs pull directly. If there is not a call to git lfs pull in the job log, then that may indicate some other problem.
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]

          No, it has it without the LFS extension enabled. With older git clients( At least i believe the only difference is an older git client since some of my agents are very old ) it does not exhibit this hanging behaviour in my internal private github enterprise instance.

          On public github with private repositories checkout scm will simply throw a bad credentials error without the lfs extension enabled.

          Perhaps checkout should be enhanced with credentials so that it might retrieve LFS blobs as well?

          Raihaan Shouhell added a comment - No, it has it without the LFS extension enabled. With older git clients( At least i believe the only difference is an older git client since some of my agents are very old ) it does not exhibit this hanging behaviour in my internal private github enterprise instance. On public github with private repositories checkout scm will simply throw a bad credentials error without the lfs extension enabled. Perhaps checkout should be enhanced with credentials so that it might retrieve LFS blobs as well?

          Mark Waite added a comment -

          Yes, checkout could be enhanced to use credentials, just as merge has been enhanced to use credentials. Merge had to be enhanced with credentials because git lfs pull can't be used reasonably in a merge environment. It can be used in the checkout step.

          Adding credentials to checkout increases the complexity of the code. It would need to wait for someone to implement it, test it, and release it. If you add the GitLFSPull extension to your job definition, you don't need to wait for those things.

          Mark Waite added a comment - Yes, checkout could be enhanced to use credentials, just as merge has been enhanced to use credentials. Merge had to be enhanced with credentials because git lfs pull can't be used reasonably in a merge environment. It can be used in the checkout step. Adding credentials to checkout increases the complexity of the code. It would need to wait for someone to implement it, test it, and release it. If you add the GitLFSPull extension to your job definition, you don't need to wait for those things.
          Mark Waite made changes -
          Summary Original: Checkout scm hanging on LFS repositories New: Checkout scm hanging on LFS repositories without GitLFSPull extension
          Mark Waite made changes -
          Summary Original: Checkout scm hanging on LFS repositories without GitLFSPull extension New: Checkout scm hangs on private LFS repos without GitLFSPull extension
          Mark Waite made changes -
          Summary Original: Checkout scm hangs on private LFS repos without GitLFSPull extension New: Checkout scm hangs or fails on private LFS repos without GitLFSPull extension
          Raihaan Shouhell made changes -
          Remote Link New: This issue links to "git-client-plugin PR-551 (Web Link)" [ 24930 ]

          Yes adding the LFSPull extension does solve this issue. We only recently ran into this issue the older git clients didn't seem to exhibit this behaviour. I've added a draft PR which fixes the issue in my environment, for your review and feedback if the approach is right.

          Raihaan Shouhell added a comment - Yes adding the LFSPull extension does solve this issue. We only recently ran into this issue the older git clients didn't seem to exhibit this behaviour. I've added a draft PR which fixes the issue in my environment, for your review and feedback if the approach is right.

            Unassigned Unassigned
            raihaan Raihaan Shouhell
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: