Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62382

Checkout scm hangs or fails on private LFS repos without GitLFSPull extension

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Jenkins 2.222.3
      Git plugin 4.2.1
      Git Client 3.0.0
      Git cli 2.26.2
      git-lfs/2.10.0 (GitHub; linux amd64; go 1.12.7; git a526ba6b)

      Checkout scm will fail or hang in repositories that

      • Contain LFS files
      • Are non-public

       

      The following repository will exhibit the issue when made private: https://github.com/res0nance/git-lfs-test

      With the error:

       > git config core.sparsecheckout # timeout=10
       > git checkout -f c146a3ca4fb0380e0e336a850a5a695b669cac3d # timeout=10
      
      GitHub has been notified of this commit’s build result
      
      hudson.plugins.git.GitException: Command "git checkout -f c146a3ca4fb0380e0e336a850a5a695b669cac3d" returned status code 128:
      stdout: 
      stderr: Downloading AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe (378 KB)
      Error downloading object: AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe (4e7d17e): Smudge error: Error downloading AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe (4e7d17e18251183ddb766620b7e1f04cf9a979184e88ddc0dba8d288c1dc3522): batch response: Bad credentials
      
      Errors logged to /mnt/data/jenkins/workspace/s0nance_git-lfs-test-priv_master/.git/lfs/logs/20200521T063801.838088036.log
      Use `git lfs logs last` to view the log.
      error: external filter 'git-lfs filter-process' failed
      fatal: AutoCAD_LT_2018_English_Win_32_64bit_wi_en-us_Setup_webinstall.exe: smudge filter lfs failed
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2372)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1000(CliGitAPIImpl.java:80)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2683)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (common-env) - ForgeCI-Linux-Agent (i-0bb64dada62d9c488)
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      		at hudson.remoting.Channel.call(Channel.java:998)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
      		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      		at java.lang.reflect.Method.invoke(Method.java:498)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
      		at com.sun.proxy.$Proxy120.execute(Unknown Source)
      		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1217)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      Caused: hudson.plugins.git.GitException: Could not checkout c146a3ca4fb0380e0e336a850a5a695b669cac3d
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2707)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

       

      The checkout scm command seems to fail because of the lack of credentials when running the checkout with the smudge filter enabled. https://github.com/jenkinsci/git-client-plugin/blob/eeec334af0b6447f3db9fb88d55728911a092d73/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2735

       

      I confirmed it manually by recompiling the git-client plugin so that it disables the smudge filter unconditionally that this results in checkout completing but the LFS file being in its "pointer" state a 131 byte file.

       

      This is not an issue on a significantly older version git 2.14.

          [JENKINS-62382] Checkout scm hangs or fails on private LFS repos without GitLFSPull extension

          Mark Waite added a comment - - edited

          I have a private GitHub LFS repository (docker-private-lfs) that I use very often from a Jenkins job. I just confirmed the the git lfs works on my Debian testing machine with CLI git 2.26.2 and git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.8).

          Does the job have the GitLFSPull extension enabled? When the GitLFSPull extension is enabled, the git checkout should not need to authenticate because the necessary large files are already available locally. Early implementations of the smudge filter were slow, so the git plugin chooses to call git lfs pull directly. If there is not a call to git lfs pull in the job log, then that may indicate some other problem.

          Mark Waite added a comment - - edited I have a private GitHub LFS repository ( docker-private-lfs ) that I use very often from a Jenkins job. I just confirmed the the git lfs works on my Debian testing machine with CLI git 2.26.2 and git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.8). Does the job have the GitLFSPull extension enabled? When the GitLFSPull extension is enabled, the git checkout should not need to authenticate because the necessary large files are already available locally. Early implementations of the smudge filter were slow, so the git plugin chooses to call git lfs pull directly. If there is not a call to git lfs pull in the job log, then that may indicate some other problem.

          No, it has it without the LFS extension enabled. With older git clients( At least i believe the only difference is an older git client since some of my agents are very old ) it does not exhibit this hanging behaviour in my internal private github enterprise instance.

          On public github with private repositories checkout scm will simply throw a bad credentials error without the lfs extension enabled.

          Perhaps checkout should be enhanced with credentials so that it might retrieve LFS blobs as well?

          Raihaan Shouhell added a comment - No, it has it without the LFS extension enabled. With older git clients( At least i believe the only difference is an older git client since some of my agents are very old ) it does not exhibit this hanging behaviour in my internal private github enterprise instance. On public github with private repositories checkout scm will simply throw a bad credentials error without the lfs extension enabled. Perhaps checkout should be enhanced with credentials so that it might retrieve LFS blobs as well?

          Mark Waite added a comment -

          Yes, checkout could be enhanced to use credentials, just as merge has been enhanced to use credentials. Merge had to be enhanced with credentials because git lfs pull can't be used reasonably in a merge environment. It can be used in the checkout step.

          Adding credentials to checkout increases the complexity of the code. It would need to wait for someone to implement it, test it, and release it. If you add the GitLFSPull extension to your job definition, you don't need to wait for those things.

          Mark Waite added a comment - Yes, checkout could be enhanced to use credentials, just as merge has been enhanced to use credentials. Merge had to be enhanced with credentials because git lfs pull can't be used reasonably in a merge environment. It can be used in the checkout step. Adding credentials to checkout increases the complexity of the code. It would need to wait for someone to implement it, test it, and release it. If you add the GitLFSPull extension to your job definition, you don't need to wait for those things.

          Yes adding the LFSPull extension does solve this issue. We only recently ran into this issue the older git clients didn't seem to exhibit this behaviour. I've added a draft PR which fixes the issue in my environment, for your review and feedback if the approach is right.

          Raihaan Shouhell added a comment - Yes adding the LFSPull extension does solve this issue. We only recently ran into this issue the older git clients didn't seem to exhibit this behaviour. I've added a draft PR which fixes the issue in my environment, for your review and feedback if the approach is right.

          ieure added a comment -

          I also ran into this behavior; wild that this issue was reported >2 years ago and is still open.

          The LFS pull extension doesn't fix the issue for me, since it does LFS pull after checkout, but the checkout is what fails.

          I'm on Jenkins 2.375.2 and Git plugin 5.0.0.

          My Jenkinsfile had no SCM configuration, but I changed it to this to try adding the LFS pull (using the pipeline snippet generator):

              stages {
                  stage('Checkout SCM') {
                      steps {
                          checkout scmGit(extensions: [lfs()])
                      }
                  }
              }
          

          ieure added a comment - I also ran into this behavior; wild that this issue was reported >2 years ago and is still open. The LFS pull extension doesn't fix the issue for me, since it does LFS pull after checkout, but the checkout is what fails. I'm on Jenkins 2.375.2 and Git plugin 5.0.0. My Jenkinsfile had no SCM configuration, but I changed it to this to try adding the LFS pull (using the pipeline snippet generator):   stages {         stage( 'Checkout SCM' ) {             steps {                 checkout scmGit(extensions: [lfs()])             }         } }

          Mark Waite added a comment - - edited

          ieure, thanks for sharing your observations. You said:

          The LFS pull extension doesn't fix the issue for me, since it does LFS pull after checkout, but the checkout is what fails.

          My Jenkinsfile had no SCM configuration, but I changed it to this to try adding the LFS pull

          You're using declarative Pipeline. If you're using it within a job that is defined as part of a multibranch Pipeline folder, then declarative Pipeline has an implicit checkout step. The implicit checkout step uses the checkout scm settings that are defined in the multibranch Pipeline. Since you didn't mention it as a change, I assume that you missed adding the declarative Pipeline option to skip the default checkout.

          I'm able to duplicate the report from raihaan with CLI git 2.34 and git lfs 3.3.0 in the declarative Pipeline (not in a multibranch Pipeline):

          pipeline {
              agent any
              options {
                  skipDefaultCheckout true
              }
              stages {
                  stage('Checkout') {
                      steps {
                          deleteDir()
                          checkout scmGit(branches: [[name: 'lts-with-plugins-add-credentials-and-nodes']], 
                                          extensions: [ ],  // Intentionally empty for JENKINS-62382 check
                                          userRemoteConfigs: [
                                              [credentialsId: 'MarkEWaite-github-token-2019-05-25', 
                                               url: 'https://github.com/MarkEWaite/docker-private-lfs.git']])
                      }
                  }
              }
          }
          

          That fails with the message that it could not run the smudge filter when I use command line git 2.34 and git LFS 3.2.0 as installed on Ubuntu 22.04. That works and performs the LFS checkout when I use command line git 2.39.1 and git LFS 3.3.0 as installed on Windows 10. It fails with command line git 2.39.0 and git LFS 3.3.0 as installed on Debian Linux unstable and Debian Linux testing. In my case, the behavior (without enabling the lfs() extension) depends on the version of command line git that is installed.

          On my Ubuntu 22.04 agent with command line git 2.34 and git LFS 3.3.0, the failure message (without the lfs() extension) was:

          15:57:35  hudson.plugins.git.GitException: Command "/usr/bin/git checkout -f 4134b0d9bb6309afe2f9cb1e57b5da2175c8a3f6" returned status code 128:
          15:57:35  stdout: 
          15:57:35  stderr: Downloading ref/plugins/PrioritySorter.jpi (114 KB)
          15:57:35  Error downloading object: ref/plugins/PrioritySorter.jpi (3a6cea3): Smudge error: Error downloading ref/plugins/PrioritySorter.jpi (3a6cea359bda210ce98e2d39e8ba3d9a1da7e08a6cf88a50094ca5e26399c8d6): batch response: Bad credentials
          

          If I enable the lfs() extension, then all the configurations that support git LFS work with that declarative Pipeline.

          I didn't check any other versions of command line git, but I would expect that there is a wide range of behaviors, depending on the version of command line git that is installed. Command line git integration of the git LFS extensions has steadily improved with each release of command line git and each release of git LFS. Your choices include:

          • Add the lfs() extension when using a repository that has large file support enabled (the preferred solution, since it makes it clear that the repository uses LFS)
          • Update command line git to 2.39.1 or later and update git LFS to 3.3.0 or later and figure out what allows that to work on Windows but causes it to fail on Debian unstable

          In reply to your first comment:

          wild that this issue was reported >2 years ago and is still open.

          When an issue has an easy, direct solution (use the lfs() extension on repositories that have large file support enabled), I prefer to not make changes in the git plugin or the git client plugin. I've broken other users too many times when I thought that I was simplifying a default behavior.

          Mark Waite added a comment - - edited ieure , thanks for sharing your observations. You said: The LFS pull extension doesn't fix the issue for me, since it does LFS pull after checkout, but the checkout is what fails. My Jenkinsfile had no SCM configuration, but I changed it to this to try adding the LFS pull You're using declarative Pipeline. If you're using it within a job that is defined as part of a multibranch Pipeline folder, then declarative Pipeline has an implicit checkout step. The implicit checkout step uses the checkout scm settings that are defined in the multibranch Pipeline. Since you didn't mention it as a change, I assume that you missed adding the declarative Pipeline option to skip the default checkout. I'm able to duplicate the report from raihaan with CLI git 2.34 and git lfs 3.3.0 in the declarative Pipeline (not in a multibranch Pipeline): pipeline { agent any options { skipDefaultCheckout true } stages { stage('Checkout') { steps { deleteDir() checkout scmGit(branches: [[name: 'lts-with-plugins-add-credentials-and-nodes']], extensions: [ ], // Intentionally empty for JENKINS-62382 check userRemoteConfigs: [ [credentialsId: 'MarkEWaite-github-token-2019-05-25', url: 'https://github.com/MarkEWaite/docker-private-lfs.git']]) } } } } That fails with the message that it could not run the smudge filter when I use command line git 2.34 and git LFS 3.2.0 as installed on Ubuntu 22.04. That works and performs the LFS checkout when I use command line git 2.39.1 and git LFS 3.3.0 as installed on Windows 10. It fails with command line git 2.39.0 and git LFS 3.3.0 as installed on Debian Linux unstable and Debian Linux testing. In my case, the behavior (without enabling the lfs() extension) depends on the version of command line git that is installed. On my Ubuntu 22.04 agent with command line git 2.34 and git LFS 3.3.0, the failure message (without the lfs() extension) was: 15:57:35 hudson.plugins.git.GitException: Command "/usr/bin/git checkout -f 4134b0d9bb6309afe2f9cb1e57b5da2175c8a3f6" returned status code 128: 15:57:35 stdout: 15:57:35 stderr: Downloading ref/plugins/PrioritySorter.jpi (114 KB) 15:57:35 Error downloading object: ref/plugins/PrioritySorter.jpi (3a6cea3): Smudge error: Error downloading ref/plugins/PrioritySorter.jpi (3a6cea359bda210ce98e2d39e8ba3d9a1da7e08a6cf88a50094ca5e26399c8d6): batch response: Bad credentials If I enable the lfs() extension, then all the configurations that support git LFS work with that declarative Pipeline. I didn't check any other versions of command line git, but I would expect that there is a wide range of behaviors, depending on the version of command line git that is installed. Command line git integration of the git LFS extensions has steadily improved with each release of command line git and each release of git LFS. Your choices include: Add the lfs() extension when using a repository that has large file support enabled (the preferred solution, since it makes it clear that the repository uses LFS) Update command line git to 2.39.1 or later and update git LFS to 3.3.0 or later and figure out what allows that to work on Windows but causes it to fail on Debian unstable In reply to your first comment: wild that this issue was reported >2 years ago and is still open. When an issue has an easy, direct solution (use the lfs() extension on repositories that have large file support enabled), I prefer to not make changes in the git plugin or the git client plugin. I've broken other users too many times when I thought that I was simplifying a default behavior.

            Unassigned Unassigned
            raihaan Raihaan Shouhell
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: