Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68555

LFS not initialized prior to checkout, at least in multibranch pipeline

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • Jenkins 2.347
      Git plugin 4.11.2

      I have a multibranch pipeline project that uses LFS.

      LFS pull is enabled in the project configuration, and LFS objects are fetched from the server, however "git lfs install" is never called in the workspace, so LFS pointers are never replaced by the actual files.

      Searching for this problem suggests that this is a known problem that people know to work around with a pre SCM step, but in a declarative multibranch pipeline, there is no way to configure this.

      I can work around the issue myself with additional build steps

      sh 'git lfs install'
      sh 'git lfs checkout
      

      but obviously that is not a proper solution.

          [JENKINS-68555] LFS not initialized prior to checkout, at least in multibranch pipeline

          Simon Richter added a comment -

          Addendum: `git lfs install` is not idempotent, `git lfs update --force` might be better.

          Simon Richter added a comment - Addendum: `git lfs install` is not idempotent, `git lfs update --force` might be better.

          Mark Waite added a comment - - edited

          I can't duplicate the problem you're describing. Steps I took:

          1. Declare a multibranch Pipeline with a GitHub branch source, a credential, and using my repository https://github.com/MarkEWaite/docker-lfs
          2. Add the "Git LFS pull after checkout" behavior to the multibranch Pipeline definition
          3. Define a Jenkinsfile in the repository for one of the branches with a declarative Pipeline that performs a single shell step to list the contents of a directory with LFS files (relies on the declarative Pipeline default checkout)
          4. Watch the log file and confirm that git lfs pull origin is included in the console log
          5. Confirm that the content of the LFS files is as expected

          Are you missing the "Git LFS pull after checkout" behavior in your multibranch Pipeline definition?

          Mark Waite added a comment - - edited I can't duplicate the problem you're describing. Steps I took: Declare a multibranch Pipeline with a GitHub branch source, a credential, and using my repository https://github.com/MarkEWaite/docker-lfs Add the "Git LFS pull after checkout" behavior to the multibranch Pipeline definition Define a Jenkinsfile in the repository for one of the branches with a declarative Pipeline that performs a single shell step to list the contents of a directory with LFS files (relies on the declarative Pipeline default checkout) Watch the log file and confirm that git lfs pull origin is included in the console log Confirm that the content of the LFS files is as expected Are you missing the "Git LFS pull after checkout" behavior in your multibranch Pipeline definition?

          Simon Richter added a comment -

          I've checked again, the "GIT LFS pull after checkout" behavior is configured, and `git lfs pull origin` is called, but the files in the workspace contain pointers instead of the expected content.

          This seems to happen on newly deployed agents that do not have the lfs filters configured in the global git config yet, which happens the first time `git lfs install` is called in any workspace.

          I'm not sure who is responsible for doing that on new agents in the first place, and I'm also wondering if this should be `git lfs install --local` to avoid modifying global agent config.

          Simon Richter added a comment - I've checked again, the "GIT LFS pull after checkout" behavior is configured, and `git lfs pull origin` is called, but the files in the workspace contain pointers instead of the expected content. This seems to happen on newly deployed agents that do not have the lfs filters configured in the global git config yet, which happens the first time `git lfs install` is called in any workspace. I'm not sure who is responsible for doing that on new agents in the first place, and I'm also wondering if this should be `git lfs install --local` to avoid modifying global agent config.

          Mark Waite added a comment -

          Thanks for the clarification. You identified the difference in our environments very nicely. My agents are all configured to run git lfs install before the agent is available to perform any operations. I've been treating git lfs install as part of the required configuration when installing git lfs. However, that is not user-friendly.

          Your first observations in the issue report were exactly right. The git plugin should detect if git lfs install has not been called and should call it to initialize. It should not rely on the user to perform initialization of the tool when it could perform

          In the interim, you can probably work around the issue on Unix based agents by using the label verifier plugin to run the git lfs install command before the agent is online. I've not had success personally with the git lfs install --local in my attempts, but it has been a while since I last attempted it.

          I run the following Pipeline job periodically in order to check that git lfs install has been performed on all my agents:

          def nodes = nodesByLabel label: 'git-lfs'
          
          Map tasks = [:]
          
          for (int i = 0; i < nodes.size(); i++) {
              def label = nodes[i]
              def stageName = 'Git LFS on ' + label
              tasks[label] = {
                  node(label) {
                      stage(stageName) {
                          def output
                          if (isUnix()) {
                              output =  sh(script: 'git --version ; git lfs install --force ; git lfs version', returnStdout: true)
                          } else {
                              output = bat(script: 'git --version & git lfs install --force & git lfs version', returnStdout: true)
                          }
                          if (output.indexOf('Git LFS initialized.') == -1) {
                              currentBuild.result = 'UNSTABLE'
                              addWarningBadge(text: "No git LFS on ${stageName}", id: "warning-badge-id-${label}")
                              echo "Output from ${stageName} did not contain 'Git LFS initialized.':\n${output}"
                          }
                      }
                  }
              }
          }
          
          timeout(time: 29, unit: 'MINUTES') {
              parallel(tasks)
          }
          

          Mark Waite added a comment - Thanks for the clarification. You identified the difference in our environments very nicely. My agents are all configured to run git lfs install before the agent is available to perform any operations. I've been treating git lfs install as part of the required configuration when installing git lfs . However, that is not user-friendly. Your first observations in the issue report were exactly right. The git plugin should detect if git lfs install has not been called and should call it to initialize. It should not rely on the user to perform initialization of the tool when it could perform In the interim, you can probably work around the issue on Unix based agents by using the label verifier plugin to run the git lfs install command before the agent is online. I've not had success personally with the git lfs install --local in my attempts, but it has been a while since I last attempted it. I run the following Pipeline job periodically in order to check that git lfs install has been performed on all my agents: def nodes = nodesByLabel label: 'git-lfs' Map tasks = [:] for (int i = 0; i < nodes.size(); i++) { def label = nodes[i] def stageName = 'Git LFS on ' + label tasks[label] = { node(label) { stage(stageName) { def output if (isUnix()) { output = sh(script: 'git --version ; git lfs install --force ; git lfs version', returnStdout: true) } else { output = bat(script: 'git --version & git lfs install --force & git lfs version', returnStdout: true) } if (output.indexOf('Git LFS initialized.') == -1) { currentBuild.result = 'UNSTABLE' addWarningBadge(text: "No git LFS on ${stageName}", id: "warning-badge-id-${label}") echo "Output from ${stageName} did not contain 'Git LFS initialized.':\n${output}" } } } } } timeout(time: 29, unit: 'MINUTES') { parallel(tasks) }

            Unassigned Unassigned
            simonrichter Simon Richter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: