Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64000

Git tag setting ignored by repository fetch in multibranch project created by GitHub org folder

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Jenkins 2.249.2
      Git plugin 4.4.4

      As requested in allowSecondFetch ** documentation I am opening a ticket.

      After upgrading git plugin to version 4.4.4 (can't recall the previous version), my Jenkins job started to fail on the checkout step because it requires the version tags, defined as bellow:

      stage('Checkout') {
        retry(3) {
          scmvars = checkout([
            $class                           : 'GitSCM',
            branches                         : scm.branches,
            doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
            extensions                       : scm.extensions +
              [$class: 'CloneOption', noTags: false, reference: '~/git/service-sourcing/', shallow: false] +
              [$class: 'LocalBranch', localBranch: "**"] +
              [$class: 'UserIdentity', email: 'dev@nezasa.com', name: 'Jenkins CI'],
            userRemoteConfigs                : scm.userRemoteConfigs
          ])
        }
      }

       

      Before the update the job used to output this during checkout:
      **

      05:16:32   > git fetch --no-tags --force --progress -- git@github.com:nezasa/tb-service-sourcing.git +refs/heads/develop:refs/remotes/origin/develop +refs/heads/*:refs/remotes/origin/* # timeout=10
      05:16:35  Fetching without tags
      05:16:35   > git config remote.origin.url git@github.com:nezasa/tb-service-sourcing.git # timeout=10
      05:16:35   > git config --add remote.origin.fetch +refs/heads/develop:refs/remotes/origin/develop # timeout=10
      05:16:35   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
      05:16:35   > git config remote.origin.url git@github.com:nezasa/tb-service-sourcing.git # timeout=10
      05:16:35  Fetching upstream changes from git@github.com:nezasa/tb-service-sourcing.git
      05:16:35  using GIT_SSH to set credentials Github SSH Key for common libraries access
      05:16:35   > git fetch --tags --force --progress -- git@github.com:nezasa/tb-service-sourcing.git +refs/heads/develop:refs/remotes/origin/develop +refs/heads/*:refs/remotes/origin/* # timeout=10

       

      After the update the output is (Git is now Avoiding second fetch):
      **

      05:16:59   > git fetch --no-tags --force --progress -- git@github.com:nezasa/tb-service-sourcing.git +refs/heads/develop:refs/remotes/origin/develop +refs/heads/*:refs/remotes/origin/* # timeout=10
      05:17:01  Avoid second fetch
      05:17:01  Checking out Revision 68628022e4553cacb97377dfd35d54c3d3d3734f (develop)
      05:17:01   > git config remote.origin.url git@github.com:nezasa/tb-service-sourcing.git # timeout=10
      05:17:01   > git config --add remote.origin.fetch +refs/heads/develop:refs/remotes/origin/develop # timeout=10
      05:17:01   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
      05:17:01   > git config core.sparsecheckout # timeout=10
      05:17:01   > git checkout -f 68628022e4553cacb97377dfd35d54c3d3d3734f # timeout=10
      05:17:01   > git branch -a -v --no-abbrev # timeout=10
      05:17:02   > git checkout -b develop 68628022e4553cacb97377dfd35d54c3d3d3734f # timeout=10

       

      Adding allowSecondFetch flag and setting it to true on the global settings seems to fix the issue but I was expecting that the Advanced Clone Setting on the job would be enough.

          [JENKINS-64000] Git tag setting ignored by repository fetch in multibranch project created by GitHub org folder

          Mark Waite added a comment - - edited

          jose_pedro_sa I can't reliably duplicate the problem. I've seen a single failure in the tests I've run. That failure was the 6th run of a multibranch pipeline that used GitHub as the provider. Builds 1-5 passed. Builds 7 and beyond passed. Only build 6 failed and reported that the --no-tags argument was used in the git fetch. I don't know what was different about that build.

          No other multi-branch pipeline failed. The job with the git plugin as provider passed. The job using JGit as the implementation passed. Various permutations of job settings all passed.

          Refer to my job definition that I used in my attempt to duplicate the problem. That attempt automatically ran the job in several different configurations of multi-branch pipeline.

          Please provide additional details that will allow others to duplicate the issue.

          Some of the questions to be answered include:

          • Version of command line git you're using
          • Operating system on the Jenkins controller
          • Operating system on the Jenkins agent
          • Context where the Pipeline is run (is it a Pipeline job with script entered directly in the Jenkins UI, a Pipeline job with script from SCM, a muilti-branch pipeline job, or a job created by an organization folder)
          • Which scm.extensions are enabled in the multi-branch pipeline

          Mark Waite added a comment - - edited jose_pedro_sa I can't reliably duplicate the problem. I've seen a single failure in the tests I've run. That failure was the 6th run of a multibranch pipeline that used GitHub as the provider. Builds 1-5 passed. Builds 7 and beyond passed. Only build 6 failed and reported that the --no-tags argument was used in the git fetch. I don't know what was different about that build. No other multi-branch pipeline failed. The job with the git plugin as provider passed. The job using JGit as the implementation passed. Various permutations of job settings all passed. Refer to my job definition that I used in my attempt to duplicate the problem. That attempt automatically ran the job in several different configurations of multi-branch pipeline. Please provide additional details that will allow others to duplicate the issue. Some of the questions to be answered include: Version of command line git you're using Operating system on the Jenkins controller Operating system on the Jenkins agent Context where the Pipeline is run (is it a Pipeline job with script entered directly in the Jenkins UI, a Pipeline job with script from SCM, a muilti-branch pipeline job, or a job created by an organization folder) Which scm.extensions are enabled in the multi-branch pipeline

          Jose Sa added a comment - - edited

          We have Jenkinsdeployed into a Kubernetes cluster using helm and the official Jenkins chart.

          • Version of command line git you're using: git version 2.20.1
          • Operating system on the Jenkins controller:  Linux/UNIX (Amazon Linux 2 image)
          • Operating system on the Jenkins agent:  Linux/UNIX (Amazon Linux 2 image) - "docker image jenkins/inbound-agent:4.3-4"
          • Context where the Pipeline is run (is it a Pipeline job with script entered directly in the Jenkins UI, a Pipeline job with script from SCM, a muilti-branch pipeline job, or a job created by an organization folder): github organization folder, configured with multibranch pipeline
          • Which scm.extensions are enabled in the multi-branch pipeline: CloneOption, LocalBranch, UserIdentity

          This might also be useful:

          Jose Sa added a comment - - edited We have Jenkinsdeployed into a Kubernetes cluster using helm and the official Jenkins chart. Version of command line git you're using: git version   2.20.1 Operating system on the Jenkins controller:   Linux/UNIX (Amazon Linux 2 image) Operating system on the Jenkins agent:   Linux/UNIX   (Amazon Linux 2 image) - "docker image jenkins/inbound-agent:4.3-4" Context where the Pipeline is run (is it a Pipeline job with script entered directly in the Jenkins UI, a Pipeline job with script from SCM, a muilti-branch pipeline job, or a job created by an organization folder): github organization folder, configured with multibranch pipeline Which scm.extensions are enabled in the multi-branch pipeline:  CloneOption, LocalBranch, UserIdentity This might also be useful:

          Mark Waite added a comment - - edited

          Thanks jose_pedro_sa. I have a GitHub organization folder with a multibranch pipeline that is showing the issue consistently.

          Your near term alternatives include:

          • Use a multibranch project created separate from the GitHub organization folder
          • Retain the second fetch globally so that the second fetch provides tags even though the first fetch incorrectly does not provide tags
          • Define the organization folder so that it retrieves tags. That should then be used by the multibranch projects

          It appears that a difference in tag retrieval configuration between the first and second fetch is not being detected by the code that removes the (potentially) redundant fetch.

          Problem is also only visible on the initial creation of a workspace. Later updates of an existing workspace do not have the issue. Since most of us are moving to ephemeral agents, it will be more and more common that the job is always performing initial creation of a workspace.

          I'm also able to consistently show the failure in a multibranch project that uses the GitHub provider and is configured to not fetch tags in the multibranch definition. My two cases that consistently pass are using the git provider and have tag fetching enabled. I suspect the issue is that the setting inside the job definition to fetch tags is being ignored or overridden on the first fetch and not ignored on the second fetch.

          Mark Waite added a comment - - edited Thanks jose_pedro_sa . I have a GitHub organization folder with a multibranch pipeline that is showing the issue consistently. Your near term alternatives include: Use a multibranch project created separate from the GitHub organization folder Retain the second fetch globally so that the second fetch provides tags even though the first fetch incorrectly does not provide tags Define the organization folder so that it retrieves tags. That should then be used by the multibranch projects It appears that a difference in tag retrieval configuration between the first and second fetch is not being detected by the code that removes the (potentially) redundant fetch. Problem is also only visible on the initial creation of a workspace. Later updates of an existing workspace do not have the issue. Since most of us are moving to ephemeral agents, it will be more and more common that the job is always performing initial creation of a workspace. I'm also able to consistently show the failure in a multibranch project that uses the GitHub provider and is configured to not fetch tags in the multibranch definition. My two cases that consistently pass are using the git provider and have tag fetching enabled. I suspect the issue is that the setting inside the job definition to fetch tags is being ignored or overridden on the first fetch and not ignored on the second fetch.

          Jose Sa added a comment - - edited

          Thank you for the feedback markewaite, setting the flag allowSecondFetch to true on global settings solved our problem*.*

          Jose Sa added a comment - - edited Thank you for the feedback markewaite , setting the flag  allowSecondFetch to true on global settings solved our problem*.*

          I'm also seeing this issue. It occurs with Bitbucket Team/Project configuration, which I guess shares code with GitHub Organization support.

          Problem is also only visible on the initial creation of a workspace. Later updates of an existing workspace do not have the issue.

          This also applies to my case.

          About this I'm not sure:

          • Define the organization folder so that it retrieves tags. That should then be used by the multibranch projects

          I have enabled Discover Tags option in Bitbucket Team/Project configs. But, unless I'm wrong, that only allows to trigger jobs based on pushed tags - it doesn't really relate to checking out the repository (via Git plugin).

          I assume there hasn't been any work related to this issue? Happy to be wrong!

          Where is allowSecondFetch documented? Googling returns nothing relevant (sure, some code references, but nothing that documents/explains it).

          Tuukka Mustonen added a comment - I'm also seeing this issue. It occurs with Bitbucket Team/Project configuration, which I guess shares code with GitHub Organization support. Problem is also only visible on the initial creation of a workspace. Later updates of an existing workspace do not have the issue. This also applies to my case. About this I'm not sure: Define the organization folder so that it retrieves tags. That should then be used by the multibranch projects I have enabled Discover Tags option in Bitbucket Team/Project configs. But, unless I'm wrong, that only allows to trigger jobs based on pushed tags - it doesn't really relate to checking out the repository (via Git plugin). I assume there hasn't been any work related to this issue? Happy to be wrong! Where is  allowSecondFetch  documented? Googling returns nothing relevant (sure, some code references, but nothing that documents/explains it).

          Mark Waite added a comment -

          tuukkamustonen The "Preserve second fetch during checkout" is documented in the global configuration section of the git plugin documentation.

          I've not done any work on this.

          Mark Waite added a comment - tuukkamustonen The "Preserve second fetch during checkout" is documented in the global configuration section of the git plugin documentation. I've not done any work on this.

          Nice. I'll enable it, too. Luckily, the bug isn't that critical as there's a workaround. Thanks! 

          Tuukka Mustonen added a comment - Nice. I'll enable it, too. Luckily, the bug isn't that critical as there's a workaround. Thanks! 

          David Navrkal added a comment - - edited

          Hello all, I guess I have found root cause of the issue, which is object of class jenkins.plugins.git.GitSCMSourceDefaults which is by default added by multi-branch pipeline and has in all cases, except when building tag, set:

          GitSCMSourceDefaults{includeTags=false}
          

          Interesting stuff is event if noTags is set to false in CloneOption like following

          [$class: 'CloneOption', noTags: false]
          

          the GitSCMSourceDefaults seems to be taking precedence.

          The ugly workaround is to manually remove GitSCMSourceDefaults from scm.extensions:

          stage('Checkout sources') {
              steps {
                  // Debug print
                  echo scm.dump()
                  script {
                      def customScm = [
                          $class: 'GitSCM',
                          branches: scm.branches,
                          extensions: scm.extensions.findAll{!(it instanceof jenkins.plugins.git.GitSCMSourceDefaults)} + [ // <---- this is the fix
                              [$class: 'CleanBeforeCheckout'],
                              [$class: 'AuthorInChangelog'],
                              [$class: 'CloneOption', noTags: false, reference: '', shallow: false, honorRefspec: false],
                              [$class: 'LocalBranch'],
                          ],
                          doGenerateSubmoduleConfigurations: false,
                          submoduleCfg: [],
                          userRemoteConfigs: scm.userRemoteConfigs
                      ]
                      // debug print
                      echo customScm.dump()
                      checkout(customScm)
                  }
              }
          }
          

          However I believe the real fix would be to change implementation of GitSCM to take precedence of CloneOption over GitSCMSourceDefaults.

          David Navrkal added a comment - - edited Hello all, I guess I have found root cause of the issue, which is object of class jenkins.plugins.git.GitSCMSourceDefaults which is by default added by multi-branch pipeline and has in all cases, except when building tag, set: GitSCMSourceDefaults{includeTags= false } Interesting stuff is event if noTags is set to false in CloneOption like following [$class: 'CloneOption' , noTags: false ] the GitSCMSourceDefaults seems to be taking precedence. The ugly workaround is to manually remove GitSCMSourceDefaults from scm.extensions : stage( 'Checkout sources' ) { steps { // Debug print echo scm.dump() script { def customScm = [ $class: 'GitSCM' , branches: scm.branches, extensions: scm.extensions.findAll{!(it instanceof jenkins.plugins.git.GitSCMSourceDefaults)} + [ // <---- this is the fix [$class: 'CleanBeforeCheckout' ], [$class: 'AuthorInChangelog' ], [$class: 'CloneOption' , noTags: false , reference: '', shallow: false , honorRefspec: false ], [$class: 'LocalBranch' ], ], doGenerateSubmoduleConfigurations: false , submoduleCfg: [], userRemoteConfigs: scm.userRemoteConfigs ] // debug print echo customScm.dump() checkout(customScm) } } } However I believe the real fix would be to change implementation of GitSCM to take precedence of CloneOption over GitSCMSourceDefaults .

          Alankrat added a comment -

          This helped me !!

          Alankrat added a comment - This helped me !!

          Brian added a comment - - edited

          navrkald you didn't link to the source where the multi-branch pipeline added GitSCMSourceDefaults. I was hit by this bug last week and did a lot of digging; the only use of GitSCMSourceDefaults I can find (in any project) is in the git plugin https://github.com/jenkinsci/git-plugin/blob/1a941e0fb78b3a57ec10248996174382a6cce0d3/src/main/java/jenkins/plugins/git/GitSCMBuilder.java#L498

           

                  for (GitSCMExtension e: extensions) {
                      if (e instanceof CloneOption) {
                          foundClone = true;
                          break;
                      }
                  }
                  if (!foundClone) {
                      // assume honour refspecs unless the clone option is added
                      // TODO revisit once we have support for TagSCMHead implemented as may need to check refspec handling then
                      extensions.add(new GitSCMSourceDefaults(head() instanceof TagSCMHead));
                  }

          The boolean parameter (head() instanceof TagSCMHead) is false unless head is an immutable tag, I believe.

           

          In our Jenkinsfile, we had code like:

                  def extraScmExtensions = [
                      ...
                      [
                          $class: 'CloneOption',
                          noTags: false
                      ],
                      ...
                  ]
                  checkout([
                           $class: 'GitSCM',
                           branches: scm.branches,
                           extensions: scm.extensions + extraScmExtensions,
                           userRemoteConfigs: scm.userRemoteConfigs
                   ]) 

          with `scm` being the SCM settings inherited from config.xml from our multibranch projects, which in turn inherit them from the org folder.  In our case, the CloneOption with noTags=false was missing from the Organization Folder, so the scm.extensions in anything created from it included GitSCMSourceDefaults. Adding CloneOption to the org folder caused a rescan of our github organisation, and quite some time later (we have 8500 branches in this instance) all of the configs in all of the projects had CloneOption set, and tags were fetched again.

          Hopefully it's clear from this why both adding CloneOption to parent projects or removing the GitSCMSourceDefaults from extensions in the Jenkinsfile work.

          And to connect up the google juice, the reason we need tags on checkout is because a lot of our projects use the `standard-version` node module. ( https://github.com/conventional-changelog/standard-version ). This attempts to discover what kind of version bump to apply by looking at "conventional commit" messages from HEAD back to the most recent tag. But when you check out with --no-tags, it doesn't find any stopping point, and builds a changelog going back to the beginning of time. If you ever had a breaking change in your history, this causes all builds to be seen as major version bumps; or if you have `feat:` commits even patches trigger minor version bumps. This was the symptom we saw - on a migration of our Jenkins instances to k8s the CloneOption at the org level had been missed, and suddenly all our builds had major version bumps.

           

          Brian added a comment - - edited navrkald you didn't link to the source where the multi-branch pipeline added GitSCMSourceDefaults. I was hit by this bug last week and did a lot of digging; the only use of GitSCMSourceDefaults I can find (in any project) is in the git plugin https://github.com/jenkinsci/git-plugin/blob/1a941e0fb78b3a57ec10248996174382a6cce0d3/src/main/java/jenkins/plugins/git/GitSCMBuilder.java#L498         for (GitSCMExtension e: extensions) {             if (e instanceof CloneOption) {                 foundClone = true ;                 break ;             }         }         if (!foundClone) {             // assume honour refspecs unless the clone option is added             // TODO revisit once we have support for TagSCMHead implemented as may need to check refspec handling then             extensions.add( new GitSCMSourceDefaults(head() instanceof TagSCMHead));         } The boolean parameter (head() instanceof TagSCMHead) is false unless head is an immutable tag, I believe.   In our Jenkinsfile, we had code like:       def extraScmExtensions = [ ...             [                 $class: 'CloneOption' ,                 noTags: false             ],             ...         ]       checkout([                  $class: 'GitSCM' ,                  branches: scm.branches,                  extensions: scm.extensions + extraScmExtensions,                  userRemoteConfigs: scm.userRemoteConfigs          ]) with `scm` being the SCM settings inherited from config.xml from our multibranch projects, which in turn inherit them from the org folder.  In our case, the CloneOption with noTags=false was missing from the Organization Folder , so the scm.extensions in anything created from it included GitSCMSourceDefaults. Adding CloneOption to the org folder caused a rescan of our github organisation, and quite some time later (we have 8500 branches in this instance) all of the configs in all of the projects had CloneOption set, and tags were fetched again. Hopefully it's clear from this why both adding CloneOption to parent projects or removing the GitSCMSourceDefaults from extensions in the Jenkinsfile work. And to connect up the google juice, the reason we need tags on checkout is because a lot of our projects use the `standard-version` node module. ( https://github.com/conventional-changelog/standard-version ). This attempts to discover what kind of version bump to apply by looking at "conventional commit" messages from HEAD back to the most recent tag. But when you check out with --no-tags, it doesn't find any stopping point, and builds a changelog going back to the beginning of time. If you ever had a breaking change in your history, this causes all builds to be seen as major version bumps; or if you have `feat:` commits even patches trigger minor version bumps. This was the symptom we saw - on a migration of our Jenkins instances to k8s the CloneOption at the org level had been missed, and suddenly all our builds had major version bumps.  

            Unassigned Unassigned
            jose_pedro_sa Jose Sa
            Votes:
            4 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: