Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43818

Git fetch fails when "Branch Specifier" uses a parameter in new version of the Git plugin

      In a pipeline job from a git repository where the "Branch Specifier" is given as a parameter, I started getting the following failure after updating the git plugin.

      hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/$\{GITREF}:refs/remotes/origin/$\{GITREF} --prune" returned status code 128:
      stdout: 
      stderr: fatal: Couldn't find remote ref refs/heads/$\{GITREF}
      
              at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1799)
              at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1525)
              at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:65)
              at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:316)
              at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:304)
              at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:196)
              at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:172)
              at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:99)
              at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:59)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:232)
              at hudson.model.ResourceController.execute(ResourceController.java:98)
              at hudson.model.Executor.run(Executor.java:404)
      Finished: FAILURE
      

      I have tested all released version of the git plugin from 3.0.1 to 3.2.0 and the failure seems to have appeared in 3.0.2.

      I don't know if it is relevant but with the Git plugin version 3.0.1, I see this at the beginning of the log:
      Lightweight checkout support not available, falling back to full checkout.
       

          [JENKINS-43818] Git fetch fails when "Branch Specifier" uses a parameter in new version of the Git plugin

          Mark Waite added a comment - - edited

          I'm unable to duplicate the problem you're reporting.  Please refer to my Jenkinsfile for the steps I'm using.

          Steps I took while trying to duplicate the problem:

          1. Copy JDK 8 tar file into my public_html/jdk directory
              $ mkdir -p ~/public_html/jdk/
              $ cp jdk-8u131-linux-*tar.gz ~/public_html/jdk/
            
          2. Copy the jenkins-bugs repo into a local bare repo for faster cloning later
              $ mkdir -p ~/git/bare/bugs/
              $ cd ~/git/bare/bugs/
              $ git clone --mirror https://github.com/MarkEWaite/jenkins-bugs
            
          3. Clone, build, and run the docker instance
              $ git lfs clone https://github.com/MarkEWaite/docker-lfs JENKINS-43818
              $ cd JENKINS-43818
              $ git lfs fetch origin origin/lts-with-plugins
              $ git checkout -b lts-with-plugins origin/lts-with-plugins
              $ docker build -t jenkins:JENKINS-43818 .
              $ VOL1=~/public_html:/var/jenkins_home/userContent/
              $ VOL2=~/git/bare:/var/lib/git/mwaite/
              $ docker run -i --rm --publish 8080:8080 --volume $VOL1 --volume $VOL2 jenkins:JENKINS-43687
            
          4. Connect a web browser to that docker instance (http://localhost:8080)
          5. Open the "Bugs - Pipeline Checks" folder
          6. Open the "jenkins-bugs" multi-branch pipeline job
          7. Click the "Scan Multibranch Pipeline" link and then the "Run Now" link to start branch indexing
          8. Confirm the JENKINS-43818 job runs successfully
          9. Confirm when I run the JENKINS-43818 job and give a parameter "master", it uses the master branch

          Mark Waite added a comment - - edited I'm unable to duplicate the problem you're reporting.  Please refer to my Jenkinsfile for the steps I'm using. Steps I took while trying to duplicate the problem: Copy JDK 8 tar file into my public_html/jdk directory $ mkdir -p ~/public_html/jdk/ $ cp jdk-8u131-linux-*tar.gz ~/public_html/jdk/ Copy the jenkins-bugs repo into a local bare repo for faster cloning later $ mkdir -p ~/git/bare/bugs/ $ cd ~/git/bare/bugs/ $ git clone --mirror https: //github.com/MarkEWaite/jenkins-bugs Clone, build, and run the docker instance $ git lfs clone https: //github.com/MarkEWaite/docker-lfs JENKINS-43818 $ cd JENKINS-43818 $ git lfs fetch origin origin/lts-with-plugins $ git checkout -b lts-with-plugins origin/lts-with-plugins $ docker build -t jenkins:JENKINS-43818 . $ VOL1=~/public_html:/ var /jenkins_home/userContent/ $ VOL2=~/git/bare:/ var /lib/git/mwaite/ $ docker run -i --rm --publish 8080:8080 --volume $VOL1 --volume $VOL2 jenkins:JENKINS-43687 Connect a web browser to that docker instance ( http://localhost:8080 ) Open the "Bugs - Pipeline Checks" folder Open the "jenkins-bugs" multi-branch pipeline job Click the "Scan Multibranch Pipeline" link and then the "Run Now" link to start branch indexing Confirm the JENKINS-43818  job runs successfully Confirm when I run the JENKINS-43818 job and give a parameter "master", it uses the master branch

          Thanks for looking at this. I am not using a multibranch pipeline but a simple pipeline job, where both the parameter and the git checkout are configured in the GUI, not in the jenkins file. I'll try to see if I can reproduce it in a docker image but I think that might already explain why the behavior is different in your experiment.

          Grégoire Détrez added a comment - Thanks for looking at this. I am not using a multibranch pipeline but a simple pipeline job, where both the parameter and the git checkout are configured in the GUI, not in the jenkins file. I'll try to see if I can reproduce it in a docker image but I think that might already explain why the behavior is different in your experiment.

          Denys Digtiar added a comment -

          gd13 Could you please give an example of git or checkout step that you use. Specifically, I am interested in what type of Groovy string you use in the `branch/branches` parameter. By default, snippet generator produces single quoted string which does not support string interpolation, therefore, the variable string you specify is used as is and not replaced by the variable value.

          Denys Digtiar added a comment - gd13 Could you please give an example of git or checkout step that you use. Specifically, I am interested in what type of Groovy string you use in the `branch/branches` parameter. By default, snippet generator produces single quoted string which does not support string interpolation, therefore, the variable string you specify is used as is and not replaced by the variable value.

          Grégoire Détrez added a comment - - edited

          duemir I don't use a checkout step, I configured both the git repo and the parameter in the "Configure" GUI. There GITREF is defined as a string parameter and in the "Branch Specifier" in the pipeline section, I simply wrote

          ${GITREF}
          

          Grégoire Détrez added a comment - - edited duemir I don't use a checkout step, I configured both the git repo and the parameter in the "Configure" GUI. There GITREF is defined as a string parameter and in the "Branch Specifier" in the pipeline section, I simply wrote ${GITREF}

          Grégoire Détrez added a comment - - edited

          I just noticed on the screen shot the "Lightweight checkout" checkbox, I just tried un-checking it and re-installing 3.1.0 and it seems to work. This can be an other indication that the problem comes from the lightweight checkout.

          In any case this makes the problem less critical for us as it doesn't block us from updating plugins.

          Grégoire Détrez added a comment - - edited I just noticed on the screen shot the "Lightweight checkout" checkbox, I just tried un-checking it and re-installing 3.1.0 and it seems to work. This can be an other indication that the problem comes from the lightweight checkout. In any case this makes the problem less critical for us as it doesn't block us from updating plugins.

          Denys Digtiar added a comment -

          gd13 I believe you found a correct way to resolve your issue since Lightweight checkout does not support parameter.

           

          Denys Digtiar added a comment - gd13 I believe you found a correct way to resolve your issue since Lightweight checkout does not support parameter.  

          Mark Waite added a comment -

          Special thanks to duemir for investigating more deeply and detecting that this is the expected behavior when using a lightweight checkout with pipeline.

          Mark Waite added a comment - Special thanks to duemir for investigating more deeply and detecting that this is the expected behavior when using a lightweight checkout with pipeline.

          Sam Duke added a comment -

          Is there any chance this could be looked at again with a view to changing Lightweight Checkout to be able to substitute in the variables?

          Our checkout is very slow so this really hurts us!

          Sam Duke added a comment - Is there any chance this could be looked at again with a view to changing Lightweight Checkout to be able to substitute in the variables? Our checkout is very slow so this really hurts us!

          Sam Duke added a comment -

          duemir Looks like Mark doesn't work on this project any more. How can I escalate this?

           

          Sam Duke added a comment - duemir Looks like Mark doesn't work on this project any more. How can I escalate this?  

          Mark Waite added a comment - - edited

          samskiter I do work on this project during my nights and weekends.

          I manage the list of issues I'm actively investigating by unassigning myself from an issue when I'm not actively investigating the issue. I'm not actively investigating this issue.

          You may be able to improve checkout performance using a reference repository or by narrowing the refspec of the repository clone or by using a shallow clone. Refer to "Git in the Large" for a presentation that I created to improve git performance with large git repositories in Jenkins.

          Mark Waite added a comment - - edited samskiter I do work on this project during my nights and weekends. I manage the list of issues I'm actively investigating by unassigning myself from an issue when I'm not actively investigating the issue. I'm not actively investigating this issue. You may be able to improve checkout performance using a reference repository or by narrowing the refspec of the repository clone or by using a shallow clone. Refer to " Git in the Large " for a presentation that I created to improve git performance with large git repositories in Jenkins.

          Sam Duke added a comment -

          Ah apologies, I must have misread your activity feed.

          We are looking into the reference repository option - is there any documentation on how this checkout method works? (does it use git alternates, for example?)

          We are already doing a depth=1 clone of a specific ref (a SHA passed from gerrit) so I don't see how we can narrow it any more. Even with that ur checkout times of the pipeline groovy script is 10 minutes currently (thanks to all the other checking out it needs to do and all our actual building happens on other machines so it really should just be a few seconds to grab the groovy file and execute it.

          It seems the Lightweight checkout is seriously stunted in its usability because of the limitation of not being able to expand variables. In fact in anything but the simplest setup ("please build this fixed branch / tag") I can't see how you would use it.

          Is there any process to escalate this ticket?

          Sam Duke added a comment - Ah apologies, I must have misread your activity feed. We are looking into the reference repository option - is there any documentation on how this checkout method works? (does it use git alternates, for example?) We are already doing a depth=1 clone of a specific ref (a SHA passed from gerrit) so I don't see how we can narrow it any more. Even with that ur checkout times of the pipeline groovy script is 10 minutes currently (thanks to all the other checking out it needs to do and all our actual building happens on other machines so it really should just be a few seconds to grab the groovy file and execute it. It seems the Lightweight checkout is seriously stunted in its usability because of the limitation of not being able to expand variables. In fact in anything but the simplest setup ("please build this fixed branch / tag") I can't see how you would use it. Is there any process to escalate this ticket?

          Mark Waite added a comment - - edited

          Sure, you could escalate through one of the organizations that provides commercial support like CloudBees. If you're a CloudBees customer, you can submit a support ticket.

          You could escalate it by implementing the desired capability including automated tests and submitting a pull request.

          You might also be able to get dramatically faster checkout by choosing a different git provider. If your source code is hosted on GitHub or Bitbucket, you can use the matching branch source provider plugin and it will use a REST API call to read the contents of the Jenkinsfile. I believe a similar technique is also used for Gitea.

          The prioritized changes that I've seen as most effective with a large repository with many branches that are mostly independent are:

          1. Use a reference repository to reduce network data transfer in git fetch
          2. Narrow the refspec when cloning the repository to only pull changes from the exact branch being built (reduce data transfer in git fetch)
          3. Shallow clone (depth=1) to only pull most recent changes (reduce data transfer in git fetch)

          Mark Waite added a comment - - edited Sure, you could escalate through one of the organizations that provides commercial support like CloudBees. If you're a CloudBees customer, you can submit a support ticket. You could escalate it by implementing the desired capability including automated tests and submitting a pull request. You might also be able to get dramatically faster checkout by choosing a different git provider. If your source code is hosted on GitHub or Bitbucket, you can use the matching branch source provider plugin and it will use a REST API call to read the contents of the Jenkinsfile. I believe a similar technique is also used for Gitea. The prioritized changes that I've seen as most effective with a large repository with many branches that are mostly independent are: Use a reference repository to reduce network data transfer in git fetch Narrow the refspec when cloning the repository to only pull changes from the exact branch being built (reduce data transfer in git fetch ) Shallow clone (depth=1) to only pull most recent changes (reduce data transfer in git fetch )

          Is the fix for this issue planned anytime soon or if the issue is not fixed how do we fix the the issue by applying this patch
          [^JENKINS-14276.patch]
           
          Any pointers greatly appreciated.

          Arun Subbaramu added a comment - Is the fix for this issue planned anytime soon or if the issue is not fixed how do we fix the the issue by applying this patch [^JENKINS-14276.patch]   Any pointers greatly appreciated.

          Mark Waite added a comment - - edited

          No fix is planned for this issue by me. I don't recognize your reference to JENKINS-14276.patch. Instructions for building the Git plugin are included in its CONTRIBUTING file. Instructions for building the git client plugin are included in its CONTRIBUTING file.

          After looking at the history of JENKINS-14276 and its related bugs, I doubt any patches from that bug or its related bugs will help with this issue. This issue is requesting parameter expansion for pipeline jobs which use lightweight checkout to obtain the Jenkinsfile. I doubt those changes from 2015 will help with lightweight checkout in pipelines.

          Mark Waite added a comment - - edited No fix is planned for this issue by me. I don't recognize your reference to JENKINS-14276 .patch. Instructions for building the Git plugin are included in its CONTRIBUTING file. Instructions for building the git client plugin are included in its CONTRIBUTING file. After looking at the history of JENKINS-14276 and its related bugs, I doubt any patches from that bug or its related bugs will help with this issue. This issue is requesting parameter expansion for pipeline jobs which use lightweight checkout to obtain the Jenkinsfile. I doubt those changes from 2015 will help with lightweight checkout in pipelines.

            Unassigned Unassigned
            gd13 Grégoire Détrez
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: