Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34603

git scm poll not working with global environment variable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None

      Steps to reproduce -

      Manage Jenkins => Configure System => Global properties => Environment variables
      Add
      Name = GITHUB_HOST
      Value = <github IP address>

      Name = GITHUB_REPO
      Value = <github repo path>

      Create a job
      Under job configuration define git repository as below
      https://$

      {GITHUB_HOST}/${GITHUB_REPO}
      provide user/pass for credential
      complete the job configuration

      Trigger SCM poll by using curl (Note - provide your Jenkins host in following command)

      curl http://${JENKINS_SERVER}/jenkins/git/notifyCommit?url=[https://\$|https://%24/]"{GITHUB_HOST}"/\$"{GITHUB_REPO}"

      It'll poll the corresponding job but if you see Git SCM poll log, it's dumping the following error
      In short - It's not able to expand the global variable

      > /usr/bin/git --version # timeout=10
      > /usr/bin/git -c core.askpass=true fetch --tags --progress https://${GITHUB_HOST}

      /${GITHUB_REPO} +refs/heads/:refs/remotes/origin/
      FATAL: hudson.plugins.git.GitException: Failed to fetch from https://$

      {GITHUB_HOST}/${GITHUB_REPO}
      hudson.util.IOException2: hudson.plugins.git.GitException: Failed to fetch from https://${GITHUB_HOST}

      /${GITHUB_REPO}
      at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:573)
      at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:381)
      at hudson.scm.SCM.poll(SCM.java:398)
      at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1468)
      at hudson.model.AbstractProject._poll(AbstractProject.java:1438)
      at hudson.model.AbstractProject.poll(AbstractProject.java:1349)
      at jenkins.triggers.SCMTriggerItem$SCMTriggerItems$Bridge.poll(SCMTriggerItem.java:119)
      at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:526)
      at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:555)
      at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: hudson.plugins.git.GitException: Failed to fetch from https://$

      {GITHUB_HOST}/${GITHUB_REPO}
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:810)
      at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:701)
      at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:571)
      ... 14 more
      Caused by: hudson.plugins.git.GitException: Command "/usr/bin/git -c core.askpass=true fetch --tags --progress https://${GITHUB_HOST}

      /${GITHUB_REPO} +refs/heads/:refs/remotes/origin/" returned status code 128:
      stdout:
      stderr: remote: Invalid username or password.
       

          [JENKINS-34603] git scm poll not working with global environment variable

          smruti sahoo added a comment - - edited

          If I'm triggering the job manually either individually or build pipeline - It's working as expected
          It's failing only for "Poll SCM"

          But because of this the notification mails are not getting triggered upon git commit which is causing big issue

          smruti sahoo added a comment - - edited If I'm triggering the job manually either individually or build pipeline - It's working as expected It's failing only for "Poll SCM" But because of this the notification mails are not getting triggered upon git commit which is causing big issue

          smruti sahoo added a comment -

          smruti sahoo added a comment - A similar issue raised earlier https://issues.jenkins-ci.org/browse/JENKINS-26315

          Brian Barker added a comment -

          I also would like this addressed. When you have hundreds of engineers browsing build jobs, it is nice to set a var like $BRANCH=develop so they can just change that and not miss the multiple places in our config we use that value.

          Brian Barker added a comment - I also would like this addressed. When you have hundreds of engineers browsing build jobs, it is nice to set a var like $BRANCH=develop so they can just change that and not miss the multiple places in our config we use that value.

          I am also seeing this issue. Environment variables are not being substituted when polling runs. I am using an environment variable for my Branch Specifier in the SCM configuration.

          I have tried this in many versions of the Git Plugin. Version 3.0 is the latest, but version 2.3.4 is the last known working version that I have found.

          Jeffrey Mendez added a comment - I am also seeing this issue. Environment variables are not being substituted when polling runs. I am using an environment variable for my Branch Specifier in the SCM configuration. I have tried this in many versions of the Git Plugin. Version 3.0 is the latest, but version 2.3.4 is the last known working version that I have found.

          I've just updated a bunch of other plugins that affect Jenkins's security, but I left the Git plugin at 2.3.4. However, this is now broken for me again, so it would seem the Git plugin is not at fault here.

          Is anyone else still affected by this problem?

          Jeffrey Mendez added a comment - I've just updated a bunch of other plugins that affect Jenkins's security, but I left the Git plugin at 2.3.4. However, this is now broken for me again, so it would seem the Git plugin is not at fault here. Is anyone else still affected by this problem?

          Brian Ray added a comment -

          jmendez, I performed a regular upgrade of our LTS core and plugins a few days ago and hit the current issue again despite it having worked with Freestyle jobs for quite some time. Some of the upgraded components:

          • LTS Core: 2.32.3 > 2.46.3
          • Git Client: 2.4.1 > 2.4.6
          • Git: 3.2.0 > 3.3.0

          And some plugins that were already at the latest versions:

          • SCM API: 2.1.1
          • Subversion: 2.7.2

          Job parameters in the Repository URL string expand fine, but not global parameters.

          Interestingly enough our polling Subversion jobs also started manifesting an analagous error in the Polling Log:

          Started on Jun 7, 2017 7:33:00 PM
          ERROR: Failed to check repository revision for ${SVN_URL}
          org.tmatesoft.svn.core.SVNException: svn: E125002: Malformed URL '${DEV_SRC_REPO_PB}/trunk/********'
          	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
          	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
          	at org.tmatesoft.svn.core.SVNURL.<init>(SVNURL.java:227)
          	at org.tmatesoft.svn.core.SVNURL.parseURIEncoded(SVNURL.java:121)
          	at hudson.scm.SubversionSCM$ModuleLocation.getSVNURL(SubversionSCM.java:2769)
          	at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1415)
          	at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:392)
          	at hudson.scm.SCM.poll(SCM.java:409)
          	at hudson.model.AbstractProject._poll(AbstractProject.java:1463)
          	at hudson.model.AbstractProject.poll(AbstractProject.java:1366)
          	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:596)
          	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:642)
          	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          	at java.util.concurrent.FutureTask.run(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	at java.lang.Thread.run(Unknown Source)
          Done. Took 34 ms
          Changes found
          

          Note how the error is nonfatal for SVN polling; changes are detected despite the failure to expand the global property. As hinted at in the Jenkins error log, I think SVN polling falls back on the last successful URL polled if available.

          WARNING: no revision found corresponding to ${DEV_SRC_REPO_PB}/trunk/*********; known: [https://*********:18080/svn/*********/trunk/*********]
          

          As we are migrating to Pipeline I have simply hardcoded those values in the jobs for now.

          Brian Ray added a comment - jmendez , I performed a regular upgrade of our LTS core and plugins a few days ago and hit the current issue again despite it having worked with Freestyle jobs for quite some time. Some of the upgraded components: LTS Core: 2.32.3 > 2.46.3 Git Client: 2.4.1 > 2.4.6 Git: 3.2.0 > 3.3.0 And some plugins that were already at the latest versions: SCM API: 2.1.1 Subversion: 2.7.2 Job parameters in the Repository URL string expand fine, but not global parameters. Interestingly enough our polling Subversion jobs also started manifesting an analagous error in the Polling Log: Started on Jun 7, 2017 7:33:00 PM ERROR: Failed to check repository revision for ${SVN_URL} org.tmatesoft.svn.core.SVNException: svn: E125002: Malformed URL '${DEV_SRC_REPO_PB}/trunk/********' at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57) at org.tmatesoft.svn.core.SVNURL.<init>(SVNURL.java:227) at org.tmatesoft.svn.core.SVNURL.parseURIEncoded(SVNURL.java:121) at hudson.scm.SubversionSCM$ModuleLocation.getSVNURL(SubversionSCM.java:2769) at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1415) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:392) at hudson.scm.SCM.poll(SCM.java:409) at hudson.model.AbstractProject._poll(AbstractProject.java:1463) at hudson.model.AbstractProject.poll(AbstractProject.java:1366) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:596) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:642) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Done. Took 34 ms Changes found Note how the error is nonfatal for SVN polling; changes are detected despite the failure to expand the global property. As hinted at in the Jenkins error log, I think SVN polling falls back on the last successful URL polled if available. WARNING: no revision found corresponding to ${DEV_SRC_REPO_PB}/trunk/*********; known: [https://*********:18080/svn/*********/trunk/*********] As we are migrating to Pipeline I have simply hardcoded those values in the jobs for now.

          Guy Knights added a comment -

          I'm having this issue as well. Using a global variable for part of the repo url (eg. ${GITLAB_URL}/codebase.git) and it fails when polling like so:

          FATAL: hudson.plugins.git.GitException: Failed to fetch from ${GITLAB_URL}/codebase.git

          Interestingly, I have another job which is a copy of this one (for a separate project that uses the same codebase) and the polling is working fine for that job despite the fact that the repo url is also using the same format.

           

          FYI this is with Jenkins v2.46.3 and Git plugin v3.5.0.

          Guy Knights added a comment - I'm having this issue as well. Using a global variable for part of the repo url (eg. ${GITLAB_URL}/codebase.git) and it fails when polling like so: FATAL: hudson.plugins.git.GitException: Failed to fetch from ${GITLAB_URL}/codebase.git Interestingly, I have another job which is a copy of this one (for a separate project that uses the same codebase) and the polling is working fine for that job despite the fact that the repo url is also using the same format.   FYI this is with Jenkins v2.46.3 and Git plugin v3.5.0.

          R. Fitzner added a comment -

          "Environment variable in Git Repository URL not accessable anymore after Jenkins restart"

          Steps to reproduce the problem:

          1. Create under configure / global properties an environment variable, e.g. "SCM_BASE_URL" with the value "https://bitbucket.company.my/scm"
          2. Create a freestyle jenkins job with:

          • SCM / Git / Repository URL, e.g. "${SCM_BASE_URL}/projectkey/myrepo.git
          • Build Triggers / Poll SCM, e.g. "H/2 6-22 * * 1-5"
            3. Start the build manually to check that git authentication works.
            4. Also check the "Git Poll Log" to see that the system starts polling frequently and that git authentication works.
            5. Restart Jenkins
            6. Check the "Git Poll Log" again to see that the system starts polling frequently and throws the following error:


          Steps to "fix" / avoid the problem:

          0. After a fresh Jenkins restart
          1. Either start a build manually OR
          2. Replace the environment variable "${SCM_BASE_URL}" in the freestyle jenkins job with a proper value, e.g. "https://bitbucket.company.my/scm" and wait until the system starts polling again

          Summary:
          It looks like that using an environment variable in the Repository URL is tricky and not resolved straight after a fresh jenkins start.
          Starting a build manually seems to make the environment variable accessable for the polling mechanism.

          Tool stack:
          Red Hat Enterprise Linux Server release 7.7 (Maipo)
          Jenkins LTS version 2.289.2 and 2.289.3
          Only Jenkins master platform, no agents
          Authentication technique: https in git url and username / password via Jenkins credential manager
          Network configuration: no proxy between Bitbucket and Jenkins
          git versions I tried out: 4.7.0 - 4.8.1
          git-client versions I tried out: 3.7.0 - 3.9.0

          R. Fitzner added a comment - "Environment variable in Git Repository URL not accessable anymore after Jenkins restart" Steps to reproduce the problem: 1. Create under configure / global properties an environment variable, e.g. "SCM_BASE_URL" with the value "https://bitbucket.company.my/scm" 2. Create a freestyle jenkins job with: SCM / Git / Repository URL, e.g. "${SCM_BASE_URL}/projectkey/myrepo.git Build Triggers / Poll SCM, e.g. "H/2 6-22 * * 1-5" 3. Start the build manually to check that git authentication works. 4. Also check the "Git Poll Log" to see that the system starts polling frequently and that git authentication works. 5. Restart Jenkins 6. Check the "Git Poll Log" again to see that the system starts polling frequently and throws the following error: Steps to "fix" / avoid the problem: 0. After a fresh Jenkins restart 1. Either start a build manually OR 2. Replace the environment variable "${SCM_BASE_URL}" in the freestyle jenkins job with a proper value, e.g. "https://bitbucket.company.my/scm" and wait until the system starts polling again Summary: It looks like that using an environment variable in the Repository URL is tricky and not resolved straight after a fresh jenkins start. Starting a build manually seems to make the environment variable accessable for the polling mechanism. Tool stack: Red Hat Enterprise Linux Server release 7.7 (Maipo) Jenkins LTS version 2.289.2 and 2.289.3 Only Jenkins master platform, no agents Authentication technique: https in git url and username / password via Jenkins credential manager Network configuration: no proxy between Bitbucket and Jenkins git versions I tried out: 4.7.0 - 4.8.1 git-client versions I tried out: 3.7.0 - 3.9.0

          Mark Waite added a comment -

          Tim Brown has implemented a proposal in PR-1511. It needs further testing from users that are affected by the issue and needs further testing by plugin maintainers. The pull request checks tab include a link to the most recent incremental build of the plugin.

          Mark Waite added a comment - Tim Brown has implemented a proposal in PR-1511 . It needs further testing from users that are affected by the issue and needs further testing by plugin maintainers. The pull request checks tab include a link to the most recent incremental build of the plugin.

            Unassigned Unassigned
            smruti_sahoo smruti sahoo
            Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: