Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31464

git-client uses proxy from plugins section of jenkins

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • git-client-plugin
    • None

      The git client appears to be using the http proxy settings from the "Manage plugins" section of Jenkins. That proxy is for the master to download plugins, not for SCM clients to access repositories. Slave machines shouldn't require the same access to the internet as the master.

      Here you can see that localhost:3128 is being used as a proxy when that slave does not have a proxy listening on port 31218
      > /usr/bin/git config --local credential.username frank # timeout=10
      > /usr/bin/git config --local credential.helper store --file=/tmp/git6157196107535640953.credentials # timeout=10
      Setting http proxy: localhost:3128
      > /usr/bin/git -c core.askpass=true fetch --tags --progress https://github.com/frank/fright.git +refs/heads/:refs/remotes/origin/
      > /usr/bin/git config --local --remove-section credential # timeout=10

      Using environmental variables should be the approach for accessing http/https on the web from a git client.

          [JENKINS-31464] git-client uses proxy from plugins section of jenkins

          Mark Waite added a comment -

          I disagree. I think the most common proxy use case for the many users of the plugin is when the master and slave are both behind the same http proxy which is blocking direct access to the outside. I can imagine scenarios where the slave is not behind the same proxy as the master (slave in the public cloud managed by a master inside a corporate firewall), but those seem much less likely than master and slaves both behind an http proxy.

          Mark Waite added a comment - I disagree. I think the most common proxy use case for the many users of the plugin is when the master and slave are both behind the same http proxy which is blocking direct access to the outside. I can imagine scenarios where the slave is not behind the same proxy as the master (slave in the public cloud managed by a master inside a corporate firewall), but those seem much less likely than master and slaves both behind an http proxy.

          While it might not be the most common use case, it's certainly a valid one, which jenkins should support.

          If the slave has the usual http_proxy environment variables defined, they should be used.

          Matthias Pretzer added a comment - While it might not be the most common use case, it's certainly a valid one, which jenkins should support. If the slave has the usual http_proxy environment variables defined, they should be used.

          Even worse, jenkins seems to be resetting the http_proxy environment variable. So i have to do

          export http_proxy=$HTTP_PROXY
          git clone http://...
          

          instead of using jenkins built in git.

          Matthias Pretzer added a comment - Even worse, jenkins seems to be resetting the http_proxy environment variable. So i have to do export http_proxy=$HTTP_PROXY git clone http: //... instead of using jenkins built in git.

          Arran added a comment -

          Mark, this isn't always true. The basis of what you say, yes, but if my master is set up to use a proxy to reach the outside world, but my git repo is inside, then the Git client is picking up the proxy from the master and using that.

          Arran added a comment - Mark, this isn't always true. The basis of what you say, yes, but if my master is set up to use a proxy to reach the outside world, but my git repo is inside , then the Git client is picking up the proxy from the master and using that .

          Mark Waite added a comment -

          ahuxtable and matthiasjenkins I agree that a Jenkins system can have slaves and master which require different HTTP proxy settings. I can imagine a private Jenkins server using slaves in a public cloud but needing access to a git repository inside the firewall, or a Jenkins server with slaves at two different locations of the same company, with a site specific http proxy at each of the locations.

          There are already some interesting techniques in other Jenkins plugins which might solve that. I understand that the Mercurial plugin can use the Jenkins master / slave communication channel to carry Mercurial source code changes between client and server. Conceptually, the Jenkins git plugin (with the JGit implementation inside it) could be extended to use a similar technique, transporting all git operations through the master node, using only the Jenkins master / slave communication channel for other git work.

          Unfortunately, I don't think it is a common enough use case for me to invest in it (at least not personally). I'm much more interested in making more authentication cases work, and in making submodule authentication work. If either of you would like to propose changes (with tests) to implement the capability, I'm willing to consider them. I don't want to sound rude, or inconsiderate of your concerns, but I want the personal time I spend on the Jenkins project to benefit as many people as I can, and submodule authentication, existing pull requests, and test improvements are already taking all of the time that I can give to the plugin.

          Mark Waite added a comment - ahuxtable and matthiasjenkins I agree that a Jenkins system can have slaves and master which require different HTTP proxy settings. I can imagine a private Jenkins server using slaves in a public cloud but needing access to a git repository inside the firewall, or a Jenkins server with slaves at two different locations of the same company, with a site specific http proxy at each of the locations. There are already some interesting techniques in other Jenkins plugins which might solve that. I understand that the Mercurial plugin can use the Jenkins master / slave communication channel to carry Mercurial source code changes between client and server. Conceptually, the Jenkins git plugin (with the JGit implementation inside it) could be extended to use a similar technique, transporting all git operations through the master node, using only the Jenkins master / slave communication channel for other git work. Unfortunately, I don't think it is a common enough use case for me to invest in it (at least not personally). I'm much more interested in making more authentication cases work, and in making submodule authentication work. If either of you would like to propose changes (with tests) to implement the capability, I'm willing to consider them. I don't want to sound rude, or inconsiderate of your concerns, but I want the personal time I spend on the Jenkins project to benefit as many people as I can, and submodule authentication, existing pull requests, and test improvements are already taking all of the time that I can give to the plugin.

          Matt Clark added a comment - - edited

          Note that below I am commenting from my perspective, I have no idea on the variations of this problem outside of my own domain so by no means do I assume the below is easy or complete.

          From my perspective, the http_proxy and https_proxy settings are set individually. For every build node, I go to node properties and then environmental variables and set values for ftp_proxy, http_proxy and https_proxy.

          It seems very strange that something is able to clobber these values.

          Who is in control of printing the line (seen in my initial cut-paste of the issue):
          Setting http proxy: localhost:3128

          As this is where the problem lies.

          I have many (20-30) different hosts all in different environments and I need to be able to set the environmental variables based on environment (and that effectively means I set this up on each node). Which is fine, it's explicit and allows all possibilities of nodes being in different environments.

          I can see how it helps people to take a default of what the master is set with, but the ordering is wrong. First it should honor the nodes environmental variables and fall back to that of the master iff one attempt fails (what if the builder can't get to the proxy that the master uses to get to the internet, like if my builders are spun up aws instances). The one attempt fails thing sucks, but you've got no other way to test if the builders are actually on the internet without the need for a proxy (I guess you could set http_proxy="" in the node properties and this would override the global proxy setting).

          The solution of not clobbering the node specific environment variables would mean every single variation of nodes being in same and different locations is 100% supported. So stopping whatever does the:
          Setting http proxy: localhost:3128
          Would allow the problem to be solved for everyone.

          IMHO.

          Happy to wrong about any of the above. I am just eager to hear your thoughts on what I see as a way out.

          Matt Clark added a comment - - edited Note that below I am commenting from my perspective, I have no idea on the variations of this problem outside of my own domain so by no means do I assume the below is easy or complete. From my perspective, the http_proxy and https_proxy settings are set individually. For every build node, I go to node properties and then environmental variables and set values for ftp_proxy, http_proxy and https_proxy. It seems very strange that something is able to clobber these values. Who is in control of printing the line (seen in my initial cut-paste of the issue): Setting http proxy: localhost:3128 As this is where the problem lies. I have many (20-30) different hosts all in different environments and I need to be able to set the environmental variables based on environment (and that effectively means I set this up on each node). Which is fine, it's explicit and allows all possibilities of nodes being in different environments. I can see how it helps people to take a default of what the master is set with, but the ordering is wrong. First it should honor the nodes environmental variables and fall back to that of the master iff one attempt fails (what if the builder can't get to the proxy that the master uses to get to the internet, like if my builders are spun up aws instances). The one attempt fails thing sucks, but you've got no other way to test if the builders are actually on the internet without the need for a proxy (I guess you could set http_proxy="" in the node properties and this would override the global proxy setting). The solution of not clobbering the node specific environment variables would mean every single variation of nodes being in same and different locations is 100% supported. So stopping whatever does the: Setting http proxy: localhost:3128 Would allow the problem to be solved for everyone. IMHO. Happy to wrong about any of the above. I am just eager to hear your thoughts on what I see as a way out.

          scattym I agree. Maybe, this is really an issue within jenkins itself, and not the git plugin?

          Matthias Pretzer added a comment - scattym I agree. Maybe, this is really an issue within jenkins itself, and not the git plugin?

          Matt Clark added a comment - - edited

          matthiasjenkins Yeah, it looks like it's part of the git-client code as follows (from src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java):

                          if (proxy != null) {
                              boolean shouldProxy = true;
                              for(Pattern p : proxy.getNoProxyHostPatterns()) {
                                  if(p.matcher(url.getHost()).matches()) {
                                      shouldProxy = false;
                                      break;
                                  }
                              }
                              if(shouldProxy) {
                                  env = new EnvVars(env);
                                  listener.getLogger().println("Setting http proxy: " + proxy.name + ":" + proxy.port);
                                  String userInfo = null;
                                  if (proxy.getUserName() != null) {
                                      userInfo = proxy.getUserName();
                                      if (proxy.getPassword() != null) {
                                          userInfo += ":" + proxy.getPassword();
                                      }
                                  }
                                  try {
                                      URI http_proxy = new URI("http", userInfo, proxy.name, proxy.port, null, null, null);
                                      env.put("http_proxy", http_proxy.toString());
                                      env.put("https_proxy", http_proxy.toString());
                                  } catch (URISyntaxException ex) {
                                      throw new GitException("Failed to create http proxy uri", ex);
                                  }
                              }
                          }
          
          

          I wonder if it's as simple adding something like the following:

          if(shouldProxy) {
                                  env = new EnvVars(env);
                                  listener.getLogger().println("Setting http proxy: " + proxy.name + ":" + proxy.port);
                                  String userInfo = null;
                                  if (proxy.getUserName() != null) {
                                      userInfo = proxy.getUserName();
                                      if (proxy.getPassword() != null) {
                                          userInfo += ":" + proxy.getPassword();
                                      }
                                  }
                                  try {
                                      URI http_proxy = new URI("http", userInfo, proxy.name, proxy.port, null, null, null);
                                      if( env.get("http_proxy") == null ) {
                                          env.put("http_proxy", http_proxy.toString());
                                      }
                                      if( env.get("https_proxy") == null ) {
                                          env.put("https_proxy", http_proxy.toString());
                                      }
                                  } catch (URISyntaxException ex) {
                                      throw new GitException("Failed to create http proxy uri", ex);
                                  }
                              }
          

          I guess it would depend on when the node level environmental variables are applied...

          Will see if I can work out how to build the plugin.

          Matt Clark added a comment - - edited matthiasjenkins Yeah, it looks like it's part of the git-client code as follows (from src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java): if (proxy != null ) { boolean shouldProxy = true ; for (Pattern p : proxy.getNoProxyHostPatterns()) { if (p.matcher(url.getHost()).matches()) { shouldProxy = false ; break ; } } if (shouldProxy) { env = new EnvVars(env); listener.getLogger().println( "Setting http proxy: " + proxy.name + ":" + proxy.port); String userInfo = null ; if (proxy.getUserName() != null ) { userInfo = proxy.getUserName(); if (proxy.getPassword() != null ) { userInfo += ":" + proxy.getPassword(); } } try { URI http_proxy = new URI( "http" , userInfo, proxy.name, proxy.port, null , null , null ); env.put( "http_proxy" , http_proxy.toString()); env.put( "https_proxy" , http_proxy.toString()); } catch (URISyntaxException ex) { throw new GitException( "Failed to create http proxy uri" , ex); } } } I wonder if it's as simple adding something like the following: if (shouldProxy) { env = new EnvVars(env); listener.getLogger().println( "Setting http proxy: " + proxy.name + ":" + proxy.port); String userInfo = null ; if (proxy.getUserName() != null ) { userInfo = proxy.getUserName(); if (proxy.getPassword() != null ) { userInfo += ":" + proxy.getPassword(); } } try { URI http_proxy = new URI( "http" , userInfo, proxy.name, proxy.port, null , null , null ); if ( env.get( "http_proxy" ) == null ) { env.put( "http_proxy" , http_proxy.toString()); } if ( env.get( "https_proxy" ) == null ) { env.put( "https_proxy" , http_proxy.toString()); } } catch (URISyntaxException ex) { throw new GitException( "Failed to create http proxy uri" , ex); } } I guess it would depend on when the node level environmental variables are applied... Will see if I can work out how to build the plugin.

          Matt Clark added a comment -

          Created a pull request with a possible solution.

          https://github.com/jenkinsci/git-client-plugin/pull/201

          Matt Clark added a comment - Created a pull request with a possible solution. https://github.com/jenkinsci/git-client-plugin/pull/201

          Matt Clark added a comment -

          markewaite I've put together a pull request for a change that makes sure the http_proxy and https_proxy values aren't already set before setting them from within git client (like if they are set at the node level). So no work for you to do there. Unfortunately the test framework is doing my head in (my fault I am sure). Any pointers on how to implement some unit tests or at least could you let me know what is a recent one that I should look at first to get my head around them. I was looking at PushTest and PushSimpleTest, being the most recently worked on, are they the best to be starting with? I couldn't see anything specifically around the environmental variables...

          Matt Clark added a comment - markewaite I've put together a pull request for a change that makes sure the http_proxy and https_proxy values aren't already set before setting them from within git client (like if they are set at the node level). So no work for you to do there. Unfortunately the test framework is doing my head in (my fault I am sure). Any pointers on how to implement some unit tests or at least could you let me know what is a recent one that I should look at first to get my head around them. I was looking at PushTest and PushSimpleTest, being the most recently worked on, are they the best to be starting with? I couldn't see anything specifically around the environmental variables...

          Mark Waite added a comment -

          GitAPITestCase.setupProxy() seems like it might be some help, since it refers to proxy configuration, uses system properties, and modifies those system properties. The GitAPITestCase is the base class which defines tests that are executed from both CliGitAPIImplTest and JGitAPIImplTest.

          I empathize with the challenge of finding a way to write a test for the case you're trying to verify. I find writing it very challenging to write tests which verify problems. Part of the challenge is (I believe) due to the "integration" nature of the client plugin. It is bridging between the file system, the git command line, the JGit implementation, and Jenkins core. That's the job it is intended to do, but that means it must interact with all those things in interesting ways.

          Mark Waite added a comment - GitAPITestCase.setupProxy() seems like it might be some help, since it refers to proxy configuration, uses system properties, and modifies those system properties. The GitAPITestCase is the base class which defines tests that are executed from both CliGitAPIImplTest and JGitAPIImplTest. I empathize with the challenge of finding a way to write a test for the case you're trying to verify. I find writing it very challenging to write tests which verify problems. Part of the challenge is (I believe) due to the "integration" nature of the client plugin. It is bridging between the file system, the git command line, the JGit implementation, and Jenkins core. That's the job it is intended to do, but that means it must interact with all those things in interesting ways.

          Was this resolved? Because I am still facing the issue

          Pratith Kanagaraj added a comment - Was this resolved? Because I am still facing the issue

          Mark Waite added a comment -

          Not resolved. The pull request is still open. You could build your own copy of the plugin which includes the master branch and the pull request. That would give you the fix and would provide an additional tester to declare if the change is working well for you.

          Proxy configuration changes are especially scary because they risk breaking users very badly. They are difficult to test due to the many different forms of proxying.

          Mark Waite added a comment - Not resolved. The pull request is still open. You could build your own copy of the plugin which includes the master branch and the pull request. That would give you the fix and would provide an additional tester to declare if the change is working well for you. Proxy configuration changes are especially scary because they risk breaking users very badly. They are difficult to test due to the many different forms of proxying.

          Mark R added a comment -

          Ended up hitting this today. It took several hours to debug and come to the root cause.

          Mark R added a comment - Ended up hitting this today. It took several hours to debug and come to the root cause.

          Christian Schneider added a comment - - edited

          There does not even seem to be a possible workaround to set the proxy per node. For me this forbids using client in other subnets (where the only option to checkout is using a different proxy).

          Christian Schneider added a comment - - edited There does not even seem to be a possible workaround to set the proxy per node. For me this forbids using client in other subnets (where the only option to checkout is using a different proxy).

          Any chance to get this PR included?
          I'm hesitating to build Jenkins on my own. But if someone can easily build with this PR, I'm happy to test and report back.

          Christian Schneider added a comment - Any chance to get this PR included? I'm hesitating to build Jenkins on my own. But if someone can easily build with this PR, I'm happy to test and report back.

          Mark Waite added a comment -

          christianschneider I've built a copy of the tip of the git client plugin master branch with this pull request included and placed it on my google drive. I've done no testing myself of that change, though I did briefly review the source code of the change.

          Mark Waite added a comment - christianschneider I've built a copy of the tip of the git client plugin master branch with this pull request included and placed it on my google drive . I've done no testing myself of that change, though I did briefly review the source code of the change.

          Thanks. Tested it, but it does not work

          Christian Schneider added a comment - Thanks. Tested it, but it does not work

          Matt Clark added a comment - - edited

          ccschneidrmarkewaite I can confirm this workaround stopped working on November 28th 2016.

          That's around the same time as we did an upgrade to Jenkins.

          The workaround falls back to current behaviour so that's what you are seeing at the moment.

          Matt Clark added a comment - - edited ccschneidr markewaite I can confirm this workaround stopped working on November 28th 2016. That's around the same time as we did an upgrade to Jenkins. The workaround falls back to current behaviour so that's what you are seeing at the moment.

          Matt Clark added a comment - - edited

          ccschneidrmarkewaite I just checked and in November we upgraded jenkins which included git-client-plugin 2.1.0 which did not have my patch. No reason to think there is anything wrong with the code then.

          Christian, when you look at the console output. Do you see the lines "Leaving http_proxy set to X".

          How are you setting http_proxy and https_proxy?

          The following is in place for our build user:

          [builduser@somemachine-vm2 ~]$ cat .profile 
          export http_proxy=http://192.168.1.1:3128
          export https_proxy=http://192.168.1.1:3128
          

          Matt Clark added a comment - - edited ccschneidr markewaite I just checked and in November we upgraded jenkins which included git-client-plugin 2.1.0 which did not have my patch. No reason to think there is anything wrong with the code then. Christian, when you look at the console output. Do you see the lines "Leaving http_proxy set to X". How are you setting http_proxy and https_proxy? The following is in place for our build user: [builduser@somemachine-vm2 ~]$ cat .profile export http_proxy=http: //192.168.1.1:3128 export https_proxy=http: //192.168.1.1:3128

          Hey Matt, thanks for looking into it.
          For me it prints the "Setting http_proxy to be ..." sentences. And yes, my setup sounds like yours: http_proxy (and https_proxy and no_proxy) are set as environment variables outside of the build node process. So calling "sh 'env'" in a pipeline script prints the desired proxy settings. In contrast to that, "env.getEnvironment()['http_proxy']" is null in a pipeline. And while env.getEnvironment() contains at least some other properties, new EnvVars() is completely empty (thus new EnvVars().get("http_proxy", "") is "").
          So, yes, the "env = new EnvVars()" line does not originate from your patch. Still it always creates an empty EnvVars, doesn't it?

          Christian Schneider added a comment - Hey Matt, thanks for looking into it. For me it prints the "Setting http_proxy to be ..." sentences. And yes, my setup sounds like yours: http_proxy (and https_proxy and no_proxy) are set as environment variables outside of the build node process. So calling "sh 'env'" in a pipeline script prints the desired proxy settings. In contrast to that, "env.getEnvironment() ['http_proxy'] " is null in a pipeline. And while env.getEnvironment() contains at least some other properties, new EnvVars() is completely empty (thus new EnvVars().get("http_proxy", "") is ""). So, yes, the "env = new EnvVars()" line does not originate from your patch. Still it always creates an empty EnvVars, doesn't it?

          Matt Clark added a comment - - edited

          ccschneidr The EnvVars must work. I just recompiled the latest git-client-plugin (2.3.0) with the change and I have just got a heap of my build servers back

          I see the following lines in the console output:
          Setting http proxy: 10.1.1.1:3128
          Leaving http_proxy set to http://192.168.1.1:3128
          Leaving https_proxy set to http://192.168.1.1:3128

          If you are setting the env variables in the same place this would work.

          These are the only things I can think of:
          1. Using HTTP_PROXY instead of http_proxy (really I should be checking both)
          2. Environmental variables are being set at a later point
          3. Plugin doesn't contain the patch or didn't upload properly
          4. git client you are using doesn't support proxy behaviour (although, if you are not seeing the Leaving lines then this is unlikely the problem yet)

          I've added some logging just before the conditional. Give this a try and check the output at the top:
          https://drive.google.com/drive/folders/0ByO3I08lkskfSUtYZzVpMGhFTUE?usp=sharing

          It will print the values of http_proxy and https_proxy from the environment before entering the new conditional logic.

          It's based off of the latest release. Hopefully that is compatible with the version of jenkins you are using.

          Matt Clark added a comment - - edited ccschneidr The EnvVars must work. I just recompiled the latest git-client-plugin (2.3.0) with the change and I have just got a heap of my build servers back I see the following lines in the console output: Setting http proxy: 10.1.1.1:3128 Leaving http_proxy set to http://192.168.1.1:3128 Leaving https_proxy set to http://192.168.1.1:3128 If you are setting the env variables in the same place this would work. These are the only things I can think of: 1. Using HTTP_PROXY instead of http_proxy (really I should be checking both) 2. Environmental variables are being set at a later point 3. Plugin doesn't contain the patch or didn't upload properly 4. git client you are using doesn't support proxy behaviour (although, if you are not seeing the Leaving lines then this is unlikely the problem yet) I've added some logging just before the conditional. Give this a try and check the output at the top: https://drive.google.com/drive/folders/0ByO3I08lkskfSUtYZzVpMGhFTUE?usp=sharing It will print the values of http_proxy and https_proxy from the environment before entering the new conditional logic. It's based off of the latest release. Hopefully that is compatible with the version of jenkins you are using.

          Christian Schneider added a comment - - edited

          Maybe this is related to pipelines? Will EnvVars behave differently there? I'm using the following pipeline script to test it:

          node('linux') {
              sh 'env'
              println "Node env:"
              println env.getEnvironment()
              println "EnvVars:"
              println new hudson.EnvVars()
              
              checkout([something from github])
          }
          

          Using your new plugin version (and Jenkins ver. 2.32.2) it prints the following:

          Running on linux1 in /home/buildchain/work/buildnode/workspace/test
          [Pipeline] {
          [Pipeline] sh
          [test] Running shell script
          + env
          [...]
          http_proxy=http://[correct.proxy].[...].de:8080
          https_proxy=http://[correct.proxy].[...].de:8080
          [...]
          no_proxy=[...].de,[...].com
          [...]
          [Pipeline] echo
          Node env:
          [Pipeline] echo
          {BUILD_DISPLAY_NAME=#7, BUILD_ID=7, BUILD_NUMBER=7, BUILD_TAG=jenkins-[...]-test [...]}
          [Pipeline] echo
          EnvVars:
          [Pipeline] echo
          {}
          [Pipeline] checkout
          Cloning the remote Git repository
          Cloning repository https://github.com/collaborationFactory/[...].git
           > git init /home/buildchain/work/buildnode/workspace/[...]/test/main # timeout=10
          Fetching upstream changes from https://github.com/collaborationFactory/[...].git
           > git --version # timeout=10
          using GIT_ASKPASS to set credentials [...] GitHubCredentials
          Setting http proxy: [wrong.proxy].[...].de:8080
          Checking if proxy already set in env.
          http_proxy in env is set to 
          Setting http_proxy to be http://[wrong.proxy].[...].de:8080
          https_proxy in env is set to 
          Setting https_proxy to be http://[wrong.proxy].[...].de:8080
           > git fetch --tags --progress https://github.com/collaborationFactory/[...].git +refs/heads/*:refs/remotes/origin/* # timeout=240
          ERROR: Error cloning remote repo 'origin'
          hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/collaborationFactory/[...].git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
          stdout: 
          stderr: fatal: unable to access 'https://github.com/[...].git/': Received HTTP code 407 from proxy after CONNECT
          

          So the environment variables are set for the process of the build node. But they are not reflected inside groovy. I guess they would need to be loaded explicitly for the current build node.

          Christian Schneider added a comment - - edited Maybe this is related to pipelines? Will EnvVars behave differently there? I'm using the following pipeline script to test it: node( 'linux' ) { sh 'env' println "Node env:" println env.getEnvironment() println "EnvVars:" println new hudson.EnvVars() checkout([something from github]) } Using your new plugin version (and Jenkins ver. 2.32.2) it prints the following: Running on linux1 in /home/buildchain/work/buildnode/workspace/test [Pipeline] { [Pipeline] sh [test] Running shell script + env [...] http_proxy=http://[correct.proxy].[...].de:8080 https_proxy=http://[correct.proxy].[...].de:8080 [...] no_proxy=[...].de,[...].com [...] [Pipeline] echo Node env: [Pipeline] echo {BUILD_DISPLAY_NAME=#7, BUILD_ID=7, BUILD_NUMBER=7, BUILD_TAG=jenkins-[...]-test [...]} [Pipeline] echo EnvVars: [Pipeline] echo {} [Pipeline] checkout Cloning the remote Git repository Cloning repository https://github.com/collaborationFactory/[...].git > git init /home/buildchain/work/buildnode/workspace/[...]/test/main # timeout=10 Fetching upstream changes from https://github.com/collaborationFactory/[...].git > git --version # timeout=10 using GIT_ASKPASS to set credentials [...] GitHubCredentials Setting http proxy: [wrong.proxy].[...].de:8080 Checking if proxy already set in env. http_proxy in env is set to Setting http_proxy to be http://[wrong.proxy].[...].de:8080 https_proxy in env is set to Setting https_proxy to be http://[wrong.proxy].[...].de:8080 > git fetch --tags --progress https://github.com/collaborationFactory/[...].git +refs/heads/*:refs/remotes/origin/* # timeout=240 ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/collaborationFactory/[...].git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: fatal: unable to access 'https://github.com/[...].git/': Received HTTP code 407 from proxy after CONNECT So the environment variables are set for the process of the build node. But they are not reflected inside groovy. I guess they would need to be loaded explicitly for the current build node.

          Well, yes, it is related to pipelines: Using the old school checkout of a single git repository I get "Leaving http_proxy set to [...]" and it works. Unfortunately that does not help me with my pipelines :/

          Christian Schneider added a comment - Well, yes, it is related to pipelines: Using the old school checkout of a single git repository I get "Leaving http_proxy set to [...] " and it works. Unfortunately that does not help me with my pipelines :/

          Matt Clark added a comment - - edited

          ccschneidr
          Oh good. That's a step closer... I haven't used pipelines yet. I was actually just looking into the concept a few days ago.

          I don't know if it's of any help to you, but a quick search shows up that you can set envars inside of your pipeline definition. Of course not having used them I don't know if it's relevant for your situation:
          http://stackoverflow.com/questions/39171341/load-file-with-environment-variables-jenkins-pipeline

          Note specifically the comment about using "withEnv" being the correct way to handle environmental variables within a scope.

          Let me know how you go as I will have the same problem as you soon!

          Matt Clark added a comment - - edited ccschneidr Oh good. That's a step closer... I haven't used pipelines yet. I was actually just looking into the concept a few days ago. I don't know if it's of any help to you, but a quick search shows up that you can set envars inside of your pipeline definition. Of course not having used them I don't know if it's relevant for your situation: http://stackoverflow.com/questions/39171341/load-file-with-environment-variables-jenkins-pipeline Note specifically the comment about using "withEnv" being the correct way to handle environmental variables within a scope. Let me know how you go as I will have the same problem as you soon!

          withEnv does not help. I think, because "These are available to any external processes spawned within that scope." (and they are available to the external processes already). Even setting env.http_proxy directly in the pipeline does not help. The question would probably be which EnvVars is passed to CliGitAPIImpl in the first place. But I can't afford the time for setting up a complete debug environment for this. Seems I will need to checkout via command line without using the git plugin.

          Christian Schneider added a comment - withEnv does not help. I think, because "These are available to any external processes spawned within that scope." (and they are available to the external processes already). Even setting env.http_proxy directly in the pipeline does not help. The question would probably be which EnvVars is passed to CliGitAPIImpl in the first place. But I can't afford the time for setting up a complete debug environment for this. Seems I will need to checkout via command line without using the git plugin.

          I am seeing this issue with Pipelines. Has anybody figured a way around this? I cannot run a pipeline with git on my repository which needs to be accessed via proxy – it works fine if I run a manual Pipeline.

          All ways to set environment variables on Jenkins also haven't helped as it seems that this ultimately gets overwritten by the Git plugin no matter what I do.

          Alex Tasioulis added a comment - I am seeing this issue with Pipelines. Has anybody figured a way around this? I cannot run a pipeline with git on my repository which needs to be accessed via proxy – it works fine if I run a manual Pipeline. All ways to set environment variables on Jenkins also haven't helped as it seems that this ultimately gets overwritten by the Git plugin no matter what I do.

          Mark Waite added a comment -

          Pull request 201 proposes a possible fix for this.  If you're willing to test drive that pull request, you could help the project and may find it solves your issue.

          Download the git-client-plugin.hpi and install it in your environment.  Review the pull request to confirm you've configured it correctly, then report your results.

          Mark Waite added a comment - Pull request 201 proposes a possible fix for this.  If you're willing to test drive that pull request, you could help the project and may find it solves your issue. Download the git-client-plugin.hpi and install it in your environment.  Review the pull request to confirm you've configured it correctly, then report your results.

          Gabe Montero added a comment -

          Alex - I've got a proposed PR https://github.com/jenkinsci/git-plugin/pull/507 that provides that allowed for this.  At least it did when I first did it back in June

          You would need to build git-plugin at the associated commit, along with building git-client-plugin with https://github.com/jenkinsci/git-client-plugin/pull/201

          If we get enough folks to try it, the maintainers have told me they will merge.

          Gabe Montero added a comment - Alex - I've got a proposed PR https://github.com/jenkinsci/git-plugin/pull/507 that provides that allowed for this.  At least it did when I first did it back in June .  You would need to build git-plugin at the associated commit, along with building git-client-plugin with https://github.com/jenkinsci/git-client-plugin/pull/201 If we get enough folks to try it, the maintainers have told me they will merge.

          Gabe Montero added a comment -

          My recollection was that https://github.com/jenkinsci/git-client-plugin/pull/201 was not sufficient for pipelines (only for classic freestyle jobs), and you needed https://github.com/jenkinsci/git-plugin/pull/507 in conjunction.

           

          But of course let me know if you all try it out and see different results.

          Gabe Montero added a comment - My recollection was that https://github.com/jenkinsci/git-client-plugin/pull/201 was not sufficient for pipelines (only for classic freestyle jobs), and you needed https://github.com/jenkinsci/git-plugin/pull/507 in conjunction.   But of course let me know if you all try it out and see different results.

          Mark Waite added a comment - - edited

          Gabe's pull request is built and ready to download from [ci.jenkins.io|https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-plugin/detail/PR-507/18/artifacts]  Combine that with the git-client plugin build, and you can help with testing.

          Mark Waite added a comment - - edited Gabe's pull request is built and ready to download from [ci.jenkins.io| https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-plugin/detail/PR-507/18/artifacts ]  Combine that with the git-client plugin build , and you can help with testing.

          Hey, thanks for the quick response and helpful comments. It looks like the 2 PRs in question address exactly what I'm seeing, however unfortunately I am not able to load custom hpi's into the environment I'm working on, so I've instead worked around this (not use git plugin, just write a script to checkout git and load my Jenkinsfile).

          Alex Tasioulis added a comment - Hey, thanks for the quick response and helpful comments. It looks like the 2 PRs in question address exactly what I'm seeing, however unfortunately I am not able to load custom hpi's into the environment I'm working on, so I've instead worked around this (not use git plugin, just write a script to checkout git and load my Jenkinsfile).

          Mark Waite added a comment -

          alex1x one of the key reasons I've been unwilling to merge those pull requests is due to insufficient testing.  I don't have a test environment to check the current proxy implementation, nor do I have a test environment to check the proposed new proxy environment. gmontero has tested it in his environment, but the plugin is used by over 120000 installations worldwide, so I'm hesitant to merge a proxy configuration change  with only a single point of validation.

          Is there any way you can construct a separate Jenkins environment to use as a test case?

          Are you allowed to run Docker images in that environment?  If so, I can provide a Docker definition which includes the two pull request builds.

          Mark Waite added a comment - alex1x one of the key reasons I've been unwilling to merge those pull requests is due to insufficient testing.  I don't have a test environment to check the current proxy implementation, nor do I have a test environment to check the proposed new proxy environment. gmontero has tested it in his environment, but the plugin is used by over 120000 installations worldwide, so I'm hesitant to merge a proxy configuration change  with only a single point of validation. Is there any way you can construct a separate Jenkins environment to use as a test case? Are you allowed to run Docker images in that environment?  If so, I can provide a Docker definition which includes the two pull request builds.

          Michael Hauf added a comment -

          I have the same problem..  and got: "Received HTTP code 403 from proxy after CONNECT..." when I try to connect to my repo.

          I have added my GIT Host to the "No Proxy Host" section under the HTTP Proxy Config.

          This seems to be acceptable workaround for the moment.

          Michael Hauf added a comment - I have the same problem..  and got: "Received HTTP code 403 from proxy after CONNECT..." when I try to connect to my repo. I have added my GIT Host to the "No Proxy Host" section under the HTTP Proxy Config. This seems to be acceptable workaround for the moment.

          Yusuke Ohashi added a comment - - edited

          Hello,

          I also had this issue, and solved by modifying git-plugin only.

          Here is my change.

          https://github.com/junkpiano/git-plugin/commit/7f86f4113bd9813ccfb2e684ecfcbea4277cc458

          In my environment, master and slave machines are located in different network zones. All machines can connect with each others. Some machines needs proxy to access external contents but the others don't. So I introduced environmental variable to toggle proxy on/off on node. This gives you flexible proxy setting on the fly.

          There might be something I missed, so your feedbacks are appreciated.

          Thanks.

          Yusuke Ohashi added a comment - - edited Hello, I also had this issue, and solved by modifying git-plugin only. Here is my change. https://github.com/junkpiano/git-plugin/commit/7f86f4113bd9813ccfb2e684ecfcbea4277cc458 In my environment, master and slave machines are located in different network zones. All machines can connect with each others. Some machines needs proxy to access external contents but the others don't. So I introduced environmental variable to toggle proxy on/off on node. This gives you flexible proxy setting on the fly. There might be something I missed, so your feedbacks are appreciated. Thanks.

          We also have different proxy settings on the master and agents. Fortunately a local .gitconfig on the build node has precedence over the setting from the master. So we distribute a /home/jenkins/.gitconfig file with the following content to the build nodes:

          /home/jenkins/.gitconfig
          [http "https://git.example.com"]
                  proxy = http://proxy.example.com:3128
          [http "https://dev.example.com"]
                  proxy = http://proxy2.example.com:3128 

          Jenkins still logs Setting http proxy: proxy.masteronly.example.com:3128 during the build - but git selects the proxy from the local .gitconfig.

          Andreas Mandel added a comment - We also have different proxy settings on the master and agents. Fortunately a local .gitconfig on the build node has precedence over the setting from the master. So we distribute a /home/jenkins/.gitconfig file with the following content to the build nodes: /home/jenkins/.gitconfig [http "https://git.example.com"] proxy = http://proxy.example.com:3128 [http "https://dev.example.com"] proxy = http://proxy2.example.com:3128 Jenkins still logs Setting http proxy: proxy.masteronly.example.com:3128 during the build - but git selects the proxy from the local .gitconfig .

            Unassigned Unassigned
            scattym Matt Clark
            Votes:
            19 Vote for this issue
            Watchers:
            27 Start watching this issue

              Created:
              Updated: