-
New Feature
-
Resolution: Unresolved
-
Major
-
None
-
Powered by SuggestiMate
The git client appears to be using the http proxy settings from the "Manage plugins" section of Jenkins. That proxy is for the master to download plugins, not for SCM clients to access repositories. Slave machines shouldn't require the same access to the internet as the master.
Here you can see that localhost:3128 is being used as a proxy when that slave does not have a proxy listening on port 31218
> /usr/bin/git config --local credential.username frank # timeout=10
> /usr/bin/git config --local credential.helper store --file=/tmp/git6157196107535640953.credentials # timeout=10
Setting http proxy: localhost:3128
> /usr/bin/git -c core.askpass=true fetch --tags --progress https://github.com/frank/fright.git +refs/heads/:refs/remotes/origin/
> /usr/bin/git config --local --remove-section credential # timeout=10
Using environmental variables should be the approach for accessing http/https on the web from a git client.
- is related to
-
JENKINS-73204 The git client plugin uses Jenkins proxy by default ( Setting http proxy: )
-
- Closed
-
[JENKINS-31464] git-client uses proxy from plugins section of jenkins
While it might not be the most common use case, it's certainly a valid one, which jenkins should support.
If the slave has the usual http_proxy environment variables defined, they should be used.
Even worse, jenkins seems to be resetting the http_proxy environment variable. So i have to do
export http_proxy=$HTTP_PROXY
git clone http://...
instead of using jenkins built in git.
Mark, this isn't always true. The basis of what you say, yes, but if my master is set up to use a proxy to reach the outside world, but my git repo is inside, then the Git client is picking up the proxy from the master and using that.
ahuxtable and matthiasjenkins I agree that a Jenkins system can have slaves and master which require different HTTP proxy settings. I can imagine a private Jenkins server using slaves in a public cloud but needing access to a git repository inside the firewall, or a Jenkins server with slaves at two different locations of the same company, with a site specific http proxy at each of the locations.
There are already some interesting techniques in other Jenkins plugins which might solve that. I understand that the Mercurial plugin can use the Jenkins master / slave communication channel to carry Mercurial source code changes between client and server. Conceptually, the Jenkins git plugin (with the JGit implementation inside it) could be extended to use a similar technique, transporting all git operations through the master node, using only the Jenkins master / slave communication channel for other git work.
Unfortunately, I don't think it is a common enough use case for me to invest in it (at least not personally). I'm much more interested in making more authentication cases work, and in making submodule authentication work. If either of you would like to propose changes (with tests) to implement the capability, I'm willing to consider them. I don't want to sound rude, or inconsiderate of your concerns, but I want the personal time I spend on the Jenkins project to benefit as many people as I can, and submodule authentication, existing pull requests, and test improvements are already taking all of the time that I can give to the plugin.
Note that below I am commenting from my perspective, I have no idea on the variations of this problem outside of my own domain so by no means do I assume the below is easy or complete.
From my perspective, the http_proxy and https_proxy settings are set individually. For every build node, I go to node properties and then environmental variables and set values for ftp_proxy, http_proxy and https_proxy.
It seems very strange that something is able to clobber these values.
Who is in control of printing the line (seen in my initial cut-paste of the issue):
Setting http proxy: localhost:3128
As this is where the problem lies.
I have many (20-30) different hosts all in different environments and I need to be able to set the environmental variables based on environment (and that effectively means I set this up on each node). Which is fine, it's explicit and allows all possibilities of nodes being in different environments.
I can see how it helps people to take a default of what the master is set with, but the ordering is wrong. First it should honor the nodes environmental variables and fall back to that of the master iff one attempt fails (what if the builder can't get to the proxy that the master uses to get to the internet, like if my builders are spun up aws instances). The one attempt fails thing sucks, but you've got no other way to test if the builders are actually on the internet without the need for a proxy (I guess you could set http_proxy="" in the node properties and this would override the global proxy setting).
The solution of not clobbering the node specific environment variables would mean every single variation of nodes being in same and different locations is 100% supported. So stopping whatever does the:
Setting http proxy: localhost:3128
Would allow the problem to be solved for everyone.
IMHO.
Happy to wrong about any of the above. I am just eager to hear your thoughts on what I see as a way out.
scattym I agree. Maybe, this is really an issue within jenkins itself, and not the git plugin?
matthiasjenkins Yeah, it looks like it's part of the git-client code as follows (from src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java):
if (proxy != null) { boolean shouldProxy = true; for(Pattern p : proxy.getNoProxyHostPatterns()) { if(p.matcher(url.getHost()).matches()) { shouldProxy = false; break; } } if(shouldProxy) { env = new EnvVars(env); listener.getLogger().println("Setting http proxy: " + proxy.name + ":" + proxy.port); String userInfo = null; if (proxy.getUserName() != null) { userInfo = proxy.getUserName(); if (proxy.getPassword() != null) { userInfo += ":" + proxy.getPassword(); } } try { URI http_proxy = new URI("http", userInfo, proxy.name, proxy.port, null, null, null); env.put("http_proxy", http_proxy.toString()); env.put("https_proxy", http_proxy.toString()); } catch (URISyntaxException ex) { throw new GitException("Failed to create http proxy uri", ex); } } }
I wonder if it's as simple adding something like the following:
if(shouldProxy) { env = new EnvVars(env); listener.getLogger().println("Setting http proxy: " + proxy.name + ":" + proxy.port); String userInfo = null; if (proxy.getUserName() != null) { userInfo = proxy.getUserName(); if (proxy.getPassword() != null) { userInfo += ":" + proxy.getPassword(); } } try { URI http_proxy = new URI("http", userInfo, proxy.name, proxy.port, null, null, null); if( env.get("http_proxy") == null ) { env.put("http_proxy", http_proxy.toString()); } if( env.get("https_proxy") == null ) { env.put("https_proxy", http_proxy.toString()); } } catch (URISyntaxException ex) { throw new GitException("Failed to create http proxy uri", ex); } }
I guess it would depend on when the node level environmental variables are applied...
Will see if I can work out how to build the plugin.
markewaite I've put together a pull request for a change that makes sure the http_proxy and https_proxy values aren't already set before setting them from within git client (like if they are set at the node level). So no work for you to do there. Unfortunately the test framework is doing my head in (my fault I am sure). Any pointers on how to implement some unit tests or at least could you let me know what is a recent one that I should look at first to get my head around them. I was looking at PushTest and PushSimpleTest, being the most recently worked on, are they the best to be starting with? I couldn't see anything specifically around the environmental variables...
GitAPITestCase.setupProxy() seems like it might be some help, since it refers to proxy configuration, uses system properties, and modifies those system properties. The GitAPITestCase is the base class which defines tests that are executed from both CliGitAPIImplTest and JGitAPIImplTest.
I empathize with the challenge of finding a way to write a test for the case you're trying to verify. I find writing it very challenging to write tests which verify problems. Part of the challenge is (I believe) due to the "integration" nature of the client plugin. It is bridging between the file system, the git command line, the JGit implementation, and Jenkins core. That's the job it is intended to do, but that means it must interact with all those things in interesting ways.
Not resolved. The pull request is still open. You could build your own copy of the plugin which includes the master branch and the pull request. That would give you the fix and would provide an additional tester to declare if the change is working well for you.
Proxy configuration changes are especially scary because they risk breaking users very badly. They are difficult to test due to the many different forms of proxying.
Ended up hitting this today. It took several hours to debug and come to the root cause.
There does not even seem to be a possible workaround to set the proxy per node. For me this forbids using client in other subnets (where the only option to checkout is using a different proxy).
Any chance to get this PR included?
I'm hesitating to build Jenkins on my own. But if someone can easily build with this PR, I'm happy to test and report back.
christianschneider I've built a copy of the tip of the git client plugin master branch with this pull request included and placed it on my google drive. I've done no testing myself of that change, though I did briefly review the source code of the change.
ccschneidrmarkewaite I can confirm this workaround stopped working on November 28th 2016.
That's around the same time as we did an upgrade to Jenkins.
The workaround falls back to current behaviour so that's what you are seeing at the moment.
ccschneidrmarkewaite I just checked and in November we upgraded jenkins which included git-client-plugin 2.1.0 which did not have my patch. No reason to think there is anything wrong with the code then.
Christian, when you look at the console output. Do you see the lines "Leaving http_proxy set to X".
How are you setting http_proxy and https_proxy?
The following is in place for our build user:
[builduser@somemachine-vm2 ~]$ cat .profile export http_proxy=http://192.168.1.1:3128 export https_proxy=http://192.168.1.1:3128
Hey Matt, thanks for looking into it.
For me it prints the "Setting http_proxy to be ..." sentences. And yes, my setup sounds like yours: http_proxy (and https_proxy and no_proxy) are set as environment variables outside of the build node process. So calling "sh 'env'" in a pipeline script prints the desired proxy settings. In contrast to that, "env.getEnvironment()['http_proxy']" is null in a pipeline. And while env.getEnvironment() contains at least some other properties, new EnvVars() is completely empty (thus new EnvVars().get("http_proxy", "") is "").
So, yes, the "env = new EnvVars()" line does not originate from your patch. Still it always creates an empty EnvVars, doesn't it?
ccschneidr The EnvVars must work. I just recompiled the latest git-client-plugin (2.3.0) with the change and I have just got a heap of my build servers back
I see the following lines in the console output:
Setting http proxy: 10.1.1.1:3128
Leaving http_proxy set to http://192.168.1.1:3128
Leaving https_proxy set to http://192.168.1.1:3128
If you are setting the env variables in the same place this would work.
These are the only things I can think of:
1. Using HTTP_PROXY instead of http_proxy (really I should be checking both)
2. Environmental variables are being set at a later point
3. Plugin doesn't contain the patch or didn't upload properly
4. git client you are using doesn't support proxy behaviour (although, if you are not seeing the Leaving lines then this is unlikely the problem yet)
I've added some logging just before the conditional. Give this a try and check the output at the top:
https://drive.google.com/drive/folders/0ByO3I08lkskfSUtYZzVpMGhFTUE?usp=sharing
It will print the values of http_proxy and https_proxy from the environment before entering the new conditional logic.
It's based off of the latest release. Hopefully that is compatible with the version of jenkins you are using.
Maybe this is related to pipelines? Will EnvVars behave differently there? I'm using the following pipeline script to test it:
node('linux') { sh 'env' println "Node env:" println env.getEnvironment() println "EnvVars:" println new hudson.EnvVars() checkout([something from github]) }
Using your new plugin version (and Jenkins ver. 2.32.2) it prints the following:
Running on linux1 in /home/buildchain/work/buildnode/workspace/test [Pipeline] { [Pipeline] sh [test] Running shell script + env [...] http_proxy=http://[correct.proxy].[...].de:8080 https_proxy=http://[correct.proxy].[...].de:8080 [...] no_proxy=[...].de,[...].com [...] [Pipeline] echo Node env: [Pipeline] echo {BUILD_DISPLAY_NAME=#7, BUILD_ID=7, BUILD_NUMBER=7, BUILD_TAG=jenkins-[...]-test [...]} [Pipeline] echo EnvVars: [Pipeline] echo {} [Pipeline] checkout Cloning the remote Git repository Cloning repository https://github.com/collaborationFactory/[...].git > git init /home/buildchain/work/buildnode/workspace/[...]/test/main # timeout=10 Fetching upstream changes from https://github.com/collaborationFactory/[...].git > git --version # timeout=10 using GIT_ASKPASS to set credentials [...] GitHubCredentials Setting http proxy: [wrong.proxy].[...].de:8080 Checking if proxy already set in env. http_proxy in env is set to Setting http_proxy to be http://[wrong.proxy].[...].de:8080 https_proxy in env is set to Setting https_proxy to be http://[wrong.proxy].[...].de:8080 > git fetch --tags --progress https://github.com/collaborationFactory/[...].git +refs/heads/*:refs/remotes/origin/* # timeout=240 ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/collaborationFactory/[...].git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: fatal: unable to access 'https://github.com/[...].git/': Received HTTP code 407 from proxy after CONNECT
So the environment variables are set for the process of the build node. But they are not reflected inside groovy. I guess they would need to be loaded explicitly for the current build node.
Well, yes, it is related to pipelines: Using the old school checkout of a single git repository I get "Leaving http_proxy set to [...]" and it works. Unfortunately that does not help me with my pipelines :/
ccschneidr
Oh good. That's a step closer... I haven't used pipelines yet. I was actually just looking into the concept a few days ago.
I don't know if it's of any help to you, but a quick search shows up that you can set envars inside of your pipeline definition. Of course not having used them I don't know if it's relevant for your situation:
http://stackoverflow.com/questions/39171341/load-file-with-environment-variables-jenkins-pipeline
Note specifically the comment about using "withEnv" being the correct way to handle environmental variables within a scope.
Let me know how you go as I will have the same problem as you soon!
withEnv does not help. I think, because "These are available to any external processes spawned within that scope." (and they are available to the external processes already). Even setting env.http_proxy directly in the pipeline does not help. The question would probably be which EnvVars is passed to CliGitAPIImpl in the first place. But I can't afford the time for setting up a complete debug environment for this. Seems I will need to checkout via command line without using the git plugin.
I am seeing this issue with Pipelines. Has anybody figured a way around this? I cannot run a pipeline with git on my repository which needs to be accessed via proxy – it works fine if I run a manual Pipeline.
All ways to set environment variables on Jenkins also haven't helped as it seems that this ultimately gets overwritten by the Git plugin no matter what I do.
Pull request 201 proposes a possible fix for this. If you're willing to test drive that pull request, you could help the project and may find it solves your issue.
Download the git-client-plugin.hpi and install it in your environment. Review the pull request to confirm you've configured it correctly, then report your results.
Alex - I've got a proposed PR https://github.com/jenkinsci/git-plugin/pull/507 that provides that allowed for this. At least it did when I first did it back in June .
You would need to build git-plugin at the associated commit, along with building git-client-plugin with https://github.com/jenkinsci/git-client-plugin/pull/201
If we get enough folks to try it, the maintainers have told me they will merge.
My recollection was that https://github.com/jenkinsci/git-client-plugin/pull/201 was not sufficient for pipelines (only for classic freestyle jobs), and you needed https://github.com/jenkinsci/git-plugin/pull/507 in conjunction.
But of course let me know if you all try it out and see different results.
Gabe's pull request is built and ready to download from [ci.jenkins.io|https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-plugin/detail/PR-507/18/artifacts] Combine that with the git-client plugin build, and you can help with testing.
Hey, thanks for the quick response and helpful comments. It looks like the 2 PRs in question address exactly what I'm seeing, however unfortunately I am not able to load custom hpi's into the environment I'm working on, so I've instead worked around this (not use git plugin, just write a script to checkout git and load my Jenkinsfile).
alex1x one of the key reasons I've been unwilling to merge those pull requests is due to insufficient testing. I don't have a test environment to check the current proxy implementation, nor do I have a test environment to check the proposed new proxy environment. gmontero has tested it in his environment, but the plugin is used by over 120000 installations worldwide, so I'm hesitant to merge a proxy configuration change with only a single point of validation.
Is there any way you can construct a separate Jenkins environment to use as a test case?
Are you allowed to run Docker images in that environment? If so, I can provide a Docker definition which includes the two pull request builds.
I have the same problem.. and got: "Received HTTP code 403 from proxy after CONNECT..." when I try to connect to my repo.
I have added my GIT Host to the "No Proxy Host" section under the HTTP Proxy Config.
This seems to be acceptable workaround for the moment.
Hello,
I also had this issue, and solved by modifying git-plugin only.
Here is my change.
https://github.com/junkpiano/git-plugin/commit/7f86f4113bd9813ccfb2e684ecfcbea4277cc458
In my environment, master and slave machines are located in different network zones. All machines can connect with each others. Some machines needs proxy to access external contents but the others don't. So I introduced environmental variable to toggle proxy on/off on node. This gives you flexible proxy setting on the fly.
There might be something I missed, so your feedbacks are appreciated.
Thanks.
We also have different proxy settings on the master and agents. Fortunately a local .gitconfig on the build node has precedence over the setting from the master. So we distribute a /home/jenkins/.gitconfig file with the following content to the build nodes:
[http "https://git.example.com"] proxy = http://proxy.example.com:3128 [http "https://dev.example.com"] proxy = http://proxy2.example.com:3128
Jenkins still logs Setting http proxy: proxy.masteronly.example.com:3128 during the build - but git selects the proxy from the local .gitconfig.
I disagree. I think the most common proxy use case for the many users of the plugin is when the master and slave are both behind the same http proxy which is blocking direct access to the outside. I can imagine scenarios where the slave is not behind the same proxy as the master (slave in the public cloud managed by a master inside a corporate firewall), but those seem much less likely than master and slaves both behind an http proxy.