Status: Closed (View Workflow)
Resolution: Not A Defect
Jenkins 2.60.1 on RedHat Enterprise Linux 5
Quite contrary to what is written on https://wiki.jenkins.io/display/JENKINS/Jenkins+SSH, X-SSH-Endpoint header is not provided on the top page when the response is 403.
$ curl -I http://localhost:8440/jenkins/ HTTP/1.1 403 Forbidden Date: Wed, 19 Jul 2017 13:43:33 GMT X-Content-Type-Options: nosniff Set-Cookie: JSESSIONID.32f49371=15lrqwbgqu5jvm4xkf7ulfomu;Path=/jenkins;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Content-Type: text/html;charset=UTF-8 X-Hudson: 1.395 X-Jenkins: 2.60.1 X-Jenkins-Session: 567160ff X-Hudson-CLI-Port: 37318 X-Jenkins-CLI-Port: 37318 X-Jenkins-CLI2-Port: 37318 X-You-Are-Authenticated-As: anonymous X-You-Are-In-Group-Disabled: JENKINS-39402: use -Dhudson.security.AccessDeniedException2.REPORT_GROUP_HEADERS=true or use /whoAmI to diagnose X-Required-Permission: hudson.model.Hudson.Read X-Permission-Implied-By: hudson.security.Permission.GenericRead X-Permission-Implied-By: hudson.model.Hudson.Administer Content-Length: 829 Server: Jetty(9.2.z-SNAPSHOT)
One needs to get response 200 (i.e. access login URL) to get the endpoint. Note that CLI ports ARE returned always, only SSH endpoint is missing.
$ curl -I http://localhost:8440/jenkins/login HTTP/1.1 200 OK Date: Wed, 19 Jul 2017 13:43:41 GMT X-Content-Type-Options: nosniff Set-Cookie: JSESSIONID.32f49371=6balu16gxdidrwpc4t258io;Path=/jenkins;HttpOnly Expires: 0 Cache-Control: no-cache,no-store,must-revalidate X-Hudson-Theme: default Content-Type: text/html;charset=UTF-8 X-Hudson: 1.395 X-Jenkins: 2.60.1 X-Jenkins-Session: 567160ff X-Hudson-CLI-Port: 37318 X-Jenkins-CLI-Port: 37318 X-Jenkins-CLI2-Port: 37318 X-Frame-Options: sameorigin X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlwNEtMd78Uf5V2p+hCgcHOazeDpoo9lKZL7YlqgzpuQeJvHfhMZ+fRCTCoIns/swiE5tG0WttqRL4z7tgA+xF8+KzKFuUtONhar+4kGDIhrJks+MuyePpEpzXplHtcB9OFUymfwTXrOBXExswjdGodqupyG9IBG1Xlx51oRHRXHEci3QgF1f/7+KfsZM3oCTOpyQxNasE5dki4nA33PvvAeZGHwxvdJ0nmLcbAVk8aSCzjuDbJZ+GnjJC+CsLR34AYWHNbxOo8rn9PCiIJid43UX/36HmYweWlUdRsnBfXtHL02/NV6OngWvWAowmFmVz9YzLj5+C4Q9QshZmvhbFwIDAQAB X-SSH-Endpoint: server:55999 Content-Length: 7304 Server: Jetty(9.2.z-SNAPSHOT)
- mentioned in
|Field||Original Value||New Value|
|Resolution||Not A Defect [ 7 ]|
|Status||Open [ 1 ]||Closed [ 6 ]|
danielbeck, sorry to say but this is clearly a bug and even one that makes the CLI usage very hard. The fact that `X-SSH-Endpoint` is not reported for unauthorized requests makes impossible to use SSH cli due to:
+ export JENKINS_HOSTNAME=rhos-ci-staging-jenkins.example.com + export JENKINS_USERNAME=ssbarnea + export JENKINS_TOKEN=xxx + export JENKINS_URL=https://rhos-ci-staging-jenkins.example.com + curl -sD - -o /dev/null -X GET https://rhos-ci-staging-jenkins.example.com --user ssbarnea:xxx HTTP/1.1 200 OK Server: nginx/1.10.2 Date: Thu, 24 Aug 2017 11:11:51 GMT Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Connection: keep-alive X-Content-Type-Options: nosniff Set-Cookie: JSESSIONID.54abbce6=1av7nndlrni8wlqzw3cbvcchv;Path=/;Secure;HttpOnly Expires: 0 Cache-Control: no-cache,no-store,must-revalidate X-Hudson-Theme: default X-Hudson: 1.395 X-Jenkins: 2.60.3 X-Jenkins-Session: b2a74a15 X-Hudson-CLI-Port: 50000 X-Jenkins-CLI-Port: 50000 X-Jenkins-CLI2-Port: 50000 X-Frame-Options: sameorigin X-Instance-Identity: ... X-SSH-Endpoint: rhos-ci-staging-jenkins.example.com:50022 Strict-Transport-Security: max-age=63072000; includeSubDomains X-Content-Type-Options: nosniff + java -jar jenkins-cli.jar -ssh -user ssbarnea -i /Users/ssbarnea/.ssh/id_rsa.rhos-jenkins -p rhos-ci-staging-jenkins.example.com:50000 -logger FINEST help Aug 24, 2017 12:11:52 PM hudson.cli.CLI _main FINE: using connection mode SSH Aug 24, 2017 12:11:53 PM hudson.cli.SSHCLI sshConnection WARNING: No header 'X-SSH-Endpoint' returned by Jenkins FAIL: 255
This means that's impossible to use SSH port without first authentication to Jenkins via HTTP.
The fact that the other ports are exposed is another reason to consider this a bug.
Do I really have to modify our NGINX to add this missing headers in order to make the CLI via SSH usable?
And correction: the header is missing even from instances that allow anonymous access. For example we are using Kerberos based authentication and we allow read-only anonymous access, the header is missing unless we make requests with BASIC-auth (tokens),... which cripples the user experience.
I even tries the nginx approach, validated that it does add the header but jenkins-cli.jar still reports the header as missing, I am wondering what kind of request it does really doing,... even if log level = FINEST it still does not tell much. This start to sound like a call for wireshark... on SSL connections.
add_header 'X-SSH-Endpoint' 'jenkins.example.com:50022';
ssbarnea My comment specifically says to use the login URL, as does the handbook. You're not doing that.
danielbeck Thanks for the hint. Still, here are the results after using the /login url and no credentials, still not sending the SSH part.
One interesting remark is that it seems that in this case nginx didn't add the header, but this is my problem which I am going to investigate.
I hope that in the end we will succeed making Jenkins inform the client about the SSH port, one solution that would not need a lot of debugging and tuning at different places. Somehow, the other 3 -Port header do seem to work correctly, so I see no reason why this one would not work.... hopefully out of the box.
+ curl -sD - -o /dev/null -X GET https://jenkins.example.com/login HTTP/1.1 401 Unauthorized Server: nginx/1.10.2 Date: Thu, 24 Aug 2017 13:05:28 GMT Content-Length: 0 Connection: keep-alive X-Content-Type-Options: nosniff Set-Cookie: JSESSIONID.54ae6=95esznpbhm36f5fbd74;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Hudson: 1.395 X-Jenkins: 2.60.3 X-Jenkins-Session: b2a74a15 X-Hudson-CLI-Port: 50000 X-Jenkins-CLI-Port: 50000 X-Jenkins-CLI2-Port: 50000 WWW-Authenticate: Negotiate WWW-Authenticate: Basic realm="EXAMPLE.COM"
I was able to convince jenkins-cli.jar to connect only after manually adding this to my nginx proxy:
add_header 'X-SSH-Endpoint' 'jenkins.example.com:50022' always;
The always part on that line is key because without it nginx will add the header only on some HTTP return codes which do NOT include 401 and 403 (the ones likely to be reported by /loging url requests).
I think that this is a confirmation that the backend doesn't work as expected and that it should always include the header regardless the response being given.
|Remote Link||This issue links to "Page (Jenkins Wiki)" [ 17515 ]|
While I understand danielbeck's arguments about this being documented, I still find it inconsistent that Jenkins announces CLI port always, but SSH port only under certain response codes.
ssbarnea So… just to be sure we're not chasing ghosts… the Jenkins internal SSH server feature, which is opt in, is enabled, right?
$ curl -sD - -o /dev/null -X GET http://localhost:8080/login HTTP/1.1 200 OK Date: Thu, 24 Aug 2017 15:25:12 GMT X-Content-Type-Options: nosniff Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache,no-store,must-revalidate X-Hudson-Theme: default Content-Type: text/html;charset=UTF-8 Set-Cookie: JSESSIONID.0f7fbfdf=1uqm12sydvz9bpds8l6850inn;Path=/;HttpOnly X-Hudson: 1.395 X-Jenkins: 2.60.1 X-Jenkins-Session: d6450f8d X-Frame-Options: sameorigin X-Instance-Identity: … X-SSH-Endpoint: localhost:56974 Content-Length: 5858 Server: Jetty(9.2.z-SNAPSHOT)
Works for me. My best guess is a bad interaction with the authentication (SSO?) plugin you're using resulting in non-200 on the login URL.
Try the /cli URL, which is unprotected. (oops, not true)
|Attachment||jenkins-global-security.png [ 39424 ]|
Indeed, I suppose that one of the authentication plugins is creating this behaviour because otherwise we would have a bigger number of people reporting this issue.
Yes, sshd is working and I succesfully used it using ssh client. See the output of list-plugins at
But I have some interesting news! It seems that Jenkins reports X-SSH-Endpoint if you ask for ROOT (with or without ending slash). If you ask for /login or /cli it will NOT return it. So your recommendation did mislead me.
What I still do not understand is how it comes that the jenkins-cli.jar reports the X-SSH-Endpoint as missing. This make me think that the test done by the java client is not using the root url and is using the /login one which does not contain the header.
So I have reasons to believe that the broken code is inside jenkins-cli.jar.
Right, the expectation is that login is always available.
If it works for the root URL, then probably only because anonymous Overall/Read is enabled.
So what should we do now as we know what happens?
From my point of view I find the use of HTTP headers to document the port a bit questionable. The industry standard for this is based on DNS SRV entries
dig _jssh._tcp.subdomain.example.com SRV
See that I didn't use the default `ssh` name because that's already used by ssh protocol and many ssh clients are already supporting it. Because Jenkins SSH is not really ssh, it may deserve its own service name, like "jssh".
So what should we do now as we know what happens?
Look into whether the SSO plugin does something sane. It might not be. Not the easiest thing to do given Jenkins' authentication process complexity.
|Summary||X-SSH-Endpoint is not provided on top page when 403 is returned||X-SSH-Endpoint is not provided on top page when 401/403 are returned|
I tried to look for all the places where X-SSH-Endpoint is used and found only 3 places and none of them look like something interesting https://github.com/search?utf8=%E2%9C%93&q=org%3Ajenkinsci+X-SSH-Endpoint&type=Code
This makes me think that this issue has nothing to do with the plugin and that's a core limitation/bug that prevents the SSHD plugin from adding a header, the response being send back before reaching the plugin code.
The only headers that seem to be working in this case are those like X-Jenkins-CLI2-Port which are added inside jenkins code, not by a plugin.
It's implemented as a page decorator, which only works for successfully served pages. Hence the problem with your auth plugin not having a 200 response for /login.
We are talking here about the ldap-plugin not some esoteric one and TBH I am not even sure that getting a 401 response from `/login` page is wrong. In fact that even more HTTP compliant than returning a 200 answer.
Somehow I have the impression that's another bug that feels between the jenkins core and its plugin. Everyone agrees thats a bug but nobody agrees which code is to blame/fix: core, ldap-plugin or sshd-plugin? ... based on the status of the ticket one could even assume that is "Not A Defect".
|Component/s||ldap-plugin [ 17122 ]|
|Component/s||ssh-plugin [ 16049 ]|
Weird, ci.jenkins.io also uses LDAP and returns 200 for /login (which could in theory be due to Overall/Read being granted, but local tests with internal user DB and without anon Overall/Read also return 200). Are you sure the reverse proxy isn't messing with that?
When I updated one of our machines to 2.60.2 or 2.60.3 my puppet client's CLI authentication through SSH key does not work anymore.
root@testmaster:/var/lib/jenkins# java -jar /usr/share/jenkins/jenkins-cli.jar -s https://testmaster:8443/login -i /etc/puppet/ssl_cli/testmaster-nonuser -ssh -user testmaster-nonuser help
returns code 255 and:
Sep 07, 2017 5:01:13 PM hudson.cli.SSHCLI sshConnection
WARNING: No header 'X-SSH-Endpoint' returned by Jenkins
Regarding a workaround I read adding that header via reverse proxy config. What to do about this if there is no reverse proxy in place? Is this really fixed like this?
Edit: I am using the matrix-auth plugin together with active-directory to authenticate against an AD server.
rst Can you please give me the link to the workaround you have mentioned , I am facing a similar issue .
in one of the older comments (Sorin Sbarnea - 2017-08-24 15:27 comment-311573) it is mentioned that the header was manually added via an nginx in front of jenkins. He mentions that after this the cli was able to connect properly.
Historically, Jenkins allowed anyone read access, to the top URL wasn't the 403. The wiki is descriptive, open to edit by anyone, and not kept well up to date, so take its content with a grain of salt.
The better maintained Jenkins handbook specifically mentions the login URL: https://jenkins.io/doc/book/managing/cli/