Historically, Jenkins allowed anyone read access, to the top URL wasn't the 403. The wiki is descriptive, open to edit by anyone, and not kept well up to date, so take its content with a grain of salt.

The better maintained Jenkins handbook specifically mentions the login URL: https://jenkins.io/doc/book/managing/cli/

danielbeck, sorry to say but this is clearly a bug and even one that makes the CLI usage very hard. The fact that `X-SSH-Endpoint` is not reported for unauthorized requests makes impossible to use SSH cli due to:

+ export JENKINS_HOSTNAME=rhos-ci-staging-jenkins.example.com
+ export JENKINS_USERNAME=ssbarnea
+ export JENKINS_TOKEN=xxx
+ export JENKINS_URL=https://rhos-ci-staging-jenkins.example.com
+ curl -sD - -o /dev/null -X GET https://rhos-ci-staging-jenkins.example.com --user ssbarnea:xxx
HTTP/1.1 200 OK
Server: nginx/1.10.2
Date: Thu, 24 Aug 2017 11:11:51 GMT
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID.54abbce6=1av7nndlrni8wlqzw3cbvcchv;Path=/;Secure;HttpOnly
Expires: 0
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
X-Hudson: 1.395
X-Jenkins: 2.60.3
X-Jenkins-Session: b2a74a15
X-Hudson-CLI-Port: 50000
X-Jenkins-CLI-Port: 50000
X-Jenkins-CLI2-Port: 50000
X-Frame-Options: sameorigin
X-Instance-Identity: ...
X-SSH-Endpoint: rhos-ci-staging-jenkins.example.com:50022
Strict-Transport-Security: max-age=63072000; includeSubDomains
X-Content-Type-Options: nosniff

+ java -jar jenkins-cli.jar -ssh -user ssbarnea -i /Users/ssbarnea/.ssh/id_rsa.rhos-jenkins -p rhos-ci-staging-jenkins.example.com:50000 -logger FINEST help
Aug 24, 2017 12:11:52 PM hudson.cli.CLI _main
FINE: using connection mode SSH
Aug 24, 2017 12:11:53 PM hudson.cli.SSHCLI sshConnection
WARNING: No header 'X-SSH-Endpoint' returned by Jenkins
FAIL: 255

This means that's impossible to use SSH port without first authentication to Jenkins via HTTP. 

The fact that the other ports are exposed is another reason to consider this a bug.

Do I really have to modify our NGINX to add this missing headers in order to make the CLI via SSH usable?

And correction: the header is missing even from instances that allow anonymous access. For example we are using Kerberos based authentication and we allow read-only anonymous access, the header is missing unless we make requests with BASIC-auth (tokens),... which cripples the user experience.

I even tries the nginx approach, validated that it does add the header but jenkins-cli.jar still reports the header as missing, I am wondering what kind of request it does really doing,... even if log level = FINEST it still does not tell much. This start to sound like a call for wireshark... on SSL connections.

add_header 'X-SSH-Endpoint' 'jenkins.example.com:50022';

ssbarnea My comment specifically says to use the login URL, as does the handbook. You're not doing that.

danielbeck Thanks for the hint. Still, here are the results after using the /login url and no credentials, still not sending the SSH part. 

One interesting remark is that it seems that in this case nginx didn't add the header, but this is my problem which I am going to investigate.

I hope that in the end we will succeed making Jenkins inform the client about the SSH port, one solution that would not need a lot of debugging and tuning at different places. Somehow, the other 3 -Port header do seem to work correctly, so I see no reason why this one would not work.... hopefully out of the box.

+ curl -sD - -o /dev/null -X GET https://jenkins.example.com/login
HTTP/1.1 401 Unauthorized
Server: nginx/1.10.2
Date: Thu, 24 Aug 2017 13:05:28 GMT
Content-Length: 0
Connection: keep-alive
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID.54ae6=95esznpbhm36f5fbd74;Path=/;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Hudson: 1.395
X-Jenkins: 2.60.3
X-Jenkins-Session: b2a74a15
X-Hudson-CLI-Port: 50000
X-Jenkins-CLI-Port: 50000
X-Jenkins-CLI2-Port: 50000
WWW-Authenticate: Negotiate
WWW-Authenticate: Basic realm="EXAMPLE.COM"

I was able to convince jenkins-cli.jar to connect only after manually adding this to my nginx proxy:

add_header 'X-SSH-Endpoint' 'jenkins.example.com:50022' always;

The always part on that line is key because without it nginx will add the header only on some HTTP return codes which do NOT include 401 and 403 (the ones likely to be reported by /loging url requests).

I think that this is a confirmation that the backend doesn't work as expected and that it should always include the header regardless the response being given.

While I understand danielbeck's arguments about this being documented, I still find it inconsistent that Jenkins announces CLI port always, but SSH port only under certain response codes.

ssbarnea So… just to be sure we're not chasing ghosts… the Jenkins internal SSH server feature, which is opt in, is enabled, right?

$ curl -sD - -o /dev/null -X GET http://localhost:8080/login
HTTP/1.1 200 OK
Date: Thu, 24 Aug 2017 15:25:12 GMT
X-Content-Type-Options: nosniff
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
Content-Type: text/html;charset=UTF-8
Set-Cookie: JSESSIONID.0f7fbfdf=1uqm12sydvz9bpds8l6850inn;Path=/;HttpOnly
X-Hudson: 1.395
X-Jenkins: 2.60.1
X-Jenkins-Session: d6450f8d
X-Frame-Options: sameorigin
X-Instance-Identity: …
X-SSH-Endpoint: localhost:56974
Content-Length: 5858
Server: Jetty(9.2.z-SNAPSHOT)

Works for me. My best guess is a bad interaction with the authentication (SSO?) plugin you're using resulting in non-200 on the login URL. Try the /cli URL, which is unprotected. (oops, not true)

Indeed, I suppose that one of the authentication plugins is creating this behaviour because otherwise we would have a bigger number of people reporting this issue.

Yes, sshd is working and I succesfully used it using ssh client. See the output of list-plugins at

https://gist.github.com/ssbarnea/49bbe45d1c3e816605701bd69bb14fcd

But I have some interesting news! It seems that Jenkins reports X-SSH-Endpoint if you ask for ROOT  (with or without ending slash). If you ask for /login or /cli it will NOT return it. So your recommendation did mislead me.

What I still do not understand is how it comes that the jenkins-cli.jar reports the X-SSH-Endpoint as missing. This make me think that the test done by the java client is not using the root url and is using the /login one which does not contain the header.

So I have reasons to believe that the broken code is inside jenkins-cli.jar.

Right, the expectation is that login is always available.

https://github.com/jenkinsci/jenkins/blob/862de8612966f0acb64e6cf1205f2bc967679547/cli/src/main/java/hudson/cli/SSHCLI.java#L64

If it works for the root URL, then probably only because anonymous Overall/Read is enabled.

So what should we do now as we know what happens?

From my point of view I find the use of HTTP headers to document the port a bit questionable. The industry standard for this is based on DNS SRV entries

dig _jssh._tcp.subdomain.example.com SRV

See that I didn't use the default `ssh` name because that's already used by ssh protocol and many ssh clients are already supporting it.  Because Jenkins SSH is not really ssh, it may deserve its own service name, like  "jssh". 

So what should we do now as we know what happens?

Look into whether the SSO plugin does something sane. It might not be. Not the easiest thing to do given Jenkins' authentication process complexity.

I tried to look for all the places where X-SSH-Endpoint  is used and found only 3 places and none of them look like something interesting https://github.com/search?utf8=%E2%9C%93&q=org%3Ajenkinsci+X-SSH-Endpoint&type=Code

This makes me think that this issue has nothing to do with the plugin and that's a core limitation/bug that prevents the SSHD plugin from adding a header, the response being send back before reaching the plugin code.

The only headers that seem to be working in this case are those like X-Jenkins-CLI2-Port which are added inside jenkins code, not by a plugin.

It's implemented as a page decorator, which only works for successfully served pages. Hence the problem with your auth plugin not having a 200 response for /login.

We are talking here about the ldap-plugin not some esoteric one and TBH I am not even sure that getting a 401 response from `/login` page is wrong. In fact that even more HTTP compliant than returning a 200 answer. 

Somehow I have the impression that's another bug that feels between the jenkins core and its plugin. Everyone agrees thats a bug but nobody agrees which code is to blame/fix: core, ldap-plugin or sshd-plugin? ... based on the status of the ticket one could even assume that is "Not A Defect".

Weird, ci.jenkins.io also uses LDAP and returns 200 for /login (which could in theory be due to Overall/Read being granted, but local tests with internal user DB and without anon Overall/Read also return 200). Are you sure the reverse proxy isn't messing with that?

When I updated one of our machines to 2.60.2 or 2.60.3 my puppet client's CLI authentication through SSH key does not work anymore.

root@testmaster:/var/lib/jenkins# java -jar /usr/share/jenkins/jenkins-cli.jar -s https://testmaster:8443/login -i /etc/puppet/ssl_cli/testmaster-nonuser -ssh -user testmaster-nonuser help

returns code 255 and:

Sep 07, 2017 5:01:13 PM hudson.cli.SSHCLI sshConnection
WARNING: No header 'X-SSH-Endpoint' returned by Jenkins

Regarding  a workaround I read adding that header via reverse proxy config. What to do about this if there is no reverse proxy in place? Is this really fixed like this?

Edit: I am using the matrix-auth plugin together with active-directory to authenticate against an AD server.

rst Can you please give me the link to the workaround you have mentioned , I am facing a similar issue .

Hey Jayesh,

in one of the older comments (Sorin Sbarnea - 2017-08-24 15:27 comment-311573) it is mentioned that the header was manually added via an nginx in front of jenkins. He mentions that after this the cli was able to connect properly.