Hey Basil - Tried today with the latest versions of both the plugin and client; no dice - still seeing the issue, when I disable EXCLUDE_SESSION_ID.

I'll try to dig more and look into implementing an integration test, but I still think this makes some sense: The commit I cited in the original bug incorporates the "Session ID" (which as far as I can tell derives from the JSESSIONID cookie) into the crumb-generation logic. If we don't pass back the JSESSIONID (which we don't appear to), then our crumb will be invalidated. Example of the Set-Cookie header from the Jenkins master:

'Set-Cookie': 'ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Secure; HttpOnly,JSESSIONID.f443e328=node01s03czg5u9gagotkcld9tm98225.node0; Path=/; Secure; HttpOnly'

Note the JESSIONID.xyz component, which we don't seem to return to the master in subsequent requests.

Anyway, as I said - I'll try to dig more on my side.

Thanks for following up with the latest versions. I stepped through the code in the Java debugger against my production Jenkins instance, and as far as I can tell I am correctly returning the JSESSIONID in Swarm. Here is a walkthrough of the code:

1. First, hudson.plugins.swarm.SwarmClient#createSwarmSlave creates an instance of HttpClientContext.
2. Next, hudson.plugins.swarm.SwarmClient#getCsrfCrumb calls the crumbIssuer API, successfully obtaining a crumb request field and a crumb. Note that this requires Overall/Read permission for the user. The server's response sets a JSESSIONID cookie, which is visible in the HttpClientContext's BasicCookieStore (e.g., JSESSIONID.472fe218).
3. Next, hudson.plugins.swarm.SwarmClient#createSwarmSlave adds the CSRF header returned by hudson.plugins.swarm.SwarmClient#getCsrfCrumb to the POST request.
4. Next, hudson.plugins.swarm.SwarmClient#getCsrfCrumb executes the POST request. During this process, org.apache.http.client.protocol.RequestAddCookies examines the HttpClientContext's BasicCookieStore, finds the JSESSIONID cookie, and adds it to the set of headers before making the HTTP POST request.

The key invariants for all this to work are that the HTTP GET in SwarmClient#getCsrfCrumb must always be called immediately before making any POST request, the HttpClientContext (and therefore the BasicCookieStore) must always be shared between the CSRF crumb retrieval GET and the subsequent POST, and the POST must contain both a CSRF header (obtained from the previous CSRF crumb retrieval GET) and a JSESSIONID header (which must be the same JSESSIONID from the previous CSRF crumb retrieval GET). As far as I can tell all of these invariants are met in the code.

To see where things are going off the rails you might want to start by turning up logging to a higher level as described here. Then see if there are any errors logged by SwarmClient#getCsrfCrumb or related methods. If you can, sanitize the log and post it here.

If you can, try to step through the above process in an IDE's debugger and see where things are going off the rails, or write a test in AuthorizationStrategyTest that reproduces the problem. If I could attach a debugger to an instance that is exhibiting this problem I'm sure I could figure it out quickly, but it is difficult to guess at the cause from far away.

Hey Basil - Increasing logging verbosity turned out to be the key here - And my apologies! As you suspected, this was on my end.

Turns out that our Swarm workers reach the Jenkins master through a minimal and very poorly built proxy served from localhost (this was implemented well before my time working on our stack, and I only discovered the existence of the proxy yesterday). The only purpose of the proxy is to append an "X-Forwarded-User: worker_node" header to identify itself to the Jenkins master; this header is typically appended to human-users' requests by our OAuth2 reverse-proxy, but Swarm agents access the master directly.

Turns out that the Jenkins master sends JSESSIONID cookies with the Secure attribute, which (correctly) forces the Swarm client to omit the cookie from http://localhost-bound requests. That solves that mystery.

If I could poke you with one more question, I am now left with the issue of, "How to move forward" - I would still like to remove that EXCLUDE_SESSION_ID override from our configuration, after all. Is there an easier way for me to inject an X-Forwarded-User header into Swarm-issued requests? If not, would you entertain adding a command-line option (--include-header or something?) to support such a use case (or maybe something to that effect already exists within the underlying Apache machinery)? Or perhaps we're going about this wrong, and you would recommend a different setup configuration altogether.

Thanks again for your time, and thanks in advance for any guidance you could give on best practices here!
---Dan

Great, I'm glad you were able to figure this out! There is some documentation for using Swarm with proxies here. I'm not very familiar with proxies or cookies with the Secure attribute, but from a quick Google search I gather that

A cookie with the Secure attribute is sent to the server only with an encrypted request over the HTTPS protocol, never with unsecured HTTP, and therefore can't easily be accessed by a man-in-the-middle attacker.

I am reticent to add support for arbitrary headers. It seems fairly outside the scope of what the Swarm client is meant to do. It is not meant to be a general purpose utility like curl(1) but rather just a wrapper around the Remoting JAR, and its proxy support matches that of Remoting. I would be willing to reconsider this position if other users express an interest and can demonstrate a common use case that requires this. As far as your specific issue is concerned, a proxy seems like the right approach to add custom headers, but it seems like your particular proxy configuration isn't compatible with how Jenkins sets the JSESSIONID cookie. So in my opinion the most appropriate solution would be to make your proxy configuration compatible by enabling TLS on your proxy server and setting -Dhttps.proxyHost and -Dhttps.proxyPort (note the "s") per the Swarm proxy documentation. Please let me know if I've misunderstood any of this.

Hey Basil - After reading up on Jenkins API tokens, I was able to make this work for our use case! I think we can safely close this - Thanks for your help with debugging!