The Jenkins project does not test Jenkins or its components with the OpenJ9 Java virtual machine. Does the same problem happen when using a recent Hotspot JDK like Adoptium 11.0.17? If not, then please switch to the hotspot virtual machine rather than the OpenJ9 virtual machine.

If it fails with the hotspot virtual machine, then please provide more details so that others can duplicate the problem. I frequently run a swarm 3.37 agent with Jenkins 2.361.3. The agent command line that I use is:

java -Dorg.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true \
     -jar ~/bin/swarm-client-3.37.jar \
     -executors 1 \
     -fsroot $agent_dir \
     -labels "$labels" \
     -name $agent_name \
     -passwordFile $passwordfile \
     -pidFile $agent_pidfile \
     --toolLocation ant-latest=$ant_dir \
     --toolLocation jdk17=$jdk17_dir \
     --toolLocation jdk11=$jdk11_dir \
     --toolLocation jdk8=$jdk8_dir \
     --toolLocation maven-latest=$maven_dir \
     --toolLocation mvn=$maven_dir \
     -username mwaite \
     -master $controller

First let me point out that when i update Jenkins master with Swarm version 3.37, then do a restart of my master and let the Slaves run Swarm client 3.36 all Slaves connect without any issue.

We run Swarm client with:
java -jar swarmcp-www527.gos.oce.net.jar -executors 1 -disableClientsUniqueId -deleteExistingClients -fsroot c:\oce\jenkins -labels "SIL000131 do_not_use" -master http://cp-www527.gos.oce.net:80 -username ovl-svc-embedded-ba -password ****** -name SIL000131 -description "SIL000131 runs do_not_use"

Just tested it with new Java:
c:\oce\Jenkins>java --version
openjdk 17.0.5 2022-10-18
OpenJDK Runtime Environment Temurin-17.0.5+8 (build 17.0.5+8)
OpenJDK 64-Bit Server VM Temurin-17.0.5+8 (build 17.0.5+8, mixed mode, sharing)

Same issue occurred:
INFO: Attempting to connect to http://cp-www527.gos.oce.net:80/
Nov 07, 2022 11:52:57 AM hudson.plugins.swarm.SwarmClient getCsrfCrumb
SEVERE: Could not obtain CSRF crumb. Response code: 400
<h1>Bad Message 400</h1><pre>reason: Bad Request</pre>
Nov 07, 2022 11:52:57 AM hudson.plugins.swarm.Client run
SEVERE: An error occurred
hudson.plugins.swarm.RetryException: Failed to create a Swarm agent on Jenkins. Response code: 400
<h1>Bad Message 400</h1><pre>reason: Bad Request</pre>
at hudson.plugins.swarm.SwarmClient.createSwarmAgent(SwarmClient.java:405)
at hudson.plugins.swarm.Client.run(Client.java:216)
at hudson.plugins.swarm.Client.main(Client.java:68)

Nov 07, 2022 11:52:57 AM hudson.plugins.swarm.Client run

Java command now is:
C:\Temp\jdk-17.0.5+8\bin\java.exe -Dorg.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true -jar swarm%MASTER%.jar -executors 1 -disableClientsUniqueId -deleteExistingClients -fsroot %JENKINS_WORKAREA% -labels "%COMPUTERNAME% %SUPPORTED_LABELS%" -master http://%MASTER%%MASTERPORT% -username %USERNAME% -password %PASSWD% -name %COMPUTERNAME% -description "%COMPUTERNAME% runs %SUPPORTED_LABELS%"

I do not see any additional logging.

I would guess that there is some networking problem between the swarm agent and the controller. I don't know what that problem might be.

Community.jenkins.io sometimes provides the following ideas to users that are seeing CSRF crumb issues:

Standard fixes to No valid crumb:

• Check jenkins error log
• Update all plugins
• Confirm hostname in /manage (or config.xml) matches hostname you are accessing jenkins with
• Confirm your load balancer/reverse proxy/etc. is doing X-Forwarded-Host, X-Forwarded-Proto and maybe X-Forwarded-Port are setup correctly

Tried the following:

• Checked Jenkins log: no log on master, only log on Slave which was added to this ticket
• Update all Plugins: In fact i upgraded Jenkins master to latest LTS then upgraded all plugins after a Jenkins restart.
  Then rebooted a slave to test if everything still worked --> Here i found that Swarm client give the above error.
  Set ONLY Swarm client back to 3.36 and did a restart of Jenkins master, then rebooted slave and it connected without any problems.
  So all Plugins are up to date only Swarm 3.37 is giving this issue.
• Checked hostname in /manage and config.xml and i confirm that it matches the hostname i am accessing jenkins with
• We do not use the reverse proxy, however i checked the X_Forwarded stuff all are ok.

To reproduce this issue is quite easy, steps are:

1. Update Swarm Plugin on master and restart
2. Login on slave and stop Jenkins service
3. Download swarm jar file from master (<master URL>/swarm/swarm-client.jar)
4. Start Jenkins service with newly downloaded swarm-client.jar

Unfortunately, I can't reproduce the issue. My steps are:

1. Update to swarm plugin 3.37 on controller and restart (using Java 11.0;.16.1)
2. Stop Jenkins swarm agent on Debian Linux 10 (using Java 11.0.16.1)
3. Download swarm jar file from controller (<controller URL>/swarm/swarm-client.jar)
4. Start Jenkins swarm agent from Debian Linux 10 with newly downloaded swarm-client.jar

I've run Jenkins 2.361.3 in that configuration recently and am now running Jenkins 2.375 in that configuration.

Please note that we use Windows 10 Pro as slave and Windows Server 2012 as master.

When i run in a cmd window a crumb GET API i get this:

$ curl -v -X GET http://cp-www527.gos.oce.net:8080/crumbIssuer/api/json --user <user>:<password>

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.95.5.43:8080...
* Connected to cp-www527.gos.oce.net (10.95.5.43) port 8080 (#0)
* Server auth using Basic with user 'ovl-svc-embedded-ba'
> GET /crumbIssuer/api/json HTTP/1.1
> Host: cp-www527.gos.oce.net:8080
> Authorization: Basic ****************************==
> User-Agent: curl/7.83.1
> Accept: /
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 404 Not Found
< Content-Type: text/plain;charset=UTF-8
< X-RBT-CLI: Name=ovl-steelhead-mgt; Ver=9.9.3a;
< Date: Tue, 08 Nov 2022 07:43:13 GMT
< Connection: close
<
Not Found
* Closing connection 0

Do you have any other things i can use/test?

Also set "Enable proxy compatibility" in "Configure Global Security" under "CSRF Protection", unfortunately same result...

Since Response code: 400 which normally is a Bad Request.

Is there a way how i can see/log which request and URL is sent ?

My curl output is quite different from yours:

$ curl -v GET http://mark-pc2.markwaite.net:8080/crumbIssuer/api/json
*   Trying 127.0.1.1:8080...
* Connected to mark-pc2.markwaite.net (127.0.1.1) port 8080 (#0)
> GET /crumbIssuer/api/json HTTP/1.1
> Host: mark-pc2.markwaite.net:8080
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 08 Nov 2022 11:09:07 GMT
< X-Content-Type-Options: nosniff
< X-Jenkins: 2.375
< X-Jenkins-Session: dd627f5a
< X-Frame-Options: deny
< Content-Type: application/json;charset=utf-8
< Set-Cookie: JSESSIONID.5508e9d8=node01tbi6ys74pdgu1gi3rpt5bicza23.node0; Path=/; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Length: 163
< Server: Jetty(10.0.12)
<
* Connection #0 to host mark-pc2.markwaite.net left intact
{"_class":"hudson.security.csrf.DefaultCrumbIssuer","crumb":"38b5a0ba5e32986a50998bf62ddb4e307bceb0b13a0185721269898a06b2a950","crumbRequestField":"Jenkins-Crumb"}mwaite@mark-pc2:~/git/jenkins/jenkins.io

I'm sure there are ways to log details of the requests and responses, though I don't know how you would do that. You might check the global configuration of the Jenkins instance and compare it with a test instance that you install on some other machine. Maybe there is some setting in your production instance that is causing the behavior?

I checked your output and mine, difference 1 is Linux versus Windows, which made me figure out the following....

So this changes the curl to:

$ curl -v GET http://cp-www527.gos.oce.net:8080/crumbIssuer/api/json --user xxx:ppp
* Could not resolve host: GET
* Closing connection 0

This gives me the following new command:

$ curl -v http://cp-www527.gos.oce.net:8080/crumbIssuer/api/json --user xxx:ppp
*   Trying 10.95.5.43:8080...
* Connected to cp-www527.gos.oce.net (10.95.5.43) port 8080 (#0)
* Server auth using Basic with user 'xxx'
> GET /crumbIssuer/api/json HTTP/1.1
> Host: cp-www527.gos.oce.net:8080
> Authorization: Basic*****************==
> User-Agent: curl/7.83.1
> Accept: /
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 404 Not Found
< Content-Type: text/plain;charset=UTF-8
< X-RBT-CLI: Name=ovl-steelhead-mgt; Ver=9.9.3a;
< Date: Tue, 08 Nov 2022 12:23:46 GMT
< Connection: close
<
Not Found
* Closing connection 0

However when i change the port from 8080 to 80 i get this response (which is quite similar to yours)

$ curl -v http://cp-www527.gos.oce.net:80/crumbIssuer/api/json --user xxx:ppp
*   Trying 10.95.5.43:80...
* Connected to cp-www527.gos.oce.net (10.95.5.43) port 80 (#0)
* Server auth using Basic with user 'xxxx'
> GET /crumbIssuer/api/json HTTP/1.1
> Host: cp-www527.gos.oce.net
{{{}> Authorization: Basic*****************==
{}}}> User-Agent: curl/7.83.1
> Accept: /
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 08 Nov 2022 12:25:40 GMT
< X-Content-Type-Options: nosniff
< X-Jenkins: 2.361.3
< X-Jenkins-Session: d535f55c
< X-Frame-Options: deny
< Content-Type: application/json;charset=utf-8
< Set-Cookie: JSESSIONID.6782108b=node04t2l0b5dikkg1rhq96ynrg7ux9.node0; Path=/; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Server: Jetty(10.0.11)
< X-RBT-CLI: Name=ovl-steelhead-mgt; Ver=9.9.3a;
< Connection: keep-alive
< Content-Length: 163
<
{{

* Connection #0 to host cp-www527.gos.oce.net left intact}}

Could this  mean that the port is wrong here ?