This documents a problem and a workaround when the controller runs a J9 JVM and the agent runs a HotSpot JVM. Jenkins is not tested with J9 JVMs, so I don't expect a fix, but want to document this for anyone else with this problem.

Symptom
s3Download on the agent fails with `Acquire operation took longer than 120000 milliseconds (or hangs indefinitely with a larger AWS_SDK_CONNECTION_ACQUISITION_TIMEOUT). The agent opens and
closes about 130–200 TCP connections per second to S3 and no data is transferred. Nothing is logged.
Semeru 21 and 25 agents work.

Build log:

hudson.remoting.ProxyException: java.util.concurrent.TimeoutException: Acquire operation took longer than 120000 milliseconds.
 at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77)
 at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda$acquire$0(HealthCheckedChannelPool.java:67)
 at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
 at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:160)
Caused: hudson.remoting.ProxyException: java.lang.Throwable: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
 at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:80)
 at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:329)
 at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:191)
 at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604)
 at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:571)
 at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:37)
 at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:517)
 at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
 at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Also:   hudson.remoting.ProxyException: hudson.remoting.Channel$CallSiteStackTrace: Remote call to agent-name
 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
 at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
 at hudson.remoting.Channel.call(Channel.java:1108)
 at hudson.FilePath.act(FilePath.java:1245)
 at hudson.FilePath.act(FilePath.java:1234)
 at PluginClassLoader for pipeline-aws//de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:194)
 at PluginClassLoader for pipeline-aws//de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:158)
 at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:49)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 at java.base/java.lang.Thread.run(Thread.java:1595)

Cause
Netty on the agent sees FlightRecorder.isAvailable() == true, so AdaptivePoolingAllocator$Chunk.<init>
uses io.netty.buffer.AllocateChunkEvent (extends jdk.jfr.Event). The class is fetched from the controller through remoting, and the controller cannot load it because its JVM has no jdk.jfr:

java.lang.LinkageError: Failed to load io.netty.buffer.AllocateChunkEvent via null
java.lang.NoClassDefFoundError: jdk.jfr.Event
at io.netty.buffer.AdaptivePoolingAllocator$Chunk.<init>(AdaptivePoolingAllocator.java:1618)
...
at io.netty.buffer.AdaptiveByteBufAllocator.newHeapBuffer(AdaptiveByteBufAllocator.java:62)
at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:2390)
at io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:2234)
at io.netty.handler.ssl.SslHandler.channelActive(SslHandler.java:2290)
...

The error closes each channel during the TLS handshake; the AWS SDK pool then opens a new connection, looping until the acquire timeout.

Workaround
Start the agent JVM with -Dio.netty.jfr.enabled=false. Confirmed to fix the problem.

Possible fixes

• netty-api: default io.netty.jfr.enabled=false on agents, or when the controller JVM lacks jdk.jfr.
• Netty: Treat a LinkageError while loading its JFR event classes as "JFR unavailable" instead of failing the allocation.
• Netty: https://github.com/netty/netty/pull/17614 might avoid this problem. Untested.