Hi Team,

We have jenkins servers(master / Agents) hosted in AWS and we setup agents(on demand based on Jobs Queue) connection through swarm-client.jar.

Problem is when we have many jobs on Queue and agents provisioned more than 10+(sometime it will be 50 as well) at a time, swarm client initiating connection to jenkins master and it is taking time to connection, may be an average 5 mins(Jenkins GUI performance good). Please find below logs from agents.

Jenkins server:

Memory: 72 GB

Cores: 32

Heap memory: 16 GB

Java Version: 11

Garbage collector used: G1GC

Below are the logs from one of the agent server:

====

Running java -jar /usr/share/jenkins/swarm-client.jar -fsroot /var/jenkins_agent_home -deleteExistingClients -disableClientsUniqueId -executors 1 -master https://xxxxxx/ -passwordEnvVariable PASSWORD -e PASSWORD=redacted -username user -fsroot /var/jenkins/0 -name xxl-linux-i-0acd8f817363d3552-0 -labels xxl_linux -mode exclusive
Aug 03, 2022 12:04:06 PM hudson.plugins.swarm.Client logArguments
INFO: Client invoked with: -deleteExistingClients true -disableClientsUniqueId true -e {PASSWORD=redacted} -executors 1 -fsroot /var/jenkins/0 -labels [xxl_linux] -mode exclusive -name xxl-linux-i-xxxxx-0 -passwordEnvVariable ***** -url https://xxxxx/ -username *****
Aug 03, 2022 12:04:06 PM hudson.plugins.swarm.Client run
INFO: Connecting to Jenkins controller
Aug 03, 2022 12:04:06 PM hudson.plugins.swarm.Client run
INFO: Attempting to connect to https://xxxxxxx/
Aug 03, 2022 12:07:07 PM hudson.plugins.swarm.Client run
SEVERE: An error occurred
java.net.SocketTimeoutException: Read timed out
        at java.base/java.net.SocketInputStream.socketRead0(Native Method)
        at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
Aug 03, 2022 12:07:07 PM hudson.plugins.swarm.Client run
INFO: Retrying in 10 seconds
Aug 03, 2022 12:07:17 PM hudson.plugins.swarm.Client run
INFO: Attempting to connect to https://xxxxxxx/
Aug 03, 2022 12:10:18 PM hudson.plugins.swarm.Client run
SEVERE: An error occurred
java.net.SocketTimeoutException: Read timed out
        at java.base/java.net.SocketInputStream.socketRead0(Native Method)
        at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
        at java.base/java.net.SocketInputStream.read(Unknown Source)

Aug 03, 2022 12:10:18 PM hudson.plugins.swarm.Client run
INFO: Retrying in 10 seconds
Aug 03, 2022 12:10:28 PM hudson.plugins.swarm.Client run
INFO: Attempting to connect to https://xxxxx/
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: xxl-linux-i-0acd8f817363d3552-0
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 03, 2022 12:11:09 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.13
Aug 03, 2022 12:11:09 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /var/jenkins/0/remoting as a remoting work directory
Aug 03, 2022 12:11:09 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /var/jenkins/0/remoting
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among https://xxxxxx/
Aug 03, 2022 12:11:09 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: xxxxxxx
  Agent port:    50000
  Identity:      4b:da:32:cc:b8:0f:c5:16:fa:07:6d:57:22:8a:22:79
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to xxxxxx:50000
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 03, 2022 12:11:09 PM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 4b:da:32:cc:b8:0f:c5:16:fa:07:6d:57:22:8a:22:79
Aug 03, 2022 12:11:09 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

======

Below is the command used to connect master:

java -jar /usr/share/jenkins/swarm-client.jar -fsroot /var/jenkins_agent_home -deleteExistingClients -disableClientsUniqueId -executors 1 -master https://xxxxxx/ -passwordEnvVariable PASSWORD -e PASSWORD=redacted -username user-fsroot /var/jenkins/0 -name linux-i-xxxxx-0 -labels linux swarm -mode exclusive

Please help us on this.

Thanks in Advance !!