• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Component/s: remoting
• Labels:

  • performance

[JENKINS-67829] JarLoaderImpl is per connection in a controller -causing repeated checking of jars checksums

James Nord created issue - 2022-02-17 10:57

James Nord made changes - 2022-02-17 10:58

Description

Original: When using ephemeral agents it was noticed even after Jenkins had been running for a day that there where still threads calculating the digest for jars with stack traces like the following in the controller {noformat}  "Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from x.x.x.x/x.x.x.x:49778 id=1547" Id=164040 Group=main RUNNABLE"Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from 100.119.213.5/100.119.213.5:49778 id=1547" Id=164040 Group=main RUNNABLE at sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147) at sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:601) at java.security.MessageDigest.update(MessageDigest.java:328) at java.security.DigestOutputStream.write(DigestOutputStream.java:147) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.Checksum.forURL(Checksum.java:83) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:92) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:64) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:1014) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:1055){noformat} it is highly unlikely that any jar that Jenkins knew about had not been already requested by at least one agent before. calcChecksum has a cache [https://github.com/jenkinsci/remoting/blob/efb2c4fbe2e3a848f39f96cb99408ea2350a3aa2/src/main/java/hudson/remoting/JarLoaderImpl.java#L88-L97] however every remoting channel has its own loader - so in effect the cache is per agent not per Jenkins.  As class loading can only work one way (an agent can load classes from a controller) the cache could be shared.  (Jenkins can not change the jars at runtime without a restart, and all agents can see exactly the same jars as all other agents)

New: When using ephemeral agents it was noticed even after Jenkins had been running for a day that there where still threads calculating the digest for jars with stack traces like the following in the controller {noformat}  "Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from x.x.x.x/x.x.x.x:49778 id=1547" Id=164040 Group=main RUNNABLE"Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from 100.119.213.5/100.119.213.5:49778 id=1547" Id=164040 Group=main RUNNABLE at sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147) at sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:601) at java.security.MessageDigest.update(MessageDigest.java:328) at java.security.DigestOutputStream.write(DigestOutputStream.java:147) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.Checksum.forURL(Checksum.java:83) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:92) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:64) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:1014) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:1055){noformat} it is highly unlikely that any jar that Jenkins knew about had not been already requested by at least one agent before. calcChecksum has a cache [https://github.com/jenkinsci/remoting/blob/efb2c4fbe2e3a848f39f96cb99408ea2350a3aa2/src/main/java/hudson/remoting/JarLoaderImpl.java#L88-L97] however every remoting channel has its own loader - so in effect the cache is per agent not per Jenkins.  As class loading can only work one way (an agent can load classes from a controller) the cache could be shared.  (Jenkins can not change the jars at runtime without a restart, and all agents can see exactly the same jars as all other agents) if you have lots of agents connecting then this imposes a scaling limit on Jenkins as it involves both IO to load the file and CPU to compute the digest.

James Nord made changes - 2022-02-17 11:02

Description

Original: When using ephemeral agents it was noticed even after Jenkins had been running for a day that there where still threads calculating the digest for jars with stack traces like the following in the controller {noformat}  "Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from x.x.x.x/x.x.x.x:49778 id=1547" Id=164040 Group=main RUNNABLE"Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from 100.119.213.5/100.119.213.5:49778 id=1547" Id=164040 Group=main RUNNABLE at sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147) at sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:601) at java.security.MessageDigest.update(MessageDigest.java:328) at java.security.DigestOutputStream.write(DigestOutputStream.java:147) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.Checksum.forURL(Checksum.java:83) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:92) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:64) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:1014) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:1055){noformat} it is highly unlikely that any jar that Jenkins knew about had not been already requested by at least one agent before. calcChecksum has a cache [https://github.com/jenkinsci/remoting/blob/efb2c4fbe2e3a848f39f96cb99408ea2350a3aa2/src/main/java/hudson/remoting/JarLoaderImpl.java#L88-L97] however every remoting channel has its own loader - so in effect the cache is per agent not per Jenkins.  As class loading can only work one way (an agent can load classes from a controller) the cache could be shared.  (Jenkins can not change the jars at runtime without a restart, and all agents can see exactly the same jars as all other agents) if you have lots of agents connecting then this imposes a scaling limit on Jenkins as it involves both IO to load the file and CPU to compute the digest.

New: When using ephemeral agents it was noticed even after Jenkins had been running for a day that there where still threads calculating the digest for jars with stack traces like the following in the controller {noformat}  "Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from x.x.x.x/x.x.x.x:49778 id=1547" Id=164040 Group=main RUNNABLE"Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from 100.119.213.5/100.119.213.5:49778 id=1547" Id=164040 Group=main RUNNABLE at sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147) at sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:601) at java.security.MessageDigest.update(MessageDigest.java:328) at java.security.DigestOutputStream.write(DigestOutputStream.java:147) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.Checksum.forURL(Checksum.java:83) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:92) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:64) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:1014) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:1055) {noformat} it is highly unlikely that any jar that Jenkins knew about had not been already requested by at least one agent before. calcChecksum has a cache [https://github.com/jenkinsci/remoting/blob/efb2c4fbe2e3a848f39f96cb99408ea2350a3aa2/src/main/java/hudson/remoting/JarLoaderImpl.java#L88-L97] however every remoting channel has its own loader - so in effect the cache is per agent not per Jenkins.  As class loading can only work one way (an agent can load classes from a controller) the cache could be shared.  (Jenkins can not change the jars at runtime without a restart, and all agents can see exactly the same jars as all other agents) if you have lots of agents connecting then this imposes a scaling limit on Jenkins as it involves both IO to load the file and CPU to compute the digest.

Jesse Glick made changes - 2022-02-17 12:52

Description

Original: When using ephemeral agents it was noticed even after Jenkins had been running for a day that there where still threads calculating the digest for jars with stack traces like the following in the controller {noformat}  "Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from x.x.x.x/x.x.x.x:49778 id=1547" Id=164040 Group=main RUNNABLE"Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from 100.119.213.5/100.119.213.5:49778 id=1547" Id=164040 Group=main RUNNABLE at sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147) at sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:601) at java.security.MessageDigest.update(MessageDigest.java:328) at java.security.DigestOutputStream.write(DigestOutputStream.java:147) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.Checksum.forURL(Checksum.java:83) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:92) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:64) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:1014) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:1055) {noformat} it is highly unlikely that any jar that Jenkins knew about had not been already requested by at least one agent before. calcChecksum has a cache [https://github.com/jenkinsci/remoting/blob/efb2c4fbe2e3a848f39f96cb99408ea2350a3aa2/src/main/java/hudson/remoting/JarLoaderImpl.java#L88-L97] however every remoting channel has its own loader - so in effect the cache is per agent not per Jenkins.  As class loading can only work one way (an agent can load classes from a controller) the cache could be shared.  (Jenkins can not change the jars at runtime without a restart, and all agents can see exactly the same jars as all other agents) if you have lots of agents connecting then this imposes a scaling limit on Jenkins as it involves both IO to load the file and CPU to compute the digest.

New: When using ephemeral agents it was noticed even after Jenkins had been running for a day that there where still threads calculating the digest for jars with stack traces like the following in the controller {code:none} "Computer.threadPoolForRemoting [#17077] for JNLP4-connect connection from x.x.x.x/x.x.x.x:49778 id=1547" Id=164040 Group=main RUNNABLE at sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147) at sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:601) at java.security.MessageDigest.update(MessageDigest.java:328) at java.security.DigestOutputStream.write(DigestOutputStream.java:147) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.Checksum.forURL(Checksum.java:83) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:92) at hudson.remoting.JarLoaderImpl.calcChecksum(JarLoaderImpl.java:64) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:1014) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:1055) {code} it is highly unlikely that any jar that Jenkins knew about had not been already requested by at least one agent before. calcChecksum has a cache [https://github.com/jenkinsci/remoting/blob/efb2c4fbe2e3a848f39f96cb99408ea2350a3aa2/src/main/java/hudson/remoting/JarLoaderImpl.java#L88-L97] however every remoting channel has its own loader - so in effect the cache is per agent not per Jenkins.  As class loading can only work one way (an agent can load classes from a controller) the cache could be shared.  (Jenkins can not change the jars at runtime without a restart, and all agents can see exactly the same jars as all other agents) if you have lots of agents connecting then this imposes a scaling limit on Jenkins as it involves both IO to load the file and CPU to compute the digest.

Jesse Glick made changes - 2022-02-17 12:53

Labels

New: performance

James Nord made changes - 2022-12-07 19:33

Assignee

Original: Jeff Thompson [ jthompson ]

Jesse Glick made changes - 2023-05-08 11:31

Link

New: This issue relates to JENKINS-71217 [ JENKINS-71217 ]