-
Bug
-
Resolution: Unresolved
-
Blocker
-
None
-
Jenkins 2.504.3
Java 17
servicenow-devops 5.1.0
Jenkins' queue maintainance is expected to be quick, and many operations rely on being able to take out the queue lock to correctly synchronize operations.
This includes but is not limited to triggering a build of a project from the UI, removing an agent etc..
The servicenow-devops-plugin however makes blocking IO calls when called from the maintenance thread when the lock is held. This can cause timeouts in the UI and other serious scalability issues.
e.g. the following thread trace was shown in an Jenkins instance that was exhibiting issues
java.base@17.0.15/sun.nio.ch.SocketDispatcher.read0(Native Method)
java.base@17.0.15/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47)
java.base@17.0.15/sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:266)
java.base@17.0.15/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:317)
java.base@17.0.15/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
java.base@17.0.15/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
java.base@17.0.15/java.net.Socket$SocketInputStream.read(Socket.java:966)
java.base@17.0.15/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
java.base@17.0.15/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
java.base@17.0.15/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
java.base@17.0.15/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
java.base@17.0.15/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
java.base@17.0.15/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
java.base@17.0.15/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
java.base@17.0.15/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
java.base@17.0.15/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:826)
java.base@17.0.15/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:761)
java.base@17.0.15/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1725)
java.base@17.0.15/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
java.base@17.0.15/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
java.base@17.0.15/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.utils.CommUtils._readResponse(CommUtils.java:312)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.utils.CommUtils._send(CommUtils.java:222)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.utils.CommUtils.call(CommUtils.java:103)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.model.DevOpsModel.sendIsUnderChgControl(DevOpsModel.java:792)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.model.DevOpsModel.handleFreestyle(DevOpsModel.java:1525)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.DevOpsQueueTaskDispatcher.canRun(DevOpsQueueTaskDispatcher.java:68)
hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1219)
hudson.model.Queue.maintain(Queue.java:1622)
canRun(Item) should be quick.
If data is not available locally or some expensive computation is needed then a placeholder blocking task can be returned whilst a lookup is performed.
Additionally CredentialProviders may not be local and looking up credentials also can involve trips to say an external Vault server or Cyberark server.
PluginClassLoader for credentials//com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentialsInItemGroup(CredentialsProvider.java:387)
PluginClassLoader for credentials//com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(CredentialsProvider.java:314)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.config.DevOpsConfigurationEntry.getCredentials(DevOpsConfigurationEntry.java:190)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.config.DevOpsConfigurationEntry.getUser(DevOpsConfigurationEntry.java:168)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.model.DevOpsModel.sendIsUnderChgControl(DevOpsModel.java:793)
PluginClassLoader for servicenow-devops//io.jenkins.plugins.model.DevOpsModel.handleFreestyle(DevOpsModel.java:1525)
Overall it has been observed that all of these lookups can lock the queue for in excess of 30 seconds
