Basil, sorry for the latency.
So, I tried to upgrade 2.365 (which was still failing) and downgrade to 2.363 (which is now failing too
)
As you said, it is probably not jenkins itself but more a extension.
Aug 29, 2022 2:14:25 PM INFO hudson.WebAppMain contextInitialized
Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Aug 29, 2022 2:14:25 PM INFO org.eclipse.jetty.server.handler.ContextHandler doStart
Started w.@5dda6f9{Jenkins v2.365,/,file:///var/cache/jenkins/war/,AVAILABLE}
Unknown macro: {/var/cache/jenkins/war}
Aug 29, 2022 2:14:25 PM INFO org.eclipse.jetty.server.AbstractConnector doStart
Started ServerConnector@7181ae3f{HTTP/1.1, (http/1.1)}
Unknown macro: {0.0.0.0}
Aug 29, 2022 2:14:25 PM INFO org.eclipse.jetty.server.Server doStart
Started Server@dd3b207{STARTING}[10.0.11,sto=0] @6004ms
Aug 29, 2022 2:14:26 PM INFO winstone.Logger logInternal
Winstone Servlet Engine running: controlPort=disabled
Aug 29, 2022 2:14:26 PM INFO jenkins.InitReactorRunner$1 onAttained
Started initialization
Aug 29, 2022 2:14:26 PM INFO hudson.ClassicPluginStrategy createPluginWrapper
Plugin project-build-times.jpi is disabled
Aug 29, 2022 2:14:26 PM INFO jenkins.InitReactorRunner$1 onAttained
Listed all plugins
Aug 29, 2022 2:14:31 PM INFO jenkins.InitReactorRunner$1 onAttained
Prepared all plugins
Aug 29, 2022 2:14:31 PM INFO jenkins.InitReactorRunner$1 onAttained
Started all plugins
Aug 29, 2022 2:14:31 PM INFO jenkins.InitReactorRunner$1 onAttained
Augmented all extensions
Aug 29, 2022 2:14:32 PM INFO hudson.plugins.build_timeout.global.GlobalTimeOutConfiguration load
global timeout not set
Aug 29, 2022 2:14:33 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect arm64-build-3
Aug 29, 2022 2:14:33 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect arm64-build-4
Aug 29, 2022 2:14:33 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect s390x-build-1
Aug 29, 2022 2:14:33 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect s390x-build-2
Aug 29, 2022 2:14:33 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect s390x-build-3
Aug 29, 2022 2:14:33 PM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect s390x-build-4
Aug 29, 2022 2:14:33 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Launch will wait 300000 for operation operation-1661781852513-5e761bd0364be-ddaeee4f-07f03d62 to complete...
Aug 29, 2022 2:14:33 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Launch will wait 300000 for operation operation-1661781857194-5e761bd4ad0f5-f5c28e43-05e7971e to complete...
Aug 29, 2022 2:14:33 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Launch will wait 300000 for operation operation-1661781856154-5e761bd3af30f-9fdd6cb1-14fb7d16 to complete...
Aug 29, 2022 2:14:33 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Launch will wait 300000 for operation operation-1661781853700-5e761bd1580d8-6bf2b015-da71c591 to complete...
Aug 29, 2022 2:14:33 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Launch will wait 300000 for operation operation-1661781855056-5e761bd2a31d1-f85e1695-fb339557 to complete...
Aug 29, 2022 2:14:33 PM INFO jenkins.InitReactorRunner$1 onAttained
System config loaded
Aug 29, 2022 2:14:40 PM INFO hudson.slaves.CommandLauncher launch
agent launched for s390x-build-1
Aug 29, 2022 2:14:40 PM INFO hudson.slaves.CommandLauncher launch
agent launched for s390x-build-2
Aug 29, 2022 2:14:40 PM INFO hudson.slaves.CommandLauncher launch
agent launched for s390x-build-3
Aug 29, 2022 2:14:42 PM INFO hudson.slaves.CommandLauncher launch
agent launched for s390x-build-4
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching instance: deb-7atrj8
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
bootstrap
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Getting keypair...
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Using autogenerated ssh keypair
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Authenticating as jenkins
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching instance: deb-b99pph
Aug 29, 2022 2:14:42 PM INFO hudson.slaves.CommandLauncher launch
agent launched for arm64-build-4
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching instance: deb-39ijgs
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
bootstrap
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Getting keypair...
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Using autogenerated ssh keypair
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Authenticating as jenkins
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
bootstrap
Aug 29, 2022 2:14:42 PM INFO hudson.slaves.CommandLauncher launch
agent launched for arm64-build-3
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Getting keypair...
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Using autogenerated ssh keypair
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Authenticating as jenkins
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching instance: deb-zw8bj6
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
bootstrap
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Getting keypair...
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Using autogenerated ssh keypair
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Authenticating as jenkins
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching instance: deb-gucyvb
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
bootstrap
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Getting keypair...
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Using autogenerated ssh keypair
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Authenticating as jenkins
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connecting to 10.132.0.24 on port 22, with timeout 10000.
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connecting to 10.132.0.13 on port 22, with timeout 10000.
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connecting to 10.132.0.41 on port 22, with timeout 10000.
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connecting to 10.132.0.16 on port 22, with timeout 10000.
Aug 29, 2022 2:14:42 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connecting to 10.132.0.26 on port 22, with timeout 10000.
Aug 29, 2022 2:14:43 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connected via SSH.
Aug 29, 2022 2:14:43 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connected via SSH.
Aug 29, 2022 2:14:43 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connected via SSH.
Aug 29, 2022 2:14:43 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connected via SSH.
Aug 29, 2022 2:14:43 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Connected via SSH.
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Verifying: java -fullversion
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Verifying: java -fullversion
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Verifying: java -fullversion
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Verifying: java -fullversion
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Verifying: java -fullversion
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Copying agent.jar to: /tmp
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Copying agent.jar to: /tmp
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Copying agent.jar to: /tmp
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Copying agent.jar to: /tmp
Aug 29, 2022 2:14:44 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Copying agent.jar to: /tmp
Aug 29, 2022 2:14:44 PM INFO jenkins.InitReactorRunner$1 onAttained
System config adapted
Aug 29, 2022 2:14:45 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar
Aug 29, 2022 2:14:45 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar
Aug 29, 2022 2:14:45 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar
Aug 29, 2022 2:14:45 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar
Aug 29, 2022 2:14:45 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
Launching Jenkins agent via plugin SSH: java -jar /tmp/agent.jar
Aug 29, 2022 2:14:45 PM INFO jenkins.InitReactorRunner$1 onAttained
Loaded all jobs
Aug 29, 2022 2:14:45 PM INFO jenkins.InitReactorRunner$1 onAttained
Configuration for all jobs updated
Aug 29, 2022 2:14:45 PM INFO hudson.model.AsyncPeriodicWork lambda$doRun$0
Started Download metadata
Aug 29, 2022 2:14:45 PM INFO hudson.model.AsyncPeriodicWork lambda$doRun$0
Finished Download metadata. 14 ms
Aug 29, 2022 2:14:45 PM SEVERE hudson.triggers.SafeTimerTask run
Timer task hudson.model.Queue$MaintainTask@42373f86 failed
java.lang.StackOverflowError
at jenkins.model.lazy.SortedIntList.get(SortedIntList.java:78)
at jenkins.model.lazy.SortedIntList.get(SortedIntList.java:37)
at java.base/java.util.AbstractList$ListItr.previous(AbstractList.java:416)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:491)
at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:428)
at jenkins.model.lazy.LazyBuildMixIn.getLastBuild(LazyBuildMixIn.java:244)
at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:944)
at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:138)
at hudson.model.Job.isLogUpdated(Job.java:314)
at hudson.model.AbstractProject.getCauseOfBlockage(AbstractProject.java:1077)
at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1240)
at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)
at hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:2630)
at hudson.model.AbstractProject.getBuildingDownstream(AbstractProject.java:1115)
at hudson.model.AbstractProject.getCauseOfBlockage(AbstractProject.java:1089)
at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1240)
at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)
at hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:2630)
at hudson.model.AbstractProject.getBuildingDownstream(AbstractProject.java:1115)
at hudson.model.AbstractProject.getCauseOfBlockage(AbstractProject.java:1089)
at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1240)
at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)
at hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:2630)
at hudson.model.AbstractProject.getBuildingDownstream(AbstractProject.java:1115)
at hudson.model.AbstractProject.getCauseOfBlockage(AbstractProject.java:1089)
at hudson.model.Queue.getCauseOfBlockageForTask(Queue.java:1240)
at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)
at hudson.model.Queue$BlockedItem.getCauseOfBlockage(Queue.java:2630)
at hudson.model.AbstractProject.getBuildingDownstream(AbstractProject.java:1115)
at hudson.model.AbstractProject.getCauseOfBlockage(AbstractProject.java:1089)
Hi sylvestre_, sorry to hear you are having this problem on 2.364. Are you also having it on 2.365? And can you share some information about the version of Java you are using in both the working and failing scenario? For example, are you using Java from the official Jenkins Docker image, or your own version? And can you share more of the log file? I don't need the full (hundreds of MiB) log, but something more detailed than what you pasted above would be helpful, especially if it illustrates the repeating pattern or the recursive portion of the stack. What I need to see is the entire portion of the stack trace that repeats over and over again resulting in the infinite loop.
The only significant change to Java code in 2.364 was jenkinsci/jenkins#6907. I have no idea how that could cause a stack overflow, but if you are able to test with that change (and its subsequent follow-up, #7014 - which was released in 2.365) reverted and verify that the problem was caused by that PR, that would be very helpful toward finding a solution. Obviously, any way to reproduce the problem from scratch would be helpful as well.