-
Bug
-
Resolution: Duplicate
-
Blocker
-
Jenkins 2.19.1, 2.19.2
java.version 1.8.0_112
java.vm.vendor Oracle Corporation
java.vm.version 25.112-b15
os.arch amd64
os.name Linux
BUILD_TIMESTAMP 20161111_1034
Maven 3.3.9
We have a similar exception as in JENKINS-35184 but this happens with builds only on master. In the hope that it is fixed we upgraded to 2.19.2 but it didn't help.
The exception occurs on a random basis and cannot be pinpointed to a specific build, plugin, time,..
When it happens, all builds after that tend to fail as well. We currently have to do a restart of Jenkins on a daily basis and sometimes multiple times.
Restarting Jenkins is the only workaround we currently have and is growing into a pain especially when we are releasing. This issue is a blocker for us.
Our setup is:
- one master Jenkins running mostly all builds
- one Windows slave for running selenium tests
- most builds are run with Maven using the Maven Project Type job (seems to be important)
StackTrace from the build logs:
java.io.IOException: Remote call on Channel to Maven [/opt/jdk1.8.0_112/bin/java, -cp, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-agent-1.8.1.jar:/opt/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar:/opt/apache-maven-3.3.9/conf/logging, jenkins.maven3.agent.Maven33Main, /opt/apache-maven-3.3.9, /var/cache/jenkins/war/WEB-INF/lib/remoting-2.62.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-interceptor-1.8.1.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar, 36506] failed at hudson.remoting.Channel.call(Channel.java:805) at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:161) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:873) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1720) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:404) Caused by: java.lang.LinkageError: Failed to load jenkins.util.SystemProperties at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:377) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.Main.<clinit>(Main.java:222) at hudson.EnvVars.initMaster(EnvVars.java:432) at hudson.EnvVars.<clinit>(EnvVars.java:427) at java.io.ObjectStreamClass.hasStaticInitializer(Native Method) at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787) at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253) at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at hudson.remoting.UserRequest.deserialize(UserRequest.java:217) at hudson.remoting.UserRequest.perform(UserRequest.java:131) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at ......remote call to Channel to Maven [/opt/jdk1.8.0_112/bin/java, -cp, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-agent-1.8.1.jar:/opt/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar:/opt/apache-maven-3.3.9/conf/logging, jenkins.maven3.agent.Maven33Main, /opt/apache-maven-3.3.9, /var/cache/jenkins/war/WEB-INF/lib/remoting-2.62.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven33-interceptor-1.8.1.jar, /opt/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar, 36506](Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:797) ... 7 more Caused by: java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:373) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.Main.<clinit>(Main.java:222) at hudson.EnvVars.initMaster(EnvVars.java:432) at hudson.EnvVars.<clinit>(EnvVars.java:427) at java.io.ObjectStreamClass.hasStaticInitializer(Native Method) at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787) at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253) at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at hudson.remoting.UserRequest.deserialize(UserRequest.java:217) at hudson.remoting.UserRequest.perform(UserRequest.java:131) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: javax.servlet.ServletContextListener (java.net.URLClassLoader@3f99bd52 did not find class file) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch2(RemoteClassLoader.java:792) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch4(RemoteClassLoader.java:842) at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch3(RemoteClassLoader.java:849) at sun.reflect.GeneratedMethodAccessor208.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:887) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at ......remote call to channel(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433) at hudson.remoting.Request.call(Request.java:172) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:255) at com.sun.proxy.$Proxy2.fetch3(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:187) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 36 more
Installed plugins
ace-editor 1.1 true false active-directory 1.49 true false analysis-core 1.79 true false android-emulator 2.15 true false android-lint 2.4 true false ant 1.4 true false antisamy-markup-formatter 1.5 true false authorize-project 1.2.2 true false bouncycastle-api 2.16.0 true false branch-api 1.11 true false build-failure-analyzer 1.17.2 true false build-monitor-plugin 1.10+build.201610041454 true false build-name-setter 1.6.5 true false build-pipeline-plugin 1.5.4 true false build-timeout 1.17.1 true false build-timestamp 1.0.1 true false claim 2.8 true false cloudbees-folder 5.13 true false conditional-buildstep 1.3.5 true false config-file-provider 2.13 true false credentials 2.1.8 true false credentials-binding 1.9 true false dashboard-view 2.9.10 true false display-url-api 0.5 true false durable-task 1.12 true false external-monitor-job 1.6 true false git 3.0.0 true false git-client 2.0.0 true false git-parameter 0.7.0 true false git-server 1.7 true false gradle 1.25 true false greenballs 1.15 true false handlebars 1.1.1 true false htmlpublisher 1.11 true false icon-shim 2.0.3 true false javadoc 1.4 true false jobConfigHistory 2.15 true false jobgenerator 1.22 true false jquery 1.11.2-0 true false jquery-detached 1.2.1 true false junit 1.19 true false ldap 1.13 true false locale 1.2 true false log-parser 2.0 true false m2release 0.14.0 true false mailer 1.18 true false matrix-auth 1.4 true false matrix-project 1.7.1 true false maven-plugin 2.14 true false momentjs 1.1.1 true false nodejs 0.2.1 true false pam-auth 1.3 true false parameterized-trigger 2.32 true false pipeline-build-step 2.3 true false pipeline-graph-analysis 1.2 true false pipeline-input-step 2.3 true false pipeline-milestone-step 1.1 true false pipeline-rest-api 2.1 true false pipeline-stage-step 2.2 true false pipeline-stage-view 2.1 true false plain-credentials 1.3 true false port-allocator 1.8 true false protex-jenkins 1.4.0 true false publish-over-ssh 1.14 true false run-condition 1.0 true false scm-api 1.3 true false script-security 1.24 true false scriptler 2.9 true false sectioned-view 1.20 true false sonar 2.5 true false ssh-agent 1.13 true false ssh-credentials 1.12 true false ssh-slaves 1.11 true false stash-pullrequest-builder 1.7.0 true false stashNotifier 1.11.4 true false structs 1.5 true false thinBackup 1.7.4 true false token-macro 2.0 true false windows-slaves 1.2 true false workflow-aggregator 2.4 true false workflow-api 2.5 true false workflow-basic-steps 2.2 true false workflow-cps 2.21 true false workflow-cps-global-lib 2.4 true false workflow-durable-task-step 2.5 true false workflow-job 2.8 true false workflow-multibranch 2.9 true false workflow-scm-step 2.2 true false workflow-step-api 2.4 true false workflow-support 2.10 true false ws-cleanup 0.30 true false
We added additional loggers: hudson.remoting.RemoteClassLoader and org.jenkinsci.plugins but they did not provide helpful information.
As discussed in JENKINS-35184 this might be caused by a plugin especially the maven plugin thus I set it as component as well.
If there is more information needed please state so, we will gladly help to get this fixed
- duplicates
-
JENKINS-46386 Agent sometimes fails to launch in custom cloud provider
-
- Resolved
-
- relates to
-
JENKINS-35184 Failed to load jenkins.util.SystemProperties on slaves due to ServletContext
-
- Resolved
-
-
JENKINS-26677 Blacklist certain classes from loading in slaves
-
- Open
-
We found a workaround - which is not nice.
The workaround is to put the servlet-api jar (containing ServletContextListener.class) on the classpath of a Maven build. We did this by copying the jar (http://search.maven.org/remotecontent?filepath=javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar) into the /lib/ext folder of the JRE.
This is not nice but I wanted to document it here in case somebody else runs into this problem.
Some explanations:
It looks like the exception stems from a remote process and we suspect it comes from Maven builds (Jobs with Maven Project Type). As these run in their own Java Process it looks like the Systemproperties are somewhat loaded in that process but don't have the necessary dependencies.
After looking in the pom.xml on Github, we saw the dependency to the javax.servlet-api in scope "provided".
The Jar itself is contained in the winstone container but not in standard Java libraries.
As cannot figure out when this happens it was worth a try to just provide the class to see if it works or produces a different error.
Builds are stable since two days - but still .. this bugs needs to be fixed.