-
Bug
-
Resolution: Fixed
-
Blocker
-
Core versions: Jenkins-1.509.2, Jenkins-1.509.3, Jenkins-1.509.4-RC
Master: Windows server 2007 SP1; RHEL6
Slave: Linux Mint 64 bits; Windows 7 (JNLP)
I am able to start slave.jar via ssh or via jnlp.
The log of the connection shows that the node is correctly started.
In a project, I have 2 main parts:
- a svn checkout
- a shell command to start cmake + compilation.
If I launch my build, it hangs. Then, I disabled the svn checkout and leave only the shell command with a "echo hello".
If I start the build, it hangs with the following error message:
Started by user anonymous
[EnvInject] - Loading node environment variables.
Building remotely on slave_linux_64 in workspace /home/crystal/jenkins/workspace/TESTLinux
FATAL: command execution failed
java.io.IOException: Remote call on slave_linux_64 failed
at hudson.remoting.Channel.call(Channel.java:723)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
at hudson.Launcher$ProcStarter.start(Launcher.java:353)
at hudson.Launcher$ProcStarter.join(Launcher.java:360)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
at hudson.model.Run.execute(Run.java:1603)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:247)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Thread.java:724)
Build step 'Execute shell' marked build as failure
Finished: FAILURE
When I looked at the jenkins directory I configured on the slave, all the required jar are present, the workspace directory is present and the build directory is correctly created by the jenkins master. But nothing in this directory.
- is blocking
-
JENKINS-15199 Slaves turns to "Dead" state after connecting to remote machine
-
- Resolved
-
- is duplicated by
-
JENKINS-15796 NoClassDefFoundError in monitoring Windows slave
-
- Resolved
-
- is related to
-
JENKINS-25316 NoClassDefFoundError for hudson.Util on Windows slave
-
- Open
-
-
JENKINS-36991 Unable to load class once the loading was interrupted
-
- Resolved
-
- relates to
-
JENKINS-26677 Blacklist certain classes from loading in slaves
-
- Open
-
[JENKINS-19453] Slave launcher fails after NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
Update to remoting-2.32 in 1.509.4 does not solve the issue.
Seems that is is being reproduced even more frequently.
According to monitoring statistics, issue has became twice more frequent after migration to 1.509.4
Seems that issue is caused by initialization of static channel instance, but I cannot catch it in the debugger.
I’ve found a reproducible scenario for the issue:
- Create a node, which has big network delays / small throughput to a master
- Reconnect a node to have a fresh classloader
- Run the job, which invokes shell/batch script
- When the jobs performs a classloading for “Execute shell/batch”, abort the job (you will see the log below)
- After that, each job utilizing remote calls will fail with a “java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer” error
Seems that issue is in incorrect behavior of remote call's during a termination of classloading processes
A stacktrace from the step #4
Run condition [Regular expression match] preventing perform for step [Inject environment variables]
Regular expression run condition: Expression=[.*linux.*], Label=[64bit cn60 cn60-custom-archw-benchmarking custom haps51 win7]
Run condition [Regular expression match] preventing perform for step [Inject environment variables]
FATAL: command execution failed
java.io.InterruptedIOException
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:864)
at com.cloudbees.jenkins.plugins.customtools.DecoratedLauncher.launch(DecoratedLauncher.java:49)
at com.cloudbees.jenkins.plugins.customtools.CustomToolInstallWrapper$2.launch(CustomToolInstallWrapper.java:218)
at hudson.Launcher$ProcStarter.start(Launcher.java:353)
at hudson.Launcher$ProcStarter.join(Launcher.java:360)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:780)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:565)
at hudson.model.Run.execute(Run.java:1592)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:237)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:146)
at hudson.remoting.Channel.call(Channel.java:714)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
... 15 more
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE
Have not tried to reproduce this yet, but I am guessing that the problem lies in RemoteClassLoader since it rethrows an InterruptedException as a ClassNotFoundException. So if some task which was doing class loading is interrupted by the user, the class loader remembers the fact that the class in progress (Jenkins?) was not found, marks the class referring to it (MasterComputer?) as uninitialized, and thereafter refuses to let any code (LocalLauncher) use the referring class.
(A similar problem might occur when an IOException is wrapped as a ClassNotFoundException, but I guess this is less likely, since an IOException probably means either that class loading on the master is broken too and Jenkins is hosed, or that the socket used by the remoting channel is down and the slave agent is as good as dead.)
If that is the right diagnosis, I am unsure how to fix it. As far as I know Java offers no way for a ClassLoader to acknowledge that initialization of a class failed earlier with a NoClassDefFoundError but that this was a transient error and it should be retried. Maybe throwing a more serious error (such as ThreadStop) up the call chain will stop it from caching the unwanted state, but I doubt it—the caching seems to be done in native code on Oracle JREs at least.
Perhaps it would be possible to notice these transient failures during class loading and simply kill off the whole RemoteClassLoader and recreate it? Not exactly sure how that would work.
Is it possible to somehow synchronize Channel::terminate() with ClassLoader operations?
It may provide a workaround for the described scenario.
I think I have reproduced this scenario in a unit test, though there are complications that I would need @kohsuke’s assistance with.
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/hudson/remoting/RemoteClassLoader.java
src/test/java/hudson/remoting/ClassRemotingTest.java
src/test/java/hudson/remoting/TestLinkage.java
http://jenkins-ci.org/commit/remoting/5532ffe6c987278cb0acbb1e6cdce96075a0e52f
Log:
JENKINS-19453 Seem to have reproduced the problem in a unit test.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/RemoteClassLoader.java
src/test/java/hudson/remoting/ClassRemotingTest.java
src/test/java/hudson/remoting/DummyClassLoader.java
src/test/java/hudson/remoting/TestLinkage.java
http://jenkins-ci.org/commit/remoting/4efb4e1dd5675a5d22993b09ebb65adab5baa9e0
Log:
JENKINS-19453 Propose fix: mask interrupt while loading a class
I think the crux of the issue is that, in general, a failed class loading
will never get retried. It is certainly the case when another class loading has triggered a
recurisve class loading as in this test case.
In https://github.com/jenkinsci/remoting/pull/19 Jesse proposed to fix
this by dropping the whole RemoteClassLoader. But this will not work
since the said RemoteClassLoader cloud have loaded other classes that
might be already running. Dropping a classloader will not drop these
classes, and we end up just creating another classloader that loads
incompatible classes. That is a disaster waiting to happen.
In this fix, we simply make findClass non-interruptible. That is, if a
blocking remote operation gets interrupted, we catch that and simply
retry and refuse to give up. We'll remember to set the interrupt flag
back on, however, so that the interrupt signal doesn't disappear.
The net effect is as if the delivery of the interrupt was bit late than
usual. The way I see it, this is the only way to fix this.
The nesting of control structures here is horrible, but I can't think of
any better way to write this.
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/hudson/remoting/RemoteClassLoader.java
src/test/java/hudson/remoting/ClassRemotingTest.java
src/test/java/hudson/remoting/TestLinkage.java
http://jenkins-ci.org/commit/remoting/b68e3e92b24942585fff517bb4bad1878883256f
Log:
JENKINS-19453 Seem to have reproduced the problem in a unit test.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/RemoteClassLoader.java
src/test/java/hudson/remoting/ClassRemotingTest.java
src/test/java/hudson/remoting/DummyClassLoader.java
src/test/java/hudson/remoting/TestLinkage.java
http://jenkins-ci.org/commit/remoting/425301511aeb5a876b224cf4e572b74447e08ebc
Log:
[FIXED JENKINS-19453] Propose fix: mask interrupt while loading a class
I think the crux of the issue is that, in general, a failed class loading
will never get retried. It is certainly the case when another class loading has triggered a
recurisve class loading as in this test case.
In https://github.com/jenkinsci/remoting/pull/19 Jesse proposed to fix
this by dropping the whole RemoteClassLoader. But this will not work
since the said RemoteClassLoader cloud have loaded other classes that
might be already running. Dropping a classloader will not drop these
classes, and we end up just creating another classloader that loads
incompatible classes. That is a disaster waiting to happen.
In this fix, we simply make findClass non-interruptible. That is, if a
blocking remote operation gets interrupted, we catch that and simply
retry and refuse to give up. We'll remember to set the interrupt flag
back on, however, so that the interrupt signal doesn't disappear.
The net effect is as if the delivery of the interrupt was bit late than
usual. The way I see it, this is the only way to fix this.
The nesting of control structures here is horrible, but I can't think of
any better way to write this.
Compare: https://github.com/jenkinsci/remoting/compare/bfbc7d693a50...425301511aeb
Not actually fixed yet, since core still uses remoting 2.32; fix is in remoting 2.33. @kohsuke including the FIXED text in the commit comment for a subcomponent is a bad idea for this reason—the JIRA updater just assumes the fix is actually integrated when this commit appears in the master branch of its repository.
Actually fixed just now in https://github.com/jenkinsci/jenkins/commit/4137c46182a0652d2eed2897a1e797bdc3f8c7f9 which did not record any issue IDs in its comment so it would not have been noticed by the JIRA link daemon.
Code changed in jenkins
User: Oliver Gondža
Path:
http://jenkins-ci.org/commit/jenkins/5cf3e28c4885a86a4dad3463952e1aae34932f9c
Log:
[FIXED JENKINS-20093 JENKINS-19453 JENKINS-19004 JENKINS-8856] Noting retroactively
Backported as 23424ea7dbced486963a73d6bf56e8367ab779dd
Compare: https://github.com/jenkinsci/jenkins/compare/7d602d52b445...5cf3e28c4885
Build of this alone applied to 1.532.1: http://maven.jenkins-ci.org:8081/content/repositories/releases/org/jenkins-ci/main/jenkins-war/1.532.1.JENKINS-19453/jenkins-war-1.532.1.JENKINS-19453.war
For the record I just saw this error in 1.553 One of the slaves started to fail the builds with this error and i had to disconnect it and then reconnect. That seemed to have resolved the issue for now.
@eguess74
Please check that slave's version is 2.33 or greater.
In some cases (e.g. for Windows services) you may have to update the slave.jar manually
I am still seeing an issue with a Windows client on 2.33. The stack trace is as follows:
hudson.remoting.Channel@7c2ac188:Win7x86
at hudson.FilePath.act(FilePath.java:907)
at hudson.FilePath.act(FilePath.java:884)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:935)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:870)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561)
at hudson.model.Run.execute(Run.java:1665)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:246)
Caused by: java.io.IOException: Remote call on Win7x86 failed
at hudson.remoting.Channel.call(Channel.java:731)
at hudson.FilePath.act(FilePath.java:900)
... 11 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins
at hudson.scm.SubversionSCM.descriptor(SubversionSCM.java:2504)
at hudson.scm.SubversionSCM.createDefaultSVNOptions(SubversionSCM.java:1084)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1074)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1006)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:986)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2439)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: FAILURE
I just upgraded to 1.532.2 and this issue is still occurring. I also manually upgraded to 2.33 on the slave.
@chotchki a particular bug was found, reproduced, and fixed. Whatever you are seeing might be related, or it might not. Better to file it separately, with whatever details can be found (steps to reproduce if possible), and link to this one.
I'm seeing this problem in another instance, and slave log says it's failing to write to local jar file cache. I wonder if that could somehow cause this problem?
I got such issue on build Jenkins ver. 1.558
slave connected by jnlp
Here is stack trace:
FATAL: command execution failed
java.io.IOException: Remote call on slave_windows failed
at hudson.remoting.Channel.call(Channel.java:731)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
at hudson.Launcher$ProcStarter.start(Launcher.java:353)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.Build$BuildExecution.build(Build.java:198)
at hudson.model.Build$BuildExecution.doRun(Build.java:159)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:523)
at hudson.model.Run.execute(Run.java:1689)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Build step 'Execute Windows batch command' marked build as failure
FATAL: Remote call on adft-temp-s failed
java.io.IOException: Remote call on adft-temp-s failed
at hudson.remoting.Channel.call(Channel.java:731)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:887)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:546)
at hudson.model.Run.execute(Run.java:1689)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
Caused by: java.lang.Error: Unable to load resource winp.dll
at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:380)
at java.lang.ClassLoader.getResource(Unknown Source)
at org.jvnet.winp.Native.load(Native.java:84)
at org.jvnet.winp.Native.<clinit>(Native.java:55)
at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:477)
at hudson.util.ProcessTree.get(ProcessTree.java:336)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unable to call writeJarTo. Invalid object ID 2
at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:75)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:372)
... 17 more
Caused by: java.lang.IllegalStateException: Unable to call writeJarTo. Invalid object ID 2
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:293)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
... 1 more
Reproduced on 1.585:
Started by user Vadym Vikulin
[EnvInject] - Loading node environment variables.
Building remotely on win7-node-01 (Win7) in workspace C:\work\workspace\Win7-ED
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
CLASSPATH=C:\work\edtest\common\libs*:C:\work\edtest\common;C:\work\edtest\all\edJava;C:\work\edtest\all\edNative/libs/*
[EnvInject] - Variables injected successfully.
[Win7-ED] $ cmd /c call C:\Users\VADYMV~1\AppData\Local\Temp\hudson142281969965340690.bat
FATAL: command execution failed
java.io.IOException: Remote call on win7-node-01 failed
at hudson.remoting.Channel.call(Channel.java:751)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:915)
at hudson.Launcher$ProcStarter.start(Launcher.java:380)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
at hudson.model.Run.execute(Run.java:1759)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class hudson.Util
at hudson.EnvVars$OverrideOrderCalculator.scan(EnvVars.java:281)
at hudson.EnvVars$OverrideOrderCalculator.<init>(EnvVars.java:223)
at hudson.EnvVars.overrideExpandingAll(EnvVars.java:329)
at hudson.Launcher.inherit(Launcher.java:1227)
at hudson.Launcher.inherit(Launcher.java:1219)
at hudson.Launcher.access$200(Launcher.java:81)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:795)
at hudson.Launcher$ProcStarter.start(Launcher.java:380)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1135)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1100)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
FATAL: Unable to delete script file C:\Users\VADYMV~1\AppData\Local\Temp\hudson142281969965340690.bat
java.io.IOException: remote file operation failed: C:\Users\VADYMV~1\AppData\Local\Temp\hudson142281969965340690.bat at hudson.remoting.Channel@5dc020b1:win7-node-01
at hudson.FilePath.act(FilePath.java:914)
at hudson.FilePath.act(FilePath.java:896)
at hudson.FilePath.delete(FilePath.java:1321)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:106)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
at hudson.model.Run.execute(Run.java:1759)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.io.IOException: Remote call on win7-node-01 failed
at hudson.remoting.Channel.call(Channel.java:751)
at hudson.FilePath.act(FilePath.java:907)
... 13 more
vikulin your problem is probably distinct and should be filed separately. Look for an earlier message explaining why hudson.Util could not be loaded.
Solution :- If you are running slave on windows VM\Server with 64 bit OS
Make sure of few things
1) You should have 32 bit JRE Installed
2) It should be default JRE - you go to configure Java and change the settings if you have 64 bit and 32 bit both change the default to 32 bit path.
3) Restart the Slave and that's it, Vola
Hi!
I have still this problem (Jenkins ver. 1.580.2):
<code>
FATAL: Remote call on win7-home-premium-x86-ru-1-f8e failed
20:33:18 java.io.IOException: Remote call on win7-home-premium-x86-ru-1-f8e failed
20:33:18 at hudson.remoting.Channel.call(Channel.java:760)
20:33:18 at hudson.Launcher$RemoteLauncher.kill(Launcher.java:941)
20:33:18 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:556)
20:33:18 at hudson.model.Run.execute(Run.java:1745)
20:33:18 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
20:33:18 at hudson.model.ResourceController.execute(ResourceController.java:89)
20:33:18 at hudson.model.Executor.run(Executor.java:240)
20:33:18 Caused by: java.lang.Error: Unable to load resource winp.dll
20:33:18 at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:381)
20:33:18 at java.lang.ClassLoader.getResource(Unknown Source)
20:33:18 at org.jvnet.winp.Native.load(Native.java:85)
20:33:18 at org.jvnet.winp.Native.<clinit>(Native.java:56)
20:33:18 at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:212)
20:33:18 at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:469)
20:33:18 at hudson.util.ProcessTree.get(ProcessTree.java:328)
20:33:18 at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:953)
20:33:18 at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:944)
20:33:18 at hudson.remoting.UserRequest.perform(UserRequest.java:121)
20:33:18 at hudson.remoting.UserRequest.perform(UserRequest.java:49)
20:33:18 at hudson.remoting.Request$2.run(Request.java:324)
20:33:18 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
20:33:18 at java.util.concurrent.FutureTask.run(Unknown Source)
20:33:18 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
20:33:18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
20:33:18 at hudson.remoting.Engine$1$1.run(Engine.java:63)
20:33:18 at java.lang.Thread.run(Unknown Source)
20:33:18 at ......remote call to yabrowser-win7-home-premium-x86-ru-1-f8e(Native Method)
20:33:18 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
20:33:18 at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
20:33:18 at hudson.remoting.Channel.call(Channel.java:752)
20:33:18 ... 6 more
20:33:18 Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Invalid object ID 2 iota=14
20:33:18 at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:75)
20:33:18 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
20:33:18 at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:373)
20:33:18 at java.lang.ClassLoader.getResource(Unknown Source)
20:33:18 at org.jvnet.winp.Native.load(Native.java:85)
20:33:18 at org.jvnet.winp.Native.<clinit>(Native.java:56)
20:33:18 at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:212)
20:33:18 at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:469)
20:33:18 at hudson.util.ProcessTree.get(ProcessTree.java:328)
20:33:18 at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:953)
20:33:18 at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:944)
20:33:18 at hudson.remoting.UserRequest.perform(UserRequest.java:121)
20:33:18 at hudson.remoting.UserRequest.perform(UserRequest.java:49)
20:33:18 at hudson.remoting.Request$2.run(Request.java:324)
20:33:18 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
20:33:18 at java.util.concurrent.FutureTask.run(Unknown Source)
20:33:18 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
20:33:18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
20:33:18 at hudson.remoting.Engine$1$1.run(Engine.java:63)
20:33:18 at java.lang.Thread.run(Unknown Source)
20:33:18 Caused by: java.lang.IllegalStateException: Invalid object ID 2 iota=14
20:33:18 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:348)
20:33:18 at hudson.remoting.ExportTable.get(ExportTable.java:324)
20:33:18 at hudson.remoting.Channel.getExportedObject(Channel.java:604)
20:33:18 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:311)
20:33:18 at hudson.remoting.Request$2.run(Request.java:324)
20:33:18 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
20:33:18 at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
20:33:18 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
20:33:18 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
20:33:18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
20:33:18 at java.lang.Thread.run(Thread.java:744)
20:33:18 at ......remote call to channel(Native Method)
20:33:18 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
20:33:18 at hudson.remoting.Request.call(Request.java:171)
20:33:18 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:176)
20:33:18 at com.sun.proxy.$Proxy5.writeJarTo(Unknown Source)
20:33:18 at hudson.remoting.FileSystemJarCache.retrieve(FileSystemJarCache.java:66)
20:33:18 at hudson.remoting.JarCacheSupport$1.run(JarCacheSupport.java:62)
20:33:18 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
20:33:18 at java.util.concurrent.FutureTask.run(Unknown Source)
20:33:18 at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
20:33:18 ... 1 more
20:33:18 Caused by: java.lang.Exception: Object was recently deallocated
20:33:18 #2 (ref.0) : [hudson.remoting.JarLoader]
20:33:18 Created at Wed Dec 24 20:28:20 MSK 2014
20:33:18 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:93)
20:33:18 at hudson.remoting.ExportTable.export(ExportTable.java:299)
20:33:18 at hudson.remoting.Channel.internalExport(Channel.java:600)
20:33:18 at hudson.remoting.Channel.export(Channel.java:591)
20:33:18 at hudson.remoting.Channel.<init>(Channel.java:472)
20:33:18 at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:280)
20:33:18 at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:290)
20:33:18 at org.jenkinsci.remoting.nio.NioChannelBuilder.build(NioChannelBuilder.java:36)
20:33:18 at org.jenkinsci.remoting.nio.NioChannelBuilder.build(NioChannelBuilder.java:52)
20:33:18 at jenkins.slaves.JnlpSlaveAgentProtocol$Handler.jnlpConnect(JnlpSlaveAgentProtocol.java:120)
20:33:18 at jenkins.slaves.DefaultJnlpSlaveReceiver.handle(DefaultJnlpSlaveReceiver.java:63)
20:33:18 at jenkins.slaves.JnlpSlaveAgentProtocol2$Handler2.run(JnlpSlaveAgentProtocol2.java:57)
20:33:18 at jenkins.slaves.JnlpSlaveAgentProtocol2.handle(JnlpSlaveAgentProtocol2.java:31)
20:33:18 at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:156)
20:33:18 Released at Wed Dec 24 20:28:20 MSK 2014
20:33:18 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:125)
20:33:18 at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:376)
20:33:18 at hudson.remoting.Channel.unexport(Channel.java:612)
20:33:18 at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
20:33:18 at hudson.remoting.Channel$2.handle(Channel.java:483)
20:33:18 at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61)
20:33:18 at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:597)
20:33:18 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
20:33:18 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
20:33:18 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
20:33:18 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
20:33:18 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
20:33:18 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
20:33:18 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
20:33:18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
20:33:18 at java.lang.Thread.run(Thread.java:744)
20:33:18 Caused by: Command hudson.remoting.UnexportCommand@1787906e created at
20:33:18 at hudson.remoting.Command.<init>(Command.java:67)
20:33:18 at hudson.remoting.Command.<init>(Command.java:50)
20:33:18 at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
20:33:18 at hudson.remoting.RemoteInvocationHandler.finalize(RemoteInvocationHandler.java:234)
20:33:18 at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
20:33:18 at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
20:33:18 at java.lang.ref.Finalizer.access$100(Unknown Source)
20:33:18 at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
20:33:18 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:341)
20:33:18 at hudson.remoting.ExportTable.get(ExportTable.java:324)
20:33:18 at hudson.remoting.Channel.getExportedObject(Channel.java:604)
20:33:18 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:311)
20:33:18 at hudson.remoting.Request$2.run(Request.java:324)
20:33:18 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
20:33:18 at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
20:33:18 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
20:33:18 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
20:33:18 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
20:33:18 at java.lang.Thread.run(Thread.java:744)
20:33:18 Caused by: Released at Wed Dec 24 20:28:20 MSK 2014
20:33:18 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:125)
20:33:18 at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:376)
20:33:18 at hudson.remoting.Channel.unexport(Channel.java:612)
20:33:18 at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
20:33:18 at hudson.remoting.Channel$2.handle(Channel.java:483)
20:33:18 at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61)
20:33:18 at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:597)
20:33:18 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
20:33:18 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
20:33:18 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
20:33:18 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
20:33:18 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
20:33:18 ... 4 more
20:33:18 Caused by: Command hudson.remoting.UnexportCommand@1539730b created at
20:33:18 at hudson.remoting.Command.<init>(Command.java:67)
20:33:18 at hudson.remoting.Command.<init>(Command.java:50)
20:33:18 at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
20:33:18 at hudson.remoting.RemoteInvocationHandler.finalize(RemoteInvocationHandler.java:234)
20:33:18 at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
20:33:18 at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
20:33:18 at java.lang.ref.Finalizer.access$100(Unknown Source)
20:33:18 at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
</code>
Statistically, one from 10 slaves break connection with such error.
svchipiga your problem (IllegalStateException: Invalid object ID 2 iota=14) looks unrelated and should be filed separately.
We experience this issue since the migration to 1.509.2. It is being reproduced in 1.509.3 as well.
Seems that Windows JNLP slaves periodically fail into the death loop