Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22428

Random crash when the job is running on a slave

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core, warnings-plugin
    • None
    • CentOS 6.4

      Sometimes the warnings plugin crashes when the job is running on an slave (created using Swarm). I can't find a reason and seems to be random at this stage.

      ERROR: Publisher hudson.plugins.warnings.WarningsPublisher aborted due to exception
      hudson.util.IOException2: remote file operation failed: /opt/jenkins-slave/workspace/retail-products-build at hudson.remoting.Channel@323ec485:ci-node-02.company.inet
      at hudson.FilePath.act(FilePath.java:908)
      at hudson.FilePath.act(FilePath.java:885)
      at hudson.plugins.warnings.WarningsPublisher.annotate(WarningsPublisher.java:461)
      at hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:407)
      at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:322)
      at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:333)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:781)
      at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
      at hudson.model.Build$BuildExecution.post2(Build.java:183)
      at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
      at hudson.model.Run.execute(Run.java:1690)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)
      Caused by: java.io.IOException: Remote call on ci-node-02.company.inet failed
      at hudson.remoting.Channel.call(Channel.java:723)
      at hudson.FilePath.act(FilePath.java:901)
      ... 14 more
      Caused by: java.lang.Error: Failed to deserialize the Callable object.
      at hudson.remoting.UserRequest.perform(UserRequest.java:104)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:287)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at hudson.remoting.Engine$1$1.run(Engine.java:60)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.ClassCastException: cannot assign instance of hudson.plugins.analysis.util.TreeString to field hudson.plugins.analysis.util.model.AbstractAnnotation.fileName of type hudson.plugins.analysis.util.TreeString in instance of hudson.plugins.warnings.parser.Warning
      at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2063)
      at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1241)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
      at java.util.HashSet.readObject(HashSet.java:308)
      at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
      at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
      at hudson.remoting.UserRequest.perform(UserRequest.java:98)
      ... 9 more

      OS: CentOS 6.4 (master and slaves)
      Jenkins version: 1.537
      Warnings version: 4.40
      Parser attached as screenshot.

      After this, every build is failing when the Warnings plugins tries to scan the log, and I need to restart the service jenkins-slave running on the slave. After the restart everything starts to work again.

        1. sbt-parser.jpg
          sbt-parser.jpg
          96 kB

          [JENKINS-22428] Random crash when the job is running on a slave

          Julio Gonzalez Gil created issue -

          Ulli Hafner added a comment -

          Seems to be a classloader bug in core. Can you please check with a more recent Jenkins version?

          Ulli Hafner added a comment - Seems to be a classloader bug in core. Can you please check with a more recent Jenkins version?
          Ulli Hafner made changes -
          Component/s New: core [ 15593 ]
          Priority Original: Major [ 3 ] New: Minor [ 4 ]

          Julio Gonzalez Gil added a comment -

          You're right!

          Now I'm checking Jenkins changelog and noticed https://issues.jenkins-ci.org/browse/JENKINS-12124 (Random class loading error mostly known to affect static analysis plugins).

          I'll talk to my IT team to see if we can just upgrade our develenv installation, and will get back with the result. It will take some days to get a date for the upgrade and to check if the plugin is still failing.

          Julio Gonzalez Gil added a comment - You're right! Now I'm checking Jenkins changelog and noticed https://issues.jenkins-ci.org/browse/JENKINS-12124 (Random class loading error mostly known to affect static analysis plugins). I'll talk to my IT team to see if we can just upgrade our develenv installation, and will get back with the result. It will take some days to get a date for the upgrade and to check if the plugin is still failing.

          Julio Gonzalez Gil added a comment -

          I upgraded last week to Jenkins 1.556 just to use the same version as the current develenv at that moment (release 29).

          I thought that Jenkins 1.556 fixed the problem, but as we still received the error, I decided to upgrade to Jenkins 1.558 this monday. I'll be back with more feedback in some days

          Julio Gonzalez Gil added a comment - I upgraded last week to Jenkins 1.556 just to use the same version as the current develenv at that moment (release 29). I thought that Jenkins 1.556 fixed the problem, but as we still received the error, I decided to upgrade to Jenkins 1.558 this monday. I'll be back with more feedback in some days

          Julio Gonzalez Gil added a comment -

          After three weeks using 1.558 we didn't get the error again.

          So it seems to me that, as Ulli pointed, it was bug in core and it's fixed with core 1.558 but not 1.556.

          Should I close the issue?

          Julio Gonzalez Gil added a comment - After three weeks using 1.558 we didn't get the error again. So it seems to me that, as Ulli pointed, it was bug in core and it's fixed with core 1.558 but not 1.556. Should I close the issue?

          Ulli Hafner added a comment -

          Please reopen if it occurs again.

          Ulli Hafner added a comment - Please reopen if it occurs again.
          Ulli Hafner made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

          Sascha Retter added a comment -

          We are having the same issue here. Unfortunately it looks like the problem is persistent and doesn't occur only sometimes.

          We are on Jenkins 1.563

          17:32:25 [WARNINGS] Parsing warnings in console log with parser MSBuild
17:32:39 ERROR: Publisher hudson.plugins.warnings.WarningsPublisher aborted due to exception
17:32:39 java.io.IOException: remote file operation failed: <path> at hudson.remoting.Channel@8da12db:<remote host>
17:32:39 	at hudson.FilePath.act(FilePath.java:916)
17:32:39 	at hudson.FilePath.act(FilePath.java:893)
17:32:39 	at hudson.plugins.warnings.WarningsPublisher.annotate(WarningsPublisher.java:461)
17:32:39 	at hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:407)
17:32:39 	at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:322)
17:32:39 	at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:333)
17:32:39 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
17:32:39 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745)
17:32:39 	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:709)
17:32:39 	at hudson.model.Build$BuildExecution.post2(Build.java:182)
17:32:39 	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:658)
17:32:39 	at hudson.model.Run.execute(Run.java:1731)
17:32:39 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
17:32:39 	at hudson.model.ResourceController.execute(ResourceController.java:88)
17:32:39 	at hudson.model.Executor.run(Executor.java:231)
17:32:39 Caused by: java.io.IOException: Remote call on <remote host> failed
17:32:39 	at hudson.remoting.Channel.call(Channel.java:748)
17:32:39 	at hudson.FilePath.act(FilePath.java:909)
17:32:39 	... 14 more
17:32:39 Caused by: java.lang.Error: Failed to deserialize the Callable object.
17:32:39 	at hudson.remoting.UserRequest.perform(UserRequest.java:104)
17:32:39 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
17:32:39 	at hudson.remoting.Request$2.run(Request.java:326)
17:32:39 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
17:32:39 	at java.util.concurrent.FutureTask.run(Unknown Source)
17:32:39 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
17:32:39 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
17:32:39 	at hudson.remoting.Engine$1$1.run(Engine.java:58)
17:32:39 	at java.lang.Thread.run(Unknown Source)
17:32:39 Caused by: java.lang.ClassCastException: cannot assign instance of hudson.plugins.analysis.util.TreeString to field hudson.plugins.analysis.util.model.AbstractAnnotation.fileName of type hudson.plugins.analysis.util.TreeString in instance of hudson.plugins.warnings.parser.Warning
17:32:39 	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(Unknown Source)
17:32:39 	at java.io.ObjectStreamClass.setObjFieldValues(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readSerialData(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject0(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject(Unknown Source)
17:32:39 	at java.util.HashSet.readObject(Unknown Source)
17:32:39 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:32:39 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
17:32:39 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
17:32:39 	at java.lang.reflect.Method.invoke(Unknown Source)
17:32:39 	at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readSerialData(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject0(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readSerialData(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject0(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readSerialData(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject0(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readSerialData(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject0(Unknown Source)
17:32:39 	at java.io.ObjectInputStream.readObject(Unknown Source)
17:32:39 	at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
17:32:39 	at hudson.remoting.UserRequest.perform(UserRequest.java:98)
17:32:39 	... 8 more

          Sascha Retter added a comment - We are having the same issue here. Unfortunately it looks like the problem is persistent and doesn't occur only sometimes. We are on Jenkins 1.563 17:32:25 [WARNINGS] Parsing warnings in console log with parser MSBuild 17:32:39 ERROR: Publisher hudson.plugins.warnings.WarningsPublisher aborted due to exception 17:32:39 java.io.IOException: remote file operation failed: <path> at hudson.remoting.Channel@8da12db:<remote host> 17:32:39 at hudson.FilePath.act(FilePath.java:916) 17:32:39 at hudson.FilePath.act(FilePath.java:893) 17:32:39 at hudson.plugins.warnings.WarningsPublisher.annotate(WarningsPublisher.java:461) 17:32:39 at hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:407) 17:32:39 at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:322) 17:32:39 at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:333) 17:32:39 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 17:32:39 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745) 17:32:39 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:709) 17:32:39 at hudson.model.Build$BuildExecution.post2(Build.java:182) 17:32:39 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:658) 17:32:39 at hudson.model.Run.execute(Run.java:1731) 17:32:39 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 17:32:39 at hudson.model.ResourceController.execute(ResourceController.java:88) 17:32:39 at hudson.model.Executor.run(Executor.java:231) 17:32:39 Caused by: java.io.IOException: Remote call on <remote host> failed 17:32:39 at hudson.remoting.Channel.call(Channel.java:748) 17:32:39 at hudson.FilePath.act(FilePath.java:909) 17:32:39 ... 14 more 17:32:39 Caused by: java.lang.Error: Failed to deserialize the Callable object. 17:32:39 at hudson.remoting.UserRequest.perform(UserRequest.java:104) 17:32:39 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 17:32:39 at hudson.remoting.Request$2.run(Request.java:326) 17:32:39 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 17:32:39 at java.util.concurrent.FutureTask.run(Unknown Source) 17:32:39 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 17:32:39 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 17:32:39 at hudson.remoting.Engine$1$1.run(Engine.java:58) 17:32:39 at java.lang. Thread .run(Unknown Source) 17:32:39 Caused by: java.lang.ClassCastException: cannot assign instance of hudson.plugins.analysis.util.TreeString to field hudson.plugins.analysis.util.model.AbstractAnnotation.fileName of type hudson.plugins.analysis.util.TreeString in instance of hudson.plugins.warnings.parser.Warning 17:32:39 at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(Unknown Source) 17:32:39 at java.io.ObjectStreamClass.setObjFieldValues(Unknown Source) 17:32:39 at java.io.ObjectInputStream.defaultReadFields(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readSerialData(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject0(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject(Unknown Source) 17:32:39 at java.util.HashSet.readObject(Unknown Source) 17:32:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 17:32:39 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 17:32:39 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 17:32:39 at java.lang.reflect.Method.invoke(Unknown Source) 17:32:39 at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readSerialData(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject0(Unknown Source) 17:32:39 at java.io.ObjectInputStream.defaultReadFields(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readSerialData(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject0(Unknown Source) 17:32:39 at java.io.ObjectInputStream.defaultReadFields(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readSerialData(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject0(Unknown Source) 17:32:39 at java.io.ObjectInputStream.defaultReadFields(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readSerialData(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject0(Unknown Source) 17:32:39 at java.io.ObjectInputStream.readObject(Unknown Source) 17:32:39 at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) 17:32:39 at hudson.remoting.UserRequest.perform(UserRequest.java:98) 17:32:39 ... 8 more
          Sascha Retter made changes -
          Resolution Original: Fixed [ 1 ]
          Status Original: Resolved [ 5 ] New: Reopened [ 4 ]

            drulli Ulli Hafner
            juliogonzalez Julio Gonzalez Gil
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: