Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33852

Exception while copying in thread - causes Slave disconnection

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • core, remoting
    • Windows Server 2008 R2 (both Slaves and Master)
      Jenkins ver. 1.654

    Description

      It's been happening several weeks already, that a Java process that being executed within Maven command, get killed randomly, as the Jenkins slave disconnects the connection unexpectedly, with error:

      WARNING: Exception while copying in thread: Copier for /127.0.0.1:{RANDOM PORT}
      java.net.SocketException: Connection reset
      at java.net.SocketInputStream.read(Unknown Source)
      at java.net.SocketInputStream.read(Unknown Source)
      at java.io.FilterInputStream.read(Unknown Source)
      at java.io.FilterInputStream.read(Unknown Source)
      at hudson.remoting.forward.CopyThread.run(CopyThread.java:30)

      Looking at Job console, we see that the Maven process completes, but the inner Java thread is terminated:

      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 02:26 h
      [INFO] Finished at: 2016-03-28T17:23:46+03:00
      [INFO] Final Memory: 33M/220M
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test (failsafe-integration-tests) on project AF: Execution failsafe-integration-tests of goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test failed: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
      [ERROR] Command was cmd.exe /X /C ""C:\Program Files\Java\jdk1.8.0_65\jre\bin\java" -Xmx4096m -jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefirebooter4689747360754285342.jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire1687639200913429704tmp c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire_07436515838238741397tmp"
      [ERROR] -> [Help 1]
      org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test (failsafe-integration-tests) on project AF: Execution failsafe-integration-tests of goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test failed: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
      Command was cmd.exe /X /C ""C:\Program Files\Java\jdk1.8.0_65\jre\bin\java" -Xmx4096m -jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefirebooter4689747360754285342.jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire1687639200913429704tmp c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire_07436515838238741397tmp"
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
      at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
      at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
      at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
      at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
      at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
      at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
      at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
      at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
      Caused by: org.apache.maven.plugin.PluginExecutionException: Execution failsafe-integration-tests of goal org.apache.maven.plugins:maven-failsafe-plugin:2.19.1:integration-test failed: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
      Command was cmd.exe /X /C ""C:\Program Files\Java\jdk1.8.0_65\jre\bin\java" -Xmx4096m -jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefirebooter4689747360754285342.jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire1687639200913429704tmp c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire_07436515838238741397tmp"
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
      at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
      ... 20 more
      Caused by: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
      Command was cmd.exe /X /C ""C:\Program Files\Java\jdk1.8.0_65\jre\bin\java" -Xmx4096m -jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefirebooter4689747360754285342.jar c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire1687639200913429704tmp c:\jenkins-slave\workspace\Automation_Matrix_New\55a7bc48\surefire\surefire_07436515838238741397tmp"
      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:590)
      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:460)
      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:229)
      at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:201)
      at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1026)
      at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:862)
      at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:755)
      at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
      ... 21 more

      Attachments

        Activity

          nmanos Noam Manos added a comment -

          Many attempts to find cause in Maven configuration, or in Java implementation, or to set any known Jenkins configuration, to workaround this problem, did not succeed, and this unpredictable error kept on happening, until...
          I asked our IT guys to increase CPU and RAM (from 4GB to 16GB), and walla - the problem disappeared.
          It's still a bug, but at least now with a workaround.

          nmanos Noam Manos added a comment - Many attempts to find cause in Maven configuration, or in Java implementation, or to set any known Jenkins configuration, to workaround this problem, did not succeed, and this unpredictable error kept on happening, until... I asked our IT guys to increase CPU and RAM (from 4GB to 16GB), and walla - the problem disappeared. It's still a bug, but at least now with a workaround.
          oleg_nenashev Oleg Nenashev added a comment -

          Should be re-tested with Jenkins 2.32.1+, which includes the https://github.com/kohsuke/winp/issues/22 fix in Windows process management library.

          oleg_nenashev Oleg Nenashev added a comment - Should be re-tested with Jenkins 2.32.1+, which includes the https://github.com/kohsuke/winp/issues/22 fix in Windows process management library.

          People

            Unassigned Unassigned
            nmanos Noam Manos
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: