Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41836

Out of memory at "exit 0" in build on Windows with IBM JVM

      On a Windows 7 slave, the slave is started with the IBM JVM,using the JNLP command. Then a build is executed; the build is an "execute Windows batch command". For test purposes I make it a simple command (dir).

      The build executes. But then after the "exit 0" command an out of memory error happens. The build is recorded as failed.

      The console output with the exception follows.

      c:\jenkins\workspace\Test Project>exit 0
      FATAL: Remote call on JNLP4-connect connection from ncb9042034033.tivlab.raleigh.ibm.com/9.42.34.33:61818 failed
      java.lang.OutOfMemoryError: Java heap space
      at java.lang.String.<init>(String.java:388)
      at java.lang.String.substring(String.java:1372)
      at org.jvnet.winp.WinProcess.parseCmdLineAndEnvVars(WinProcess.java:144)
      at org.jvnet.winp.WinProcess.getEnvironmentVariables(WinProcess.java:121)
      at hudson.util.ProcessTree$Windows$1.getEnvironmentVariables(ProcessTree.java:456)
      at hudson.util.ProcessTree$OSProcess.hasMatchingEnvVars(ProcessTree.java:280)
      at hudson.util.ProcessTree$Windows.killAll(ProcessTree.java:482)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:996)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:987)
      at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      at hudson.remoting.Request$2.run(Request.java:336)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      at java.util.concurrent.FutureTask.run(FutureTask.java:277)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at hudson.remoting.Engine$1$1.run(Engine.java:94)
      at java.lang.Thread.run(Thread.java:785)
      at ......remote call to JNLP4-connect connection from ncb9042034033.tivlab.raleigh.ibm.com/9.42.34.33:61818(Native Method)
      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
      at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      at hudson.remoting.Channel.call(Channel.java:822)
      Caused: java.io.IOException: Remote call on JNLP4-connect connection from ncb9042034033.tivlab.raleigh.ibm.com/9.42.34.33:61818 failed
      at hudson.remoting.Channel.call(Channel.java:830)
      at hudson.Launcher$RemoteLauncher.kill(Launcher.java:984)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540)
      at hudson.model.Run.execute(Run.java:1728)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:98)
      at hudson.model.Executor.run(Executor.java:405)
      Finished: FAILURE

      Same error discussed on Google Groups https://groups.google.com/forum/#!topic/jenkinsci-users/HjBud36mA2w (as WebSphere is mentioned, probably the JVM is an IBM version, too)

          [JENKINS-41836] Out of memory at "exit 0" in build on Windows with IBM JVM

          Oleg Nenashev added a comment -

          ramendik please provide info about your memory settings and the heapdump link. Without this information I cannot analyze the issue

          Oleg Nenashev added a comment - ramendik please provide info about your memory settings and the heapdump link. Without this information I cannot analyze the issue

          Oleg Nenashev added a comment -

          No response from the requester. If somebody from the voters has the same issue, please attach the requested information and reopen the ticket

          Oleg Nenashev added a comment - No response from the requester. If somebody from the voters has the same issue, please attach the requested information and reopen the ticket

          Rick Patterson added a comment - - edited

          Hi

          We are getting the same error reported here. We need to upgrade to JRE8 before upgrade to Jenkins server 2.60.x. Presently using IBM JRE 7 no issues.

          IBM JRE:
          >.\java.exe -version
          java version "1.8.0"
          Java(TM) SE Runtime Environment (build pwa6480sr3fp12-20160919_01(SR3 FP12))
          IBM J9 VM (build 2.8, JRE 1.8.0 Windows Server 2008 R2 amd64-64 Compressed Refer
          ences 20160915_318796 (JIT enabled, AOT enabled)
          J9VM - R28_Java8_SR3_20160915_0912_B318796
          JIT - tr.r14.java.green_20160818_122998
          GC - R28_Java8_SR3_20160915_0912_B318796_CMPRSS
          J9CL - 20160915_318796)
          JCL - 20160914_01 based on Oracle jdk8u101-b13

          Slave operating system is Windows 2008 R2.

          using the slave.jar from Jenkins 2.46.3, Jenkins slave log says: Slave.jar version: 3.7

          Rick Patterson added a comment - - edited Hi We are getting the same error reported here. We need to upgrade to JRE8 before upgrade to Jenkins server 2.60.x. Presently using IBM JRE 7 no issues. IBM JRE: >.\java.exe -version java version "1.8.0" Java(TM) SE Runtime Environment (build pwa6480sr3fp12-20160919_01(SR3 FP12)) IBM J9 VM (build 2.8, JRE 1.8.0 Windows Server 2008 R2 amd64-64 Compressed Refer ences 20160915_318796 (JIT enabled, AOT enabled) J9VM - R28_Java8_SR3_20160915_0912_B318796 JIT - tr.r14.java.green_20160818_122998 GC - R28_Java8_SR3_20160915_0912_B318796_CMPRSS J9CL - 20160915_318796) JCL - 20160914_01 based on Oracle jdk8u101-b13 Slave operating system is Windows 2008 R2. using the slave.jar from Jenkins 2.46.3, Jenkins slave log says: Slave.jar version: 3.7

          Oleg Nenashev - have attached a heapdump file. Was not using any java memory settings, just default. java -jar slave.jar .....

          Rick Patterson added a comment - Oleg Nenashev - have attached a heapdump file. Was not using any java memory settings, just default. java -jar slave.jar .....

          Oleg Nenashev added a comment -

          Will try to take a look this month

          Oleg Nenashev added a comment - Will try to take a look this month

          Great, thanks.

          Rick Patterson added a comment - Great, thanks.

          Oleg Nenashev added a comment -

          Didn't find time for it, sorry

          Oleg Nenashev added a comment - Didn't find time for it, sorry

          Oleg Nenashev added a comment -

          Taking other IBM Java issues into account (https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%20java-ibm), I am about to say that Remoting/Jenkins does not really support this JDK/JRE well. Not sure if it is a Jenkins or JVM fault, the latter one seems to be plausible

          Oleg Nenashev added a comment - Taking other IBM Java issues into account ( https://issues.jenkins-ci.org/issues/?jql=labels%20%3D%20java-ibm), I am about to say that Remoting/Jenkins does not really support this JDK/JRE well. Not sure if it is a Jenkins or JVM fault, the latter one seems to be plausible

          Rick Patterson added a comment - - edited

          Rick Patterson added a comment - - edited This issue seems to be discussed here as well: https://stackoverflow.com/questions/45359604/java-substring-operation-seems-to-cause-out-of-memory-error-in-java-1-8 A "fix" is documented here:  https://stackoverflow.com/questions/45377008/how-do-i-set-the-disablecopyinsubstring-system-property-from-the-string-class-in   Which involves adding the following define to the java used for the slave invocation: -Djava.lang.string.substring.nocopy=true     Related articles:   https://stackoverflow.com/questions/33893655/string-substring-making-a-copy-of-the-underlying-char-value https://stackoverflow.com/questions/20260140/how-to-detect-whether-string-substring-copies-the-character-data/20275133#20275133  

          Mark Waite added a comment -

          The Jenkins platform SIG decided in August 2021 that they would stop testing the OpenJ9 virtual machine. There are not enough people available to continue testing OpenJ9 with Jenkins, especially considering the availability of a Hotstpot JVM for s390x with Java 11 and with Java 17.

          Jenkins Platform SIG is currently using a Jenkins Enhancement Proposal pull request to plan and discuss Java 11 as the required minimum JVM for Jenkins controllers and Jenkins agents.

          Mark Waite added a comment - The Jenkins platform SIG decided in August 2021 that they would stop testing the OpenJ9 virtual machine. There are not enough people available to continue testing OpenJ9 with Jenkins, especially considering the availability of a Hotstpot JVM for s390x with Java 11 and with Java 17. Jenkins Platform SIG is currently using a Jenkins Enhancement Proposal pull request to plan and discuss Java 11 as the required minimum JVM for Jenkins controllers and Jenkins agents.

            Unassigned Unassigned
            ramendik Mikhail Ramendik
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: