Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-4058

winp consistently causing OutOfMemoryError

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: other
    • Labels:
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      I'm running a unit test class with 2 tests, and consistently getting this error:

      ATAL: Java heap space
      java.lang.OutOfMemoryError: Java heap space
      at org.jvnet.winp.Native.getCmdLineAndEnvVars(Native Method)
      at org.jvnet.winp.WinProcess.parseCmdLineAndEnvVars(WinProcess.java:114)
      at org.jvnet.winp.WinProcess.getEnvironmentVariables(WinProcess.java:109)
      at hudson.util.ProcessTree$Windows$1.getEnvironmentVariables(ProcessTree.java:321)
      at hudson.util.ProcessTree$OSProcess.hasMatchingEnvVars(ProcessTree.java:205)
      at hudson.util.ProcessTree$OSProcess.access$200(ProcessTree.java:138)
      at hudson.util.ProcessTree$Windows.killAll(ProcessTree.java:341)
      at hudson.Launcher$LocalLauncher.kill(Launcher.java:651)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:276)
      at hudson.model.Run.run(Run.java:948)
      at hudson.model.Build.run(Build.java:112)
      at hudson.model.ResourceController.execute(ResourceController.java:93)
      at hudson.model.Executor.run(Executor.java:118)

      This does not always happen with the same process.

      Setting ProcessTreeKiller.disabled fixes it.

      Setting -Xmx128M fixes it too (but this was not necessary before).

        Attachments

        1. leak.zip
          190 kB
        2. winp.patch
          4 kB

          Activity

          Hide
          huybrechts huybrechts added a comment -

          Previous tests were on Vista (32-bit).
          I tried this on another PC with XP, and there are no problems there.

          Show
          huybrechts huybrechts added a comment - Previous tests were on Vista (32-bit). I tried this on another PC with XP, and there are no problems there.
          Hide
          kir0 kir0 added a comment -

          Hello,

          I have the similar problem on Windows Vista 32bit machines. I am running Hudson
          inside tomcat, and I have 20 jobs that executes batch files. After few execution
          I get the OOM: Unable to create new native threads. I am able to reproduce the
          same behaviour on another Vista machine. The problem seems not to affect any
          Windows XP/2003 machines, but only Vista.

          WARNUNG: hudson.triggers.SCMTrigger.run() failed for MyJob
          java.lang.OutOfMemoryError: unable to create new native thread
          at java.lang.Thread.start0(Native Method)
          at java.lang.Thread.start(Thread.java:597)
          at
          java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
          at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
          at
          java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
          at hudson.triggers.SCMTrigger.run(SCMTrigger.java:115)
          at hudson.triggers.Trigger.checkTriggers(Trigger.java:229)
          at hudson.triggers.Trigger$Cron.doRun(Trigger.java:177)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)

          I have profiled the tomcat and find-out that the error occurs somewhere in winp
          library. It creates 1.5Gb of char[] objects.

          I used the Microsoft UMDH tool (http://support.microsoft.com/kb/268343) and was
          able to produce more accurate location where the error occurs.

          +32c30cfe ( 32c30cfe - 0) 6d4 allocs BackTrace1E3D9FC
          + 6d4 ( 6d4 - 0) BackTrace1E3D9FC allocations

          ntdll!RtlAllocateHeap+0000021D
          kernel32!LocalAlloc+00000052
          winp!Java_org_jvnet_winp_Native_getCmdLineAndEnvVars+00000144

          meaning that native method getCmdLineAndEnvVars had allocated 32c30cfe bytes
          (approx 812 Mb).

          I am attaching the full UMDH logs.

          Unfortunately only the java part of winp is available but the sources of
          winp.dll are missing.

          Can you please provide the debug symbols for the dll, so I can create more
          accurate stack traces, or provide the sources, so I can eventually dig into the
          problem and propose a patch?

          Best regards
          Kiril

          Show
          kir0 kir0 added a comment - Hello, I have the similar problem on Windows Vista 32bit machines. I am running Hudson inside tomcat, and I have 20 jobs that executes batch files. After few execution I get the OOM: Unable to create new native threads. I am able to reproduce the same behaviour on another Vista machine. The problem seems not to affect any Windows XP/2003 machines, but only Vista. WARNUNG: hudson.triggers.SCMTrigger.run() failed for MyJob java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:597) at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78) at hudson.triggers.SCMTrigger.run(SCMTrigger.java:115) at hudson.triggers.Trigger.checkTriggers(Trigger.java:229) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:177) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) I have profiled the tomcat and find-out that the error occurs somewhere in winp library. It creates 1.5Gb of char[] objects. I used the Microsoft UMDH tool ( http://support.microsoft.com/kb/268343 ) and was able to produce more accurate location where the error occurs. +32c30cfe ( 32c30cfe - 0) 6d4 allocs BackTrace1E3D9FC + 6d4 ( 6d4 - 0) BackTrace1E3D9FC allocations ntdll!RtlAllocateHeap+0000021D kernel32!LocalAlloc+00000052 winp!Java_org_jvnet_winp_Native_getCmdLineAndEnvVars+00000144 meaning that native method getCmdLineAndEnvVars had allocated 32c30cfe bytes (approx 812 Mb). I am attaching the full UMDH logs. Unfortunately only the java part of winp is available but the sources of winp.dll are missing. Can you please provide the debug symbols for the dll, so I can create more accurate stack traces, or provide the sources, so I can eventually dig into the problem and propose a patch? Best regards Kiril
          Hide
          kir0 kir0 added a comment -

          Created an attachment (id=831)
          UMDH logs

          Show
          kir0 kir0 added a comment - Created an attachment (id=831) UMDH logs
          Hide
          jgoehringer jgoehringer added a comment -

          We see a similar problem, where the slave process is leaking process handles, as
          shown by Process Explorer. Looking at the source from this repository
          (https://winp.dev.java.net/source/browse/winp/), there are places in the native
          code where handles are not released prior to a returns from error conditions.

          There are also places where calls to LocalAlloc are not paired with LocalFree.

          I'll attach the patch to the code we have been testing.

          Show
          jgoehringer jgoehringer added a comment - We see a similar problem, where the slave process is leaking process handles, as shown by Process Explorer. Looking at the source from this repository ( https://winp.dev.java.net/source/browse/winp/ ), there are places in the native code where handles are not released prior to a returns from error conditions. There are also places where calls to LocalAlloc are not paired with LocalFree. I'll attach the patch to the code we have been testing.
          Hide
          jgoehringer jgoehringer added a comment -

          Created an attachment (id=921)
          Patch to winp native code - release handles and memory prior to error returns

          Show
          jgoehringer jgoehringer added a comment - Created an attachment (id=921) Patch to winp native code - release handles and memory prior to error returns
          Hide
          kir0 kir0 added a comment -

          I also created similar patch two weeks ago:
          https://winp.dev.java.net/issues/show_bug.cgi?id=8 (however it doesn't include
          patches for enableDebugPrivilege.cpp and java-interface.cpp

          Show
          kir0 kir0 added a comment - I also created similar patch two weeks ago: https://winp.dev.java.net/issues/show_bug.cgi?id=8 (however it doesn't include patches for enableDebugPrivilege.cpp and java-interface.cpp
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in hudson
          User: : kohsuke
          Path:
          trunk/hudson/main/core/pom.xml
          trunk/www/changelog.html
          http://fisheye4.cenqua.com/changelog/hudson/?cs=23632
          Log:
          [FIXED JENKINS-4058] Integrating a newer version of Winp that fixes memory leak problems.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in hudson User: : kohsuke Path: trunk/hudson/main/core/pom.xml trunk/www/changelog.html http://fisheye4.cenqua.com/changelog/hudson/?cs=23632 Log: [FIXED JENKINS-4058] Integrating a newer version of Winp that fixes memory leak problems.

            People

            Assignee:
            kohsuke Kohsuke Kawaguchi
            Reporter:
            huybrechts huybrechts
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: