Windows 7 Professional 64-bit with SP1
Jenkins 1.498
Winstone Servlet Engine v0.9.10
Active Directory plugin 1.30
one master, no slaves
JVM: Java HotSpot(TM) Server VM (20.1-b02, mixed mode) (32-bit)
Java: version 1.6.0_26, vendor Sun Microsystems Inc.
JVM arguments:
-Xrs
-Xms512m
-Xmx768m
-XX:PermSize=64M
-XX:MaxPermSize=128M
-Xss256k
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=F:\JENKINS_HOME\OOM.hprof
-Dcom.sun.management.jmxremote.port=8010
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle
Exception:
FATAL: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at hudson.Proc$LocalProc.<init>(Proc.java:264)
at hudson.Proc$LocalProc.<init>(Proc.java:216)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:763)
at hudson.Launcher$ProcStarter.start(Launcher.java:353)
at hudson.plugins.perforce.HudsonP4DefaultExecutor.exec(HudsonP4DefaultExecutor.java:79)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:321)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:61)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1467)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1428)
at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:792)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1325)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
at hudson.model.Run.execute(Run.java:1516)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
The URL http:/jenkinsserver/threadDump shows that there are hundreds of ComThreads:
ComThread for RequestHandlerThread[#301]
"ComThread for RequestHandlerThread[#301]" Id=50942 Group=main RUNNABLE (in native)
at com4j.Win32Lock.suspend0(Native Method)
at com4j.Win32Lock.suspend(Win32Lock.java:37)
at com4j.ComThread.run0(ComThread.java:146)
at com4j.ComThread.run(ComThread.java:134)
As far as I understand the threads belong to the Active Directory plugin but I do not understand why they are created / never finish.
I have activated logging for the Active Directory plugin by enabling the following loggers (all with log level all):
hudson.plugins.active_directory
org.acegisecurity
com4j
The log contains a lot of entries like this:
Jan 16, 2013 12:20:27 PM org.acegisecurity.context.HttpSessionContextIntegrationFilter doFilter
FINE: SecurityContextHolder now cleared, as request processing completed
Jan 16, 2013 12:20:27 PM org.acegisecurity.providers.anonymous.AnonymousProcessingFilter doFilter
FINE: Populated SecurityContextHolder with anonymous token: 'org.acegisecurity.providers.anonymous.AnonymousAuthenticationToken@7ceb964f: Username: anonymous; Password: [PROTECTED]; Authenticated: true; Details: org.acegisecurity.ui.WebAuthenticationDetails@1de6: RemoteIpAddress: ##.##.##.##; SessionId: null; Granted Authorities: '
Jan 16, 2013 12:20:27 PM org.acegisecurity.context.HttpSessionContextIntegrationFilter storeSecurityContextInSession
FINE: The HttpSession is currently null, and the HttpSessionContextIntegrationFilter is prohibited from creating an HttpSession (because the allowSessionCreation property is false) - SecurityContext thus not stored for next request
Jan 16, 2013 12:20:27 PM org.acegisecurity.ui.basicauth.BasicProcessingFilter doFilter
FINE: Authorization header: null
Jan 16, 2013 12:20:27 PM org.acegisecurity.ui.ExceptionTranslationFilter doFilter
FINE: Chain processed normally
Jan 16, 2013 12:20:27 PM org.acegisecurity.context.HttpSessionContextIntegrationFilter doFilter
FINE: New SecurityContext instance will be associated with SecurityContextHolder
Jan 16, 2013 12:20:27 PM org.acegisecurity.context.HttpSessionContextIntegrationFilter readSecurityContextFromSession
FINE: No HttpSession currently exists
Jan 16, 2013 12:20:27 PM org.acegisecurity.context.HttpSessionContextIntegrationFilter doFilter
FINE: SecurityContextHolder now cleared, as request processing completed
Jan 16, 2013 12:20:27 PM org.acegisecurity.context.HttpSessionContextIntegrationFilter storeSecurityContextInSession
FINE: The HttpSession is currently null, and the HttpSessionContextIntegrationFilter is prohibited from creating an HttpSession (because the allowSessionCreation property is false) - SecurityContext thus not stored for next request
Jan 16, 2013 12:20:27 PM org.acegisecurity.ui.ExceptionTranslationFilter doFilter
FINE: Chain processed normally
A lot of traffic (about four requests per second) is caused by the desktop application Hudson Tray Tracker (https://code.google.com/p/hudson-tray-tracker/). This application does not log in (it uses anonymous access).
I have also
enabled access logging, but found no unusual URL requests
connected with VisualVM (http://visualvm.java.net/) to the Jenkins JVM, but got no more informations
created a PowerShell script that logs the number of threads, handles, working set and the virtual size of the Java process, but it only shows that all numbers are constantly increasing:
Date Time
Pid
Thread
Handle Count
WorkingSet
VirtualSize
2013-01-20 22:01:31
740
86
1380
474.285 MB
1,291.566 MB
2013-01-20 23:01:31
740
111
1719
520.957 MB
1,309.094 MB
2013-01-21 00:01:31
740
138
2094
576.602 MB
1,322.531 MB
2013-01-21 01:01:30
740
161
2381
630.742 MB
1,334.031 MB
2013-01-21 02:01:30
740
177
2563
635.363 MB
1,342.094 MB
2013-01-21 03:01:30
740
203
2884
638.371 MB
1,355.031 MB
2013-01-21 04:01:31
740
222
3192
660.984 MB
1,396.156 MB
2013-01-21 05:01:30
740
243
3443
641.645 MB
1,406.609 MB
2013-01-21 06:01:30
740
264
3719
648.918 MB
1,417.219 MB
2013-01-21 07:01:31
740
294
4114
661.613 MB
1,432.156 MB
2013-01-21 08:01:31
740
313
4357
647.117 MB
1,441.719 MB
2013-01-21 09:01:30
740
344
4758
667.797 MB
1,457.219 MB
2013-01-21 10:01:31
740
414
5701
742.559 MB
1,575.898 MB
2013-01-21 11:01:31
740
502
6777
729.488 MB
1,620.813 MB
2013-01-21 12:01:52
740
562
7533
740.715 MB
1,650.813 MB
2013-01-21 13:01:31
740
616
8313
763.828 MB
1,677.922 MB
The problem is related to system load: at weekend the numbers are increasing at a lower rate than at work days. Between Christmas and New Year it didn't happened at all as most developers have been on vacation.
created a Groovey script:
def threads=com4j.ComThread.threads.toArray(new com4j.ComThread[com4j.ComThread.threads.size()]);
def numberOfTerminatedPeerThreads=0
println("The follwing ComThreads have a peer thread which has been terminated")
for (t in threads) {
if (t.getName().startsWith("ComThread for RequestHandlerThread[#") &&
t.peer.getState().equals(Thread.State.TERMINATED) ) {
println("ComThread: " + t.getName() + " Peer: " + t.peer )
numberOfTerminatedPeerThreads++
//t.kill()
}
}
println("\nnumber of terminated peer threads: " + numberOfTerminatedPeerThreads)
result:
The follwing ComThreads have a peer thread which has been terminated
ComThread: ComThread for RequestHandlerThread[#357] Peer: Thread[RequestHandlerThread[#357],5,]
ComThread: ComThread for RequestHandlerThread[#258] Peer: Thread[RequestHandlerThread[#258],5,]
...
number of terminated peer threads: 506
I found com4j-x86.dll is in %JENKINS_HOME%\plugins\active-directory\WEB-INF\lib, but it still doesn't work. And I try to put com4j-20120426-2.jar (extract from Activedirectory plugin) into jenkins.war (in WEB-INF\lib)
After restart Jenkins, it works... is this a good solution ?
Dennys Hsieh
added a comment - I found com4j-x86.dll is in %JENKINS_HOME%\plugins\active-directory\WEB-INF\lib, but it still doesn't work. And I try to put com4j-20120426-2.jar (extract from Activedirectory plugin) into jenkins.war (in WEB-INF\lib)
After restart Jenkins, it works... is this a good solution ?
Is @Dennys Hsieh's workaround on 19/Nov/13 9:24AM a valid solution? Not sure I understand it. We like(d) the AD plugin so much better than the LDAP plugin but this issue continues to be a reason why we can't use it any more.
Jeremy Van Haren
added a comment - Is @Dennys Hsieh's workaround on 19/Nov/13 9:24AM a valid solution? Not sure I understand it. We like(d) the AD plugin so much better than the LDAP plugin but this issue continues to be a reason why we can't use it any more.
Integrated a newer version of com4j that fixes the leak
SCM/JIRA link daemon
added a comment - Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
pom.xml
http://jenkins-ci.org/commit/active-directory-plugin/5fbb298d9e66f256a8549fa3e0ed81e8c1cb100b
Log:
[FIXED JENKINS-16429]
Integrated a newer version of com4j that fixes the leak
I just upgrade to active directory plugin 1.36 with com4j 2.1
It seems the leak issue is not fixed yet.
But my groovy script runs every 60mins, I'm not sure will the new com4j clear the leak thread in batch or any dependency?
Dennys Hsieh
added a comment - I just upgrade to active directory plugin 1.36 with com4j 2.1
It seems the leak issue is not fixed yet.
But my groovy script runs every 60mins, I'm not sure will the new com4j clear the leak thread in batch or any dependency?
It seems that the issue is not fixed completely.
Now the number of threads are ok, but java memory usage is excessively high and rises constantly as before.
I think there is a remaining memory leak.
Frank Seifart
added a comment - - edited It seems that the issue is not fixed completely.
Now the number of threads are ok, but java memory usage is excessively high and rises constantly as before.
I think there is a remaining memory leak.
Sorry, I think the thread issue is fixed. Because I forget to remove the old com4j in WEB-INF\lib of jenkins.war (in order to run the Groovy script). After remove it, the thread is normal now.
Dennys Hsieh
added a comment - Sorry, I think the thread issue is fixed. Because I forget to remove the old com4j in WEB-INF\lib of jenkins.war (in order to run the Groovy script). After remove it, the thread is normal now.
@franks then you may be seeing an unrelated issue; file it separately (and use for example jmap -histo:live $pid to diagnose).
Jesse Glick
added a comment - @franks then you may be seeing an unrelated issue; file it separately (and use for example jmap -histo:live $pid to diagnose).
I was looking at how this plugin is packaged and there is a copy of com4j-amd64.dll in Jenkins\plugins\active-directory\WEB-INF\lib and there is also a copy inside com4j-2.1.jar\com4j\
Is this expected?
Walter Kacynski
added a comment - I was looking at how this plugin is packaged and there is a copy of com4j-amd64.dll in Jenkins\plugins\active-directory\WEB-INF\lib and there is also a copy inside com4j-2.1.jar\com4j\
Is this expected?
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Bug
Blocker
Jeremy Van Haren
added a comment - 
Dennys Hsieh
added a comment - 
Frank Seifart
added a comment - - edited It seems that the issue is not fixed completely.
Now the number of threads are ok, but java memory usage is excessively high and rises constantly as before.
I think there is a remaining memory leak. 
Walter Kacynski
added a comment - 
I found com4j-x86.dll is in %JENKINS_HOME%\plugins\active-directory\WEB-INF\lib, but it still doesn't work. And I try to put com4j-20120426-2.jar (extract from Activedirectory plugin) into jenkins.war (in WEB-INF\lib)
After restart Jenkins, it works... is this a good solution ?