-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins Version = 1.546
PTC Integrity Plugin Version =1.22
Integration Point Server = Integrity Client Installation on the same server machine that runs Jenkins
Integrity Server / Client Version = 10.4
Using the PTC Integrity Plugin leads to a Out of Memory exceptions- and effectively kills the underlying Integrity Client that serves as "Integration Point Server" (within hours).
As recommended by the PTC support team, we do not use our Integrity Server directly but let a locally installed Integrity Client serve as "Integration Point Server".
This used to work fine for earlier versions, where a known bug in the Integrity API Implementation might lead to memory issues. Utilizing the server directly had even killed our Integrity server several times and in our current setup, at least only the "Integration Point Server" aka local client dies.
It still worked with Plugin 1.18
---------------------------------------------------------------------------
Since Plugin Version 1.20, the local Integrity Client's memory usage grows with every polling and every workspace update job performed!
This improved with version 1.22, as the memory consumption is now growing slower.
But still the Client won't last 8 hour.
---------------------------------------------------------------------------
Monitoring the client with "Java Visual VM" shows that the garbage collection is running, but seems to find nothing to delete.
It also shows that, for each SCM Update Job started by jenkins, the number of threads called "Client Instance Logger" increases by the "Checkout Thread Pool Size" defined in the plugin.
These threads wait for ever and never die.
at ~300 waiting threads like these the client dies.
[JENKINS-21587] memory leak in Plugin 1.22 -orphaned Logger Threads
Matthias Rump
added a comment - Hi Cletus,
Yes, the threads are are in the integrity client's VM, NOT in Jenkins, so you're right, the plugin itself does not have the problem.
But nevertheless the plugin seems to be the course of this issue, as we did not see this number of daemon threads on the same client, same API, with plugin version 1.18.
So what's you proposal?
- connecting the plugin directly to the Server and keeping my fingers crossed it won't kill my production system?
- schedule a restart of my integration server client every hour?
- stepping back to plugin version 1.18 (with all other known issues)?
Matthias Rump
added a comment - Hi Cletus,
Yes, the threads are are in the integrity client's VM, NOT in Jenkins, so you're right, the plugin itself does not have the problem.
But nevertheless the plugin seems to be the course of this issue, as we did not see this number of daemon threads on the same client, same API, with plugin version 1.18.
So what's you proposal?
connecting the plugin directly to the Server and keeping my fingers crossed it won't kill my production system?
schedule a restart of my integration server client every hour?
stepping back to plugin version 1.18 (with all other known issues)?
Matthias Rump
added a comment - see last comment 
Cletus D'Souza
added a comment - I highly doubt that this wasn't the problem in v1.18. Nothing has changed in terms of the interaction with the API since v.18 (except for the sessions bug fix).
Seems like this is an API issue, so connecting to the Server directly will just move the problem from the Client Integration Point to the Server Integration Point.
My suggestion is you should monitor the threads (in the Integrity Client) like you've done in 1.22 and compare those results with 1.18 and see if there is a difference.
Cletus D'Souza
added a comment - I highly doubt that this wasn't the problem in v1.18. Nothing has changed in terms of the interaction with the API since v.18 (except for the sessions bug fix).
Seems like this is an API issue, so connecting to the Server directly will just move the problem from the Client Integration Point to the Server Integration Point.
My suggestion is you should monitor the threads (in the Integrity Client) like you've done in 1.22 and compare those results with 1.18 and see if there is a difference. Matthias Rump
added a comment - I've been inspecting my Client with dynaTrace and compared memory dumps over time.
What I found is that corresponding to the number of "Client Instance Logger" threads the number of "mks.si.api.SISession" instances in the memory increases as well.
It seems to be related to the SCM Polling because when i lowered the "Max # of concurrent polling" setting, the growth slowed down.
Matthias Rump
added a comment - I've been inspecting my Client with dynaTrace and compared memory dumps over time.
What I found is that corresponding to the number of "Client Instance Logger" threads the number of "mks.si.api.SISession" instances in the memory increases as well.
It seems to be related to the SCM Polling because when i lowered the "Max # of concurrent polling" setting, the growth slowed down.
Cletus D'Souza
added a comment - Hi Matthias,
This is all great debugging... I hate to say this but you are barking up the wrong tree 
While, I'm not in the least surprised that the number of threads directly contribute to the number of 'API Sessions' and hence 'Client Instance Logger' threads. The fact of the matter remains, this is exclusively an Integrity API problem.
The part where the plugin is responsible is to "terminate" the API Session after the thread is done. Which I have done and now there are no more lingering API Sessions (connections) to the server after the threads have completed.
Obviously, calling the terminate sequence is not killing the 'Client Instance Logger' threads which the API starts up on its own (and in its own JVM server or client). I've no way to control that. I'm doing my part as it relates to the "api contract"... now its up to R&D to fix their API issue.
Thanks!
Cletus
Cletus D'Souza
added a comment - Hi Matthias,
This is all great debugging... I hate to say this but you are barking up the wrong tree
While, I'm not in the least surprised that the number of threads directly contribute to the number of 'API Sessions' and hence 'Client Instance Logger' threads. The fact of the matter remains, this is exclusively an Integrity API problem.
The part where the plugin is responsible is to "terminate" the API Session after the thread is done. Which I have done and now there are no more lingering API Sessions (connections) to the server after the threads have completed.
Obviously, calling the terminate sequence is not killing the 'Client Instance Logger' threads which the API starts up on its own (and in its own JVM server or client). I've no way to control that. I'm doing my part as it relates to the "api contract"... now its up to R&D to fix their API issue.
Thanks!
Cletus Matthias Rump
added a comment - Hi Cletus,
sorry for being so obstrusive, but this issue kind of drives me crazy 
I hope you can consider my pull-request for the next release:
https://github.com/maschuru/integrity-plugin/commit/c8bea4be0ea72bd50a921828e982cd263c85ff08
Regards Matthias
Matthias Rump
added a comment - Hi Cletus,
sorry for being so obstrusive, but this issue kind of drives me crazy
I hope you can consider my pull-request for the next release:
https://github.com/maschuru/integrity-plugin/commit/c8bea4be0ea72bd50a921828e982cd263c85ff08
Regards Matthias Cletus D'Souza
added a comment - No, not at all! You code changes seem reasonable to me and I can certainly incorporate it in the next release. I've not seen any exceptions thrown during the terminate process, but glad you've caught it.
Thanks for your persistence! Its paid of
Cletus
Cletus D'Souza
added a comment - No, not at all! You code changes seem reasonable to me and I can certainly incorporate it in the next release. I've not seen any exceptions thrown during the terminate process, but glad you've caught it.
Thanks for your persistence! Its paid of
Cletus Code changed in jenkins
User: Cletus D'Souza
Path:
src/main/java/hudson/scm/APISession.java
src/main/java/hudson/scm/IntegrityCMProject.java
src/main/java/hudson/scm/IntegrityCheckoutTask.java
src/main/java/hudson/scm/IntegritySCM.java
http://jenkins-ci.org/commit/integrity-plugin/9c4605abb92d6adebd8618828bd8c49e3d0cb526
Log:
Resolved the following issues:
JENKINS-21587: memory leak in Plugin 1.22 -orphaned Logger Threads (accepted patch from Matthias Rump - thank you!)
JENKINS-21479: Poll SCM trigger no functioning during the first execution
JENKINS-21260: PTC integrity plugin: version 1.19 instantly triggers new build though corresponding files in repository have not been changed!
Cletus D'Souza
added a comment - Patch accepted and released! 
I've confirmed that the threads in the Jenkins process terminate and are cleaned up. Hence, this is an issue with the client/api itself. Please report the problem to PTC Integrity Support team.