-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins Version = 1.546
PTC Integrity Plugin Version =1.22
Integration Point Server = Integrity Client Installation on the same server machine that runs Jenkins
Integrity Server / Client Version = 10.4
Using the PTC Integrity Plugin leads to a Out of Memory exceptions- and effectively kills the underlying Integrity Client that serves as "Integration Point Server" (within hours).
As recommended by the PTC support team, we do not use our Integrity Server directly but let a locally installed Integrity Client serve as "Integration Point Server".
This used to work fine for earlier versions, where a known bug in the Integrity API Implementation might lead to memory issues. Utilizing the server directly had even killed our Integrity server several times and in our current setup, at least only the "Integration Point Server" aka local client dies.
It still worked with Plugin 1.18
---------------------------------------------------------------------------
Since Plugin Version 1.20, the local Integrity Client's memory usage grows with every polling and every workspace update job performed!
This improved with version 1.22, as the memory consumption is now growing slower.
But still the Client won't last 8 hour.
---------------------------------------------------------------------------
Monitoring the client with "Java Visual VM" shows that the garbage collection is running, but seems to find nothing to delete.
It also shows that, for each SCM Update Job started by jenkins, the number of threads called "Client Instance Logger" increases by the "Checkout Thread Pool Size" defined in the plugin.
These threads wait for ever and never die.
at ~300 waiting threads like these the client dies.
[JENKINS-21587] memory leak in Plugin 1.22 -orphaned Logger Threads
Matthias Rump
created issue -
Matthias Rump
made changes -
| Description |
Original:
Using the PTC Integrity Plugin leads to a Out of Memory exceptions- and effectively kills the underlying Integrity Client that serves as "Integration Point Server" (within hours). As recommended by the PTC support team, we do not use our Integrity Server directly but let a locally installed Integrity Client serve as "Integration Point Server". This used to work fine for earlier versions, where a known bug in the Integrity API Implementation might lead to memory issues. Utilizing the server directly had even killed our Integrity server several times and in our current setup, at least only the "Integration Point Server" aka local client dies. It still worked with Plugin 1.18 Since Plugin Version 1.20, the local Integrity Client's memory usage grows with every polling and every workspace update job performed! Monitoring the client shows that the garbage collection is running, but seems to find nothing to delete. Looking at the Integriy Server diag, we find and increasing number of sessions our CIUser's account. So obviously there seem's to be a problem creating and releasing API sessions by the plugin, which would IMHO explain the memory issue. |
New:
Using the PTC Integrity Plugin leads to a Out of Memory exceptions- and effectively kills the underlying Integrity Client that serves as "Integration Point Server" (within hours). As recommended by the PTC support team, we do not use our Integrity Server directly but let a locally installed Integrity Client serve as "Integration Point Server". This used to work fine for earlier versions, where a known bug in the Integrity API Implementation might lead to memory issues. Utilizing the server directly had even killed our Integrity server several times and in our current setup, at least only the "Integration Point Server" aka local client dies. It still worked with Plugin 1.18 --------------------------------------------------------------------------- Since Plugin Version 1.20, the local Integrity Client's memory usage grows with every polling and every workspace update job performed! This improved with version 1.22, as the memory consumption is now growing slower. But still the Client won't last 8 hour. --------------------------------------------------------------------------- Monitoring the client with "Java Visual VM" shows that the garbage collection is running, but seems to find nothing to delete. It also shows that, for each SCM Update Job started by jenkins, the number of threads called "Client Instance Logger" increases by the "Checkout Thread Pool Size" defined in the plugin. These threads wait for ever and never die. at ~300 waiting threads like these the client dies. |
| Environment |
Original:
Jenkins Version = 1.546 PTC Integrity Plugin Version =1.20 Integration Point Server = Integrity Client Installation on the same server machine that runs Jenkins Integrity Server / Client Version = 10.4 |
New:
Jenkins Version = 1.546 PTC Integrity Plugin Version =1.22 Integration Point Server = Integrity Client Installation on the same server machine that runs Jenkins Integrity Server / Client Version = 10.4 |
Cletus D'Souza
added a comment - I've confirmed that the threads in the Jenkins process terminate and are cleaned up. Hence, this is an issue with the client/api itself. Please report the problem to PTC Integrity Support team.
Cletus D'Souza
added a comment - I've confirmed that the threads in the Jenkins process terminate and are cleaned up. Hence, this is an issue with the client/api itself. Please report the problem to PTC Integrity Support team. Cletus D'Souza
made changes -
| Resolution | New: Won't Fix [ 2 ] | |
| Status | Original: Open [ 1 ] | New: Resolved [ 5 ] |
Matthias Rump
added a comment - Hi Cletus,
Yes, the threads are are in the integrity client's VM, NOT in Jenkins, so you're right, the plugin itself does not have the problem.
But nevertheless the plugin seems to be the course of this issue, as we did not see this number of daemon threads on the same client, same API, with plugin version 1.18.
So what's you proposal?
- connecting the plugin directly to the Server and keeping my fingers crossed it won't kill my production system?
- schedule a restart of my integration server client every hour?
- stepping back to plugin version 1.18 (with all other known issues)?
Matthias Rump
added a comment - Hi Cletus,
Yes, the threads are are in the integrity client's VM, NOT in Jenkins, so you're right, the plugin itself does not have the problem.
But nevertheless the plugin seems to be the course of this issue, as we did not see this number of daemon threads on the same client, same API, with plugin version 1.18.
So what's you proposal?
connecting the plugin directly to the Server and keeping my fingers crossed it won't kill my production system?
schedule a restart of my integration server client every hour?
stepping back to plugin version 1.18 (with all other known issues)?
Matthias Rump
added a comment - see last comment Matthias Rump
made changes -
| Resolution | Original: Won't Fix [ 2 ] | |
| Status | Original: Resolved [ 5 ] | New: Reopened [ 4 ] |
Cletus D'Souza
added a comment - I highly doubt that this wasn't the problem in v1.18. Nothing has changed in terms of the interaction with the API since v.18 (except for the sessions bug fix).
Seems like this is an API issue, so connecting to the Server directly will just move the problem from the Client Integration Point to the Server Integration Point.
My suggestion is you should monitor the threads (in the Integrity Client) like you've done in 1.22 and compare those results with 1.18 and see if there is a difference.
Cletus D'Souza
added a comment - I highly doubt that this wasn't the problem in v1.18. Nothing has changed in terms of the interaction with the API since v.18 (except for the sessions bug fix).
Seems like this is an API issue, so connecting to the Server directly will just move the problem from the Client Integration Point to the Server Integration Point.
My suggestion is you should monitor the threads (in the Integrity Client) like you've done in 1.22 and compare those results with 1.18 and see if there is a difference. Matthias Rump
added a comment - I've been inspecting my Client with dynaTrace and compared memory dumps over time.
What I found is that corresponding to the number of "Client Instance Logger" threads the number of "mks.si.api.SISession" instances in the memory increases as well.
It seems to be related to the SCM Polling because when i lowered the "Max # of concurrent polling" setting, the growth slowed down.
Matthias Rump
added a comment - I've been inspecting my Client with dynaTrace and compared memory dumps over time.
What I found is that corresponding to the number of "Client Instance Logger" threads the number of "mks.si.api.SISession" instances in the memory increases as well.
It seems to be related to the SCM Polling because when i lowered the "Max # of concurrent polling" setting, the growth slowed down.
Just to confirm the ~300 threads are waiting in the IntegrityClient process or in Jenkins?
If it is the IntegrityClient process, then this is really an issue in the IntegrityClient as all API connections get terminated from the plugin.
If this is in Jenkins process, then while its not a good thing and perhaps the plugin can be updated, it has no bearing on the Integrity Client process.