-
Bug
-
Resolution: Fixed
-
Critical
-
influxdb-plugin: 3.0.0
Jenkins version: 2.289
-
-
3.0.2
After upgrading influxDB from 2.5 to 3.0.0 we started to have a huge amount of thread leak causing our jenkins instance to fail. We had to revert and redeploy Jenkins, and I was not able to capture a thread dump. I will update this ticket if/when I am able to have more information on the thread leak.
It happened about ~6h after startup, so for us this means after about 15,000 jobs, all with post-deploy influxDb publisher.
We used telegraf to have an influxDB listener v2, and configured the plugin to use the v2 HTTP client.
- causes
-
-
- Resolved
-
[JENKINS-65622] Thread leak after upgrade to 3.0.0
Joe Athman
added a comment - I think the bug is that this line: https://github.com/jenkinsci/influxdb-plugin/blob/970319deaf87b848edafa6b5056ab6ba6ce88bc8/src/main/java/jenkinsci/plugins/influxdb/InfluxDbPublicationService.java#L354 calls the `getWriteApi` for each metric. This is not how the API is intended to work, the influx client and the write api are intended to be singletons. The ticket you link to shows that many people have this same problem. It is not clear at all that calling the getWriteApi will cause problems when done from a loop.
Joe Athman
added a comment - I think the bug is that this line: https://github.com/jenkinsci/influxdb-plugin/blob/970319deaf87b848edafa6b5056ab6ba6ce88bc8/src/main/java/jenkinsci/plugins/influxdb/InfluxDbPublicationService.java#L354 calls the `getWriteApi` for each metric. This is not how the API is intended to work, the influx client and the write api are intended to be singletons. The ticket you link to shows that many people have this same problem. It is not clear at all that calling the getWriteApi will cause problems when done from a loop. Hi jjathman
The getWriteApi() is not called for each metric. It's called for each separate target. The measurement points are batched into a list, which is written for multiple InfluxDB databases (you're able to have global listeners that write data from every job without specifying it directly in the job). And since you have the option to write to multiple targets, you need a separate InfluxDBClient for each target.
Jakub
added a comment - Hi All,
The getWriteApi() is supposed to run as a singleton because it is use a background thread to ingesting data into InfluxDB. The better choice for influxdb-pluging is getWriteApiBlocking(), also the InfluxDBClient should be closed to free-up resources.
I've prepare the PR with fixes: https://github.com/jenkinsci/influxdb-plugin/pull/121
Regards
Jakub
added a comment - Hi All,
The getWriteApi() is supposed to run as a singleton because it is use a background thread to ingesting data into InfluxDB. The better choice for influxdb-pluging is getWriteApiBlocking() , also the InfluxDBClient should be closed to free-up resources.
I've prepare the PR with fixes: https://github.com/jenkinsci/influxdb-plugin/pull/121
Regards Joe Athman
added a comment - aleksisimell - you are correct, it's not called for each metric, but is called for each invocation of the step/target.
Looks like jakub_bednar has a nice fix though. Hope we can get that PR merged in quickly. Thanks!
Joe Athman
added a comment - aleksisimell - you are correct, it's not called for each metric, but is called for each invocation of the step/target.
Looks like jakub_bednar has a nice fix though. Hope we can get that PR merged in quickly. Thanks!
Hi!
If this strated after upgrade and your 15 000 jobs have run perfectly until then, I believe the issue is the changed API. influxdb-java doesn't support InfluxDB 2.0, so I had to change it to influxdb-java-client and at least there is a ticket about threading. Very likely it's caused by that. I have to check if this can be avoided by calling some other functions.