Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30333

Thread Leak due to use of deprecated JiraSize.createSession

    XMLWordPrintable

Details

    Description

      We use the 2.0 Jira Plugin to add comments to Jiras. We have the problem that every such request causes a Thread Leak - see the "diff" output of "jstack" below.

      Seems that this is caused by using JiraSite.createSession instead of getSession() in Updater class. Changing it to getSession worked for me.
      Mind that I havent yet tested for the "// TODO: we should check for session timeout, too" in JiraSite.


      Output of the thread-list diff - we can see many additional "httpclient-" threads

      glangs@cgm-1194:/opt/docker_volumes/jenkins3$ diff *tid
      1c1,20
      < "httpclient-callbacks:thread-1" daemon prio=10 tid=0x0000000000c83800 nid=0x5364 waiting on condition [0x00007ff7a73f2000]
      ---
      > "RequestHandlerThread[#12]" daemon prio=10 tid=0x00007ff7a800b800 nid=0x53c2 waiting on condition [0x00007ff7a67e6000]
      > "httpclient-callbacks:thread-1" daemon prio=10 tid=0x0000000000bf6000 nid=0x53c0 waiting on condition [0x00007ff7a68e7000]
      > "httpclient-io:thread-10" daemon prio=10 tid=0x0000000000ae8800 nid=0x53bf runnable [0x00007ff7a69e8000]
      > "httpclient-io:thread-9" daemon prio=10 tid=0x0000000000913000 nid=0x53be runnable [0x00007ff7a6ae9000]
      > "httpclient-io:thread-8" daemon prio=10 tid=0x0000000000910800 nid=0x53bd runnable [0x00007ff7a6bea000]
      > "httpclient-io:thread-7" daemon prio=10 tid=0x000000000090e800 nid=0x53bc runnable [0x00007ff7a6ceb000]
      > "httpclient-io:thread-6" daemon prio=10 tid=0x000000000090d000 nid=0x53bb runnable [0x00007ff7a6dec000]
      > "httpclient-io:thread-5" daemon prio=10 tid=0x0000000000909800 nid=0x53ba runnable [0x00007ff7a6eed000]
      > "httpclient-io:thread-4" daemon prio=10 tid=0x0000000000908000 nid=0x53b9 runnable [0x00007ff7a6fee000]
      > "httpclient-io:thread-3" daemon prio=10 tid=0x0000000000904800 nid=0x53b6 runnable [0x00007ff7a70ef000]
      > "httpclient-io:thread-2" daemon prio=10 tid=0x0000000000906800 nid=0x53b5 runnable [0x00007ff7a71f0000]
      > "httpclient-io:thread-1" daemon prio=10 tid=0x0000000000967000 nid=0x53b4 runnable [0x00007ff7a72f1000]
      > "httpclient-req-killer" prio=10 tid=0x00007ff7d80d8000 nid=0x53b3 waiting on condition [0x00007ff7ac122000]
      > "Thread-333" prio=10 tid=0x00007ff7d837c000 nid=0x53ae runnable [0x00007ff7a7cfb000]
      > "Jenkins-Remoting-Thread-312" daemon prio=10 tid=0x000000000098d000 nid=0x53ad waiting on condition [0x00007ff7a73f2000]
      > "Jenkins-Remoting-Thread-311" daemon prio=10 tid=0x0000000000c83800 nid=0x53ac waiting on condition [0x00007ff7a7ffe000]
      > "Jenkins-Remoting-Thread-310" daemon prio=10 tid=0x0000000000986800 nid=0x53ab waiting on condition [0x00007ff7a7dfc000]
      > "Jenkins-Remoting-Thread-309" daemon prio=10 tid=0x00007ff7b807a800 nid=0x53aa waiting on condition [0x00007ff7a7bfa000]
      > "process reaper" daemon prio=10 tid=0x00007ff7d8075000 nid=0x53a2 waiting on condition [0x00007ff7acc74000]
      > "pool-16-thread-375" daemon prio=10 tid=0x0000000000de9000 nid=0x5399 waiting on condition [0x00007ff7ae1e1000]
      14,17d32
      < "RequestHandlerThread[#11]" daemon prio=10 tid=0x00007ff7a800b800 nid=0x5355 waiting on condition [0x00007ff7ae1e1000]
      < "Jenkins-Remoting-Thread-308" daemon prio=10 tid=0x0000000000986800 nid=0x5352 waiting on condition [0x00007ff7a7bfa000]
      < "Jenkins-Remoting-Thread-307" daemon prio=10 tid=0x0000000000967800 nid=0x5351 waiting on condition [0x00007ff7a7cfb000]
      < "Jenkins-Remoting-Thread-306" daemon prio=10 tid=0x000000000098d000 nid=0x5350 waiting on condition [0x00007ff7a7dfc000]
      19d33
      < "process reaper" daemon prio=10 tid=0x0000000000de9000 nid=0x5339 waiting on condition [0x00007ff7acc74000]
      21d34
      < "Jenkins-Remoting-Thread-305" daemon prio=10 tid=0x00007ff7bcb79800 nid=0x5332 waiting on condition [0x00007ff7a7ffe000]
      23d35
      < "pool-16-thread-372" daemon prio=10 tid=0x00007ff7d8056000 nid=0x52b6 waiting on condition [0x00007ff7ac122000]
      glangs@cgm-1194:/opt/docker_volumes/jenkins3$
      
      

      Attachments

        Activity

          gerhard6 gerhard6 created issue -
          warden Radek Antoniuk made changes -
          Field Original Value New Value
          Assignee Radek Antoniuk [ warden ]

          Code changed in jenkins
          User: Radek Antoniuk
          Path:
          src/main/java/hudson/plugins/jira/JiraCreateIssueNotifier.java
          src/main/java/hudson/plugins/jira/JiraMailAddressResolver.java
          src/main/java/hudson/plugins/jira/JiraSite.java
          src/main/java/hudson/plugins/jira/Updater.java
          src/main/java/hudson/plugins/jira/listissuesparameter/JiraIssueParameterDefinition.java
          src/main/java/hudson/plugins/jira/versionparameter/JiraVersionParameterDefinition.java
          http://jenkins-ci.org/commit/jira-plugin/00e439d3e45eda4f51bef85d8ddf0b27f5e7f694
          Log:
          JENKINS-30333 - Using site.getSession() instead of site.createSession()

          Compare: https://github.com/jenkinsci/jira-plugin/compare/286a5320efb6...00e439d3e45e

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Radek Antoniuk Path: src/main/java/hudson/plugins/jira/JiraCreateIssueNotifier.java src/main/java/hudson/plugins/jira/JiraMailAddressResolver.java src/main/java/hudson/plugins/jira/JiraSite.java src/main/java/hudson/plugins/jira/Updater.java src/main/java/hudson/plugins/jira/listissuesparameter/JiraIssueParameterDefinition.java src/main/java/hudson/plugins/jira/versionparameter/JiraVersionParameterDefinition.java http://jenkins-ci.org/commit/jira-plugin/00e439d3e45eda4f51bef85d8ddf0b27f5e7f694 Log: JENKINS-30333 - Using site.getSession() instead of site.createSession() Compare: https://github.com/jenkinsci/jira-plugin/compare/286a5320efb6...00e439d3e45e
          gerhard6 gerhard6 added a comment -

          Thx for working on this. also noticed during usage on our production system, that the WeakReferences are GC'd quite often, causing repeated "createSession" calls, which in turn then again leak threads. Will try to switch to SoftReference for my tests, so that this happens hopefully much less often.

          gerhard6 gerhard6 added a comment - Thx for working on this. also noticed during usage on our production system, that the WeakReferences are GC'd quite often, causing repeated "createSession" calls, which in turn then again leak threads. Will try to switch to SoftReference for my tests, so that this happens hopefully much less often.

          Yes, that's known as commented in the source code by the previous implementer
          If you can fix it and test it out, please submit a pull request via github - possibly with a test, but this might be a bit hard to cover.

          warden Radek Antoniuk added a comment - Yes, that's known as commented in the source code by the previous implementer If you can fix it and test it out, please submit a pull request via github - possibly with a test, but this might be a bit hard to cover.
          warden Radek Antoniuk made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          warden Radek Antoniuk made changes -
          Labels jira-plugin-2.0.1 jira-plugin-2.1
          warden Radek Antoniuk made changes -
          Labels jira-plugin-2.0.1 jira-plugin-2.1 release-queue
          warden Radek Antoniuk added a comment - - edited

          I double checked and I think this is actually not fixed yet.
          I can still see the threads created every time any JIRA connection is used, tested on "Update relevant issues" and just accessing build results where the issues are linked in the Changelog - in this case, when one of the linked issues is having permission problem - so is not cached.
          This needs more investigation I guess...

          gerhard6 out of curiosity, can you double check your instance if it really fixed it in your case? I was just checking with :

          jstack $(ps aux | grep java | grep 9090 | awk '{print $2;}') | grep httpclient-io | wc -l
          

          and I can see the result being increased by 10 every build when JIRA issue is commented.

          warden Radek Antoniuk added a comment - - edited I double checked and I think this is actually not fixed yet. I can still see the threads created every time any JIRA connection is used, tested on "Update relevant issues" and just accessing build results where the issues are linked in the Changelog - in this case, when one of the linked issues is having permission problem - so is not cached. This needs more investigation I guess... gerhard6 out of curiosity, can you double check your instance if it really fixed it in your case? I was just checking with : jstack $(ps aux | grep java | grep 9090 | awk '{print $2;}' ) | grep httpclient-io | wc -l and I can see the result being increased by 10 every build when JIRA issue is commented.
          warden Radek Antoniuk made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          warden Radek Antoniuk made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          warden Radek Antoniuk made changes -
          Labels release-queue
          gerhard6 gerhard6 added a comment -

          What I've seen so far is that the usage of WeakReference seems to be not sufficient, as it gets GC'd too often, on my machine every few minutes, and on the production server I ended up in the roughly same situation as before.
          As mentioned, yesterday I changed to SoftReference, and the situation on the test and production server is better now - on production server I just saw 6 "createSession" (added a LOGGER) calls instead of 100 or so.

          If time permits, I will test your recent changes together with SoftReference today or tomorrow.

          gerhard6 gerhard6 added a comment - What I've seen so far is that the usage of WeakReference seems to be not sufficient, as it gets GC'd too often, on my machine every few minutes, and on the production server I ended up in the roughly same situation as before. As mentioned, yesterday I changed to SoftReference, and the situation on the test and production server is better now - on production server I just saw 6 "createSession" (added a LOGGER) calls instead of 100 or so. If time permits, I will test your recent changes together with SoftReference today or tomorrow.

          Code changed in jenkins
          User: Radek Antoniuk
          Path:
          src/main/java/hudson/plugins/jira/JiraCreateIssueNotifier.java
          src/main/java/hudson/plugins/jira/JiraMailAddressResolver.java
          src/main/java/hudson/plugins/jira/JiraSite.java
          src/main/java/hudson/plugins/jira/Updater.java
          src/main/java/hudson/plugins/jira/listissuesparameter/JiraIssueParameterDefinition.java
          src/main/java/hudson/plugins/jira/versionparameter/JiraVersionParameterDefinition.java
          http://jenkins-ci.org/commit/jira-plugin/57b3d8b1f7103e2e64f066e48b00bb927767c228
          Log:
          JENKINS-30333 - Using site.getSession() instead of site.createSession()

          Conflicts:
          src/main/java/hudson/plugins/jira/Updater.java

          Compare: https://github.com/jenkinsci/jira-plugin/compare/55e53851e4e5^...57b3d8b1f710

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Radek Antoniuk Path: src/main/java/hudson/plugins/jira/JiraCreateIssueNotifier.java src/main/java/hudson/plugins/jira/JiraMailAddressResolver.java src/main/java/hudson/plugins/jira/JiraSite.java src/main/java/hudson/plugins/jira/Updater.java src/main/java/hudson/plugins/jira/listissuesparameter/JiraIssueParameterDefinition.java src/main/java/hudson/plugins/jira/versionparameter/JiraVersionParameterDefinition.java http://jenkins-ci.org/commit/jira-plugin/57b3d8b1f7103e2e64f066e48b00bb927767c228 Log: JENKINS-30333 - Using site.getSession() instead of site.createSession() Conflicts: src/main/java/hudson/plugins/jira/Updater.java Compare: https://github.com/jenkinsci/jira-plugin/compare/55e53851e4e5 ^...57b3d8b1f710
          gerhard6 gerhard6 added a comment -

          Using SoftReference instead of WeakReference helped a bit on our production server, it is however not sufficient.
          Im wondering, why it is that important to use WhateverReferences at all.
          I will test with a direct reference...

          gerhard6 gerhard6 added a comment - Using SoftReference instead of WeakReference helped a bit on our production server, it is however not sufficient. Im wondering, why it is that important to use WhateverReferences at all. I will test with a direct reference...
          gerhard6 gerhard6 added a comment -

          tested with direct reference (and without Soft/WeakReferences), seems to work - we also dont get hit by any timeout of authentication (see that cc1 below was committed after more that 2 days).
          Will install on our production server this evening...

          Looking for JIRA ID in HISTL-8625 bb2
          Sep 11, 2015 10:52:58 AM FINE hudson.plugins.jira.Updater
          Looking for JIRA ID in HISTL-8625 aa3
          Sep 14, 2015 8:13:00 AM FINE hudson.plugins.jira.Updater
          Looking for JIRA ID in HISTL-8625 cc1
          
          gerhard6 gerhard6 added a comment - tested with direct reference (and without Soft/WeakReferences), seems to work - we also dont get hit by any timeout of authentication (see that cc1 below was committed after more that 2 days). Will install on our production server this evening... Looking for JIRA ID in HISTL-8625 bb2 Sep 11, 2015 10:52:58 AM FINE hudson.plugins.jira.Updater Looking for JIRA ID in HISTL-8625 aa3 Sep 14, 2015 8:13:00 AM FINE hudson.plugins.jira.Updater Looking for JIRA ID in HISTL-8625 cc1

          Sounds good, do you want to submit PR for this to contribute?

          warden Radek Antoniuk added a comment - Sounds good, do you want to submit PR for this to contribute?
          warden Radek Antoniuk made changes -
          Labels jira-plugin-2.0.2
          gerhard6 gerhard6 added a comment -

          I can try to do so (would be my first PR ever....) - but lets first wait till I verified it on my production system.....(tomorrow evening or so...)

          gerhard6 gerhard6 added a comment - I can try to do so (would be my first PR ever....) - but lets first wait till I verified it on my production system.....(tomorrow evening or so...)

          Everybody has his first time Sure, that sounds good, if you need any help, just shout!

          warden Radek Antoniuk added a comment - Everybody has his first time Sure, that sounds good, if you need any help, just shout!
          warden Radek Antoniuk made changes -
          Assignee Radek Antoniuk [ warden ] gerhard6 [ gerhard6 ]
          gerhard6 gerhard6 added a comment - here is my first PR ever: https://github.com/jenkinsci/jira-plugin/pull/62
          gerhard6 gerhard6 added a comment -

          note regarding the "TODO timeout" - If I understand corerrectly, then with REST, there cant be any such session-timeout, as every request carries along the authentication information.

          gerhard6 gerhard6 added a comment - note regarding the "TODO timeout" - If I understand corerrectly, then with REST, there cant be any such session-timeout, as every request carries along the authentication information.
          gerhard6 gerhard6 added a comment -

          As I'm new here and meanwhile "assignee". Is there anything left to do here for me ?

          • resolve the issue
          • who then pulls the outstanding request ?
          • when is a build / new release triggered ?
          gerhard6 gerhard6 added a comment - As I'm new here and meanwhile "assignee". Is there anything left to do here for me ? resolve the issue who then pulls the outstanding request ? when is a build / new release triggered ?
          warden Radek Antoniuk made changes -
          Assignee gerhard6 [ gerhard6 ] Radek Antoniuk [ warden ]

          Great, thanks for the PR. Yes, that's correct and should be taken care by the rest api client.
          I will take over and release this after verification.

          warden Radek Antoniuk added a comment - Great, thanks for the PR. Yes, that's correct and should be taken care by the rest api client. I will take over and release this after verification.

          Code changed in jenkins
          User: Gerhard Langs
          Path:
          src/main/java/hudson/plugins/jira/JiraSite.java
          http://jenkins-ci.org/commit/jira-plugin/a17aab4b71e5311082817c1273dbaeb4be63bc7c
          Log:
          JENKINS-30333 use direct references instead of WeakReference, added createSession logging

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Gerhard Langs Path: src/main/java/hudson/plugins/jira/JiraSite.java http://jenkins-ci.org/commit/jira-plugin/a17aab4b71e5311082817c1273dbaeb4be63bc7c Log: JENKINS-30333 use direct references instead of WeakReference, added createSession logging

          Code changed in jenkins
          User: Gerhard Langs
          Path:
          src/main/java/hudson/plugins/jira/JiraSite.java
          http://jenkins-ci.org/commit/jira-plugin/f6d0f1fc927fcd373dd05eada22dd87a24a2fe0b
          Log:
          JENKINS-30333 use direct references instead of WeakReference, added createSession logging

          Conflicts:
          src/main/java/hudson/plugins/jira/JiraSite.java

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Gerhard Langs Path: src/main/java/hudson/plugins/jira/JiraSite.java http://jenkins-ci.org/commit/jira-plugin/f6d0f1fc927fcd373dd05eada22dd87a24a2fe0b Log: JENKINS-30333 use direct references instead of WeakReference, added createSession logging Conflicts: src/main/java/hudson/plugins/jira/JiraSite.java

          Code changed in jenkins
          User: Gerhard Langs
          Path:
          src/main/java/hudson/plugins/jira/JiraSite.java
          http://jenkins-ci.org/commit/jira-plugin/f238b971bca8f682040e0344dc4e8778fd7da112
          Log:
          JENKINS-30333 use direct references instead of WeakReference, added createSession logging

          Conflicts:
          src/main/java/hudson/plugins/jira/JiraSite.java

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Gerhard Langs Path: src/main/java/hudson/plugins/jira/JiraSite.java http://jenkins-ci.org/commit/jira-plugin/f238b971bca8f682040e0344dc4e8778fd7da112 Log: JENKINS-30333 use direct references instead of WeakReference, added createSession logging Conflicts: src/main/java/hudson/plugins/jira/JiraSite.java
          warden Radek Antoniuk made changes -
          Labels jira-plugin-2.0.2 jira-plugin-2.0.2 release-queue
          warden Radek Antoniuk made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]

          What happens if the session expires and a new login token has to be acquired? It seems like the WeakReference caused this to be refreshed at a random interval.

          walterk82 Walter Kacynski added a comment - What happens if the session expires and a new login token has to be acquired? It seems like the WeakReference caused this to be refreshed at a random interval.
          gerhard6 gerhard6 added a comment - @Walter: my knowledge/assumption and related test is described in the comments above: https://issues.jenkins-ci.org/browse/JENKINS-30333?focusedCommentId=236862&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-236862 https://issues.jenkins-ci.org/browse/JENKINS-30333?focusedCommentId=236762&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-236762

          Sorry I missed that, thanks.

          walterk82 Walter Kacynski added a comment - Sorry I missed that, thanks.
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 165434 ] JNJira + In-Review [ 197717 ]
          warden Radek Antoniuk made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            warden Radek Antoniuk
            gerhard6 gerhard6
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: