• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • sonar-plugin
    • None

      The sonar plugin seems to be causing hung threads on our jenkins server. When requesting certain pages, like the Pull Request list on multibranch-pipeline jobs, or the blue ocean view for some jobs, the browser page will load forever and eventually time out in apache. I tested bypassing apache reverse proxy and it just hung indefinitely.

      I ran a thread dump on the jenkins server and I see hung threads for each time I attempted to load the page, that are waiting on code in the sonar plugin. These threads have persisted indefinitely, until the Jenkins server was restarted.

      The jobs that have issues have large (~80) parallel build matrixes, each with a sonar stage, so we may be overloading the sonar server when trying to get the status of all of the jobs. However it seems like the requests should have a client timeout/retry on the jenkins side so that the threads don't get stuck forever.

       

      I have attached one of the stack traces of the threads as a txt file

          [JENKINS-70694] Sonar Plugin Causes Hung Threads

          Chao added a comment - - edited

          we are facing the same issue. 

          Jenkins version: 2.361.4

          sonar plugin version: 2.15

          sonar server version: 9.9.0.65466

          jenkins-jstack-pid-675.txt

          Chao added a comment - - edited we are facing the same issue.  Jenkins version: 2.361.4 sonar plugin version: 2.15 sonar server version: 9.9.0.65466 jenkins-jstack-pid-675.txt

          Heidi Rechek added a comment -

          Jenkins 2.387.1
          sonar plugin version: 2.15
          sonar server version:  sonarqube:10.0.0-developer  running as an Azure app service with a custom domain. 

           

          When migrating from a public .azurewebsites.net to a custom domain available only to vpn clients we had this issue. 

           

          Heidi Rechek added a comment - Jenkins 2.387.1 sonar plugin version: 2.15 sonar server version:  sonarqube:10.0.0-developer  running as an Azure app service with a custom domain.    When migrating from a public .azurewebsites.net to a custom domain available only to vpn clients we had this issue.   

          ipleten added a comment -

          We also have this problem, also it happens than SQ server is just down. Jenkins BlueOcean UI is not usable at all during outage.

          ipleten added a comment - We also have this problem, also it happens than SQ server is just down. Jenkins BlueOcean UI is not usable at all during outage.

          We are having the same issue. Regardless of what the actual root cause is, network calls should always have a timeout so that they don't block the thread indefinitly.

          Thorsten Meinl added a comment - We are having the same issue. Regardless of what the actual root cause is, network calls should always have a timeout so that they don't block the thread indefinitly.

          Ben added a comment -

          We are also having the problem, for what it's worth.  We believe it is caused by the plugin calling out to sonarcloud to get the version.  We suspect that there is throttling involved since this is not an endpoint that requires authentication.  It appears that the plugin gets stuck trying

          To fix it, we forked the plugin and simply hard-coded a sonar version where it normally would call out to sonar to get the version, which I believe is right here.  I know this isn't a long term fix, but the only long term fix at the moment that we know of is to not use the plugin at all.

          Ben added a comment - We are also having the problem, for what it's worth.  We believe it is caused by the plugin calling out to sonarcloud to get the version.  We suspect that there is throttling involved since this is not an endpoint that requires authentication.  It appears that the plugin gets stuck trying To fix it, we forked the plugin and simply hard-coded a sonar version where it normally would call out to sonar to get the version, which I believe is right here .  I know this isn't a long term fix, but the only long term fix at the moment that we know of is to not use the plugin at all.

          The libraries for talking to Sonar that are bundled with the plug-in are pretty old (from around 2017!). I updated sonar-ws to the latest version which also updates okhttp to something much more recent and now the issue is gone. Therefore the easiest solution is likely to simply update dependencies.

          Thorsten Meinl added a comment - The libraries for talking to Sonar that are bundled with the plug-in are pretty old (from around 2017!). I updated sonar-ws to the latest version which also updates okhttp to something much more recent and now the issue is gone. Therefore the easiest solution is likely to simply update dependencies.

          Yoann added a comment -

          sithmein , Would you mind sharing the pom.xml you updated to recompile the plugin ? We basically came to the same conclusion on that issue here : BlueOcean sends a lot of requests to sonar, one times out, blocks a shared ressource, causing all the subsequent requests and users to hung on the same ressource. There's no timeout indeed, and nothing we tried worked. I will also try recompiling the plugin.

          Yoann added a comment - sithmein , Would you mind sharing the pom.xml you updated to recompile the plugin ? We basically came to the same conclusion on that issue here : BlueOcean sends a lot of requests to sonar, one times out, blocks a shared ressource, causing all the subsequent requests and users to hung on the same ressource. There's no timeout indeed, and nothing we tried worked. I will also try recompiling the plugin.

          Thorsten Meinl added a comment - - edited

          Here's a patch that bumps the Sonar client library. Some minor code changes in the tests are required due to the major update of okhttp.

          After we patched the plug-in the issue didn't occur any more.

          Updated-Sonar-client.patch

          Thorsten Meinl added a comment - - edited Here's a patch that bumps the Sonar client library. Some minor code changes in the tests are required due to the major update of okhttp. After we patched the plug-in the issue didn't occur any more. Updated-Sonar-client.patch

            sonarteam Sonar Team
            msedlik Mason
            Votes:
            7 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: