Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21902

Add Waiting / Blocked on information in textual thread dump

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • monitoring-plugin
    • None

      A waiting thread will show in the Monitoring plugin like this:
      "http-bio-9029-exec-3493" daemon prio=5 TIMED_WAITING
      But on host-url/threadDump screen it'll give more information on the object that is being waited for:
      "http-bio-9029-exec-3493" Id=194353 Group=main TIMED_WAITING on com.sun.jndi.ldap.LdapRequest@10f324c

      This goes the same for Blocked state.
      Without this information the thread-dump feature becomes kind of useless in these situations.

          [JENKINS-21902] Add Waiting / Blocked on information in textual thread dump

          I managed to find a BLOCKED state example that clearly shows that the monitoring plugin is not helping in these cases.

          Monitoring plugin:
          "Thread-127432" daemon prio=5 BLOCKED
          com.sun.jndi.ldap.Connection.getInputStream(Connection.java:710)
          com.sun.jndi.ldap.Connection.run(Connection.java:832)
          java.lang.Thread.run(Thread.java:722)

          jenkins-url/threadDump:
          "Thread-127432" Id=193884 Group=main BLOCKED on com.sun.jndi.ldap.Connection@77b338dc owned by "Jenkins cron thread" Id=23
          at com.sun.jndi.ldap.Connection.getInputStream(Connection.java:710)

          • blocked on com.sun.jndi.ldap.Connection@77b338dc
            at com.sun.jndi.ldap.Connection.run(Connection.java:832)
            at java.lang.Thread.run(Thread.java:722)

          Sagi Sinai-Glazer added a comment - I managed to find a BLOCKED state example that clearly shows that the monitoring plugin is not helping in these cases. Monitoring plugin: "Thread-127432" daemon prio=5 BLOCKED com.sun.jndi.ldap.Connection.getInputStream(Connection.java:710) com.sun.jndi.ldap.Connection.run(Connection.java:832) java.lang.Thread.run(Thread.java:722) jenkins-url/threadDump: "Thread-127432" Id=193884 Group=main BLOCKED on com.sun.jndi.ldap.Connection@77b338dc owned by "Jenkins cron thread" Id=23 at com.sun.jndi.ldap.Connection.getInputStream(Connection.java:710) blocked on com.sun.jndi.ldap.Connection@77b338dc at com.sun.jndi.ldap.Connection.run(Connection.java:832) at java.lang.Thread.run(Thread.java:722)

          evernat added a comment -

          First, as said above, use http://host/threadDump to display locks in Jenkins.

          Then, adding "Waiting / Blocked on" information in the textual thread dump [1] is currently not a high priority for javamelody (used in the monitoring plugin), because:

          • Having lock issues in a webapp, such as Jenkins, is a critical issue.
          • So, I think personnally that developers using synchronized / locks are supposed to know what they are doing or otherwise to use lock-free patterns: it's much better to not have lock issues than to fight a critical fire with just a report of locks in a thread dump.
          • In the worst case, those developers are supposed to know and recommend tools like jstack, or http://host/threadDump in Jenkins.
            That said, the plugin displays the names of the deadlocked threads, when needed.

          [1] https://code.google.com/p/javamelody/source/browse/trunk/javamelody-core/src/main/java/net/bull/javamelody/HtmlThreadInformationsReport.java#91

          evernat added a comment - First, as said above, use http://host/threadDump to display locks in Jenkins. Then, adding "Waiting / Blocked on" information in the textual thread dump [1] is currently not a high priority for javamelody (used in the monitoring plugin), because: Having lock issues in a webapp, such as Jenkins, is a critical issue. So, I think personnally that developers using synchronized / locks are supposed to know what they are doing or otherwise to use lock-free patterns: it's much better to not have lock issues than to fight a critical fire with just a report of locks in a thread dump. In the worst case, those developers are supposed to know and recommend tools like jstack, or http://host/threadDump in Jenkins. That said, the plugin displays the names of the deadlocked threads, when needed. [1] https://code.google.com/p/javamelody/source/browse/trunk/javamelody-core/src/main/java/net/bull/javamelody/HtmlThreadInformationsReport.java#91

          The problem with using /threadDump is that it uses the main Jenkins UI to display it.
          This means that in case you have the Queue is locked, the page will not be generated.
          The nice thing about the monitoring plugin is the fact that it doesn't use any Jenkins UI, and therefore isn't influenced by such problems and actually help detect them... but again, without the locking information - it's not really helpful.

          One more note - in many cases the administrators of the instance do not have access to the backend VM and therefore it's hard or almost impossible to get a jstack thread-dump.

          And as for Jenkins development - I don't think that in the near future we'll see the enough lock-free patterns to replace all that is currently used.

          Sagi Sinai-Glazer added a comment - The problem with using /threadDump is that it uses the main Jenkins UI to display it. This means that in case you have the Queue is locked, the page will not be generated. The nice thing about the monitoring plugin is the fact that it doesn't use any Jenkins UI, and therefore isn't influenced by such problems and actually help detect them... but again, without the locking information - it's not really helpful. One more note - in many cases the administrators of the instance do not have access to the backend VM and therefore it's hard or almost impossible to get a jstack thread-dump. And as for Jenkins development - I don't think that in the near future we'll see the enough lock-free patterns to replace all that is currently used.

          evernat added a comment -

          About your last comment:
          1) Thanks for the plugin. But a logical change, for everyone not using the monitoring plugin, would be that http://host/threadDump of Jenkins does not use Jenkins UI, in order to be independent of queue locks. Waiting for that change in Jenkins, your argument is valid anyway for the plugin.

          2) A bit sad. I suppose that a Jenkins job, created just to print jstack output, does not help for the same reason. So, I have no solution to this point except the one above.

          3) I am well aware that Jenkins has gone very far on too many "public in the api" locks.

          Said otherwise:

          • Points 2 and 3 will not change anything to Jenkins.
          • Point 1 should motivate to create a related core issue (and a pull request) in Jenkins, speaking about the difficulty to diagnose the (queue) locks in Jenkins. And anyway, it is a valid argument for the plugin.

          evernat added a comment - About your last comment: 1) Thanks for the plugin. But a logical change, for everyone not using the monitoring plugin, would be that http://host/threadDump of Jenkins does not use Jenkins UI, in order to be independent of queue locks. Waiting for that change in Jenkins, your argument is valid anyway for the plugin. 2) A bit sad. I suppose that a Jenkins job, created just to print jstack output, does not help for the same reason. So, I have no solution to this point except the one above. 3) I am well aware that Jenkins has gone very far on too many "public in the api" locks. Said otherwise: Points 2 and 3 will not change anything to Jenkins. Point 1 should motivate to create a related core issue (and a pull request) in Jenkins, speaking about the difficulty to diagnose the (queue) locks in Jenkins. And anyway, it is a valid argument for the plugin.

            evernat evernat
            esinsag Sagi Sinai-Glazer
            Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: