Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52374

Issue with unclosed LDAP connections

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      On our environment we are observing a high number of threads waiting with the following stack:

      Thread-169"Thread-169" Id=277 Group=main WAITING on java.lang.Object@64e59be7
      	at java.base@10.0.1/java.lang.Object.wait(Native Method)
      	-  waiting on java.lang.Object@64e59be7
      	at java.base@10.0.1/java.lang.Object.wait(Object.java:328)
      	at java.naming@10.0.1/com.sun.jndi.ldap.Connection.pauseReader(Connection.java:771)
      	at java.naming@10.0.1/com.sun.jndi.ldap.Connection.run(Connection.java:911)
      	at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
      
      Thread-175"Thread-175" Id=283 Group=main WAITING on java.lang.Object@156aef27
      	at java.base@10.0.1/java.lang.Object.wait(Native Method)
      	-  waiting on java.lang.Object@156aef27
      	at java.base@10.0.1/java.lang.Object.wait(Object.java:328)
      	at java.naming@10.0.1/com.sun.jndi.ldap.Connection.pauseReader(Connection.java:771)
      	at java.naming@10.0.1/com.sun.jndi.ldap.Connection.run(Connection.java:911)
      	at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
      
      Thread-177"Thread-177" Id=285 Group=main WAITING on java.lang.Object@3df5e68a
      	at java.base@10.0.1/java.lang.Object.wait(Native Method)
      	-  waiting on java.lang.Object@3df5e68a
      	at java.base@10.0.1/java.lang.Object.wait(Object.java:328)
      	at java.naming@10.0.1/com.sun.jndi.ldap.Connection.pauseReader(Connection.java:771)
      	at java.naming@10.0.1/com.sun.jndi.ldap.Connection.run(Connection.java:911)
      	at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
      

      (dump from http://jenkinsurl/threadDump )

      The amount of similar waiting threads increases by 2 on every login and are not being closed leading to a "Too many open files" after a couple days, at that point 378 were waiting with the stack above.

      Please let me know if any further information is required to help solve/reproduce this issue.

       

        Attachments

          Issue Links

            Activity

            Hide
            nridgway Nick Ridgway added a comment -

            I am also seeing this issue.  Switching STARTTLS off isn't an option for me, is there any information I can provide to help diagnose/fix the issue? 

            Show
            nridgway Nick Ridgway added a comment - I am also seeing this issue.  Switching STARTTLS off isn't an option for me, is there any information I can provide to help diagnose/fix the issue? 
            Hide
            gradol Oliver Grad added a comment -

            We discovered the same issue. On 3/10/2020 we upgraded our productive Jenkins (2.204.5) from Java 8 to Java 11 and the missbehaviour started immediatly.

            We use AD-Plugin version 2.16.
            As soon as we disabled StartTLS the problem disappeared (on the right side of the graph).

            Show
            gradol Oliver Grad added a comment - We discovered the same issue. On 3/10/2020 we upgraded our productive Jenkins (2.204.5) from Java 8 to Java 11 and the missbehaviour started immediatly. We use AD-Plugin version 2.16. As soon as we disabled StartTLS the problem disappeared (on the right side of the graph).
            Hide
            mramonleon Ramon Leon added a comment - - edited

            The issue is related to LdapContext#reconnect always opens a new connection https://bugs.openjdk.java.net/browse/JDK-8217606 We do a reconnect at https://github.com/jenkinsci/active-directory-plugin/blob/d3a94592176108701e72ad8726f462c6f0e8b606/src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java#L721

            It's worth to check whether it's fixed with Java 11.0.8 (2020-07-14). Seems to be fixed there.

            Versions where the fix was backported:

            Issue Fix Version
            JDK-8245802 13.0.4 
            JDK-8237876 11.0.8-oracle 
            JDK-8240434 11.0.8 
            JDK-8249807 openjdk8u272 
            JDK-8248118 8u271 
            JDK-8248718 8u261 
            JDK-8251719 emb-8u271 
            Show
            mramonleon Ramon Leon added a comment - - edited The issue is related to LdapContext#reconnect always opens a new connection https://bugs.openjdk.java.net/browse/JDK-8217606 We do a reconnect at https://github.com/jenkinsci/active-directory-plugin/blob/d3a94592176108701e72ad8726f462c6f0e8b606/src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java#L721 It's worth to check whether it's fixed with Java 11.0.8 (2020-07-14). Seems to be fixed there. Versions where the fix was backported: Issue Fix Version JDK-8245802 13.0.4  JDK-8237876 11.0.8-oracle  JDK-8240434 11.0.8  JDK-8249807 openjdk8u272  JDK-8248118 8u271  JDK-8248718 8u261  JDK-8251719 emb-8u271 
            Hide
            mramonleon Ramon Leon added a comment -

            I put it on review to gather feedback from affected people to see whether the Java update works

            Show
            mramonleon Ramon Leon added a comment - I put it on review to gather feedback from affected people to see whether the Java update works
            Hide
            timja Tim Jacomb added a comment -

            should be fixed in newer java, no new reports in many months

            Show
            timja Tim Jacomb added a comment - should be fixed in newer java, no new reports in many months

              People

              Assignee:
              mramonleon Ramon Leon
              Reporter:
              felipebrnd Felipe Nascimento
              Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: