Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31192

Subversion HTTPS + PKCS12 Certificate in CPU infinite loop

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • subversion-plugin
    • Jenkins LTS 1.625.1, Subversion plugin 2.5.3 (svnkit 1.8.6), Java 8u60, Debian GNU/Linux Jessie 8.2

      After upgrading to Jenkins 1.625.1 and switching to Java 8u60,
      one of our job configured to use mutual SSL authentication against a HTTPS Subversion repository produces the following behaviour:

      • the polling SCM thread consumes 100% in "endless" sun.security.pkcs12.PKCS12KeyStore.engineGetKey method
      • because of the lock hudson.scm.SubversionSCM$ModuleLocation held there, all SCM commit notification threads get stuck
      • as a result, the SCM thread pool gets exhausted and management page displays a warning

      In production, everything get stuck for 12 hours before we had to restart Jenkins master.
      In test environment, we reproduced with both Java 7u80 and Java 8u60.
      The issue persists even after replacing default "Sun" security provider by "BouncyCastle" to use alternate PKCS#12, SHA and DES implementations.

          [JENKINS-31192] Subversion HTTPS + PKCS12 Certificate in CPU infinite loop

          Yves Martin created issue -
          Yves Martin made changes -

          Yves Martin added a comment -

          Running with -Xint to disable JIT leads to same high CPU consumption and stack trace in thread dump.

          Yves Martin added a comment - Running with -Xint to disable JIT leads to same high CPU consumption and stack trace in thread dump.
          Yves Martin made changes -
          Description Original: After upgrading to Jenkins 1.625.1 and switching to Java 8u60,
          one of our job configured to use mutual SSL authentication against a HTTPS Subversion repository produces the following behaviour:
          - the polling SCM thread consumes 100% in "endless" {{sun.security.pkcs12.PKCS12KeyStore.engineGetKey}} method
          - because of the lock {{hudson.scm.SubversionSCM$ModuleLocation}} held there, all SCM commit notification gets stuck
          - as a result, the SCM thread pool gets exhausted and management page displays a warning

          In production, everything get stuck for 12 hours before we had to restart Jenkins master.
          In test environment, we reproduced with both Java 7u80 and Java 8u60.
          The issue persists even after replacing default "Sun" security provider by "BouncyCastle" to use alternate PKCS#12, SHA and DES implementations.
          New: After upgrading to Jenkins 1.625.1 and switching to Java 8u60,
          one of our job configured to use mutual SSL authentication against a HTTPS Subversion repository produces the following behaviour:
          - the polling SCM thread consumes 100% in "endless" {{sun.security.pkcs12.PKCS12KeyStore.engineGetKey}} method
          - because of the lock {{hudson.scm.SubversionSCM$ModuleLocation}} held there, all SCM commit notification threads get stuck
          - as a result, the SCM thread pool gets exhausted and management page displays a warning

          In production, everything get stuck for 12 hours before we had to restart Jenkins master.
          In test environment, we reproduced with both Java 7u80 and Java 8u60.
          The issue persists even after replacing default "Sun" security provider by "BouncyCastle" to use alternate PKCS#12, SHA and DES implementations.
          Yves Martin made changes -
          Environment Original: Jenkins LTS 1.625.1, Java 8u60, Debian GNU/Linux Jessie 8.2 New: Jenkins LTS 1.625.1, Subversion plugin Java 8u60, Debian GNU/Linux Jessie 8.2
          Yves Martin made changes -
          Environment Original: Jenkins LTS 1.625.1, Subversion plugin Java 8u60, Debian GNU/Linux Jessie 8.2 New: Jenkins LTS 1.625.1, Subversion plugin 2.5.3 (svnkit 1.8.6), Java 8u60, Debian GNU/Linux Jessie 8.2

          Yves Martin added a comment - - edited

          After investigation, the issue appears when Subversion plugin has moved to standard SVNKit 1.8.6, where SVNSSLAuthentication constructor with certificate as byte[] does not exist. This constructor was probably only available in Jenkins-specific svnkit 1.7 version.
          http://javadox.com/org.tmatesoft.svnkit/svnkit/1.8.3/org/tmatesoft/svn/core/auth/SVNSSLAuthentication.html

          As a result, the String.valueOf(certificateFile) introduced in CredentialsSVNAuthenticationProviderImpl.build when "porting" to SVNKit 1.8.6 is a big mistake and is not expected to produce any correct SSL certificat authentication mechanism.

          By the way, it is abnormal such an authentication failure leads to infinite authentication loop in plugin. Still investigating, patches should come soon.

          Yves Martin added a comment - - edited After investigation, the issue appears when Subversion plugin has moved to standard SVNKit 1.8.6, where SVNSSLAuthentication constructor with certificate as byte[] does not exist. This constructor was probably only available in Jenkins-specific svnkit 1.7 version. http://javadox.com/org.tmatesoft.svnkit/svnkit/1.8.3/org/tmatesoft/svn/core/auth/SVNSSLAuthentication.html As a result, the String.valueOf(certificateFile) introduced in CredentialsSVNAuthenticationProviderImpl.build when "porting" to SVNKit 1.8.6 is a big mistake and is not expected to produce any correct SSL certificat authentication mechanism. By the way, it is abnormal such an authentication failure leads to infinite authentication loop in plugin. Still investigating, patches should come soon.

          ymartin1040, Could you try this PR where SVNKit was updated?

          Manuel Recena Soto added a comment - ymartin1040 , Could you try this PR where SVNKit was updated?

          Yves Martin added a comment - - edited

          My collegue Sylvain Pasche and I have done debugging session in Jenkins and find out that the String.valueOf(certificateFile) expression which returns [B@xxxxx prevents the SVNSSLAuthentication object to be compared as equal even when generated from the same certificate entity. As a result, the "getNextAuthentication" method is endlessly invoked in loop, explaining the high CPU symptom.

          Definitely the issue is in Subversion Plugin only. I have designed and tested a working patch - without any change in SVNKit 1.8 itself (so not as implemented previously with 1.7.x). I am preparing the pull request.

          Yves Martin added a comment - - edited My collegue Sylvain Pasche and I have done debugging session in Jenkins and find out that the String.valueOf(certificateFile) expression which returns [B@xxxxx prevents the SVNSSLAuthentication object to be compared as equal even when generated from the same certificate entity. As a result, the "getNextAuthentication" method is endlessly invoked in loop, explaining the high CPU symptom. Definitely the issue is in Subversion Plugin only. I have designed and tested a working patch - without any change in SVNKit 1.8 itself (so not as implemented previously with 1.7.x). I am preparing the pull request.

          ymartin1040, Great. I'll review the PR. Please, assign yourself the ticket.

          Manuel Recena Soto added a comment - ymartin1040 , Great. I'll review the PR. Please, assign yourself the ticket.

            ymartin1040 Yves Martin
            ymartin1040 Yves Martin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: