Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31192

Subversion HTTPS + PKCS12 Certificate in CPU infinite loop

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • subversion-plugin
    • Jenkins LTS 1.625.1, Subversion plugin 2.5.3 (svnkit 1.8.6), Java 8u60, Debian GNU/Linux Jessie 8.2

      After upgrading to Jenkins 1.625.1 and switching to Java 8u60,
      one of our job configured to use mutual SSL authentication against a HTTPS Subversion repository produces the following behaviour:

      • the polling SCM thread consumes 100% in "endless" sun.security.pkcs12.PKCS12KeyStore.engineGetKey method
      • because of the lock hudson.scm.SubversionSCM$ModuleLocation held there, all SCM commit notification threads get stuck
      • as a result, the SCM thread pool gets exhausted and management page displays a warning

      In production, everything get stuck for 12 hours before we had to restart Jenkins master.
      In test environment, we reproduced with both Java 7u80 and Java 8u60.
      The issue persists even after replacing default "Sun" security provider by "BouncyCastle" to use alternate PKCS#12, SHA and DES implementations.

          [JENKINS-31192] Subversion HTTPS + PKCS12 Certificate in CPU infinite loop

          Yves Martin added a comment -

          Running with -Xint to disable JIT leads to same high CPU consumption and stack trace in thread dump.

          Yves Martin added a comment - Running with -Xint to disable JIT leads to same high CPU consumption and stack trace in thread dump.

          Yves Martin added a comment - - edited

          After investigation, the issue appears when Subversion plugin has moved to standard SVNKit 1.8.6, where SVNSSLAuthentication constructor with certificate as byte[] does not exist. This constructor was probably only available in Jenkins-specific svnkit 1.7 version.
          http://javadox.com/org.tmatesoft.svnkit/svnkit/1.8.3/org/tmatesoft/svn/core/auth/SVNSSLAuthentication.html

          As a result, the String.valueOf(certificateFile) introduced in CredentialsSVNAuthenticationProviderImpl.build when "porting" to SVNKit 1.8.6 is a big mistake and is not expected to produce any correct SSL certificat authentication mechanism.

          By the way, it is abnormal such an authentication failure leads to infinite authentication loop in plugin. Still investigating, patches should come soon.

          Yves Martin added a comment - - edited After investigation, the issue appears when Subversion plugin has moved to standard SVNKit 1.8.6, where SVNSSLAuthentication constructor with certificate as byte[] does not exist. This constructor was probably only available in Jenkins-specific svnkit 1.7 version. http://javadox.com/org.tmatesoft.svnkit/svnkit/1.8.3/org/tmatesoft/svn/core/auth/SVNSSLAuthentication.html As a result, the String.valueOf(certificateFile) introduced in CredentialsSVNAuthenticationProviderImpl.build when "porting" to SVNKit 1.8.6 is a big mistake and is not expected to produce any correct SSL certificat authentication mechanism. By the way, it is abnormal such an authentication failure leads to infinite authentication loop in plugin. Still investigating, patches should come soon.

          ymartin1040, Could you try this PR where SVNKit was updated?

          Manuel Recena Soto added a comment - ymartin1040 , Could you try this PR where SVNKit was updated?

          Yves Martin added a comment - - edited

          My collegue Sylvain Pasche and I have done debugging session in Jenkins and find out that the String.valueOf(certificateFile) expression which returns [B@xxxxx prevents the SVNSSLAuthentication object to be compared as equal even when generated from the same certificate entity. As a result, the "getNextAuthentication" method is endlessly invoked in loop, explaining the high CPU symptom.

          Definitely the issue is in Subversion Plugin only. I have designed and tested a working patch - without any change in SVNKit 1.8 itself (so not as implemented previously with 1.7.x). I am preparing the pull request.

          Yves Martin added a comment - - edited My collegue Sylvain Pasche and I have done debugging session in Jenkins and find out that the String.valueOf(certificateFile) expression which returns [B@xxxxx prevents the SVNSSLAuthentication object to be compared as equal even when generated from the same certificate entity. As a result, the "getNextAuthentication" method is endlessly invoked in loop, explaining the high CPU symptom. Definitely the issue is in Subversion Plugin only. I have designed and tested a working patch - without any change in SVNKit 1.8 itself (so not as implemented previously with 1.7.x). I am preparing the pull request.

          ymartin1040, Great. I'll review the PR. Please, assign yourself the ticket.

          Manuel Recena Soto added a comment - ymartin1040 , Great. I'll review the PR. Please, assign yourself the ticket.

          ymartin1040, I could not reproduce this bug. Probably different Subversion Server configuration. I used SVN + HTTPs offered by Assembla.

          Manuel Recena Soto added a comment - ymartin1040 , I could not reproduce this bug. Probably different Subversion Server configuration. I used SVN + HTTPs offered by Assembla.

          Yves Martin added a comment - - edited

          Yves Martin added a comment - - edited Pending pull request: https://github.com/jenkinsci/subversion-plugin/pull/147

          Code changed in jenkins
          User: Yves Martin
          Path:
          src/main/java/hudson/scm/CredentialsSVNAuthenticationProviderImpl.java
          src/main/java/hudson/scm/SubversionSCM.java
          http://jenkins-ci.org/commit/subversion-plugin/86e18b3e1b340fb1fe88296a8bf070fbc4ae2cdf
          Log:
          JENKINS-31192 Fix HTTPS SSL client certificate authentication

          Take benefits of new SVNKit 1.8.11 SVNSSLAuthentication methods to properly
          support byte array PKCS#12 certificate usage.

          Resolves: JENKINS-31192

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Yves Martin Path: src/main/java/hudson/scm/CredentialsSVNAuthenticationProviderImpl.java src/main/java/hudson/scm/SubversionSCM.java http://jenkins-ci.org/commit/subversion-plugin/86e18b3e1b340fb1fe88296a8bf070fbc4ae2cdf Log: JENKINS-31192 Fix HTTPS SSL client certificate authentication Take benefits of new SVNKit 1.8.11 SVNSSLAuthentication methods to properly support byte array PKCS#12 certificate usage. Resolves: JENKINS-31192

          Code changed in jenkins
          User: Manuel Recena
          Path:
          pom.xml
          src/main/java/hudson/scm/CredentialsSVNAuthenticationProviderImpl.java
          src/main/java/hudson/scm/SubversionSCM.java
          http://jenkins-ci.org/commit/subversion-plugin/ae4ae6408801cc6e14aa3f8f1586e2a19929224c
          Log:
          Merge pull request #147 from ymartin59/JENKINS-31192

          JENKINS-31192 HTTPS + PKCS12 Certificate in CPU infinite loop

          Compare: https://github.com/jenkinsci/subversion-plugin/compare/3141a0947ac9...ae4ae6408801

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Manuel Recena Path: pom.xml src/main/java/hudson/scm/CredentialsSVNAuthenticationProviderImpl.java src/main/java/hudson/scm/SubversionSCM.java http://jenkins-ci.org/commit/subversion-plugin/ae4ae6408801cc6e14aa3f8f1586e2a19929224c Log: Merge pull request #147 from ymartin59/ JENKINS-31192 JENKINS-31192 HTTPS + PKCS12 Certificate in CPU infinite loop Compare: https://github.com/jenkinsci/subversion-plugin/compare/3141a0947ac9...ae4ae6408801

          Thanks to ymartin1040 to solve this bug.

          Manuel Recena Soto added a comment - Thanks to ymartin1040 to solve this bug.

            ymartin1040 Yves Martin
            ymartin1040 Yves Martin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: