Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34121

IBM Java doesn't support AES/CTR/PKCS5Padding, required for JNLP3

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • remoting
    • None

      It is not possible to run slaves anymore using IBM Java.

      This is due to the new encrypted communication introduced in Jenkins 1.653, where the handshake is done using "AES/CTR/PKCS5Padding".
      I couldn't find what ciphers IBM Java does or doesn't support (maybe nothing else than the default ones), but I created a quick test to check (see below).

      Would it be possible to switch to a cipher supported by IBM Java?

      Test:
      import java.security.NoSuchAlgorithmException;
      import javax.crypto.Cipher;
      import javax.crypto.NoSuchPaddingException;
      public class PaddingIssue {
      private static final String CIPHER = "AES/CTR/PKCS5Padding";
      public static void main(String[] args) throws NoSuchAlgorithmException, NoSuchPaddingException

      { Cipher encryptCipher = Cipher.getInstance(CIPHER); System.out.println("Fine!"); }

      }

      Executions:
      IBM JAVA 1.6
      /usr/lib/j2re1.6-ibm/jre/bin/java PaddingIssue
      Exception in thread "main" java.security.NoSuchAlgorithmException: Cannot find any provider supporting AES/CTR/PKCS5Padding
      at javax.crypto.Cipher.getInstance(Unknown Source)
      at PaddingIssue.main(PaddingIssue.java:10)
      Caused by: javax.crypto.NoSuchPaddingException: CTR mode must be used with ISO10126Padding or NoPadding
      at com.ibm.crypto.provider.AESCipher.engineSetPadding(Unknown Source)
      at javax.crypto.Cipher$a.a(Unknown Source)
      ... 2 more

      IBM JAVA 1.7
      /usr/lib/j2re1.7-ibm/jre/bin/java PaddingIssue
      Exception in thread "main" java.security.NoSuchAlgorithmException: Cannot find any provider supporting AES/CTR/PKCS5Padding
      at javax.crypto.Cipher.getInstance(Unknown Source)
      at PaddingIssue.main(PaddingIssue.java:10)
      Caused by: javax.crypto.NoSuchPaddingException: CTR mode must be used with ISO10126Padding or NoPadding
      at com.ibm.crypto.provider.AESCipher.engineSetPadding(Unknown Source)
      at javax.crypto.Cipher$a.a(Unknown Source)
      ... 2 more

      IBM JAVA 1.8
      /usr/lib/jvm/java-ibm-x86_64-80/jre/bin/java PaddingIssue
      Exception in thread "main" java.security.NoSuchAlgorithmException: Cannot find any provider supporting AES/CTR/PKCS5Padding
      at javax.crypto.Cipher.getInstance(Unknown Source)
      at PaddingIssue.main(PaddingIssue.java:10)
      Caused by: javax.crypto.NoSuchPaddingException: CTR mode must be used with ISO10126Padding or NoPadding
      at com.ibm.crypto.provider.AbstractBufferingCipher.engineSetPadding(Unknown Source)
      at javax.crypto.Cipher$a.a(Unknown Source)
      ... 2 more

      OpenJDK 7
      /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java PaddingIssue
      Fine!

      Full stacktrace:
      Mar 22, 2016 3:54:13 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP3-connect
      Mar 22, 2016 3:54:14 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Failed to create handshake ciphers
      java.lang.AssertionError: Failed to create handshake ciphers
      at org.jenkinsci.remoting.engine.HandshakeCiphers.create(HandshakeCiphers.java:116)
      at org.jenkinsci.remoting.engine.JnlpProtocol3.performHandshake(JnlpProtocol3.java:138)
      at org.jenkinsci.remoting.engine.JnlpProtocol.establishChannel(JnlpProtocol.java:77)
      at hudson.remoting.Engine.run(Engine.java:308)
      Caused by: java.security.NoSuchAlgorithmException: Cannot find any provider supporting AES/CTR/PKCS5Padding
      at javax.crypto.Cipher.getInstance(Unknown Source)
      at org.jenkinsci.remoting.engine.HandshakeCiphers.create(HandshakeCiphers.java:109)
      ... 3 more
      Caused by: javax.crypto.NoSuchPaddingException: CTR mode must be used with ISO10
      126Padding or NoPadding
      at com.ibm.crypto.provider.AESCipher.engineSetPadding(Unknown Source)
      at javax.crypto.Cipher$a_.a(Unknown Source)
      ... 5 more

          [JENKINS-34121] IBM Java doesn't support AES/CTR/PKCS5Padding, required for JNLP3

          Daniel Beck added a comment -

          Daniel Beck added a comment - oleg_nenashev FYI

          nyoung02 added a comment -

          Hitting the same problem. As I work on platforms where the IBM JRE is the only option, this will stop me from upgrading my Jenkins instance.

          If it helps, I reverted the slave.jar file to an old one from another instance of Jenkins running on the same machine and this connects successfully using Jenkins 1.656, giving me a workaround of sorts.

          The version number of the slave that will connect is at 2.33.

          nyoung02 added a comment - Hitting the same problem. As I work on platforms where the IBM JRE is the only option, this will stop me from upgrading my Jenkins instance. If it helps, I reverted the slave.jar file to an old one from another instance of Jenkins running on the same machine and this connects successfully using Jenkins 1.656, giving me a workaround of sorts. The version number of the slave that will connect is at 2.33.

          Daniel Beck added a comment -

          What happens when you remove the JNLP3 protocol from the list of supported protocols on Jenkins master?

          CC stephenconnolly who's planning to implement JNLP4, maybe this can be considered.

          Daniel Beck added a comment - What happens when you remove the JNLP3 protocol from the list of supported protocols on Jenkins master? CC stephenconnolly who's planning to implement JNLP4, maybe this can be considered.

          So the JNLP4 protocol is implemented already. I am just awaiting approval to release the code. As it uses TLS rather than a home grown cipher system this should not be an issue for JNLP4. (The other change with JNLP4 is that it uses NIO whereas JNLP3 dropped the NIO support)

          If you remove JNLP3 from the list of supported protocols then it should fall back to JNLP2 if I recall the code correctly

          Stephen Connolly added a comment - So the JNLP4 protocol is implemented already. I am just awaiting approval to release the code. As it uses TLS rather than a home grown cipher system this should not be an issue for JNLP4. (The other change with JNLP4 is that it uses NIO whereas JNLP3 dropped the NIO support) If you remove JNLP3 from the list of supported protocols then it should fall back to JNLP2 if I recall the code correctly

          nyoung02 added a comment -

          I have a system I can try it on so I'm happy to give it a try and see as a workaround - but how do I remove JNLP3 from the list of supported protocols?!

          nyoung02 added a comment - I have a system I can try it on so I'm happy to give it a try and see as a workaround - but how do I remove JNLP3 from the list of supported protocols?!

          Daniel Beck added a comment -
          java -Djenkins.slaves.JnlpSlaveAgentProtocol3.enabled=false -jar jenkins.war

          should do it.

          Daniel Beck added a comment - java -Djenkins.slaves.JnlpSlaveAgentProtocol3.enabled=false -jar jenkins.war should do it.

          nyoung02 added a comment -

          I started Jenkins using that property, all started ok. This appeared in the console output for the master when I tried to connect a slave:

          WARNING: Connection #1 failed
          java.io.EOFException
                  at java.io.DataInputStream.readUnsignedShort(Unknown Source)
                  at java.io.DataInputStream.readUTF(Unknown Source)
                  at java.io.DataInputStream.readUTF(Unknown Source)
                  at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListe
          ner.java:175)
          

          And the slave still fails to connect:

          May 10, 2016 9:41:11 AM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Trying protocol: JNLP3-connect
          May 10, 2016 9:41:12 AM hudson.remoting.jnlp.Main$CuiListener error
          SEVERE: Failed to create handshake ciphers
          java.lang.AssertionError: Failed to create handshake ciphers
                  at org.jenkinsci.remoting.engine.HandshakeCiphers.create(HandshakeCipher
          s.java:116)
                  at org.jenkinsci.remoting.engine.JnlpProtocol3.performHandshake(JnlpProt
          ocol3.java:138)
                  at org.jenkinsci.remoting.engine.JnlpProtocol.establishChannel(JnlpProto
          col.java:77)
                  at hudson.remoting.Engine.run(Engine.java:308)
          Caused by: java.security.NoSuchAlgorithmException: Cannot find any provider supp
          orting AES/CTR/PKCS5Padding
                  at javax.crypto.Cipher.getInstance(Unknown Source)
                  at org.jenkinsci.remoting.engine.HandshakeCiphers.create(HandshakeCipher
          s.java:109)
                  ... 3 more
          Caused by: javax.crypto.NoSuchPaddingException: CTR mode must be used with ISO10
          126Padding or NoPadding
                  at com.ibm.crypto.provider.AESCipher.engineSetPadding(Unknown Source)
                  at javax.crypto.Cipher$a_.a(Unknown Source)
                  ... 5 more
          

          So it looks like the slave is still trying to connect with JNLP3. I tried passing the same property to the slave.jar in case the intention was to use it there, with no effect.

          nyoung02 added a comment - I started Jenkins using that property, all started ok. This appeared in the console output for the master when I tried to connect a slave: WARNING: Connection #1 failed java.io.EOFException at java.io.DataInputStream.readUnsignedShort(Unknown Source) at java.io.DataInputStream.readUTF(Unknown Source) at java.io.DataInputStream.readUTF(Unknown Source) at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListe ner.java:175) And the slave still fails to connect: May 10, 2016 9:41:11 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP3-connect May 10, 2016 9:41:12 AM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Failed to create handshake ciphers java.lang.AssertionError: Failed to create handshake ciphers at org.jenkinsci.remoting.engine.HandshakeCiphers.create(HandshakeCipher s.java:116) at org.jenkinsci.remoting.engine.JnlpProtocol3.performHandshake(JnlpProt ocol3.java:138) at org.jenkinsci.remoting.engine.JnlpProtocol.establishChannel(JnlpProto col.java:77) at hudson.remoting.Engine.run(Engine.java:308) Caused by: java.security.NoSuchAlgorithmException: Cannot find any provider supp orting AES/CTR/PKCS5Padding at javax.crypto.Cipher.getInstance(Unknown Source) at org.jenkinsci.remoting.engine.HandshakeCiphers.create(HandshakeCipher s.java:109) ... 3 more Caused by: javax.crypto.NoSuchPaddingException: CTR mode must be used with ISO10 126Padding or NoPadding at com.ibm.crypto.provider.AESCipher.engineSetPadding(Unknown Source) at javax.crypto.Cipher$a_.a(Unknown Source) ... 5 more So it looks like the slave is still trying to connect with JNLP3. I tried passing the same property to the slave.jar in case the intention was to use it there, with no effect.

          Yeah looks like there is no way to disable the client protocol: https://github.com/jenkinsci/remoting/blob/master/src/main/java/org/jenkinsci/remoting/engine/JnlpProtocolFactory.java#L54

          So just use an older slave.jar as that will not try JNLP3 at all.

          I will be fixing that code up to allow control of the individual protocols when I merge by JNLP4 protocol

          Stephen Connolly added a comment - Yeah looks like there is no way to disable the client protocol: https://github.com/jenkinsci/remoting/blob/master/src/main/java/org/jenkinsci/remoting/engine/JnlpProtocolFactory.java#L54 So just use an older slave.jar as that will not try JNLP3 at all. I will be fixing that code up to allow control of the individual protocols when I merge by JNLP4 protocol

          (by fixing up I mean refactoring into something else)

          Stephen Connolly added a comment - (by fixing up I mean refactoring into something else)

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/Engine.java
          src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol.java
          http://jenkins-ci.org/commit/remoting/04037ec8ee633109c557d8f5f35922f2e2f495fc
          Log:
          [Related to JENKINS-34121] Allow disabling the protocols individually

          • Also ensure that a fatal unexpected error in one protocol does not prevent a fall-back

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/Engine.java src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol.java http://jenkins-ci.org/commit/remoting/04037ec8ee633109c557d8f5f35922f2e2f495fc Log: [Related to JENKINS-34121] Allow disabling the protocols individually Also ensure that a fatal unexpected error in one protocol does not prevent a fall-back

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/Engine.java
          src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol.java
          http://jenkins-ci.org/commit/remoting/43793e5b46f223c57f0a7b137cd983903fd63446
          Log:
          [Related to JENKINS-34121] Allow disabling the protocols individually (#83)

          • Also ensure that a fatal unexpected error in one protocol does not prevent a fall-back

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/Engine.java src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol.java http://jenkins-ci.org/commit/remoting/43793e5b46f223c57f0a7b137cd983903fd63446 Log: [Related to JENKINS-34121] Allow disabling the protocols individually (#83) Also ensure that a fatal unexpected error in one protocol does not prevent a fall-back

          Oleg Nenashev added a comment -

          Workaround - Disable JNLP3 protocol.
          This workaround has been implemented by Stephen and released as remoting 2.59.
          Should become available in the next weekly, may be a subject for backporting into the next LTS release (1.651.3)

          Oleg Nenashev added a comment - Workaround - Disable JNLP3 protocol. This workaround has been implemented by Stephen and released as remoting 2.59. Should become available in the next weekly, may be a subject for backporting into the next LTS release (1.651.3)

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4
          Log:
          [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

          • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

          Changes:

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25
          Log:
          [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

          • [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58.

          Changes:

          (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25 Log: [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344) [JENKINS-19445, JENKINS-34213, JENKINS-34808] Bump remoting to 2.58. Changes: JENKINS-34213 ( https://issues.jenkins-ci.org/browse/JENKINS-34213 ) - Ensure that the unexporter cleans up whatever it can each sweep ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-19445 ( https://issues.jenkins-ci.org/browse/JENKINS-19445 ) Force class load on UserRequest in order to prevent deadlock on windows nodes when using JNA and Subversion ( https://github.com/jenkinsci/remoting/pull/81 ) JENKINS-34808 ( https://issues.jenkins-ci.org/browse/JENKINS-34808 ) - Allow user to adjust socket timeout ( https://github.com/jenkinsci/remoting/pull/68 ) JENKINS-34121 - Upgrade remoting to 2.59 (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

          The workaround will be available in 1.651.3 LTS.

          Oliver Gondža added a comment - The workaround will be available in 1.651.3 LTS.

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          docs/protocols.md
          src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java
          http://jenkins-ci.org/commit/remoting/fe2587b7f9d78334e0ab05ab0b95f39b4b600a25
          Log:
          Docs - Noting JENKINS-37302, JENKINS-33886, and JENKINS-34121 in Errata

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: docs/protocols.md src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java http://jenkins-ci.org/commit/remoting/fe2587b7f9d78334e0ab05ab0b95f39b4b600a25 Log: Docs - Noting JENKINS-37302 , JENKINS-33886 , and JENKINS-34121 in Errata

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          docs/protocols.md
          src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java
          http://jenkins-ci.org/commit/remoting/86e13055079fd679a46b06fc7ce54ea1eb33ac1f
          Log:
          Merge pull request #155 from oleg-nenashev/doc/jnlp3_errata

          [Docs] - Noting JENKINS-37302, JENKINS-33886, and JENKINS-34121 in JNLP3 Errata

          Compare: https://github.com/jenkinsci/remoting/compare/b8f10d809829...86e13055079f

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: docs/protocols.md src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java http://jenkins-ci.org/commit/remoting/86e13055079fd679a46b06fc7ce54ea1eb33ac1f Log: Merge pull request #155 from oleg-nenashev/doc/jnlp3_errata [Docs] - Noting JENKINS-37302 , JENKINS-33886 , and JENKINS-34121 in JNLP3 Errata Compare: https://github.com/jenkinsci/remoting/compare/b8f10d809829...86e13055079f

          Oleg Nenashev added a comment -

          Closing as "Won't fix", upgrade to JNLP4 is the recommended solution.

          If somebody wants to fix it, please feel free to reopen it and to create a pull request

          Oleg Nenashev added a comment - Closing as "Won't fix", upgrade to JNLP4 is the recommended solution. If somebody wants to fix it, please feel free to reopen it and to create a pull request

          Can someone please provide instructions to upgrade to JNLP4 ?

          We prefer to use IBM JRE.

          Mallikarjuna Manoj Kandala added a comment - Can someone please provide instructions to upgrade to JNLP4 ? We prefer to use IBM JRE.

          Oleg Nenashev added a comment -

          srimanoj_kandala

          1) Upgrade to Jenkins 2.32.1 or above
          2) Go to the Global Security Configuration in Manage Jenkins
          3) Ensure that JNLP4 is allowed in protocol security settings

          Oleg Nenashev added a comment - srimanoj_kandala 1) Upgrade to Jenkins 2.32.1 or above 2) Go to the Global Security Configuration in Manage Jenkins 3) Ensure that JNLP4 is allowed in protocol security settings

            Unassigned Unassigned
            gloparm Gabriel Lopez
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: