-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Jenkins Master (v 2.164.1) running directly (no container) on a Windows 7 (64bit) host. Several Jenkins clients running in Windows 10 (64bit) VMs on different machines. Clients connect via swarm client (v3.17). Both master and clients are started as batch files, and are running in a console window (no Windows service).
Issue seen both with Oracle Java 8u141 and OpenJDK 8u232.Jenkins Master (v 2.164.1) running directly (no container) on a Windows 7 (64bit) host. Several Jenkins clients running in Windows 10 (64bit) VMs on different machines. Clients connect via swarm client (v3.17). Both master and clients are started as batch files, and are running in a console window (no Windows service). Issue seen both with Oracle Java 8u141 and OpenJDK 8u232.
We are sporadically seeing jobs failing with this callstack shown in the log:
javax.crypto.BadPaddingException: Invalid TLS padding data at sun.security.ssl.CipherBox.removePadding(CipherBox.java:802) at sun.security.ssl.CipherBox.decrypt(CipherBox.java:574) at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200) at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:963) Caused: javax.net.ssl.SSLException: Invalid TLS padding data at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:970) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:347) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668) at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from <server name>/<server ip>:54103 failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:950) at hudson.FilePath.act(FilePath.java:1068) at hudson.FilePath.act(FilePath.java:1057) at hudson.FilePath.mkdirs(FilePath.java:1242) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:181) at org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:93) at org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:91) at org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:60) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:106) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810) at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) at buildscript.call(C:\Jenkins_master\workspace\jobs\NightlyBuild_SDK_QNX\builds\365\libs\pipeline-common\vars\buildscript.groovy:62) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55) at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45) at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
There are days when this is seen several times a day, and others when it is not seen at all. It happens on different Windows10 clients, but so far I have not seen it on any of our Linux clients. When restarting the failing job (with the same parameters) it usually runs fine.
[JENKINS-60280] Jobs failing sporadically with javax.crypto.BadPaddingException: Invalid TLS padding data
Stephan Vogt
added a comment - Hi, thanks for the suggestions. Especially the one about activating additional logging might come in handy.
As for the Java versions: Both the master and the client were using the exact same Java version - every time I tried a new one I installed it on all machines at the same time.
In the meantime however I have upgraded our environment to Windows 10, OpenJDK 11.0.6_10, and Jenkins master 2.204.1. So far I have not seen this issue again in several weeks of usage. I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.
Stephan Vogt
added a comment - Hi, thanks for the suggestions. Especially the one about activating additional logging might come in handy.
As for the Java versions: Both the master and the client were using the exact same Java version - every time I tried a new one I installed it on all machines at the same time.
In the meantime however I have upgraded our environment to Windows 10, OpenJDK 11.0.6_10, and Jenkins master 2.204.1. So far I have not seen this issue again in several weeks of usage. I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket. 
Basil Crow
added a comment - I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.
Done.
Basil Crow
added a comment - I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.
Done. 
Hey stephanvogt, I'm not sure what the root cause of this problem is. May I suggest a few things: