Major CLI commands hangs with no console output but following Exception in Jenkins.err.log:
>> java -jar jenkins-cli.jar -s http://ciserver:8080/ help
_
Aug 22, 2014 6:47:11 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #3 from /192.168.1.200:58954
Exception in thread "TCP slave agent connection handler #3 with /192.168.1.200:58954" java.lang.Error: java.lang.reflect.InvocationTargetException
at hudson.cli.CliProtocol2$Handler2.run(CliProtocol2.java:76)
at hudson.cli.CliProtocol2.handle(CliProtocol2.java:32)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:157)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.cli.CliProtocol2$Handler2.run(CliProtocol2.java:63)
... 2 more
Caused by: java.lang.NullPointerException
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.get(InstanceIdentity.java:126)
... 7 more
[JENKINS-24385] Jenkins CLI hangs with InvocationTargetException
Please check the Jenkins logs from the last restart (or restart now) and look for anything warnings/errors related to 'instance identity' or 'PageDecoratorImpl'.
You could also try new org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl() and see what exception gets thrown your way (although I'd still like to know whether anything is logged upon Jenkins start).
Marcel Richter
added a comment - Yes, there is an error after restart:
Aug 31, 2014 7:39:52 PM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
WARNING: Failed to instantiate Key[type=org.jenkinsci.main.modules.sshd.SSHD, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Guice provision errors:
1) Error injecting constructor, java.io.IOException: javax.crypto.BadPaddingException: Given final block not properly padded
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(Unknown Source)
while locating org.jenkinsci.main.modules.instance_identity.InstanceIdentity
for field at org.jenkinsci.main.modules.sshd.SSHD.identity(Unknown Source)
1 error
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:117)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:391)
at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:382)
at hudson.ExtensionFinder._find(ExtensionFinder.java:151)
at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:341)
at hudson.ExtensionList.load(ExtensionList.java:295)
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248)
at hudson.ExtensionList.iterator(ExtensionList.java:138)
at jenkins.model.Jenkins.<init>(Jenkins.java:860)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.model.Hudson.<init>(Hudson.java:78)
at hudson.WebAppMain$3.run(WebAppMain.java:222)
Caused by: java.io.IOException: javax.crypto.BadPaddingException: Given final block not properly padded
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:62)
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:38)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:86)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:117)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
... 28 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:811)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:676)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:313)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:60)
... 45 more
Aug 31, 2014 7:39:52 PM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
WARNING: Failed to instantiate Key[type=org.jenkinsci.main.modules.sshd.ItemListenerImpl, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Guice provision errors:
1) null returned by binding at hudson.ExtensionFinder$GuiceFinder$SezpozModule.configure(ExtensionFinder.java:513)
but org.jenkinsci.main.modules.sshd.ItemListenerImpl.sshd is not @Nullable
while locating org.jenkinsci.main.modules.sshd.SSHD
for field at org.jenkinsci.main.modules.sshd.ItemListenerImpl.sshd(Unknown Source)
1 error
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:391)
at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:382)
at hudson.ExtensionFinder._find(ExtensionFinder.java:151)
at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:341)
at hudson.ExtensionList.load(ExtensionList.java:295)
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248)
at hudson.ExtensionList.iterator(ExtensionList.java:138)
at jenkins.model.Jenkins.<init>(Jenkins.java:860)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.model.Hudson.<init>(Hudson.java:78)
at hudson.WebAppMain$3.run(WebAppMain.java:222)
Aug 31, 2014 7:39:52 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
Marcel Richter
added a comment - Yes, there is an error after restart:
Aug 31, 2014 7:39:52 PM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
WARNING: Failed to instantiate Key[type=org.jenkinsci.main.modules.sshd.SSHD, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Guice provision errors:
1) Error injecting constructor, java.io.IOException: javax.crypto.BadPaddingException: Given final block not properly padded
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(Unknown Source)
while locating org.jenkinsci.main.modules.instance_identity.InstanceIdentity
for field at org.jenkinsci.main.modules.sshd.SSHD.identity(Unknown Source)
1 error
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:117)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:391)
at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:382)
at hudson.ExtensionFinder._find(ExtensionFinder.java:151)
at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:341)
at hudson.ExtensionList.load(ExtensionList.java:295)
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248)
at hudson.ExtensionList.iterator(ExtensionList.java:138)
at jenkins.model.Jenkins.<init>(Jenkins.java:860)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.model.Hudson.<init>(Hudson.java:78)
at hudson.WebAppMain$3.run(WebAppMain.java:222)
Caused by: java.io.IOException: javax.crypto.BadPaddingException: Given final block not properly padded
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:62)
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:38)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:86)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:108)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:117)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
... 28 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:811)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:676)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:313)
at javax.crypto.Cipher.doFinal(Cipher.java:2087)
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.<init>(InstanceIdentity.java:60)
... 45 more
Aug 31, 2014 7:39:52 PM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
WARNING: Failed to instantiate Key[type=org.jenkinsci.main.modules.sshd.ItemListenerImpl, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Guice provision errors:
1) null returned by binding at hudson.ExtensionFinder$GuiceFinder$SezpozModule.configure(ExtensionFinder.java:513)
but org.jenkinsci.main.modules.sshd.ItemListenerImpl.sshd is not @Nullable
while locating org.jenkinsci.main.modules.sshd.SSHD
for field at org.jenkinsci.main.modules.sshd.ItemListenerImpl.sshd(Unknown Source)
1 error
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:429)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:391)
at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:382)
at hudson.ExtensionFinder._find(ExtensionFinder.java:151)
at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:341)
at hudson.ExtensionList.load(ExtensionList.java:295)
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:248)
at hudson.ExtensionList.iterator(ExtensionList.java:138)
at jenkins.model.Jenkins.<init>(Jenkins.java:860)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.model.Hudson.<init>(Hudson.java:78)
at hudson.WebAppMain$3.run(WebAppMain.java:222)
Aug 31, 2014 7:39:52 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
Back up the files identity.key and/or identity.key.enc in the Jenkins home directory on the master node, and then delete them. Restart.
From the documentation, it doesn't look like it's really used that much:
https://wiki.jenkins-ci.org/display/JENKINS/Instance+Identity
Still, make sure that e.g. SCM credentials (or any other passwords stored in Jenkins) still work with the newly generated instance identity.
If you could search older jenkins.logs for errors related to instance_identity to help identify the cause of the broken key file that would be great.
Related: https://groups.google.com/d/msg/jenkinsci-users/q9C1m8O0ruE/lrlSllwH-T8J
Resolving as Cannot Reproduce after a week without response by the reporter.
The reported issue should be straightforward to resolve with the provided instruction; and no information has been provided to help the investigation into the original file corruption cause.
This issue is happening now again.
When running jenkins_cli command it hangs.
Every couple of days the identity file is getting corrupted.
Moving it and restarting Jenkins resolving the problem as a new identity file is created.
This happen when Jenkins is running and not on shutdown/restart of the platform.
Jenkins Version 1.565.3
Java build 1.7.0_75-b13
Saar Netzer
added a comment - - edited This issue is happening now again.
When running jenkins_cli command it hangs.
Every couple of days the identity file is getting corrupted.
Moving it and restarting Jenkins resolving the problem as a new identity file is created.
This happen when Jenkins is running and not on shutdown/restart of the platform.
Jenkins Version 1.565.3
Java build 1.7.0_75-b13 1.565.x has been out of community support for over a year. For this issue to be relevant, needs to happen with a recent release (1.609.x preferably, 1.596.x if absolutely necessary).
If the identify file gets corrupted while Jenkins is running, consider making it readonly for the user Jenkins is running as to see what fails to work because of that.
Hi there,
Unfortunately I've just seen this issue with the previous LTS release:
- jenkins version: 2.7.4
- Java: 1.7.0_111
command
$ telnet 127.0.0.1 8080 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. ^Connection closed by foreign host. $ java -jar /usr/share/jenkins/jenkins-cli.jar -s http://127.0.0.1:8080 help
jenkins.log
Oct 13, 2016 2:25:08 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #4 from /127.0.0.1:46816
Oct 13, 2016 2:25:08 PM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (TCP agent connection handler #4 with /127.0.0.1:46816/1601) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.Error: java.lang.reflect.InvocationTargetException
at hudson.cli.CliProtocol2$Handler2.run(CliProtocol2.java:74)
at hudson.cli.CliProtocol2.handle(CliProtocol2.java:33)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:182)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hudson.cli.CliProtocol2$Handler2.run(CliProtocol2.java:65)
... 2 more
Caused by: java.lang.NullPointerException
at org.jenkinsci.main.modules.instance_identity.InstanceIdentity.get(InstanceIdentity.java:152)
... 7 more
I'll double check whether with the latest release 2.19.X works fine
It works after upgrading to the latest version: 2.19.1 and the below plugins:
- credentials = '2.1.0'
- ssh_credentials = '1.12'
- build_pipeline_plugin = '1.4.4'
- findbugs = '4.62'
- jquery = '1.11.2-0'
- token_macro = '1.12.1'
Cheers
Brenna Flood
added a comment - I experienced this issue. My setup:
Jenkins 2.7.4 - 2.11.11
Docker - CentOS 6.8
JDK 1.8.111
No security setup (this is a test instance - chef + kitchen)
The cause of the behavior in my case was that the identity file in the Jenkins root could not be loaded. To correct the issue, I simply removed the file and restarted the Jenkins service.
Brenna Flood
added a comment - I experienced this issue. My setup:
Jenkins 2.7.4 - 2.11.11
Docker - CentOS 6.8
JDK 1.8.111
No security setup (this is a test instance - chef + kitchen)
The cause of the behavior in my case was that the identity file in the Jenkins root could not be loaded. To correct the issue, I simply removed the file and restarted the Jenkins service. 
I used version 1.532 before.
Here the output of each:
Jenkins.getInstance().getExtensionList(hudson.model.PageDecorator.class).get(org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl.class).identity
java.lang.NullPointerException: Cannot get property 'identity' on null object at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:56) at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:156) at org.codehaus.groovy.runtime.callsite.NullCallSite.getProperty(NullCallSite.java:44) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:227) at Script1.run(Script1.groovy:1) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:580) . . .Jenkins.getInstance().getExtensionList(hudson.model.PageDecorator.class).get(org.jenkinsci.main.modules.instance_identity.PageDecoratorImpl.class)
Jenkins.getInstance().getExtensionList(hudson.model.PageDecorator.class)
Jenkins.getInstance()
Seems so as if that PageDecoratorImpl.class object is missing, right?
THX