Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12302

Remote call on CLI channel from [ip] failed

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • cli, groovy-plugin
    • None
    • Jenkins 1.446, Linux CentOS

    Description

      I've had a KillStuckPolling.groovy job running for about 6 months without problems, but on January 4th it stopped working with the below stack trace. This matches with the release of Jenkins 1.446, which is currently installed. I have tried to restart the server without any effect.

      Killing all stuck SCM polls using ~/bin/KillStuckPolling.groovy
      log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
      log4j:WARN Please initialize the log4j system properly.
      java.io.IOException: Remote call on CLI channel from /[ip] failed
      	at hudson.remoting.Channel.call(Channel.java:690)
      	at hudson.cli.GroovyCommand.loadScript(GroovyCommand.java:106)
      	at hudson.cli.GroovyCommand.run(GroovyCommand.java:93)
      	at hudson.cli.CLICommand.main(CLICommand.java:205)
      	at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:66)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:274)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:287)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.ExceptionInInitializerError
      	at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
      	at java.io.ObjectStreamClass.computeDefaultSUID(Unknown Source)
      	at java.io.ObjectStreamClass.access$100(Unknown Source)
      	at java.io.ObjectStreamClass$1.run(Unknown Source)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.io.ObjectStreamClass.getSerialVersionUID(Unknown Source)
      	at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
      	at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
      	at java.io.ObjectInputStream.readClassDesc(Unknown Source)
      	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
      	at java.io.ObjectInputStream.readObject0(Unknown Source)
      	at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
      	at java.io.ObjectInputStream.readSerialData(Unknown Source)
      	at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
      	at java.io.ObjectInputStream.readObject0(Unknown Source)
      	at java.io.ObjectInputStream.readObject(Unknown Source)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:98)
      	... 8 more
      Caused by: java.lang.NullPointerException
      	at hudson.cli.CLICommand.<clinit>(CLICommand.java:448)
      	... 26 more
      Build step 'Execute shell' marked build as failure
      

      The groovy script is as follows:

      KillStuckPolling.groovy
       Thread.getAllStackTraces().keySet().each() { 
         item ->
         println "Checking item" + item.getName();
         if (item.getName().contains("SCM polling") && item.getName().contains("waiting for hudson.remoting")) { 
           println "   Interrupting thread " + item.getId(); 
           item.interrupt() 
         }
       }
      

      And the bash script used to start it is as follows

      KillStuckPolling.sh
      #!/bin/bash -e
      echo "## Kills stuck polling jobs on Jenkins"
      
      if [ -z "$JENKINS_URL" ]; then
        echo "ERROR: This script must be run on Jenkins. It cannot be run manually"
        exit 1
      fi
      
      echo "Getting latest Jenkins CLI from ${JENKINS_URL}"
      if [ -w jenkins-cli.jar ]; then
        rm jenkins-cli.jar
      fi
      wget ${JENKINS_URL}jnlpJars/jenkins-cli.jar
      echo "Killing all stuck SCM polls using ~/bin/KillStuckPolling.groovy"
      java -jar jenkins-cli.jar -s ${JENKINS_URL}/ groovy ~/bin/KillStuckPolling.groovy
      

      Attachments

        Issue Links

          Activity

            mabahj Markus created issue -
            mabahj Markus added a comment -

            The problem is still there in Jenkins version 1.447.

            mabahj Markus added a comment - The problem is still there in Jenkins version 1.447.
            mabahj Markus made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-5413 [ JENKINS-5413 ]
            greghmerrill Greg Merrill added a comment -

            I am seeing the same with any groovy script i try to run through the "groovy" option on jenkins-cli.jar.

            java.io.IOException: Remote call on CLI channel from /10.4.9.145 failed
            at hudson.remoting.Channel.call(Channel.java:690)
            at hudson.cli.GroovyCommand.loadScript(GroovyCommand.java:106)
            at hudson.cli.GroovyCommand.run(GroovyCommand.java:93)
            at hudson.cli.CLICommand.main(CLICommand.java:205)
            at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:66)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:274)
            at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255)
            at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:287)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:662)
            Caused by: java.lang.ExceptionInInitializerError
            at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
            at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1696)
            at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52)
            at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:205)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:202)
            at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:558)
            at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
            at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
            at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
            at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
            at hudson.remoting.UserRequest.perform(UserRequest.java:98)
            ... 8 more
            Caused by: java.lang.NullPointerException
            at hudson.cli.CLICommand.<clinit>(CLICommand.java:448)
            ... 26 more

            greghmerrill Greg Merrill added a comment - I am seeing the same with any groovy script i try to run through the "groovy" option on jenkins-cli.jar. java.io.IOException: Remote call on CLI channel from /10.4.9.145 failed at hudson.remoting.Channel.call(Channel.java:690) at hudson.cli.GroovyCommand.loadScript(GroovyCommand.java:106) at hudson.cli.GroovyCommand.run(GroovyCommand.java:93) at hudson.cli.CLICommand.main(CLICommand.java:205) at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:274) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:287) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.ExceptionInInitializerError at java.io.ObjectStreamClass.hasStaticInitializer(Native Method) at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1696) at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52) at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:205) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:202) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:558) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) ... 8 more Caused by: java.lang.NullPointerException at hudson.cli.CLICommand.<clinit>(CLICommand.java:448) ... 26 more

            This bug is still there in Jenkins ver. 1.450

            franck Franck Gilliers added a comment - This bug is still there in Jenkins ver. 1.450

            Just ran into the same with 1.451 trying to use CLI to execute a groovy script by url.

            thekaptain Kelly Robinson added a comment - Just ran into the same with 1.451 trying to use CLI to execute a groovy script by url.
            ingorichter Ingo Richter added a comment -

            As a workaround you can pipe the content of the file to jenkins-cli.jar.
            cat myscript.groovy | java -jar jenkins-cli.jar http://localhost:8080 groovy =
            This helped me in the short term.
            I debugged through the code and found the reason for the NPE. But I don't have a resolution yet.

            ingorichter Ingo Richter added a comment - As a workaround you can pipe the content of the file to jenkins-cli.jar. cat myscript.groovy | java -jar jenkins-cli.jar http://localhost:8080 groovy = This helped me in the short term. I debugged through the code and found the reason for the NPE. But I don't have a resolution yet.

            Code changed in jenkins
            User: Vojtech Juranek
            Path:
            core/src/main/java/hudson/cli/GroovyCommand.java
            core/src/main/java/hudson/cli/util/ScriptLoader.java
            http://jenkins-ci.org/commit/jenkins/7bdc1790166198de93b2b7c58286d9554be967b4
            Log:
            [FIXED JENKINS-12302] Refactor anonymout class for loading groovy script into separate class as static block (due to inheritance of outer class from CLICommand) initialization leads to NPE (namely in Jenkins.getInstance Jenkins.getInstance().getPluginManager())

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vojtech Juranek Path: core/src/main/java/hudson/cli/GroovyCommand.java core/src/main/java/hudson/cli/util/ScriptLoader.java http://jenkins-ci.org/commit/jenkins/7bdc1790166198de93b2b7c58286d9554be967b4 Log: [FIXED JENKINS-12302] Refactor anonymout class for loading groovy script into separate class as static block (due to inheritance of outer class from CLICommand) initialization leads to NPE (namely in Jenkins.getInstance Jenkins.getInstance().getPluginManager())
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #1540
            [FIXED JENKINS-12302] Refactor anonymout class for loading groovy script into separate class as static block (due to inheritance of outer class from CLICommand) initialization leads to NPE (namely in Jenkins.getInstance Jenkins.getInstance().getPluginManager()) (Revision 7bdc1790166198de93b2b7c58286d9554be967b4)

            Result = SUCCESS
            Vojtech Juranek : 7bdc1790166198de93b2b7c58286d9554be967b4
            Files :

            • core/src/main/java/hudson/cli/GroovyCommand.java
            • core/src/main/java/hudson/cli/util/ScriptLoader.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1540 [FIXED JENKINS-12302] Refactor anonymout class for loading groovy script into separate class as static block (due to inheritance of outer class from CLICommand) initialization leads to NPE (namely in Jenkins.getInstance Jenkins.getInstance().getPluginManager()) (Revision 7bdc1790166198de93b2b7c58286d9554be967b4) Result = SUCCESS Vojtech Juranek : 7bdc1790166198de93b2b7c58286d9554be967b4 Files : core/src/main/java/hudson/cli/GroovyCommand.java core/src/main/java/hudson/cli/util/ScriptLoader.java

            Code changed in jenkins
            User: Vojtech Juranek
            Path:
            core/src/main/java/hudson/cli/GroovyCommand.java
            core/src/main/java/hudson/cli/util/ScriptLoader.java
            http://jenkins-ci.org/commit/jenkins/99385dcd1c8a2dc84d51133f1f4b06d4bc1cf653
            Log:
            [FIXED JENKINS-12302] Refactor anonymout class for loading groovy script into separate class as static block (due to inheritance of outer class from CLICommand) initialization leads to NPE (namely in Jenkins.getInstance Jenkins.getInstance().getPluginManager())
            (cherry picked from commit 7bdc1790166198de93b2b7c58286d9554be967b4)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vojtech Juranek Path: core/src/main/java/hudson/cli/GroovyCommand.java core/src/main/java/hudson/cli/util/ScriptLoader.java http://jenkins-ci.org/commit/jenkins/99385dcd1c8a2dc84d51133f1f4b06d4bc1cf653 Log: [FIXED JENKINS-12302] Refactor anonymout class for loading groovy script into separate class as static block (due to inheritance of outer class from CLICommand) initialization leads to NPE (namely in Jenkins.getInstance Jenkins.getInstance().getPluginManager()) (cherry picked from commit 7bdc1790166198de93b2b7c58286d9554be967b4)

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/cli/CLICommand.java
            http://jenkins-ci.org/commit/jenkins/7110ddb7d2a2f141a6887ffe352526db91046867
            Log:
            JENKINS-12302

            Make sure that we don't require Jenkins.getInstance() on CLICommand, since it can run on slaves.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/cli/CLICommand.java http://jenkins-ci.org/commit/jenkins/7110ddb7d2a2f141a6887ffe352526db91046867 Log: JENKINS-12302 Make sure that we don't require Jenkins.getInstance() on CLICommand, since it can run on slaves.
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #1605
            JENKINS-12302 (Revision 7110ddb7d2a2f141a6887ffe352526db91046867)

            Result = SUCCESS
            Kohsuke Kawaguchi : 7110ddb7d2a2f141a6887ffe352526db91046867
            Files :

            • core/src/main/java/hudson/cli/CLICommand.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #1605 JENKINS-12302 (Revision 7110ddb7d2a2f141a6887ffe352526db91046867) Result = SUCCESS Kohsuke Kawaguchi : 7110ddb7d2a2f141a6887ffe352526db91046867 Files : core/src/main/java/hudson/cli/CLICommand.java
            dogfood dogfood added a comment -

            Integrated in jenkins_ui-changes_branch #21
            JENKINS-12302 (Revision 7110ddb7d2a2f141a6887ffe352526db91046867)

            Result = SUCCESS
            Kohsuke Kawaguchi : 7110ddb7d2a2f141a6887ffe352526db91046867
            Files :

            • core/src/main/java/hudson/cli/CLICommand.java
            dogfood dogfood added a comment - Integrated in jenkins_ui-changes_branch #21 JENKINS-12302 (Revision 7110ddb7d2a2f141a6887ffe352526db91046867) Result = SUCCESS Kohsuke Kawaguchi : 7110ddb7d2a2f141a6887ffe352526db91046867 Files : core/src/main/java/hudson/cli/CLICommand.java
            mabahj Markus made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 142625 ] JNJira + In-Review [ 205584 ]

            People

              vjuranek vjuranek
              mabahj Markus
              Votes:
              6 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: