Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26866

jenkins java.io.ioexception remote file operation failed

      Jenkins jobs assigned to java web-start slaves.

      The jobs in question were all working fine.
      Suddenly I get:

      jenkins java.io.ioexception remote file operation failed

      The call stack is essentially useless, and points to an exception handler in some Java class without indicating what exactly was attempted.

      Unfortunately this occurred just as I was preparing for a holiday. I do not have exact call stack now (I'm on a train).

      I fixed the problem by making the script explicitly perform a svn checkout/update rather than the normal method where one provides the URL and credentials in the fields for the "configure" page.

      That should be a good indication of where the issue lies.

      I really would like some kind of feedback on this issue, even though I will be away for a week. I would like to have the "normal" functionality and am willing to help investigate.

      Environment is:

      Fedora 21 x86_64 - all updates current as of 09.Feb.2015
      Master and slaves are identical.

      I appreciate any help.

          [JENKINS-26866] jenkins java.io.ioexception remote file operation failed

          Daniel Beck added a comment -

          Please note: I am on holiday and have no access to this server other than by an e-mail chain with a colleague. Please forgive me if this takes some back and forth to get the necessary info.

          Please note that this is an issue tracker, and not a support site. So it doesn't matter if it takes a few weeks for you to provide additional information.

          Daniel Beck added a comment - Please note: I am on holiday and have no access to this server other than by an e-mail chain with a colleague. Please forgive me if this takes some back and forth to get the necessary info. Please note that this is an issue tracker, and not a support site. So it doesn't matter if it takes a few weeks for you to provide additional information.

          Smouch Smouch added a comment -

          Thanks Daniel.

          Sorry, I am doing my best to get this resolved, because I have left our build system in an unknown state - even though the jobs are all running successfully - while I am away on holiday, during a fairly important milestone for our project. Needless to say, for any conscientious person, that makes me feel ill.

          Anyway, apparently I haven't managed to provide the Jenkins version. My initial installation of Jenkins on this server was via the download link for Fedora on the Jenkins site here: http://pkg.jenkins-ci.org/redhat/ Then I have updated via the web interface to the running Master server. I am fairly confident that means I am running 1.598. Please tell me how I check.

          Smouch Smouch added a comment - Thanks Daniel. Sorry, I am doing my best to get this resolved, because I have left our build system in an unknown state - even though the jobs are all running successfully - while I am away on holiday, during a fairly important milestone for our project. Needless to say, for any conscientious person, that makes me feel ill. Anyway, apparently I haven't managed to provide the Jenkins version. My initial installation of Jenkins on this server was via the download link for Fedora on the Jenkins site here: http://pkg.jenkins-ci.org/redhat/ Then I have updated via the web interface to the running Master server. I am fairly confident that means I am running 1.598. Please tell me how I check.

          Daniel Beck added a comment -

          Please tell me how I check.

          Get your colleague to open any web page served by Jenkins. Tell him to scroll down, and look to the bottom right corner of the browser window.

          Daniel Beck added a comment - Please tell me how I check. Get your colleague to open any web page served by Jenkins. Tell him to scroll down, and look to the bottom right corner of the browser window.

          Smouch Smouch added a comment -

          Hello,

          Thanks again. I am back from holiday. Here is the version information:

          Jenkins ver. 1.598

          Smouch Smouch added a comment - Hello, Thanks again. I am back from holiday. Here is the version information: Jenkins ver. 1.598

          Have the same issue on Jenkins version 1.618. Appreciate any help.

          Started by user Jenkins System Builder
          ...
          Commencing build of Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master)
          Checking out Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master)
          java.io.IOException: remote file operation failed: /var/lib/openshift/55f0547b46dcfd56f5000011/app-root/runtime/repo at hudson.remoting.Channel@551b97:netztestbldr: java.io.IOException: Remote call on netztestbldr failed
          	at hudson.FilePath.act(FilePath.java:987)
          	at hudson.FilePath.act(FilePath.java:969)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1178)
          	at hudson.model.AbstractProject.checkout(AbstractProject.java:1282)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
          	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
          	at hudson.model.Run.execute(Run.java:1741)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:98)
          	at hudson.model.Executor.run(Executor.java:381)
          Caused by: java.io.IOException: Remote call on netztestbldr failed
          	at hudson.remoting.Channel.call(Channel.java:761)
          	at hudson.FilePath.act(FilePath.java:980)
          	... 10 more
          Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins
          	at hudson.ExtensionList.lookup(ExtensionList.java:400)
          	at hudson.FilePath.act(FilePath.java:977)
          	at hudson.FilePath.act(FilePath.java:969)
          	at hudson.FilePath.write(FilePath.java:1883)
          	at hudson.plugins.git.GitSCM.computeChangeLog(GitSCM.java:1262)
          	at hudson.plugins.git.GitSCM.access$700(GitSCM.java:79)
          	at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1233)
          	at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1178)
          	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2691)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          	at hudson.remoting.Request$2.run(Request.java:325)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:745)
          	at ......remote call to netztestbldr(Native Method)
          	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1361)
          	at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
          	at hudson.remoting.Channel.call(Channel.java:753)
          	... 11 more
          Archiving artifacts
          Finished: FAILURE
          

          Dmitry Miroshnichenko added a comment - Have the same issue on Jenkins version 1.618. Appreciate any help. Started by user Jenkins System Builder ... Commencing build of Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master) Checking out Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master) java.io.IOException: remote file operation failed: /var/lib/openshift/55f0547b46dcfd56f5000011/app-root/runtime/repo at hudson.remoting.Channel@551b97:netztestbldr: java.io.IOException: Remote call on netztestbldr failed at hudson.FilePath.act(FilePath.java:987) at hudson.FilePath.act(FilePath.java:969) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1178) at hudson.model.AbstractProject.checkout(AbstractProject.java:1282) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:381) Caused by: java.io.IOException: Remote call on netztestbldr failed at hudson.remoting.Channel.call(Channel.java:761) at hudson.FilePath.act(FilePath.java:980) ... 10 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins at hudson.ExtensionList.lookup(ExtensionList.java:400) at hudson.FilePath.act(FilePath.java:977) at hudson.FilePath.act(FilePath.java:969) at hudson.FilePath.write(FilePath.java:1883) at hudson.plugins.git.GitSCM.computeChangeLog(GitSCM.java:1262) at hudson.plugins.git.GitSCM.access$700(GitSCM.java:79) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1233) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1178) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2691) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:325) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to netztestbldr(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1361) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:753) ... 11 more Archiving artifacts Finished: FAILURE

          Ian Tait added a comment -

          I am also having this issue on various machines.

          Ian Tait added a comment - I am also having this issue on various machines.

          dmir1988 I could not reproduce the bug. Could you provide more information? /cc smouch

          Manuel Recena Soto added a comment - dmir1988 I could not reproduce the bug. Could you provide more information? /cc smouch

          Peter Varga added a comment -

          Hi,

          we are facing a similar problem. We are using Jenkins version 1.591, and have two slaves configured. In case the job uses our slave-01, we are getting the NoClassDefFoundError shown below. If I configure the job, to use our slave-02, it works fine. We were not able to find any configuration differences between our slaves.

          [EnvInject] - Loading node environment variables.
          Building remotely on slave-01 (wfm-trunk linux tasktrack) in workspace /srv/jenkins-home/workspace/deployment-int-shutdown-instance-abc
          java.io.IOException: remote file operation failed: /srv/jenkins-home/workspace/deployment-int-shutdown-instance-abc at hudson.remoting.Channel@205b38e:slave-01: java.io.IOException: Remote call on slave-01 failed
          at hudson.FilePath.act(FilePath.java:977)
          at hudson.FilePath.act(FilePath.java:959)
          at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910)
          at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845)
          at hudson.model.AbstractProject.checkout(AbstractProject.java:1259)
          at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
          at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
          at hudson.model.Run.execute(Run.java:1759)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:89)
          at hudson.model.Executor.run(Executor.java:240)
          Caused by: java.io.IOException: Remote call on slave-01 failed
          at hudson.remoting.Channel.call(Channel.java:760)
          at hudson.FilePath.act(FilePath.java:970)
          ... 11 more
          Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins
          at hudson.scm.SubversionSCM.descriptor(SubversionSCM.java:2509)
          at hudson.scm.SubversionSCM.createDefaultSVNOptions(SubversionSCM.java:1059)
          at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1049)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:981)
          at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:961)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2665)
          at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          at hudson.remoting.Request$2.run(Request.java:324)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          at ......remote call to slave-01(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
          at hudson.remoting.Channel.call(Channel.java:752)
          ... 12 more

          Peter Varga added a comment - Hi, we are facing a similar problem. We are using Jenkins version 1.591, and have two slaves configured. In case the job uses our slave-01, we are getting the NoClassDefFoundError shown below. If I configure the job, to use our slave-02, it works fine. We were not able to find any configuration differences between our slaves. [EnvInject] - Loading node environment variables. Building remotely on slave-01 (wfm-trunk linux tasktrack) in workspace /srv/jenkins-home/workspace/deployment-int-shutdown-instance-abc java.io.IOException: remote file operation failed: /srv/jenkins-home/workspace/deployment-int-shutdown-instance-abc at hudson.remoting.Channel@205b38e:slave-01: java.io.IOException: Remote call on slave-01 failed at hudson.FilePath.act(FilePath.java:977) at hudson.FilePath.act(FilePath.java:959) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845) at hudson.model.AbstractProject.checkout(AbstractProject.java:1259) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) at hudson.model.Run.execute(Run.java:1759) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) Caused by: java.io.IOException: Remote call on slave-01 failed at hudson.remoting.Channel.call(Channel.java:760) at hudson.FilePath.act(FilePath.java:970) ... 11 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins at hudson.scm.SubversionSCM.descriptor(SubversionSCM.java:2509) at hudson.scm.SubversionSCM.createDefaultSVNOptions(SubversionSCM.java:1059) at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1049) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:981) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:961) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2665) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:324) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to slave-01(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:752) ... 12 more

          Oleg Nenashev added a comment -

          From what I see Jenkins class is not being propagated to slaves. It may happen if you enable Slave2Master security AFAIK.

          Anyway,SubversionSCM:: createDefaultSVNOptions() is misdesigned. It should not access descriptior() on its own. Once it is being called on slave, the method should retrieve the data from master via another remoting call. If the class is serializable, such double propagatiion will work correctly.

          Some bits of the issue have been introduced in https://github.com/jenkinsci/subversion-plugin/commit/208f10ba16f8317f64060262fa82477db18be299 , but I think the real cause is even older.

          Oleg Nenashev added a comment - From what I see Jenkins class is not being propagated to slaves. It may happen if you enable Slave2Master security AFAIK. Anyway,SubversionSCM:: createDefaultSVNOptions() is misdesigned. It should not access descriptior() on its own. Once it is being called on slave, the method should retrieve the data from master via another remoting call. If the class is serializable, such double propagatiion will work correctly. Some bits of the issue have been introduced in https://github.com/jenkinsci/subversion-plugin/commit/208f10ba16f8317f64060262fa82477db18be299 , but I think the real cause is even older.

          Peter Varga added a comment -

          oleg_nenashev thanks for your answer.

          The option "Enable Slave → Master Access Control" is not selected in our Jenkins configuration. When it had been selected it would affect all slaves, if I understand it correctly. As we have this problem only with one of our slaves, I don't think that this setting is relevant in our case.

          Peter Varga added a comment - oleg_nenashev thanks for your answer. The option "Enable Slave → Master Access Control" is not selected in our Jenkins configuration. When it had been selected it would affect all slaves, if I understand it correctly. As we have this problem only with one of our slaves, I don't think that this setting is relevant in our case.

            Unassigned Unassigned
            smouch Smouch Smouch
            Votes:
            3 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: