Details
-
Type:
Bug
-
Status: Reopened (View Workflow)
-
Priority:
Critical
-
Resolution: Unresolved
-
Component/s: subversion-plugin
-
Labels:
-
Environment:Jenkins ver. 1.598
-
Similar Issues:
Description
Jenkins jobs assigned to java web-start slaves.
The jobs in question were all working fine.
Suddenly I get:
jenkins java.io.ioexception remote file operation failed
The call stack is essentially useless, and points to an exception handler in some Java class without indicating what exactly was attempted.
Unfortunately this occurred just as I was preparing for a holiday. I do not have exact call stack now (I'm on a train).
I fixed the problem by making the script explicitly perform a svn checkout/update rather than the normal method where one provides the URL and credentials in the fields for the "configure" page.
That should be a good indication of where the issue lies.
I really would like some kind of feedback on this issue, even though I will be away for a week. I would like to have the "normal" functionality and am willing to help investigate.
Environment is:
Fedora 21 x86_64 - all updates current as of 09.Feb.2015
Master and slaves are identical.
I appreciate any help.
Attachments
Activity
Please note: I am on holiday and have no access to this server other than by an e-mail chain with a colleague. Please forgive me if this takes some back and forth to get the necessary info.
Jenkins master configuration
#################################################################
awt.toolkit sun.awt.X11.XToolkit
com.sun.akuma.Daemon daemonized
executable-war /usr/lib/jenkins/jenkins.war
file.encoding UTF-8
file.encoding.pkg sun.io
file.separator /
hudson.diyChunking true
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.headless true
java.awt.printerjob sun.print.PSPrinterJob
java.class.path /usr/lib/jenkins/jenkins.war
java.class.version 51.0
java.endorsed.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/endorsed
java.ext.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/ext:/usr/java/packages/lib/ext
java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre
java.io.tmpdir /tmp
java.library.path /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.7.0_75-mockbuild_2015_01_21_08_43-b00
java.specification.name Java Platform API Specification
java.specification.vendor Oracle Corporation
java.specification.version 1.7
java.vendor Oracle Corporation
java.vendor.url http://java.oracle.com/
java.vendor.url.bug http://bugreport.sun.com/bugreport/
java.version 1.7.0_75
java.vm.info mixed mode
java.vm.name OpenJDK 64-Bit Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Oracle Corporation
java.vm.specification.version 1.7
java.vm.vendor Oracle Corporation
java.vm.version 24.75-b04
JENKINS_HOME /var/lib/jenkins
jna.platform.library.path /usr/lib64:/lib64:/usr/lib:/lib
jnidispatch.path /tmp/jna--1712433994/jna2593038690531263893.tmp
line.separator
mail.smtp.sendpartial true
mail.smtps.sendpartial true
os.arch amd64
os.name Linux
os.version 3.18.5-101.fc20.x86_64
path.separator :
sun.arch.data.model 64
sun.boot.class.path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/classes
sun.boot.library.path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.2.fc20.x86_64/jre/lib/amd64
sun.cpu.endian little
sun.cpu.isalist
sun.font.fontmanager sun.awt.X11FontManager
sun.io.unicode.encoding UnicodeLittle
sun.java.command /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --ajp13Port=8009 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
sun.java.launcher SUN_STANDARD
sun.jnu.encoding UTF-8
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.os.patch.level unknown
svnkit.http.methods Digest,Basic,NTLM,Negotiate
svnkit.ssh2.persistent false
user.country US
user.dir /
user.home /var/lib/jenkins
user.language en
user.name jenkins
user.timezone Europe/Berlin
Environment Variables
Name ↓
Value
HOME /var/lib/jenkins
LANG en_US.UTF-8
LOGNAME jenkins
NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
PATH /sbin:/usr/sbin:/bin:/usr/bin
PWD /
SHELL /bin/bash
SHLVL 2
USER jenkins
XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
Plugins
Name ↓
Version
Enabled
Pinned
ant 1.2 false false
antisamy-markup-formatter 1.1 false false
build-monitor-plugin 1.6+build.138 true false
credentials 1.18 true false
cvs 2.11 false false
email-ext 2.39 true false
external-monitor-job 1.4 false false
hudson-wsclean-plugin 1.0.4 true false
javadoc 1.1 false false
jenkinswalldisplay 0.6.27 true false
junit 1.2-beta-4 true false
ldap 1.8 false true
mailer 1.15 true true
matrix-auth 1.1 true false
matrix-project 1.3 true false
maven-plugin 2.7.1 false false
multi-slave-config-plugin 1.0.0 true false
pam-auth 1.1 false false
parameterized-trigger 2.25 true false
scm-api 0.2 true false
script-security 1.10 true false
ssh-credentials 1.10 true false
ssh-slaves 1.9 false false
subversion 2.2 true true
token-macro 1.10 true false
translation 1.11 false true
windows-slaves 1.0 false false
xunit 1.93 true false
#################################################################
Call stack:
Started by user anonymous
Building remotely on builder1 in workspace /data/workspace/test_subversion
java.io.IOException: remote file operation failed: /data/workspace/test_subversion at hudson.remoting.Channel@727be102:builder1: java.io.IOException: Remote call on builder1 failed
at hudson.FilePath.act(FilePath.java:985)
at hudson.FilePath.act(FilePath.java:967)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:935)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:870)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1270)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:531)
at hudson.model.Run.execute(Run.java:1718)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.io.IOException: Remote call on builder1 failed
at hudson.remoting.Channel.call(Channel.java:760)
at hudson.FilePath.act(FilePath.java:978)
... 11 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins
at hudson.scm.SubversionSCM.descriptor(SubversionSCM.java:2504)
at hudson.scm.SubversionSCM.createDefaultSVNOptions(SubversionSCM.java:1084)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1074)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1006)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:986)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2685)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:324)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to builder1(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
at hudson.remoting.Channel.call(Channel.java:752)
... 12 more
Finished: FAILURE
Still no mention of Jenkins version. Since there are two release lines (LTS and weekly), "current" is unhelpful.
I've seen this (or a similar) issue before, it's probably due to the design of Subversion Plugin that causes Jenkins to try to transfer unremotable classes to be transferred via the remoting channel.
The log (either master's jenkins.log or the slave's log) should contain another related error.
Please note: I am on holiday and have no access to this server other than by an e-mail chain with a colleague. Please forgive me if this takes some back and forth to get the necessary info.
Please note that this is an issue tracker, and not a support site. So it doesn't matter if it takes a few weeks for you to provide additional information.
Thanks Daniel.
Sorry, I am doing my best to get this resolved, because I have left our build system in an unknown state - even though the jobs are all running successfully - while I am away on holiday, during a fairly important milestone for our project. Needless to say, for any conscientious person, that makes me feel ill.
Anyway, apparently I haven't managed to provide the Jenkins version. My initial installation of Jenkins on this server was via the download link for Fedora on the Jenkins site here: http://pkg.jenkins-ci.org/redhat/ Then I have updated via the web interface to the running Master server. I am fairly confident that means I am running 1.598. Please tell me how I check.
Please tell me how I check.
Get your colleague to open any web page served by Jenkins. Tell him to scroll down, and look to the bottom right corner of the browser window.
Hello,
Thanks again. I am back from holiday. Here is the version information:
Jenkins ver. 1.598
Have the same issue on Jenkins version 1.618. Appreciate any help.
Started by user Jenkins System Builder ... Commencing build of Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master) Checking out Revision d2a9fafcda6f9c7b446280bc7e108817d1d734b0 (origin/HEAD, origin/master) java.io.IOException: remote file operation failed: /var/lib/openshift/55f0547b46dcfd56f5000011/app-root/runtime/repo at hudson.remoting.Channel@551b97:netztestbldr: java.io.IOException: Remote call on netztestbldr failed at hudson.FilePath.act(FilePath.java:987) at hudson.FilePath.act(FilePath.java:969) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1178) at hudson.model.AbstractProject.checkout(AbstractProject.java:1282) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:381) Caused by: java.io.IOException: Remote call on netztestbldr failed at hudson.remoting.Channel.call(Channel.java:761) at hudson.FilePath.act(FilePath.java:980) ... 10 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins at hudson.ExtensionList.lookup(ExtensionList.java:400) at hudson.FilePath.act(FilePath.java:977) at hudson.FilePath.act(FilePath.java:969) at hudson.FilePath.write(FilePath.java:1883) at hudson.plugins.git.GitSCM.computeChangeLog(GitSCM.java:1262) at hudson.plugins.git.GitSCM.access$700(GitSCM.java:79) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1233) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1178) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2691) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:325) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to netztestbldr(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1361) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:753) ... 11 more Archiving artifacts Finished: FAILURE
Dmitry Miroshnichenko I could not reproduce the bug. Could you provide more information? /cc Smouch Smouch
Hi,
we are facing a similar problem. We are using Jenkins version 1.591, and have two slaves configured. In case the job uses our slave-01, we are getting the NoClassDefFoundError shown below. If I configure the job, to use our slave-02, it works fine. We were not able to find any configuration differences between our slaves.
[EnvInject] - Loading node environment variables.
Building remotely on slave-01 (wfm-trunk linux tasktrack) in workspace /srv/jenkins-home/workspace/deployment-int-shutdown-instance-abc
java.io.IOException: remote file operation failed: /srv/jenkins-home/workspace/deployment-int-shutdown-instance-abc at hudson.remoting.Channel@205b38e:slave-01: java.io.IOException: Remote call on slave-01 failed
at hudson.FilePath.act(FilePath.java:977)
at hudson.FilePath.act(FilePath.java:959)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1259)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
at hudson.model.Run.execute(Run.java:1759)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Caused by: java.io.IOException: Remote call on slave-01 failed
at hudson.remoting.Channel.call(Channel.java:760)
at hudson.FilePath.act(FilePath.java:970)
... 11 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins
at hudson.scm.SubversionSCM.descriptor(SubversionSCM.java:2509)
at hudson.scm.SubversionSCM.createDefaultSVNOptions(SubversionSCM.java:1059)
at hudson.scm.SubversionSCM.createClientManager(SubversionSCM.java:1049)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:981)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:961)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2665)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:324)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to slave-01(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
at hudson.remoting.Channel.call(Channel.java:752)
... 12 more
From what I see Jenkins class is not being propagated to slaves. It may happen if you enable Slave2Master security AFAIK.
Anyway,SubversionSCM:: createDefaultSVNOptions() is misdesigned. It should not access descriptior() on its own. Once it is being called on slave, the method should retrieve the data from master via another remoting call. If the class is serializable, such double propagatiion will work correctly.
Some bits of the issue have been introduced in https://github.com/jenkinsci/subversion-plugin/commit/208f10ba16f8317f64060262fa82477db18be299 , but I think the real cause is even older.
Oleg Nenashev thanks for your answer.
The option "Enable Slave → Master Access Control" is not selected in our Jenkins configuration. When it had been selected it would affect all slaves, if I understand it correctly. As we have this problem only with one of our slaves, I don't think that this setting is relevant in our case.
Report needs much more information, such as the exact error message and stack trace. Relevant jenkins.log excerpts would be useful as well. And "version is whatever is latest as of (date)" is just lazy.
Please read https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue