[JENKINS-39880] Jenkins agents should offer better Diagnostics remote operation failures due to JDK-8017777

Type: Improvement
Resolution: Won't Fix
Priority: Critical
Component/s: remoting
Labels:
- newbie-friendly
Environment:
Ubuntu 14.04, OpenJDK 1.7

Similar Issues:
Powered by SuggestiMate

Show

We have a Jenkins setup with slaves on several other machines.
Each slave shares the same home directory on a mounted network drive.
Since some update in July (we dont know what caused it), the parallel building on multiple machines always crashes with "remote file operation failed" and most of the time caused by "Caused by: java.util.zip.ZipException: error reading zip file".

If only one slave is used (thus no parallel builds), everything works fine.

This might be in relation to this java issue: https://support.cloudbees.com/hc/en-us/articles/204374874-Jenkins-JVM-Crash-in-libzip-so-JDK-8017777

The full backtrace looks like this:

[WARNINGS] Parsing warnings in console log with parser GNU C Compiler 4 (gcc)
[WARNINGS] Computing warning deltas based on reference build #396
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing BoostTest-1.x (default)
[xUnit] [INFO] - [BoostTest-1.x (default)] - 23 test report file(s) were found with the pattern 'build/bin/*.xml' relative to '/common/homes/students/jenkins1/jenkins-workspace/workspace/armarx-core/DISTRIB_CODENAME/trusty' for the testing framework 'BoostTest-1.x (default)'.
[xUnit] [INFO] - Converting '/common/homes/students/jenkins1/jenkins-workspace/workspace/armarx-core/DISTRIB_CODENAME/trusty/build/bin/ArmarX::Core::DataPath.xml' .
[xUnit] [ERROR] - Conversion error remote file operation failed: /var/lib/jenkins/userContent/xunit/BoostTest/1.x at hudson.remoting.Channel@21d73ff5:channel: java.io.IOException: Remote call on channel failed
ERROR: Step 'Publish xUnit test result report' aborted due to exception: 
java.io.IOException: remote file operation failed: /common/homes/students/jenkins1/jenkins-workspace/workspace/armarx-core/DISTRIB_CODENAME/trusty at hudson.remoting.Channel@3c281bc4:i61pc014: hudson.util.IOException2: There are some problems during the conversion into JUnit reports: 
    at hudson.FilePath.act(FilePath.java:1014)
    at hudson.FilePath.act(FilePath.java:996)
    at org.jenkinsci.plugins.xunit.XUnitProcessor.performTests(XUnitProcessor.java:138)
    at org.jenkinsci.plugins.xunit.XUnitProcessor.performXUnit(XUnitProcessor.java:81)
    at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:112)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
    at hudson.model.Build$BuildExecution.post2(Build.java:185)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
    at hudson.model.Run.execute(Run.java:1745)
    at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:404)
Caused by: hudson.util.IOException2: There are some problems during the conversion into JUnit reports: 
    at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:183)
    at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:44)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
    at hudson.remoting.UserRequest.perform(UserRequest.java:153)
    at hudson.remoting.UserRequest.perform(UserRequest.java:50)
    at hudson.remoting.Request$2.run(Request.java:332)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to i61pc014(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
    at hudson.remoting.Channel.call(Channel.java:781)
    at hudson.FilePath.act(FilePath.java:1007)
    ... 13 more
Caused by: org.jenkinsci.plugins.xunit.exception.XUnitException: Conversion error remote file operation failed: /var/lib/jenkins/userContent/xunit/BoostTest/1.x at hudson.remoting.Channel@21d73ff5:channel: java.io.IOException: Remote call on channel failed
    at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:91)
    at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:146)
    at org.jenkinsci.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:44)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
    at hudson.remoting.UserRequest.perform(UserRequest.java:153)
    at hudson.remoting.UserRequest.perform(UserRequest.java:50)
    at hudson.remoting.Request$2.run(Request.java:332)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: remote file operation failed: /var/lib/jenkins/userContent/xunit/BoostTest/1.x at hudson.remoting.Channel@21d73ff5:channel: java.io.IOException: Remote call on channel failed
    at hudson.FilePath.act(FilePath.java:1014)
    at hudson.FilePath.act(FilePath.java:996)
    at hudson.FilePath.exists(FilePath.java:1464)
    at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convertInputMetricXSL(XUnitConversionService.java:106)
    at org.jenkinsci.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:80)
    ... 11 more
Caused by: java.io.IOException: Remote call on channel failed
    at hudson.remoting.Channel.call(Channel.java:789)
    at hudson.FilePath.act(FilePath.java:1007)
    ... 15 more
Caused by: java.lang.NoClassDefFoundError: hudson/remoting/RemoteClassLoader$RemoteIClassLoader
    at hudson.remoting.RemoteClassLoader.export(RemoteClassLoader.java:675)
    at hudson.remoting.UserRequest.<init>(UserRequest.java:71)
    at hudson.remoting.Channel.call(Channel.java:779)
    ... 16 more
Caused by: java.lang.ClassNotFoundException: hudson.remoting.RemoteClassLoader$RemoteIClassLoader
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 19 more
Caused by: java.util.zip.ZipException: error reading zip file
    at java.util.zip.ZipFile.read(Native Method)
    at java.util.zip.ZipFile.access$1400(ZipFile.java:56)
    at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:679)
    at java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:415)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
    at sun.misc.IOUtils.readFully(IOUtils.java:65)
    at java.util.jar.JarFile.getBytes(JarFile.java:390)
    at java.util.jar.JarFile.getManifestFromReference(JarFile.java:180)
    at java.util.jar.JarFile.getManifest(JarFile.java:167)
    at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:779)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:416)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    ... 25 more
Archiviere Artefakte

duplicates

JENKINS-32403 xUnit NoClassDefFoundError when converting CTest output on Windows slave

Resolved

is related to

JENKINS-42910 After update of slave.jar to 3.5 xUnit an Plot plugin no more working

Resolved

Oleg Nenashev added a comment - 2016-11-19 19:56

From what I see it's a duplicate of ~~JENKINS-32403~~

Oleg Nenashev added a comment - 2016-11-19 19:56 From what I see it's a duplicate of JENKINS-32403

Mirko Wächter added a comment - 2016-11-20 13:23 - edited

I dont think it's the same problem.

I now used different workspace directories on each slave and it works now.
So it seems to be that jdk problem (https://support.cloudbees.com/hc/en-us/articles/204374874-Jenkins-JVM-Crash-in-libzip-so-JDK-8017777) is the cause.
Question is if Jenkins can and wants to workaround that.

Mirko Wächter added a comment - 2016-11-20 13:23 - edited I dont think it's the same problem. I now used different workspace directories on each slave and it works now. So it seems to be that jdk problem ( https://support.cloudbees.com/hc/en-us/articles/204374874-Jenkins-JVM-Crash-in-libzip-so-JDK-8017777 ) is the cause. Question is if Jenkins can and wants to workaround that.

Oleg Nenashev added a comment - 2016-12-22 11:13

I agree that it deserves some discussion.
Usage of the same workspace for multiple agents is not a case we are going to support. But I agree JDK-8017777 deserves some diagnostics/workaround.

Oleg Nenashev added a comment - 2016-12-22 11:13 I agree that it deserves some discussion. Usage of the same workspace for multiple agents is not a case we are going to support. But I agree JDK-8017777 deserves some diagnostics/workaround.

Mark Waite added a comment - 2023-09-09 12:55

Won't be fixed in Jenkins

Mark Waite added a comment - 2023-09-09 12:55 Won't be fixed in Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Oleg Nenashev added a comment - 2016-11-19 19:56

Expand comment: Oleg Nenashev added a comment - 2016-11-19 19:56

Collapse comment: Mirko Wächter added a comment - 2016-11-20 13:23, Edited by Mirko Wächter - 2016-11-20 13:25

Expand comment: Mirko Wächter added a comment - 2016-11-20 13:23, Edited by Mirko Wächter - 2016-11-20 13:25

Collapse comment: Oleg Nenashev added a comment - 2016-12-22 11:13

Expand comment: Oleg Nenashev added a comment - 2016-12-22 11:13

Collapse comment: Mark Waite added a comment - 2023-09-09 12:55

Expand comment: Mark Waite added a comment - 2023-09-09 12:55

People

Dates