Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7871

"java.io.IOException: Bad file descriptor" when file copied from slave

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Critical
    • Resolution: Fixed
    • core
    • None
    • Linux, 1.381, all plugins updated to latest and greatest.

    Description

      Just updated from 1.377 to 1.381.

      Large reporting jobs, where Cobertura output is large (many files around 1MB, totaling 18MB), will have issues while copying "coverage.xml" files from slave machine to main Hudson server.

      No error in logs on Slave machine, but we find the following error on the Hudson Server:


      Oct 21, 2010 4:40:03 AM hudson.remoting.ProxyOutputStream$Chunk$1 run
      WARNING: Failed to write to stream
      java.io.IOException: Bad file descriptor
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:247)
      at hudson.remoting.ProxyOutputStream$Chunk$1.run(ProxyOutputStream.java:185)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)


      The build actually fails then with following stack trace:


      Publishing Cobertura coverage report...
      Publishing Cobertura coverage results...
      FATAL: Unable to parse /home/hudson/hudson/server_installation/jobs/projectname/builds/2010-10-21_03-00-32/coverage35.xml
      hudson.util.IOException2: Cannot parse coverage results
      at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:89)
      at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:52)
      at hudson.plugins.cobertura.CoberturaPublisher.perform(CoberturaPublisher.java:262)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
      at hudson.model.Build$RunnerImpl.post2(Build.java:157)
      at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
      at hudson.model.Run.run(Run.java:1303)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:139)
      Caused by: org.xml.sax.SAXParseException: XML document structures must start and end within the same entity.
      at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:195)
      at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:174)
      at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:388)
      at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1411)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:902)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:605)
      at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1393)
      at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1763)
      at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1492)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanAttribute(XMLDocumentFragmentScannerImpl.java:1520)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1316)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2747)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
      at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
      at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
      at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
      at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:81)
      ... 12 more
      Email was triggered for: Failure
      Sending email for trigger: Failure
      Sending email to: xx@xx.com
      Finished: FAILURE


      Looking at the file "coverage35.xml" on the Hudson server, we see that the size is 983040 bytes. However, in the slave workspace, this file is 1023819 bytes.

      => As the file size 983040 is a multiple of 64k, and the difference between the original and copied is less than 64k, it is tempting to say that the error must be caused by some buffering issue and/or lack of a output "flush"

      Attachments

        Issue Links

          Activity

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/FilePath.java
            core/src/test/java/hudson/FilePathTest.java
            remoting/src/main/java/hudson/remoting/Channel.java
            remoting/src/main/java/hudson/remoting/LocalChannel.java
            remoting/src/main/java/hudson/remoting/VirtualChannel.java
            http://jenkins-ci.org/commit/jenkins/341fd4bb056216eb3fd14add5d8c72ea83329498
            Log:
            [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo
            that results in missing tail bits when the master is heavily loaded.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/FilePath.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/Channel.java remoting/src/main/java/hudson/remoting/LocalChannel.java remoting/src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/jenkins/341fd4bb056216eb3fd14add5d8c72ea83329498 Log: [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo that results in missing tail bits when the master is heavily loaded.
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #734
            [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo

            Kohsuke Kawaguchi : 957ffd30096a4309fc7da779d03086edb46e4fe3
            Files :

            • core/src/main/java/hudson/FilePath.java
            • remoting/src/main/java/hudson/remoting/LocalChannel.java
            • core/src/test/java/hudson/FilePathTest.java
            • remoting/src/main/java/hudson/remoting/VirtualChannel.java
            • remoting/src/main/java/hudson/remoting/Channel.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #734 [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo Kohsuke Kawaguchi : 957ffd30096a4309fc7da779d03086edb46e4fe3 Files : core/src/main/java/hudson/FilePath.java remoting/src/main/java/hudson/remoting/LocalChannel.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/VirtualChannel.java remoting/src/main/java/hudson/remoting/Channel.java

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/FilePath.java
            core/src/test/java/hudson/FilePathTest.java
            remoting/src/main/java/hudson/remoting/Channel.java
            remoting/src/main/java/hudson/remoting/LocalChannel.java
            remoting/src/main/java/hudson/remoting/VirtualChannel.java
            http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3
            Log:
            [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo
            that results in missing tail bits when the master is heavily loaded.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/FilePath.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/Channel.java remoting/src/main/java/hudson/remoting/LocalChannel.java remoting/src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3 Log: [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo that results in missing tail bits when the master is heavily loaded.

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/FilePath.java
            core/src/test/java/hudson/FilePathTest.java
            remoting/src/main/java/hudson/remoting/Channel.java
            remoting/src/main/java/hudson/remoting/LocalChannel.java
            remoting/src/main/java/hudson/remoting/VirtualChannel.java
            http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3
            Log:
            [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo
            that results in missing tail bits when the master is heavily loaded.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/FilePath.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/Channel.java remoting/src/main/java/hudson/remoting/LocalChannel.java remoting/src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3 Log: [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo that results in missing tail bits when the master is heavily loaded.
            pancake pancake added a comment -

            Oh, thank you for fixing that nasty bug we've been experiencing for several recent Jenkins releases!
            Which Jenkins version will the fix be included to?

            pancake pancake added a comment - Oh, thank you for fixing that nasty bug we've been experiencing for several recent Jenkins releases! Which Jenkins version will the fix be included to?

            People

              lacostej lacostej
              terjestrand terjestrand
              Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: