Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12235

FATAL, Unable to delete script file, IOException2, remote file operation failed, unexpected termination of channel

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • core, remoting
    • None

      Below is the stacktrace.

      It happened when I ran two jobs on a master. After running a while, both jobs crashed with this exception.
      I think this might be caused by a small flip-flop connectivity of the network, but I didn't noticed any disconnection.
      Another cause may be the huge load of jenkins:

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      25942 hudson 15 0 6902m 5.8g 5720 S 0.3 74.3 401:22.30 java

      Does the jenkins runs its own garbage collector at some specified time?
      We have to restart every few days because it's getting slower and slower until hangs out.

      FATAL: Unable to delete script file /tmp/hudson8303731085225956739.sh
      hudson.util.IOException2: remote file operation failed: /tmp/hudson8303731085225956739.sh at hudson.remoting.Channel@30e472f4:build@autom-1
      at hudson.FilePath.act(FilePath.java:781)
      at hudson.FilePath.act(FilePath.java:767)
      at hudson.FilePath.delete(FilePath.java:1022)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:695)
      at hudson.model.Build$RunnerImpl.build(Build.java:178)
      at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:461)
      at hudson.model.Run.run(Run.java:1404)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)
      Caused by: hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.remoting.Channel.send(Channel.java:499)
      at hudson.remoting.Request.call(Request.java:110)
      at hudson.remoting.Channel.call(Channel.java:681)
      at hudson.FilePath.act(FilePath.java:774)
      ... 13 more
      Caused by: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.call(Request.java:149)
      at hudson.remoting.Channel.call(Channel.java:681)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy29.join(Unknown Source)
      at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:859)
      at hudson.Launcher$ProcStarter.join(Launcher.java:345)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:695)
      at hudson.model.Build$RunnerImpl.build(Build.java:178)
      at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:461)
      at hudson.model.Run.run(Run.java:1404)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.abort(Request.java:273)
      at hudson.remoting.Channel.terminate(Channel.java:732)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
      Caused by: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)

        1. jenkins_1.471.txt
          5 kB
        2. stacktrace.txt
          4 kB
        3. jenkins_fatal_io_exception.txt
          4 kB

          [JENKINS-12235] FATAL, Unable to delete script file, IOException2, remote file operation failed, unexpected termination of channel

          Ghenadie Dumitru created issue -

          Ghenadie Dumitru added a comment -

          This issues is bothering, in special, when running on windows slaves connected via JNLP agent.

          In windows slaves case, it seems that the jnlp socket connection is quite sensitive to connection even it's not used at 100%.

          Maybe the solution for this is to use a ssh server on windows?

          Thanks

          Ghenadie Dumitru added a comment - This issues is bothering, in special, when running on windows slaves connected via JNLP agent. In windows slaves case, it seems that the jnlp socket connection is quite sensitive to connection even it's not used at 100%. Maybe the solution for this is to use a ssh server on windows? Thanks

          Erik Purins added a comment -

          Hitting this frequently on windows jenkins slaves. Similar call stack attached.

          Erik Purins added a comment - Hitting this frequently on windows jenkins slaves. Similar call stack attached.
          Erik Purins made changes -
          Attachment New: jenkins_fatal_io_exception.txt [ 21474 ]
          Erik Purins made changes -
          Link New: This issue duplicates JENKINS-1948 [ JENKINS-1948 ]

          brianharris added a comment -

          Suspected duplicate: JENKINS-6817

          brianharris added a comment - Suspected duplicate: JENKINS-6817

          Richard Taylor added a comment -

          We have started getting this reciently. Difficult to search but I think about version 1.455.

          • If there is no obvious fix can the exception be caught so that it does fail an otherwise successful build?

          I think this would be an adequate workaround for most people, atm this issue is causing random builds to fail which is a significant annoyance for the developers.

          Thanks.
          Rich.

          Richard Taylor added a comment - We have started getting this reciently. Difficult to search but I think about version 1.455. If there is no obvious fix can the exception be caught so that it does fail an otherwise successful build? I think this would be an adequate workaround for most people, atm this issue is causing random builds to fail which is a significant annoyance for the developers. Thanks. Rich.

          Kristian Karl added a comment - - edited

          I get this problem even so often on Windows machines. I run Jenkins ver. 1.463
          See attached stacktrace.txt

          Kristian Karl added a comment - - edited I get this problem even so often on Windows machines. I run Jenkins ver. 1.463 See attached stacktrace.txt
          Kristian Karl made changes -
          Attachment New: stacktrace.txt [ 21834 ]

          Brian Harris added a comment -

          For us, the cause of this error was our build slaves (VMs) running out of memory and self-rebooting.

          Brian Harris added a comment - For us, the cause of this error was our build slaves (VMs) running out of memory and self-rebooting.

            Unassigned Unassigned
            dumghen Ghenadie Dumitru
            Votes:
            38 Vote for this issue
            Watchers:
            47 Start watching this issue

              Created:
              Updated:
              Resolved: