• Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Critical Critical
    • docker-plugin
    • None
    • Jenkins 2.140
      Docker Plugin 1.1.5
      Docker version 18.06.1-ce, build e68fc7a
      Ubuntu 18.04.1 LTS

      We started using Docker Plugin 1.1.5 with Jenkins 2.140 and we are seeing this "channel closed down" message an almost every job.
      [WORK] Cannot contact docker-000cu0ya5igsh: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on docker-000cu0ya5igsh failed. The channel is closing down or has closed down
      It always feels like this is while a larger amount of work is being done in the container (for example running "mvn test" where we have thousands of tests).

       

          [JENKINS-53668] hudson.remoting.ChannelClosedException

          pjdarton added a comment -

          I've experienced this kind of issue where I work.  In our case, it was caused by the host running out of memory, triggering the oom-killer, and the oom-killer then decided that the Jenkins "java -jar slave.jar" process (the one responsible for keeping the slave connected to the master) was the least important process and killed it.

          The result was that, when things got busy, slaves died at random, despite doing nothing wrong themselves.

          This was particularly caused by our use of certain software packages that decide how much memory they're going to allocate to themselves based on the amount of memory available ... and that look at the whole host's memory instead of the container's fair share of that memory.  It doesn't take many processes to each allocate themselves half of the host's entire RAM before things get tight and the oom-killer gets invoked.

           

          Try turning off memory overcommit in your docker host, limiting the amount of memory available to each container, and limiting the number of containers you run concurrently.

          pjdarton added a comment - I've experienced this kind of issue where I work.  In our case, it was caused by the host running out of memory, triggering the oom-killer, and the oom-killer then decided that the Jenkins "java -jar slave.jar" process (the one responsible for keeping the slave connected to the master) was the least important process and killed it. The result was that, when things got busy, slaves died at random, despite doing nothing wrong themselves. This was particularly caused by our use of certain software packages that decide how much memory they're going to allocate to themselves based on the amount of memory available ... and that look at the whole host's memory instead of the container's fair share of that memory.  It doesn't take many processes to each allocate themselves half of the host's entire RAM before things get tight and the oom-killer gets invoked.   Try turning off memory overcommit in your docker host, limiting the amount of memory available to each container, and limiting the number of containers you run concurrently.

          pjdarton added a comment -

          Without further information, it won't be possible to debug this.

          If you're still experiencing this issue, and getting it with a recent version of the plugin, please add log information (see https://github.com/jenkinsci/docker-plugin/blob/master/CONTRIBUTING.md for hints about this) and re-open the issue.

          pjdarton added a comment - Without further information, it won't be possible to debug this. If you're still experiencing this issue, and getting it with a recent version of the plugin, please add log information (see https://github.com/jenkinsci/docker-plugin/blob/master/CONTRIBUTING.md for hints about this) and re-open the issue.

            Unassigned Unassigned
            mgreco2k Michael Greco
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: