Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66025

Builds lost with large projects repository

    XMLWordPrintable

Details

    Description

      Hello,

      When using docker-plugin with hundred of simultaneous builds, sometimes the connection is lost.

      docker-plugin should be able to restore the connection, it doesn't, instead the job is broken and can't be canceled. This is a bug.

       

      We are using this docker-plugin in a heavy load installation of Jenkins.

      Thanks,

      Jean-Sébastien

      Attachments

        Issue Links

          Activity

            This issue is linked to the 65872 https://issues.jenkins.io/browse/JENKINS-65872

            jsbevilacqua Jean-Sébastien added a comment - This issue is linked to the 65872 https://issues.jenkins.io/browse/JENKINS-65872

            The problem is linked to the utilization of netty as transport layer, Http resolves the problem

            jsbevilacqua Jean-Sébastien added a comment - The problem is linked to the utilization of netty as transport layer, Http resolves the problem
            pjdarton pjdarton added a comment -

            OK, without a mass of additional logging/diagnostics it'll be difficult to independently confirm this.
            e.g. we use this plugin where I work, under heavy load across thousands of Jenkins instances, but we've not seen this to be a problem.
            ... but if you're saying that you've already traced the issue to the netty transport layer and found that replacing it with Apache HttpClient then that pretty much takes it out of the hands of the docker-plugin code - the docker-plugin code doesn't control the netty transport layer.
            FYI the "chain of command" here is that the docker-plugin uses the docker-java-api-plugin to provide it with a java library that gives it access to the docker API. The docker-java-api-plugin's job is to provide a java library (and with its dependencies) to the rest of Jenkins, and it uses the docker-java library. The docker-java library is able to use a few different transports, but back when the docker-plugin was designed the only option was Netty, so that's what it did (and did so successfully).
            i.e. if you're saying that switching the transport layer from netty to http resolves this issue then that's it's not a bug in the docker-plugin as the docker plugin is "working as designed" here (even if it is, as you claim, being let down by underlying layers of code).

            I am aware that switching transport layers to Apache's http library may provide other benefits that would be welcomed (e.g. better support for Microsoft Npipes - Windows had no docker support back when the docker-plugin was designed) so that wasn't even a consideration back then), but that'd be a (non-trivial) enhancement rather than a bugfix.

            i.e. As a maintainer of the docker-plugin, I'd welcome a PR (or, more likely, a set of PRs) that made such an enhancement (because Apache HttpClient 5 seems to be the docker-java library's preferred transport these days, and especially if the PR adds support for Microsoft pipes - now Microsoft have finally started to support docker, it'd be nice to improve the docker-plugin's support for their docker variant too), but beware that it may be a non-trivial bit of work and it would be well outside the scope of a mere "bugfix".

            pjdarton pjdarton added a comment - OK, without a mass of additional logging/diagnostics it'll be difficult to independently confirm this. e.g. we use this plugin where I work, under heavy load across thousands of Jenkins instances, but we've not seen this to be a problem. ... but if you're saying that you've already traced the issue to the netty transport layer and found that replacing it with Apache HttpClient then that pretty much takes it out of the hands of the docker-plugin code - the docker-plugin code doesn't control the netty transport layer. FYI the "chain of command" here is that the docker-plugin uses the docker-java-api-plugin to provide it with a java library that gives it access to the docker API. The docker-java-api-plugin's job is to provide a java library (and with its dependencies) to the rest of Jenkins, and it uses the docker-java library. The docker-java library is able to use a few different transports, but back when the docker-plugin was designed the only option was Netty, so that's what it did (and did so successfully). i.e. if you're saying that switching the transport layer from netty to http resolves this issue then that's it's not a bug in the docker-plugin as the docker plugin is "working as designed" here (even if it is, as you claim, being let down by underlying layers of code). I am aware that switching transport layers to Apache's http library may provide other benefits that would be welcomed (e.g. better support for Microsoft Npipes - Windows had no docker support back when the docker-plugin was designed) so that wasn't even a consideration back then), but that'd be a (non-trivial) enhancement rather than a bugfix. i.e. As a maintainer of the docker-plugin, I'd welcome a PR (or, more likely, a set of PRs) that made such an enhancement (because Apache HttpClient 5 seems to be the docker-java library's preferred transport these days, and especially if the PR adds support for Microsoft pipes - now Microsoft have finally started to support docker, it'd be nice to improve the docker-plugin's support for their docker variant too), but beware that it may be a non-trivial bit of work and it would be well outside the scope of a mere "bugfix".
            jsbevilacqua Jean-Sébastien added a comment - Pull request: https://github.com/jenkinsci/docker-plugin/pull/843
            pjdarton pjdarton added a comment -

            Sitrep:

            • Switching transport layers is reported to improve stability.
            • ...but requires matching changes to both the docker-java-api-plugin and the docker-plugin.
            • Changes to the latter are blocked upon the former.
            pjdarton pjdarton added a comment - Sitrep: Switching transport layers is reported to improve stability. ...but requires matching changes to both the docker-java-api-plugin and the docker-plugin. Changes to the latter are blocked upon the former.

            People

              ndeloof Nicolas De Loof
              jsbevilacqua Jean-Sébastien
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: