Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73869

Cloud setup - docker connection refused over unix domain socket

      Hello,

      this is a regression but I'm not able to find the old bug which was solved months ago.

      I just upgraded jenkins and its plugins, and while trying to run the build on dynamic slaves containers, i get this error:

      org.apache.hc.client5.http.HttpHostConnectException: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused

      before the upgrade, everything was working properly.

      My cloud config:

       

      Docker host URI: unix:///var/run/docker.sock

       

      Here the versions:

      Jenkins 2.462.1 -> 2.462.3

      Docker API Plugin 3.4.0-94.v65ced49b_a_7d5

      Docker Commons Plugin 443.v921729d5611d

      Docker Pipeline 580.vc0c340686b_54

      Docker plugin 1.6.2

      Thank you!

          [JENKINS-73869] Cloud setup - docker connection refused over unix domain socket

          Mark Waite added a comment - - edited

          The Apache HTTP client 5 library has a breaking change that harms the docker-java API library that is used in the Docker plugin. More details are in the following GitHub issues:

          Apache httpclient 5 library 5.3.1-110.v77252fb_d4da_5 seems to work well, while Apache httpclient 5 library 5.4-118.v199115451c4d is known to break compatibility.

          In future reports, it is a great help if you include the precise plugins installed. There is a system groovy script that generates the list from your controller.

          Mark Waite added a comment - - edited The Apache HTTP client 5 library has a breaking change that harms the docker-java API library that is used in the Docker plugin. More details are in the following GitHub issues: Newer httpclient 5 API breaks the docker plugin Connect to unix://localhost:2375 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused /var/run/docker.sock is broken Can't make plugin work over Unix socket Update org.apache.httpcomponents.client5:httpclient5 to >= 5.3.1 Apache httpclient 5 library 5.3.1-110.v77252fb_d4da_5 seems to work well, while Apache httpclient 5 library 5.4-118.v199115451c4d is known to break compatibility. In future reports, it is a great help if you include the precise plugins installed. There is a system groovy script that generates the list from your controller.

          TheUnableDeveloper added a comment - - edited

          Thank you for the quick reply.
          I added the list with plugins installed as requested.

          Is there a way to update Jenkins and plugins now?

          Or should I wait for new version of the http5 library?

          Thank you!

          TheUnableDeveloper added a comment - - edited Thank you for the quick reply. I added the list with plugins installed as requested. Is there a way to update Jenkins and plugins now? Or should I wait for new version of the http5 library? Thank you!

          Mark Waite added a comment -

          You can resolve the issue immediately by downgrading the Apache httpclient 5 API plugin from 5.4-118.v199115451c4d to 5.3.1-110.v77252fb_d4da_5.

          You'll need to remain on that older version of Apache httpclient 5 API plugin until the upstream library is fixed.

          Mark Waite added a comment - You can resolve the issue immediately by downgrading the Apache httpclient 5 API plugin from 5.4-118.v199115451c4d to 5.3.1-110.v77252fb_d4da_5. You'll need to remain on that older version of Apache httpclient 5 API plugin until the upstream library is fixed.

          Greg Whiteley added a comment -

          Apologies if this is the wrong place to ask this question.

          Are the the details in the comment above definitely true?

          • Is this an Apache regression?
          • Is there a matching bug logged against the httpcomponents library?
          • Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change?

          Greg Whiteley added a comment - Apologies if this is the wrong place to ask this question. Are the the details in the comment above definitely true? Is this an Apache regression? Is there a matching bug logged against the httpcomponents library? Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change?

          Mark Waite added a comment -

          Are the the details in the comment above definitely true?

          The details that are posted are true as far as I have been able to determine. The Jenkins docker-plugin fails automated tests when those tests are configured to run with Apache httpclient 5 API plugin 5.4-118.v199115451c4d. They pass when those tests are configured to run with Apache httpclient 5 library 5.3.1-110.v77252fb_d4da_5. Users have reported that the docker-plugin fails in production when attempting to use a Unix domain socket if the Apache httpclient 5 API plugin 5.4-118.v199115451c4d is used.

          * Is this an Apache regression?

          Yes, as far as I can tell, this is a regression in the Apache httpclient 5 API plugin 5.4-118.v199115451c4d, since that is the only component that changed from the working case to the non-working case.

          * Is there a matching bug logged against the httpcomponents library?

          Not for this case. I've not yet taken the time to extract enough details for a bug report to the httpclient issue tracker. Are you interested in doing that investigation and submitting the issue? I don't have an account on the Apache issue tracker.

          * Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change?

          I assume there is a missing test because this is the second release of Apache httpclient 5 that has broken the Jenkins docker-plugin use through the docker-java library. Are you offering to investigate further?

          Mark Waite added a comment - Are the the details in the comment above definitely true? The details that are posted are true as far as I have been able to determine. The Jenkins docker-plugin fails automated tests when those tests are configured to run with Apache httpclient 5 API plugin 5.4-118.v199115451c4d. They pass when those tests are configured to run with Apache httpclient 5 library 5.3.1-110.v77252fb_d4da_5. Users have reported that the docker-plugin fails in production when attempting to use a Unix domain socket if the Apache httpclient 5 API plugin 5.4-118.v199115451c4d is used. * Is this an Apache regression? Yes, as far as I can tell, this is a regression in the Apache httpclient 5 API plugin 5.4-118.v199115451c4d, since that is the only component that changed from the working case to the non-working case. * Is there a matching bug logged against the httpcomponents library? Not for this case. I've not yet taken the time to extract enough details for a bug report to the httpclient issue tracker . Are you interested in doing that investigation and submitting the issue? I don't have an account on the Apache issue tracker. * Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change? I assume there is a missing test because this is the second release of Apache httpclient 5 that has broken the Jenkins docker-plugin use through the docker-java library. Are you offering to investigate further?

          Greg Whiteley added a comment -

          > Are you offering to investigate further?

          I'll see if I can get some time this weekend - I'm a C++ developer, not a Java developer and only did some poking around with Jenkins plugin development for a few weeks a few years back. I'm not promising anything productive

          Greg Whiteley added a comment - > Are you offering to investigate further? I'll see if I can get some time this weekend - I'm a C++ developer, not a Java developer and only did some poking around with Jenkins plugin development for a few weeks a few years back. I'm not promising anything productive

          Mark Waite added a comment -

          Apache httpcomponents 5.4-124.v31e2987e48f4 has released with a revert to httpclient library 5.3.1. That was done with PR-62.

          No upstream issue has been submitted yet. I hope to spend time on that tomorrow.

          Mark Waite added a comment - Apache httpcomponents 5.4-124.v31e2987e48f4 has released with a revert to httpclient library 5.3.1. That was done with PR-62 . No upstream issue has been submitted yet. I hope to spend time on that tomorrow.

          Mark Waite added a comment -

          Upstream issue has been submitted by Basil and a pull request has been submitted to the docker-java library to resolve the issue. Waiting for a response from the maintainers of the docker-java library.

          Mark Waite added a comment - Upstream issue has been submitted by Basil and a pull request has been submitted to the docker-java library to resolve the issue. Waiting for a response from the maintainers of the docker-java library.

            Unassigned Unassigned
            theunabledeveloper TheUnableDeveloper
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: