Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73869

Cloud setup - docker connection refused over unix domain socket

      Hello,

      this is a regression but I'm not able to find the old bug which was solved months ago.

      I just upgraded jenkins and its plugins, and while trying to run the build on dynamic slaves containers, i get this error:

      org.apache.hc.client5.http.HttpHostConnectException: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused

      before the upgrade, everything was working properly.

      My cloud config:

       

      Docker host URI: unix:///var/run/docker.sock

       

      Here the versions:

      Jenkins 2.462.1 -> 2.462.3

      Docker API Plugin 3.4.0-94.v65ced49b_a_7d5

      Docker Commons Plugin 443.v921729d5611d

      Docker Pipeline 580.vc0c340686b_54

      Docker plugin 1.6.2

      Thank you!

          [JENKINS-73869] Cloud setup - docker connection refused over unix domain socket

          Greg Whiteley added a comment -

          Apologies if this is the wrong place to ask this question.

          Are the the details in the comment above definitely true?

          • Is this an Apache regression?
          • Is there a matching bug logged against the httpcomponents library?
          • Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change?

          Greg Whiteley added a comment - Apologies if this is the wrong place to ask this question. Are the the details in the comment above definitely true? Is this an Apache regression? Is there a matching bug logged against the httpcomponents library? Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change?

          Mark Waite added a comment -

          Are the the details in the comment above definitely true?

          The details that are posted are true as far as I have been able to determine. The Jenkins docker-plugin fails automated tests when those tests are configured to run with Apache httpclient 5 API plugin 5.4-118.v199115451c4d. They pass when those tests are configured to run with Apache httpclient 5 library 5.3.1-110.v77252fb_d4da_5. Users have reported that the docker-plugin fails in production when attempting to use a Unix domain socket if the Apache httpclient 5 API plugin 5.4-118.v199115451c4d is used.

          * Is this an Apache regression?

          Yes, as far as I can tell, this is a regression in the Apache httpclient 5 API plugin 5.4-118.v199115451c4d, since that is the only component that changed from the working case to the non-working case.

          * Is there a matching bug logged against the httpcomponents library?

          Not for this case. I've not yet taken the time to extract enough details for a bug report to the httpclient issue tracker. Are you interested in doing that investigation and submitting the issue? I don't have an account on the Apache issue tracker.

          * Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change?

          I assume there is a missing test because this is the second release of Apache httpclient 5 that has broken the Jenkins docker-plugin use through the docker-java library. Are you offering to investigate further?

          Mark Waite added a comment - Are the the details in the comment above definitely true? The details that are posted are true as far as I have been able to determine. The Jenkins docker-plugin fails automated tests when those tests are configured to run with Apache httpclient 5 API plugin 5.4-118.v199115451c4d. They pass when those tests are configured to run with Apache httpclient 5 library 5.3.1-110.v77252fb_d4da_5. Users have reported that the docker-plugin fails in production when attempting to use a Unix domain socket if the Apache httpclient 5 API plugin 5.4-118.v199115451c4d is used. * Is this an Apache regression? Yes, as far as I can tell, this is a regression in the Apache httpclient 5 API plugin 5.4-118.v199115451c4d, since that is the only component that changed from the working case to the non-working case. * Is there a matching bug logged against the httpcomponents library? Not for this case. I've not yet taken the time to extract enough details for a bug report to the httpclient issue tracker . Are you interested in doing that investigation and submitting the issue? I don't have an account on the Apache issue tracker. * Is someone (Apache?) missing a regression test? Or is this an unsupported use of that library and docker-plugin should change? I assume there is a missing test because this is the second release of Apache httpclient 5 that has broken the Jenkins docker-plugin use through the docker-java library. Are you offering to investigate further?

          Greg Whiteley added a comment -

          > Are you offering to investigate further?

          I'll see if I can get some time this weekend - I'm a C++ developer, not a Java developer and only did some poking around with Jenkins plugin development for a few weeks a few years back. I'm not promising anything productive

          Greg Whiteley added a comment - > Are you offering to investigate further? I'll see if I can get some time this weekend - I'm a C++ developer, not a Java developer and only did some poking around with Jenkins plugin development for a few weeks a few years back. I'm not promising anything productive

          Mark Waite added a comment -

          Apache httpcomponents 5.4-124.v31e2987e48f4 has released with a revert to httpclient library 5.3.1. That was done with PR-62.

          No upstream issue has been submitted yet. I hope to spend time on that tomorrow.

          Mark Waite added a comment - Apache httpcomponents 5.4-124.v31e2987e48f4 has released with a revert to httpclient library 5.3.1. That was done with PR-62 . No upstream issue has been submitted yet. I hope to spend time on that tomorrow.

          Mark Waite added a comment -

          Upstream issue has been submitted by Basil and a pull request has been submitted to the docker-java library to resolve the issue. Waiting for a response from the maintainers of the docker-java library.

          Mark Waite added a comment - Upstream issue has been submitted by Basil and a pull request has been submitted to the docker-java library to resolve the issue. Waiting for a response from the maintainers of the docker-java library.

          Giuliano added a comment -

          Hi there.

          The problem persists in Jenkins 2.479.3: https://issues.jenkins.io/browse/JENKINS-75460.

          But even worse is that in Jenkins 2.479.3, the minumum versions of the plugins are:

          DockerVersione1.6.2 which requires a minimum of
          Docker API PluginVersione3.4.1-96.v77147a_de67f8 which requires a minimum of
          Apache HttpComponents Client 5.x API PluginVersione5.4-136.v5a_21779c63f8
          So, downgrading to Apache HttpComponents Client 5.x API PluginVersione5.3.1-110.v77252fb_d4da_5 is not possible:

          Giuliano added a comment - Hi there. The problem persists in Jenkins 2.479.3: https://issues.jenkins.io/browse/JENKINS-75460 . But even worse is that in Jenkins 2.479.3, the minumum versions of the plugins are: DockerVersione1.6.2 which requires a minimum of Docker API PluginVersione3.4.1-96.v77147a_de67f8 which requires a minimum of Apache HttpComponents Client 5.x API PluginVersione5.4-136.v5a_21779c63f8 So, downgrading to Apache HttpComponents Client 5.x API PluginVersione5.3.1-110.v77252fb_d4da_5 is not possible:

          Mark Waite added a comment -

          giuliano I think that you are confusing two different conditions.

          The issue reported by this bug is that connections to the Docker agent are not allowed over a Unix domain socket. Unix domain sockets are only available when communication is on a single host. This issue is still resolved in the Apache HTTPComponents Client 5 API release 5.4-136.v5a_21779c63f8 because that API plugin provides Apache HTTPComponents Client 5 API release 5.3.3. It has intentionally been held at 5.3.3 inside the API library because of the change in behavior.

          The issue you are reporting in JENKINS-75460 does not use a Unix domain socket. JENKINS-75460 is using a TCP connection, not a Unix domain socket. It is a different issue.

          Mark Waite added a comment - giuliano I think that you are confusing two different conditions. The issue reported by this bug is that connections to the Docker agent are not allowed over a Unix domain socket. Unix domain sockets are only available when communication is on a single host. This issue is still resolved in the Apache HTTPComponents Client 5 API release 5.4-136.v5a_21779c63f8 because that API plugin provides Apache HTTPComponents Client 5 API release 5.3.3. It has intentionally been held at 5.3.3 inside the API library because of the change in behavior. The issue you are reporting in JENKINS-75460 does not use a Unix domain socket. JENKINS-75460 is using a TCP connection, not a Unix domain socket. It is a different issue.

          Hello,

          as for my original issue, it is now solved because I waited some months and new versions of the plugin have been released.

          So we can close the issue, at least on my side.

          thank you

          TheUnableDeveloper added a comment - Hello, as for my original issue, it is now solved because I waited some months and new versions of the plugin have been released. So we can close the issue, at least on my side. thank you

          Mark Waite added a comment -

          theunabledeveloper Thanks for confirming that the plugin is working for your case. I've kept the issue open while we wait for a permanent fix in the docker java API library. A pull request has been submitted that uses a different API but it needs to be merged and released.

          Mark Waite added a comment - theunabledeveloper Thanks for confirming that the plugin is working for your case. I've kept the issue open while we wait for a permanent fix in the docker java API library. A pull request has been submitted that uses a different API but it needs to be merged and released.

          Mark Waite added a comment - Fixed in Docker Java API https://plugins.jenkins.io/docker-java-api/releases/#version_3.5.0-108.v211cdd21c383 and in https://github.com/jenkinsci/apache-httpcomponents-client-5-api-plugin/releases/tag/5.4.3-140.v2516ccde99e7

            Unassigned Unassigned
            theunabledeveloper TheUnableDeveloper
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: