Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32542

Maven project failed with "Connection refused" when built in a docker Container

      Building a "Maven project" in a Docker container fails with a "connection refused" error message whereas the same build is successful in a "Freestyle project" (also build in a Docker container plugin).

      [ael.validate-in-docker-mvnproject] $ docker exec --tty --user 500: dc69be1aed975dbff977348d060329d75d1377ecbefaf929a269a11fc5e40373 env ANT_HOME=/opt/apache-ant-1.8.2 CLASSPATH= EXECUTOR_NUMBER=0 JAVA_HOME=/opt/jdk1.7.0_45 JENKINS_HOME=/data/EDT/jenkins M2_HOME=/opt/apache-maven-3.0.4 MAVEN_HOME=/opt/apache-maven-3.0.4 "MAVEN_OPTS=-Xms1g -Xmx1g -XX:MaxPermSize=256m -Dfile.encoding=Cp1252" no_proxy=/var/run/docker.sock ORACLE_HOME=/opt/oracle11/product/11.2.0/client/ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/jdk1.7.0_45/bin:/opt/apache-ant-1.8.2/bin:/opt/apache-maven-3.0.4/bin TERM=xterm WORKSPACE=/data/EDT/jenkinsSlave_recette_1/c3f621fe/workspace/ael.validate-in-docker-mvnproject java -cp /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-agent.jar:/opt/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /opt/apache-maven-3.0.4 /data/EDT/jenkinsSlave_recette_1/slave.jar /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-interceptor.jar /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-interceptor-commons.jar 44848
      Exception in thread "main" java.net.ConnectException: Connection refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:579)
      at java.net.Socket.connect(Socket.java:528)
      at java.net.Socket.<init>(Socket.java:425)
      at java.net.Socket.<init>(Socket.java:208)
      at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:136)
      at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:70)
      Stopping Docker container after build completion

          [JENKINS-32542] Maven project failed with "Connection refused" when built in a docker Container

          Steven Dwyer added a comment -

          Having same issue.

          Steven Dwyer added a comment - Having same issue.

          is your jenkins slave itself running in a docker container ?

          Nicolas De Loof added a comment - is your jenkins slave itself running in a docker container ?

          No the jenkins slave is "classic".
          Jenkins master (Linux RHEL) => jenkins slave (another Linux RHEL)

          Thomas Collignon added a comment - No the jenkins slave is "classic". Jenkins master (Linux RHEL) => jenkins slave (another Linux RHEL)

          Aaron Corley added a comment -

          My company almost exclusively uses Maven projects, and we are seeing this error. I'd really like to use this plugin but without Maven job support it is pretty much useless to us.

          Aaron Corley added a comment - My company almost exclusively uses Maven projects, and we are seeing this error. I'd really like to use this plugin but without Maven job support it is pretty much useless to us.

          Aaron Corley added a comment -

          I discovered a workaround: changing the advanced setting "Network bridge" to "host" caused the error to go away and the build to proceed normally.

          Aaron Corley added a comment - I discovered a workaround: changing the advanced setting "Network bridge" to "host" caused the error to go away and the build to proceed normally.

          Zane Cahill added a comment -

          We're having the same issue... In our case, as ndeloof considered, the slave is running within our JNLP docker container deployed upon Amazon EC2 Container Service, using his plugin.

          The problem, it seems, stems from the network bridging; each container being in its own network and no "advanced" option within the CloudBees Docker Custom Build Environment Plugin to link containers. Although with a dynamically produced slave container name by the Amazon EC2 Container Service plugin, I guess this may need to utilise an environment variable if/when this feature becomes available?

          Setting the "Network bridge" to host as aaryn101 suggest does not help in our case as the containers are still on separate networks. However, I'm presuming if ndeloof's plugin had an advanced option to configure the "Network Bridge" for the Slave and setting it to "host" could cure this issue. However there may be other consequences to this...

          For the time being we can build with a Freestyle project, but if I get a chance to review the plugin(s) sources I'll see if I can contribute a fix.

          Zane Cahill added a comment - We're having the same issue... In our case, as ndeloof considered, the slave is running within our JNLP docker container deployed upon Amazon EC2 Container Service, using his plugin . The problem, it seems, stems from the network bridging; each container being in its own network and no "advanced" option within the CloudBees Docker Custom Build Environment Plugin to link containers. Although with a dynamically produced slave container name by the Amazon EC2 Container Service plugin, I guess this may need to utilise an environment variable if/when this feature becomes available? Setting the "Network bridge" to host as aaryn101 suggest does not help in our case as the containers are still on separate networks. However, I'm presuming if ndeloof 's plugin had an advanced option to configure the "Network Bridge" for the Slave and setting it to "host" could cure this issue. However there may be other consequences to this... For the time being we can build with a Freestyle project, but if I get a chance to review the plugin(s) sources I'll see if I can contribute a fix.

          petermount added a comment -

          I've had a docker java slave running happily for a few weeks now but after adding the Cloudbees docker build environment plugin (may be a coincidence) it failed with this same issue, with one minor difference in that it included a hostname in the logs:

          Modules changed, recalculating dependency graph
          Established TCP socket on dockerhost:48877
          maven32-agent.jar already up to date
          maven32-interceptor.jar already up to date
          maven3-interceptor-commons.jar already up to date
          [opendata-common] $ /opt/jdk//bin/java -cp /opt/jenkins/maven32-agent.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/boot/plexus-classworlds-2.5.2.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/conf/logging jenkins.maven3.agent.Maven32Main /opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3 /opt/jenkins/slave.jar /opt/jenkins/maven32-interceptor.jar /opt/jenkins/maven3-interceptor-commons.jar dockerhost:48877
          Exception in thread "main" java.net.ConnectException: Connection refused
          	at java.net.PlainSocketImpl.socketConnect(Native Method)
          	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
          	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
          	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
          	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
          	at java.net.Socket.connect(Socket.java:589)
          	at java.net.Socket.connect(Socket.java:538)
          	at java.net.Socket.<init>(Socket.java:434)
          	at java.net.Socket.<init>(Socket.java:211)
          	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:141)
          	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:71)
          ERROR: Failed to launch Maven. Exit code = 1
          

          Whilst looking for other possible causes for this I found JENKINS-29674 which stated:

          This is by intention, maven job type won't work with a docker container as it open an arbitrary socket for communication between slave agent and maven process, which we can't discover at container launch time

          So as the exception gave me a hostname & as this was using my own docker images (both jenkins server and slaves) I added as part of the slaves entrypoint script the following:

          echo "127.0.0.1 dockerhost" >>/etc/hosts
          

          Now this may be a fluke that it's working & I don't know where dockerhost is being defined (it's not in any of my config) but this may help in finding the underlying issue.

          Environment details: This was with Jenkins 2.5 (first spotted with 2.3) and an SSH based slave, both running within separate docker images. My docker images that have this fix in are area51/jenkins:2.5 & area51/jenkins-slave:2.5

          petermount added a comment - I've had a docker java slave running happily for a few weeks now but after adding the Cloudbees docker build environment plugin (may be a coincidence) it failed with this same issue, with one minor difference in that it included a hostname in the logs: Modules changed, recalculating dependency graph Established TCP socket on dockerhost:48877 maven32-agent.jar already up to date maven32-interceptor.jar already up to date maven3-interceptor-commons.jar already up to date [opendata-common] $ /opt/jdk//bin/java -cp /opt/jenkins/maven32-agent.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/boot/plexus-classworlds-2.5.2.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/conf/logging jenkins.maven3.agent.Maven32Main /opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3 /opt/jenkins/slave.jar /opt/jenkins/maven32-interceptor.jar /opt/jenkins/maven3-interceptor-commons.jar dockerhost:48877 Exception in thread "main" java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.<init>(Socket.java:434) at java.net.Socket.<init>(Socket.java:211) at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:141) at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:71) ERROR: Failed to launch Maven. Exit code = 1 Whilst looking for other possible causes for this I found JENKINS-29674 which stated: This is by intention, maven job type won't work with a docker container as it open an arbitrary socket for communication between slave agent and maven process, which we can't discover at container launch time So as the exception gave me a hostname & as this was using my own docker images (both jenkins server and slaves) I added as part of the slaves entrypoint script the following: echo "127.0.0.1 dockerhost" >>/etc/hosts Now this may be a fluke that it's working & I don't know where dockerhost is being defined (it's not in any of my config) but this may help in finding the underlying issue. Environment details: This was with Jenkins 2.5 (first spotted with 2.3) and an SSH based slave, both running within separate docker images. My docker images that have this fix in are area51/jenkins:2.5 & area51/jenkins-slave:2.5

          Manuel Ryan added a comment -

          I'm using the kubernetes plugin to provision fungible slaves in a cluster for each build and I have this issue too.

          Maybe the "dockerhost" hostname should actually be "jenkinshost" and be obtained by running somethig like "hostname -i" on the jenkins host performing the build (the slave container in my case) instead of the "ip route in an alpine:3.2 container" technique.

          Some other issues remain as the port needs to be exposed in the slave pod.

          • The kubernetes plugin needs to be able to specify port exposition in slave templates
          • The jenkins-maven plugin needs to support using a fixed port for the jvm agent -> build host connection. (this would cause collisions in most cases, but not when using the kubernetes plugin as each build has it's own IP address).

          Manuel Ryan added a comment - I'm using the kubernetes plugin to provision fungible slaves in a cluster for each build and I have this issue too. Maybe the "dockerhost" hostname should actually be "jenkinshost" and be obtained by running somethig like "hostname -i" on the jenkins host performing the build (the slave container in my case) instead of the "ip route in an alpine:3.2 container" technique. Some other issues remain as the port needs to be exposed in the slave pod. The kubernetes plugin needs to be able to specify port exposition in slave templates The jenkins-maven plugin needs to support using a fixed port for the jvm agent -> build host connection. (this would cause collisions in most cases, but not when using the kubernetes plugin as each build has it's own IP address).

          Manuel Ryan added a comment -

          It all boils down to the "getDocker0" method in the custom-build-plugin. We should not assume that the address of the jenkins (master or slave) is the default gateway of the container it starts. The basic example is jenkins running in a container with a shared docker socket.

          I think a portable solution could be to allow master/slave provisionners to specify and environment variable (for the master/slave process) with the value computed depending on their networking environment (there are so many different scenarios, we can give sensible defaults, but we also need to allow full customization).

          The environment variable could be something like JENKINS_MAVEN_PLUGIN_HOST. I'll try to submit a PR tomorrow for this.

          Manuel Ryan added a comment - It all boils down to the "getDocker0" method in the custom-build-plugin. We should not assume that the address of the jenkins (master or slave) is the default gateway of the container it starts. The basic example is jenkins running in a container with a shared docker socket. I think a portable solution could be to allow master/slave provisionners to specify and environment variable (for the master/slave process) with the value computed depending on their networking environment (there are so many different scenarios, we can give sensible defaults, but we also need to allow full customization). The environment variable could be something like JENKINS_MAVEN_PLUGIN_HOST. I'll try to submit a PR tomorrow for this.

          Tobias Prinz added a comment -

          Have the same issue. Any update on the PR?

          Tobias Prinz added a comment - Have the same issue. Any update on the PR?

            Unassigned Unassigned
            scarrier Sylvie Carrier
            Votes:
            10 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: