Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29798

Failed to listen to incoming slave connection after fixing port through init.groovy.d

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core
    • None

      When using a init.groovy.d script to set the slave agent port to a fixed value (like the official docker image for Jenkins does), Jenkins shows an error message that seem to tell us that it failed to listen to the agent port.

      java.net.BindException: Failed to listen on port 10000 because it's already in use.
      at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:74)
      at jenkins.model.Jenkins.<init>(Jenkins.java:825)
      at hudson.model.Hudson.<init>(Hudson.java:83)
      at hudson.model.Hudson.<init>(Hudson.java:79)
      at hudson.WebAppMain$3.run(WebAppMain.java:225)
      Caused by: java.net.BindException: Address already in use
      at sun.nio.ch.Net.bind0(Native Method)
      at sun.nio.ch.Net.bind(Net.java:436)
      at sun.nio.ch.Net.bind(Net.java:428)
      at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
      at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
      at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
      at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:72)
      ... 4 more

      As a workaround, people have been adding delays, but really the issue is that we have the following sequence of events occurring :

      • Jenkins init
      • init.groovy.d
        • calls Jenkins.getInstance(). setSlaveAgentPort(xxx)
          • behind the scenes, this call starts the TcpSlaveAgentListener
      • initializes TcpSlaveAgentListener, but doesn't suppose that it may have been started before => the port is already taken, hence the error message above.

          [JENKINS-29798] Failed to listen to incoming slave connection after fixing port through init.groovy.d

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          changelog.html
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/7dfa765d8b7562ea25b8eab6c26d8c7a5ca6ef1b
          Log:
          [FIXED JENKINS-29798] Failed to listen to incoming slave connection

          Factor tcpSlaveAgentListener logic to a single method able to shutdown
          existing listener if required, then launch a new one if it needs to.

          Also, the administrative monitor gets cleared if we had previously a
          taken port and changed to a valid port.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/7dfa765d8b7562ea25b8eab6c26d8c7a5ca6ef1b Log: [FIXED JENKINS-29798] Failed to listen to incoming slave connection Factor tcpSlaveAgentListener logic to a single method able to shutdown existing listener if required, then launch a new one if it needs to. Also, the administrative monitor gets cleared if we had previously a taken port and changed to a valid port.

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/03bba487296dd7d0f44a0c235724c3b7fb41cd69
          Log:
          JENKINS-29798 Add a lock to serialize updates of tcpSlaveAgentListener

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/03bba487296dd7d0f44a0c235724c3b7fb41cd69 Log: JENKINS-29798 Add a lock to serialize updates of tcpSlaveAgentListener

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          changelog.html
          cli/pom.xml
          core/pom.xml
          core/src/main/java/hudson/FilePath.java
          core/src/main/java/hudson/FileSystemProvisioner.java
          core/src/main/java/hudson/FileSystemProvisionerDescriptor.java
          core/src/main/java/hudson/Functions.java
          core/src/main/java/hudson/PluginWrapper.java
          core/src/main/java/hudson/cli/CloneableCLICommand.java
          core/src/main/java/hudson/diagnosis/OldDataMonitor.java
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/ChoiceParameterDefinition.java
          core/src/main/java/hudson/model/Computer.java
          core/src/main/java/hudson/model/Executor.java
          core/src/main/java/hudson/model/UsageStatistics.java
          core/src/main/java/hudson/os/SU.java
          core/src/main/java/hudson/scheduler/CronTab.java
          core/src/main/java/hudson/security/ACL.java
          core/src/main/java/hudson/security/SecurityRealm.java
          core/src/main/java/hudson/slaves/NodeProvisioner.java
          core/src/main/java/hudson/tasks/Maven.java
          core/src/main/java/hudson/util/ChartUtil.java
          core/src/main/java/hudson/util/CopyOnWriteList.java
          core/src/main/java/hudson/util/DoubleLaunchChecker.java
          core/src/main/java/hudson/util/QuotedStringTokenizer.java
          core/src/main/java/hudson/util/RobustCollectionConverter.java
          core/src/main/java/hudson/util/RobustMapConverter.java
          core/src/main/java/hudson/util/RobustReflectionConverter.java
          core/src/main/java/hudson/util/jna/DotNet.java
          core/src/main/java/hudson/util/xstream/ImmutableListConverter.java
          core/src/main/java/jenkins/model/BlockedBecauseOfBuildInProgress.java
          core/src/main/java/jenkins/model/JenkinsLocationConfiguration.java
          core/src/main/java/jenkins/slaves/restarter/SlaveRestarter.java
          core/src/main/java/jenkins/util/xstream/CriticalXStreamException.java
          core/src/main/resources/hudson/model/AbstractBuild/changes.jelly
          core/src/main/resources/hudson/model/AbstractBuild/index_pt_BR.properties
          core/src/main/resources/hudson/model/AbstractBuild/index_pt_PT.properties
          core/src/main/resources/hudson/model/AbstractBuild/sidepanel_pt_PT.properties
          core/src/main/resources/hudson/model/Messages.properties
          core/src/main/resources/hudson/model/Messages_da.properties
          core/src/main/resources/hudson/model/Messages_de.properties
          core/src/main/resources/hudson/model/Messages_es.properties
          core/src/main/resources/hudson/model/Messages_fr.properties
          core/src/main/resources/hudson/model/Messages_it.properties
          core/src/main/resources/hudson/model/Messages_ja.properties
          core/src/main/resources/hudson/model/Messages_nl.properties
          core/src/main/resources/hudson/model/Messages_pt_BR.properties
          core/src/main/resources/hudson/model/Messages_ru.properties
          core/src/main/resources/hudson/model/Messages_tr.properties
          core/src/main/resources/hudson/model/Messages_zh_CN.properties
          core/src/main/resources/hudson/model/Messages_zh_TW.properties
          core/src/main/resources/jenkins/model/Messages.properties
          core/src/main/resources/jenkins/model/Messages_da.properties
          core/src/main/resources/jenkins/model/Messages_de.properties
          core/src/main/resources/jenkins/model/Messages_es.properties
          core/src/main/resources/jenkins/model/Messages_fr.properties
          core/src/main/resources/jenkins/model/Messages_it.properties
          core/src/main/resources/jenkins/model/Messages_ja.properties
          core/src/main/resources/jenkins/model/Messages_nl.properties
          core/src/main/resources/jenkins/model/Messages_pt_BR.properties
          core/src/main/resources/jenkins/model/Messages_ru.properties
          core/src/main/resources/jenkins/model/Messages_tr.properties
          core/src/main/resources/jenkins/model/Messages_zh_CN.properties
          core/src/main/resources/jenkins/model/Messages_zh_TW.properties
          core/src/main/resources/jenkins/security/s2m/filepath-filter.conf
          core/src/main/resources/lib/hudson/scriptConsole.properties
          core/src/main/resources/lib/hudson/scriptConsole_da.properties
          core/src/main/resources/lib/hudson/scriptConsole_de.properties
          core/src/main/resources/lib/hudson/scriptConsole_es.properties
          core/src/main/resources/lib/hudson/scriptConsole_fr.properties
          core/src/main/resources/lib/hudson/scriptConsole_ja.properties
          core/src/main/resources/lib/hudson/scriptConsole_ko.properties
          core/src/main/resources/lib/hudson/scriptConsole_nb_NO.properties
          core/src/main/resources/lib/hudson/scriptConsole_nl.properties
          core/src/main/resources/lib/hudson/scriptConsole_pl.properties
          core/src/main/resources/lib/hudson/scriptConsole_pt_BR.properties
          core/src/main/resources/lib/hudson/scriptConsole_ru.properties
          core/src/main/resources/lib/hudson/scriptConsole_sv_SE.properties
          core/src/main/resources/lib/hudson/scriptConsole_zh_TW.properties
          core/src/site/markdown/index.md
          core/src/test/java/hudson/UtilTest.java
          core/src/test/java/hudson/model/ParametersActionTest.java
          core/src/test/java/hudson/util/ArgumentListBuilderTest.java
          plugins/pom.xml
          pom.xml
          test/pom.xml
          test/src/main/java/org/jvnet/hudson/test/ExtractResourceSCM.java
          test/src/test/java/hudson/cli/SetBuildDisplayNameCommandTest.java
          test/src/test/java/hudson/model/JobQueueTest.java
          test/src/test/java/hudson/model/MyViewsPropertyTest.java
          test/src/test/java/hudson/model/ProjectTest.java
          test/src/test/java/hudson/model/SimpleJobTest.java
          test/src/test/java/hudson/model/UserTest.java
          test/src/test/java/hudson/model/ViewTest.java
          test/src/test/java/hudson/model/queue/WideExecutionTest.java
          test/src/test/java/hudson/util/RobustReflectionConverterTest.java
          test/src/test/java/lib/form/RepeatableTest.java
          translation-tool.pl
          war/pom.xml
          war/src/main/webapp/help/project-config/custom-workspace.html
          http://jenkins-ci.org/commit/jenkins/d3f53b9304fc4c505c05f9be2e1abc3c4985f488
          Log:
          Merge branch 'master' into JENKINS-29798

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html cli/pom.xml core/pom.xml core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/FileSystemProvisioner.java core/src/main/java/hudson/FileSystemProvisionerDescriptor.java core/src/main/java/hudson/Functions.java core/src/main/java/hudson/PluginWrapper.java core/src/main/java/hudson/cli/CloneableCLICommand.java core/src/main/java/hudson/diagnosis/OldDataMonitor.java core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/ChoiceParameterDefinition.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/Executor.java core/src/main/java/hudson/model/UsageStatistics.java core/src/main/java/hudson/os/SU.java core/src/main/java/hudson/scheduler/CronTab.java core/src/main/java/hudson/security/ACL.java core/src/main/java/hudson/security/SecurityRealm.java core/src/main/java/hudson/slaves/NodeProvisioner.java core/src/main/java/hudson/tasks/Maven.java core/src/main/java/hudson/util/ChartUtil.java core/src/main/java/hudson/util/CopyOnWriteList.java core/src/main/java/hudson/util/DoubleLaunchChecker.java core/src/main/java/hudson/util/QuotedStringTokenizer.java core/src/main/java/hudson/util/RobustCollectionConverter.java core/src/main/java/hudson/util/RobustMapConverter.java core/src/main/java/hudson/util/RobustReflectionConverter.java core/src/main/java/hudson/util/jna/DotNet.java core/src/main/java/hudson/util/xstream/ImmutableListConverter.java core/src/main/java/jenkins/model/BlockedBecauseOfBuildInProgress.java core/src/main/java/jenkins/model/JenkinsLocationConfiguration.java core/src/main/java/jenkins/slaves/restarter/SlaveRestarter.java core/src/main/java/jenkins/util/xstream/CriticalXStreamException.java core/src/main/resources/hudson/model/AbstractBuild/changes.jelly core/src/main/resources/hudson/model/AbstractBuild/index_pt_BR.properties core/src/main/resources/hudson/model/AbstractBuild/index_pt_PT.properties core/src/main/resources/hudson/model/AbstractBuild/sidepanel_pt_PT.properties core/src/main/resources/hudson/model/Messages.properties core/src/main/resources/hudson/model/Messages_da.properties core/src/main/resources/hudson/model/Messages_de.properties core/src/main/resources/hudson/model/Messages_es.properties core/src/main/resources/hudson/model/Messages_fr.properties core/src/main/resources/hudson/model/Messages_it.properties core/src/main/resources/hudson/model/Messages_ja.properties core/src/main/resources/hudson/model/Messages_nl.properties core/src/main/resources/hudson/model/Messages_pt_BR.properties core/src/main/resources/hudson/model/Messages_ru.properties core/src/main/resources/hudson/model/Messages_tr.properties core/src/main/resources/hudson/model/Messages_zh_CN.properties core/src/main/resources/hudson/model/Messages_zh_TW.properties core/src/main/resources/jenkins/model/Messages.properties core/src/main/resources/jenkins/model/Messages_da.properties core/src/main/resources/jenkins/model/Messages_de.properties core/src/main/resources/jenkins/model/Messages_es.properties core/src/main/resources/jenkins/model/Messages_fr.properties core/src/main/resources/jenkins/model/Messages_it.properties core/src/main/resources/jenkins/model/Messages_ja.properties core/src/main/resources/jenkins/model/Messages_nl.properties core/src/main/resources/jenkins/model/Messages_pt_BR.properties core/src/main/resources/jenkins/model/Messages_ru.properties core/src/main/resources/jenkins/model/Messages_tr.properties core/src/main/resources/jenkins/model/Messages_zh_CN.properties core/src/main/resources/jenkins/model/Messages_zh_TW.properties core/src/main/resources/jenkins/security/s2m/filepath-filter.conf core/src/main/resources/lib/hudson/scriptConsole.properties core/src/main/resources/lib/hudson/scriptConsole_da.properties core/src/main/resources/lib/hudson/scriptConsole_de.properties core/src/main/resources/lib/hudson/scriptConsole_es.properties core/src/main/resources/lib/hudson/scriptConsole_fr.properties core/src/main/resources/lib/hudson/scriptConsole_ja.properties core/src/main/resources/lib/hudson/scriptConsole_ko.properties core/src/main/resources/lib/hudson/scriptConsole_nb_NO.properties core/src/main/resources/lib/hudson/scriptConsole_nl.properties core/src/main/resources/lib/hudson/scriptConsole_pl.properties core/src/main/resources/lib/hudson/scriptConsole_pt_BR.properties core/src/main/resources/lib/hudson/scriptConsole_ru.properties core/src/main/resources/lib/hudson/scriptConsole_sv_SE.properties core/src/main/resources/lib/hudson/scriptConsole_zh_TW.properties core/src/site/markdown/index.md core/src/test/java/hudson/UtilTest.java core/src/test/java/hudson/model/ParametersActionTest.java core/src/test/java/hudson/util/ArgumentListBuilderTest.java plugins/pom.xml pom.xml test/pom.xml test/src/main/java/org/jvnet/hudson/test/ExtractResourceSCM.java test/src/test/java/hudson/cli/SetBuildDisplayNameCommandTest.java test/src/test/java/hudson/model/JobQueueTest.java test/src/test/java/hudson/model/MyViewsPropertyTest.java test/src/test/java/hudson/model/ProjectTest.java test/src/test/java/hudson/model/SimpleJobTest.java test/src/test/java/hudson/model/UserTest.java test/src/test/java/hudson/model/ViewTest.java test/src/test/java/hudson/model/queue/WideExecutionTest.java test/src/test/java/hudson/util/RobustReflectionConverterTest.java test/src/test/java/lib/form/RepeatableTest.java translation-tool.pl war/pom.xml war/src/main/webapp/help/project-config/custom-workspace.html http://jenkins-ci.org/commit/jenkins/d3f53b9304fc4c505c05f9be2e1abc3c4985f488 Log: Merge branch 'master' into JENKINS-29798

          Code changed in jenkins
          User: James Nord
          Path:
          changelog.html
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/b8f5342c73843b530ccbcd8bb3748d8365b9d762
          Log:
          Merge pull request #1784 from Vlatombe/JENKINS-29798

          [FIXED JENKINS-29798] Failed to listen to incoming slave connection

          Compare: https://github.com/jenkinsci/jenkins/compare/d81cd2bbf156...b8f5342c7384

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: changelog.html core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/b8f5342c73843b530ccbcd8bb3748d8365b9d762 Log: Merge pull request #1784 from Vlatombe/ JENKINS-29798 [FIXED JENKINS-29798] Failed to listen to incoming slave connection Compare: https://github.com/jenkinsci/jenkins/compare/d81cd2bbf156...b8f5342c7384

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4257
          [FIXED JENKINS-29798] Failed to listen to incoming slave connection (Revision 7dfa765d8b7562ea25b8eab6c26d8c7a5ca6ef1b)
          JENKINS-29798 Add a lock to serialize updates of tcpSlaveAgentListener (Revision 03bba487296dd7d0f44a0c235724c3b7fb41cd69)

          Result = UNSTABLE
          vincent latombe : 7dfa765d8b7562ea25b8eab6c26d8c7a5ca6ef1b
          Files :

          • changelog.html
          • core/src/main/java/jenkins/model/Jenkins.java

          vincent latombe : 03bba487296dd7d0f44a0c235724c3b7fb41cd69
          Files :

          • core/src/main/java/jenkins/model/Jenkins.java

          dogfood added a comment - Integrated in jenkins_main_trunk #4257 [FIXED JENKINS-29798] Failed to listen to incoming slave connection (Revision 7dfa765d8b7562ea25b8eab6c26d8c7a5ca6ef1b) JENKINS-29798 Add a lock to serialize updates of tcpSlaveAgentListener (Revision 03bba487296dd7d0f44a0c235724c3b7fb41cd69) Result = UNSTABLE vincent latombe : 7dfa765d8b7562ea25b8eab6c26d8c7a5ca6ef1b Files : changelog.html core/src/main/java/jenkins/model/Jenkins.java vincent latombe : 03bba487296dd7d0f44a0c235724c3b7fb41cd69 Files : core/src/main/java/jenkins/model/Jenkins.java

          ovi craciun added a comment - - edited

          I can still see this happening
          Jenkins ver. 2.190.3, windows OS

          Failed to listen to incoming agent connections
          java.net.BindException: Address already in use: bind
          at sun.nio.ch.Net.bind0(Native Method)
          at sun.nio.ch.Net.bind(Unknown Source)
          at sun.nio.ch.Net.bind(Unknown Source)
          at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
          at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
          at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
          at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:103)
          Caused: java.net.BindException: Failed to listen on port 49994 because it's already in use.
          at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:105)
          at jenkins.model.Jenkins.launchTcpSlaveAgentListener(Jenkins.java:1287)
          at jenkins.model.Jenkins.<init>(Jenkins.java:983)
          at hudson.model.Hudson.<init>(Hudson.java:85)
          at hudson.model.Hudson.<init>(Hudson.java:81)
          at hudson.WebAppMain$3.run(WebAppMain.java:233)

          the netstat reveals the port is occupied by a process id which points to services.exe
          I did not installed any new app on this server, it is just another aws instance spawned. It used to work just fine before.
          I have serious reasons to believe this is a problem with how the port is binded first and then maybe the code flow is losing it and tries to bind again without releases the initial connection?...

          ovi craciun added a comment - - edited I can still see this happening Jenkins ver. 2.190.3, windows OS Failed to listen to incoming agent connections java.net.BindException: Address already in use: bind at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Unknown Source) at sun.nio.ch.Net.bind(Unknown Source) at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:103) Caused: java.net.BindException: Failed to listen on port 49994 because it's already in use. at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:105) at jenkins.model.Jenkins.launchTcpSlaveAgentListener(Jenkins.java:1287) at jenkins.model.Jenkins.<init>(Jenkins.java:983) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233) the netstat reveals the port is occupied by a process id which points to services.exe I did not installed any new app on this server, it is just another aws instance spawned. It used to work just fine before. I have serious reasons to believe this is a problem with how the port is binded first and then maybe the code flow is losing it and tries to bind again without releases the initial connection?...

          Brendan Holmes added a comment - - edited

          Did you find a solution iamveritas? We're experiencing this port 50000 conflict too running in docker. I expect it's a race condition. Our test jenkins with the same configuration, but fewer jobs & no persistent storage doesn't experience it.

          Brendan Holmes added a comment - - edited Did you find a solution iamveritas ? We're experiencing this port 50000 conflict too running in docker. I expect it's a race condition. Our test jenkins with the same configuration, but fewer jobs & no persistent storage doesn't experience it.

          ovi craciun added a comment -

          brendanh It's been several months and I really do not remember exactly how I got around this/fixed this. I think I just changed the port to the next one and it stopped having this problem.
          We encountered numerous problems with jenkins, to my unpleasant surprise. I am still astounded at how many problems one can encounter with jenkins vs its reputation in the wild.

          ovi craciun added a comment - brendanh It's been several months and I really do not remember exactly how I got around this/fixed this. I think I just changed the port to the next one and it stopped having this problem. We encountered numerous problems with jenkins, to my unpleasant surprise. I am still astounded at how many problems one can encounter with jenkins vs its reputation in the wild.

          Yes changing the port to something other than 50000 seems to have fixed for us too. Thanks

          Brendan Holmes added a comment - Yes changing the port to something other than 50000 seems to have fixed for us too. Thanks

            vlatombe Vincent Latombe
            vlatombe Vincent Latombe
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: