Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67404

Jenkins Debian/Ubuntu Start Timeout Failure Mode

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • packaging
    • None
    • Any Debian / Ubuntu
    • 2.335

      Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

      I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

      I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

      Aforesaid 8 minutes is longer than systemd 5 minute start timeout causing systemd to abandon the start and report Jenkins service failure.

      Issue opened to

      1. alert Jenkins project supporters
      2. alert users who may have this issue
      3. propose timeout prevention edits

      Proposed revisions:

      1. Drop "check_tcp_port" function; a start when port in use fails with similar message.
      2. Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
      3. Replace old "netstat" with new "ss"; drops "net-utils" package dependency.

      The "check_tcp_port" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "netstat" output " [::]:8080 " doesn't match "grep" argument "0.0.0.0\:8080[[:space:]]" value; "systemd" still shows service as active.

          [JENKINS-67404] Jenkins Debian/Ubuntu Start Timeout Failure Mode

          Mark Waite added a comment -

          I confirmed with the switch from System V init to systemd that is being evaluated in pull request 266 that the winstone java.net.BindException that is reported when the port is already busy is sufficient to stop the service. The systemctl start jenkins command correctly reports that the service did not start and journalctl -u jenkins.service reports:

          Feb 05 19:28:06 debian10-c jenkins[23189]: java.net.BindException: Address already in use
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.Net.bind0(Native Method)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.Net.bind(Net.java:459)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.Net.bind(Net.java:448)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
          Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.Server.doStart(Server.java:401)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at winstone.Launcher.<init>(Launcher.java:192)
          Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to start Jetty
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at winstone.Launcher.<init>(Launcher.java:194)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at winstone.Launcher.main(Launcher.java:369)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at Main._main(Main.java:304)
          Feb 05 19:28:06 debian10-c jenkins[23189]:         at Main.main(Main.java:108)
          Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE
          Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Failed with result 'exit-code'.
          Feb 05 19:28:06 debian10-c systemd[1]: Failed to start Jenkins Continuous Integration Server.
          

          conrad_t_pino could you verify that the most recent PR-266 build resolves it for you as well? We'd like more validation of the switch from System V init to systemd and this issue is a good chance to check another aspect of that transition.

          Mark Waite added a comment - I confirmed with the switch from System V init to systemd that is being evaluated in pull request 266 that the winstone java.net.BindException that is reported when the port is already busy is sufficient to stop the service. The systemctl start jenkins command correctly reports that the service did not start and journalctl -u jenkins.service reports: Feb 05 19:28:06 debian10-c jenkins[23189]: java.net.BindException: Address already in use Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.Net.bind0(Native Method) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.Net.bind(Net.java:459) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.Net.bind(Net.java:448) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344) Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080 Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.Server.doStart(Server.java:401) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Feb 05 19:28:06 debian10-c jenkins[23189]: at winstone.Launcher.<init>(Launcher.java:192) Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to start Jetty Feb 05 19:28:06 debian10-c jenkins[23189]: at winstone.Launcher.<init>(Launcher.java:194) Feb 05 19:28:06 debian10-c jenkins[23189]: at winstone.Launcher.main(Launcher.java:369) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Feb 05 19:28:06 debian10-c jenkins[23189]: at Main._main(Main.java:304) Feb 05 19:28:06 debian10-c jenkins[23189]: at Main.main(Main.java:108) Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Failed with result 'exit-code'. Feb 05 19:28:06 debian10-c systemd[1]: Failed to start Jenkins Continuous Integration Server. conrad_t_pino could you verify that the most recent PR-266 build resolves it for you as well? We'd like more validation of the switch from System V init to systemd and this issue is a good chance to check another aspect of that transition.

          Basil Crow added a comment -

          As of 2.335, the systemd service unit no longer calls netstat(1).

          Basil Crow added a comment - As of 2.335, the systemd service unit no longer calls netstat(1) .

            basil Basil Crow
            conrad_t_pino Conrad T. Pino
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: