Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67404

Jenkins Debian/Ubuntu Start Timeout Failure Mode

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • packaging
    • None
    • Any Debian / Ubuntu
    • 2.335

    Description

      Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

      I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

      I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

      Aforesaid 8 minutes is longer than systemd 5 minute start timeout causing systemd to abandon the start and report Jenkins service failure.

      Issue opened to

      1. alert Jenkins project supporters
      2. alert users who may have this issue
      3. propose timeout prevention edits

      Proposed revisions:

      1. Drop "check_tcp_port" function; a start when port in use fails with similar message.
      2. Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
      3. Replace old "netstat" with new "ss"; drops "net-utils" package dependency.

      The "check_tcp_port" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "netstat" output " [::]:8080 " doesn't match "grep" argument "0.0.0.0\:8080[[:space:]]" value; "systemd" still shows service as active.

      Attachments

        Issue Links

          Activity

            conrad_t_pino Conrad T. Pino created issue -
            conrad_t_pino Conrad T. Pino made changes -
            Field Original Value New Value
            Description Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than systemd 5 minute start timeout causing systemd to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.
            Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.
            conrad_t_pino Conrad T. Pino made changes -
            Description Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.
            Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.
            conrad_t_pino Conrad T. Pino made changes -
            Link This issue is related to JENKINS-41218 [ JENKINS-41218 ]
            conrad_t_pino Conrad T. Pino made changes -
            Comment [ common function ]
            conrad_t_pino Conrad T. Pino made changes -
            Summary Jenkins Debian/Ubuntu Start Failure Mode Jenkins Debian/Ubuntu Start Timeout Failure Mode
            conrad_t_pino Conrad T. Pino made changes -
            Description Debian/Ubuntu Jenkins init.d script has unique function "check_tcp_port" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.
            Debian/Ubuntu Jenkins init.d script has unique function "{{check_tcp_port}}" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.

            The "{{check_tcp_port}}" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "{{netstat}}" output "{{ [::]:8080 }}" doesn't match "{{grep}}" argument "{{0.0.0.0\:8080[[:space:]]}}" value; "{{systemd}}" still shows service as active.
            conrad_t_pino Conrad T. Pino made changes -
            Description Debian/Ubuntu Jenkins init.d script has unique function "{{check_tcp_port}}" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.

            The "{{check_tcp_port}}" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "{{netstat}}" output "{{ [::]:8080 }}" doesn't match "{{grep}}" argument "{{0.0.0.0\:8080[[:space:]]}}" value; "{{systemd}}" still shows service as active.
            Debian/Ubuntu Jenkins init.d script has unique function "{{check_tcp_port}}" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.

            The "{{check_tcp_port}}" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "{{netstat}}" output "{{ [::]:8080}} " doesn't match "{{grep}}" argument "{{0.0.0.0\:8080[[:space:]]}}" value; "{{systemd}}" still shows service as active.
            conrad_t_pino Conrad T. Pino made changes -
            Description Debian/Ubuntu Jenkins init.d script has unique function "{{check_tcp_port}}" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "check_tcp_port" causes startup failure because "netstat --listen --numeric-ports" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.

            The "{{check_tcp_port}}" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "{{netstat}}" output "{{ [::]:8080}} " doesn't match "{{grep}}" argument "{{0.0.0.0\:8080[[:space:]]}}" value; "{{systemd}}" still shows service as active.
            Debian/Ubuntu Jenkins init.d script has unique function "{{check_tcp_port}}" absent from RPM and SUSE scripts to detect listener on host:port pair.

            I have Ubuntu 20.04 LTS host with 13 IPv4 and 14 IPv6 addresses where DNS reverse lookups timeout for many addresses.

            I find Jenkins init "{{check_tcp_port}}" causes startup failure because "{{netstat --listen --numeric-ports}}" takes 8 minutes to complete.

            Aforesaid 8 minutes is longer than {{systemd}} 5 minute start timeout causing {{systemd}} to abandon the start and report Jenkins service failure.

            Issue opened to
             # alert Jenkins project supporters
             # alert users who may have this issue
             # propose timeout prevention edits

            Proposed revisions:
             # Drop "check_tcp_port" function; a start when port in use fails with similar message.
             # Revise listener enumeration for numeric mode to avoid DNS timeouts; details later.
             # Replace old "{{netstat}}" with new "{{ss}}"; drops "{{net-utils}}" package dependency.

            The "{{check_tcp_port}}" function fails to detect used port on Ubuntu 20.04 LTS OpenJDK 1.8.0_292 since "{{netstat}}" output "{{ [::]:8080}} " doesn't match "{{grep}}" argument "{{0.0.0.0\:8080[[:space:]]}}" value; "{{systemd}}" still shows service as active.
            markewaite Mark Waite added a comment -

            I confirmed with the switch from System V init to systemd that is being evaluated in pull request 266 that the winstone java.net.BindException that is reported when the port is already busy is sufficient to stop the service. The systemctl start jenkins command correctly reports that the service did not start and journalctl -u jenkins.service reports:

            Feb 05 19:28:06 debian10-c jenkins[23189]: java.net.BindException: Address already in use
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.Net.bind0(Native Method)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.Net.bind(Net.java:459)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.Net.bind(Net.java:448)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
            Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.server.Server.doStart(Server.java:401)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at winstone.Launcher.<init>(Launcher.java:192)
            Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to start Jetty
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at winstone.Launcher.<init>(Launcher.java:194)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at winstone.Launcher.main(Launcher.java:369)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at Main._main(Main.java:304)
            Feb 05 19:28:06 debian10-c jenkins[23189]:         at Main.main(Main.java:108)
            Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE
            Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Failed with result 'exit-code'.
            Feb 05 19:28:06 debian10-c systemd[1]: Failed to start Jenkins Continuous Integration Server.
            

            conrad_t_pino could you verify that the most recent PR-266 build resolves it for you as well? We'd like more validation of the switch from System V init to systemd and this issue is a good chance to check another aspect of that transition.

            markewaite Mark Waite added a comment - I confirmed with the switch from System V init to systemd that is being evaluated in pull request 266 that the winstone java.net.BindException that is reported when the port is already busy is sufficient to stop the service. The systemctl start jenkins command correctly reports that the service did not start and journalctl -u jenkins.service reports: Feb 05 19:28:06 debian10-c jenkins[23189]: java.net.BindException: Address already in use Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.Net.bind0(Native Method) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.Net.bind(Net.java:459) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.Net.bind(Net.java:448) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344) Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080 Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.server.Server.doStart(Server.java:401) Feb 05 19:28:06 debian10-c jenkins[23189]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) Feb 05 19:28:06 debian10-c jenkins[23189]: at winstone.Launcher.<init>(Launcher.java:192) Feb 05 19:28:06 debian10-c jenkins[23189]: Caused: java.io.IOException: Failed to start Jetty Feb 05 19:28:06 debian10-c jenkins[23189]: at winstone.Launcher.<init>(Launcher.java:194) Feb 05 19:28:06 debian10-c jenkins[23189]: at winstone.Launcher.main(Launcher.java:369) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Feb 05 19:28:06 debian10-c jenkins[23189]: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Feb 05 19:28:06 debian10-c jenkins[23189]: at Main._main(Main.java:304) Feb 05 19:28:06 debian10-c jenkins[23189]: at Main.main(Main.java:108) Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Main process exited, code=exited, status=1/FAILURE Feb 05 19:28:06 debian10-c systemd[1]: jenkins.service: Failed with result 'exit-code'. Feb 05 19:28:06 debian10-c systemd[1]: Failed to start Jenkins Continuous Integration Server. conrad_t_pino could you verify that the most recent PR-266 build resolves it for you as well? We'd like more validation of the switch from System V init to systemd and this issue is a good chance to check another aspect of that transition.
            basil Basil Crow added a comment -

            As of 2.335, the systemd service unit no longer calls netstat(1).

            basil Basil Crow added a comment - As of 2.335, the systemd service unit no longer calls netstat(1) .
            basil Basil Crow made changes -
            Released As 2.335
            Assignee Basil Crow [ basil ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            markewaite Mark Waite made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              basil Basil Crow
              conrad_t_pino Conrad T. Pino
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: