Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52166

Jenkins unresponsive when using native https

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core, winstone-jetty
    • None

      We are trying to run jenkins using https as described here : https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777

       At first it appeared to work properly, but now the Jenkins web server is unresponsive. Https requests timeout, both when accessing the web pages through a browser and using the http API from other tools.

      We tried taking out the https related java parameters and it appears to work with http only.

      However, when using both http and https on different ports, Jenkins is unresponsive on both ports.

       

      Jenkins does not appear to be running out of memory. It is running dozens of threads but they don't seem to be consuming much CPU time.

      Looking at the thread dump does not seem to indicate a deadlock, at least some thread are still running.

      Furthermore, we run builds every night and they appear to be running properly. However, since we use http requests to fetch the logs, we have little more then the build result to go on.

      Outside of Jenkins, our build is not very stable but has comparable stability to what Jenkins seems to be reporting.

       

      Update 1:

      Jenkins version is 2.121.1

       

      Update 2:

      Looking at various logs. It appears that:

       - Native Https was working on May 29, which is when we started using https instead of http.

       - Jenkins was Upgraded from version 2.107.3 to version 2.121.1 on June 07.

       - We observed that Jenkins was unresponsive on June 08.

       

      Update 3:

      After reverting Jenkins to version  2.107.3,

      I can confirm that Native Https support was broken in version 2.121.1.

        1. jenkins.log
          74 kB

          [JENKINS-52166] Jenkins unresponsive when using native https

          Oleg Nenashev added a comment -

          Ugh, I didn't know that we have documentation for it. I've always thought that we support HTTPS only via external services.
          tdelisle which Jenkins version do you use?

          Oleg Nenashev added a comment - Ugh, I didn't know that we have documentation for it. I've always thought that we support HTTPS only via external services. tdelisle which Jenkins version do you use?

          Oleg Nenashev added a comment -

          The wiki page (https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777) has been created in 2017 by jthornsen and then edited by jpi.

          olamy danielbeck WDYT? Should actually support it? Or should we add a kind of disclaimer to the Wiki page ("available but not supported" or so).

          Oleg Nenashev added a comment - The wiki page ( https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777 ) has been created in 2017 by jthornsen and then edited by jpi . olamy danielbeck WDYT? Should actually support it? Or should we add a kind of disclaimer to the Wiki page ("available but not supported" or so).

          Jeff Thornsen added a comment -

          I wrote those instructions back when I was required to deploy Jenkins over HTTPS on Windows for one of my customers.  This was on a network without internet access, and they forced me to use Windows, so I figured out a way to make the Jenkins embedded webserver listen over HTTPS on port 443 natively and documented it.  Normally I would just deploy Jenkins on Linux and throw haproxy or nginx or apache httpd in front to handle the SSL.

          AFAIK that server had no issues with the web service going down.  It is likely running Jenkins LTS 2.60.3 or maybe something slightly earlier than that.

          I do notice in the the Jenkins LTS Changelog that Jenkins LTS 2.73 introduced a new version of Winstone and again in 2.121.1 so it's possible the different Winstone/Jetty versions are causing the issue?

          Jeff Thornsen added a comment - I wrote those instructions back when I was required to deploy Jenkins over HTTPS on Windows for one of my customers.  This was on a network without internet access, and they forced me to use Windows, so I figured out a way to make the Jenkins embedded webserver listen over HTTPS on port 443 natively and documented it.  Normally I would just deploy Jenkins on Linux and throw haproxy or nginx or apache httpd in front to handle the SSL. AFAIK that server had no issues with the web service going down.  It is likely running Jenkins LTS 2.60.3 or maybe something slightly earlier than that. I do notice in the the Jenkins LTS Changelog that Jenkins LTS 2.73 introduced a new version of Winstone and again in 2.121.1 so it's possible the different Winstone/Jetty versions are causing the issue?

          Oleg Nenashev added a comment -

          Thanks jthornsen!

          Yes, it may be related to Jetty upgrades. That's why I CCed olamy. We do not have test automation for such configuration, so it's hard to say when particularly it became unstable (if it is a wide issue unrelated to the reporter's configuration).

          Oleg Nenashev added a comment - Thanks jthornsen ! Yes, it may be related to Jetty upgrades. That's why I CCed olamy . We do not have test automation for such configuration, so it's hard to say when particularly it became unstable (if it is a wide issue unrelated to the reporter's configuration).

          Thierry Delisle added a comment -

          I would like to add that using native HTTPS is much more convenient in our case. 

          We run Jenkins on a university network, therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections. 

          Thierry Delisle added a comment - I would like to add that using native HTTPS is much more convenient in our case.  We run Jenkins on a university network, therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections. 

          Thierry Delisle added a comment -

          To add to jthornsen's hypothesis, it appears that the problem appear immediately after upgrading to 2.121.1, and was not present before.

          We will try to revert to the previous version and see if the problem goes away.

          Thierry Delisle added a comment - To add to jthornsen 's hypothesis, it appears that the problem appear immediately after upgrading to 2.121.1, and was not present before. We will try to revert to the previous version and see if the problem goes away.

          Daniel Beck added a comment - - edited

          therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections

          https://github.com/jenkinsci/winstone/#command-line-options has httpListenAddress, make that 127.0.0.1.

          Daniel Beck added a comment - - edited therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections https://github.com/jenkinsci/winstone/#command-line-options has httpListenAddress, make that 127.0.0.1.

            Unassigned Unassigned
            tdelisle Thierry Delisle
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: