• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • Jenkins 2.261 and 2.262
    • Jenkins 2.263

      Updating to 2.261 and even 2.262 the URL for some links are injecting the port number ":80" which in our instance is invalid.  This prevents pages from loading. For example the "search" at top of screen when finding a job and trying to go to the job will change the URL from "https://jenkins.company.com/" to "https://jenkins.company.com:80/job/jobname/".  This makes pages not render properly and user needs to remove ":80" from URL to continue.

      Another instance is clicking Manage Jenkins -> Manage Plugins goes from "https://jenkins.company.com/manage" to "https://jenkins.company.com:80/pluginManager/".

      These are just a couple of instances but have run into more.  I couldn't easily identity what change in 2.261 that affected this behavior.

       

          [JENKINS-63958] Invalid URL containing port

          Richard Fearn added a comment - - edited

          This only seems to occur when Jenkins is being accessed (a) over HTTPS and (b) on port 443.

          This can be reproduced quite easily without nginx. I use iptables to forward 443 to 8443, but even that isn't required if Jenkins is allowed to bind to 443, e.g. by doing:

          sudo sysctl -w net.ipv4.ip_unprivileged_port_start=443

          Then:

          java -jar jenkins.war --httpPort=-1 --httpsPort=443

          will start it up listening on port 443 (no nginx, no iptables redirect).

          The problem seems to originate in org.eclipse.jetty.server.ForwardedRequestCustomizer.customize. It's trying to figure out what the real protocol/host/port are. proto starts off set to "http" and never changes to "https". port is temporarily set to 0 (the HttpURI object contains 0, since it isn't specified in a URL like "https://machinename/"). Finally with port set to 0 and proto still "http" it decides to use the default port for HTTP - 80. The port number in the request is set to this value.

          This doesn't seem to cause problems except if Jenkins needs to send a redirect - this is why "Manage Plugins" is broken. The URL requested when you click that is /pluginManager (no trailing slash). Jenkins redirects to /pluginManager/ (with a trailing slash), but the URL sent back to the browser will include the now-incorrect port number from the request.

          Richard Fearn added a comment - - edited This only seems to occur when Jenkins is being accessed (a) over HTTPS and (b) on port 443. This can be reproduced quite easily without nginx. I use iptables to forward 443 to 8443, but even that isn't required if Jenkins is allowed to bind to 443, e.g. by doing: sudo sysctl -w net.ipv4.ip_unprivileged_port_start=443 Then: java -jar jenkins.war --httpPort=-1 --httpsPort=443 will start it up listening on port 443 (no nginx, no iptables redirect). The problem seems to originate in org.eclipse.jetty.server.ForwardedRequestCustomizer.customize . It's trying to figure out what the real protocol/host/port are. proto starts off set to "http" and never changes to "https". port is temporarily set to 0 (the HttpURI object contains 0, since it isn't specified in a URL like "https://machinename/"). Finally with port set to 0 and proto still "http" it decides to use the default port for HTTP - 80. The port number in the request is set to this value. This doesn't seem to cause problems except if Jenkins needs to send a redirect - this is why "Manage Plugins" is broken. The URL requested when you click that is /pluginManager (no trailing slash). Jenkins redirects to /pluginManager/ (with a trailing slash), but the URL sent back to the browser will include the now-incorrect port number from the request.

          Richard Fearn added a comment -

          As zbynek pointed out yesterday, https://github.com/eclipse/jetty.project/pull/5419 may be related to this. https://github.com/eclipse/jetty.project/issues/5443 looks more relevant, however, because after that PR ForwardedRequestCustomizer doesn't do anything if there aren't any Forwarded / X-Forwarded-* / etc. headers.

          I've tried running Jenkins 2.262 using the latest version of ForwardedRequestCustomizer from Jetty's 9.4.x branch, and it does seem to fix this problem.

          With any luck there will be a 9.4.33 release soon...

          Richard Fearn added a comment - As zbynek pointed out yesterday, https://github.com/eclipse/jetty.project/pull/5419 may be related to this. https://github.com/eclipse/jetty.project/issues/5443 looks more relevant, however, because after that PR ForwardedRequestCustomizer doesn't do anything if there aren't any Forwarded / X-Forwarded-* / etc. headers. I've tried running Jenkins 2.262 using the latest version of ForwardedRequestCustomizer from Jetty's 9.4.x branch, and it does seem to fix this problem. With any luck there will be a 9.4.33 release soon...

          Olivier Lamy added a comment -

           new release might be very soon

          Olivier Lamy added a comment -  new release might be very soon

          Bruce Coveny added a comment -

          olamy

          We have the instance installed on Windows and using the service to start Jenkins.  Only changes is in the jenkins.xml in the home folder for the arguments to be as follows:

          <arguments>-Xrs -Xmx2048m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=80 --httpsPort=443 --webroot="%BASE%\war"</arguments>
          

          Bruce Coveny added a comment - olamy We have the instance installed on Windows and using the service to start Jenkins.  Only changes is in the jenkins.xml in the home folder for the arguments to be as follows: <arguments>-Xrs -Xmx2048m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=80 --httpsPort=443 --webroot= "%BASE%\war" </arguments>

          Oleg Nenashev added a comment -

           We plan to either release a fix or rollback Jetty in the tomorrow's weekly

          Oleg Nenashev added a comment -  We plan to either release a fix or rollback Jetty in the tomorrow's weekly

          Oleg Nenashev added a comment -

          https://github.com/jenkinsci/jenkins/pull/5017 as a last resort rollback fix. Taking the confirmation by richardfearn, I believe our best option is to pick up the Jetty release which is expected soon. If it does not happen tomorrow until the weekly release, we might cut another out of order weekly release once new Jetty is available.

           

          It worth mentioning that yet another Jetty upgrade would increase risk of additional reressions in the weekly. Also, it might be impediment for the LTS baseline selection which is expected to happen this week.

           

          Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/5017  as a last resort rollback fix. Taking the confirmation by richardfearn , I believe our best option is to pick up the Jetty release which is expected soon. If it does not happen tomorrow until the weekly release, we might cut another out of order weekly release once new Jetty is available.   It worth mentioning that yet another Jetty upgrade would increase risk of additional reressions in the weekly. Also, it might be impediment for the LTS baseline selection which is expected to happen this week.  

          Richard Fearn added a comment -

          Everything seems to be working fine again with 2.263. Thanks to everyone who has worked on this

          Richard Fearn added a comment - Everything seems to be working fine again with 2.263. Thanks to everyone who has worked on this

          Can confirm it's working again.

          Fabian Grutschus added a comment - Can confirm it's working again.

          Bruce Coveny added a comment -

          Also confirming that 2.263 is back to functioning as expected. Thanks oleg_nenashev

          Bruce Coveny added a comment - Also confirming that 2.263 is back to functioning as expected. Thanks oleg_nenashev ! 

          Oleg Nenashev added a comment -

          Marking it as resolved based on the feedback. Thanks all!

          Oleg Nenashev added a comment - Marking it as resolved based on the feedback. Thanks all!

            Unassigned Unassigned
            bcoveny Bruce Coveny
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: