• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • Jenkins 2.261 and 2.262
    • Jenkins 2.263

      Updating to 2.261 and even 2.262 the URL for some links are injecting the port number ":80" which in our instance is invalid.  This prevents pages from loading. For example the "search" at top of screen when finding a job and trying to go to the job will change the URL from "https://jenkins.company.com/" to "https://jenkins.company.com:80/job/jobname/".  This makes pages not render properly and user needs to remove ":80" from URL to continue.

      Another instance is clicking Manage Jenkins -> Manage Plugins goes from "https://jenkins.company.com/manage" to "https://jenkins.company.com:80/pluginManager/".

      These are just a couple of instances but have run into more.  I couldn't easily identity what change in 2.261 that affected this behavior.

       

          [JENKINS-63958] Invalid URL containing port

          Oleg Nenashev added a comment -

          bcoveny what Jenkins version did you use before the upgrade?

          Oleg Nenashev added a comment - bcoveny what Jenkins version did you use before the upgrade?

          Bruce Coveny added a comment -

          oleg_nenashev we were using 2.260 which is where I know it was a change in 2.261 which caused the issue.  We have 3 other instances running...2 with 2.260 and 1 with 2.258. 

          Bruce Coveny added a comment - oleg_nenashev we were using 2.260 which is where I know it was a change in 2.261 which caused the issue.  We have 3 other instances running...2 with 2.260 and 1 with 2.258. 

          Oleg Nenashev added a comment -

          Could it be somehow related to the Jetty upgrade olamy? I do not see anything else which would be immediately suspicious

          bcoveny could you please confirm that you are using the embedded Jetty web server?

          Oleg Nenashev added a comment - Could it be somehow related to the Jetty upgrade olamy ? I do not see anything else which would be immediately suspicious bcoveny could you please confirm that you are using the embedded Jetty web server?

          Olivier Lamy added a comment -

          sound possible but I'm not sure how the url is injected (different injection in different places)

          Olivier Lamy added a comment - sound possible but I'm not sure how the url is injected (different injection in different places)

          Zbynek Konecny added a comment - - edited

          I can confirm this happens on Jenkins installed via debian package, running without any proxy on HTTPS port 443. No problem with 2.260. The process is started like this

           /usr/bin/java -Dhudson.DNSMultiCast.disabled=true -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpsPort=443 --ajp13Port=-1 --httpsCertificate=/foo/bar.pem --httpsPrivateKey=/foo/bar.key 

          Also the "reverse proxy broken" administrative monitor became active. Since this was a production Jenkins install I had to roll back to 260, but if more info is needed I can check again.

          Zbynek Konecny added a comment - - edited I can confirm this happens on Jenkins installed via debian package, running without any proxy on HTTPS port 443. No problem with 2.260. The process is started like this /usr/bin/java -Dhudson.DNSMultiCast.disabled= true -Djava.awt.headless= true -jar /usr/share/jenkins/jenkins.war --webroot=/ var /cache/jenkins/war --httpsPort=443 --ajp13Port=-1 --httpsCertificate=/foo/bar.pem --httpsPrivateKey=/foo/bar.key Also the "reverse proxy broken" administrative monitor became active. Since this was a production Jenkins install I had to roll back to 260, but if more info is needed I can check again.

          Olivier Lamy added a comment -

          something not clear. Are you guys using proxy? how is it configured?

          Olivier Lamy added a comment - something not clear. Are you guys using proxy? how is it configured?

          Richard Fearn added a comment - - edited

          Same issue here. 2.260 was fine; 2.261 has this problem. I have Jenkins running on port 8443 with an iptables redirect from 443 to 8443. So I always access the instance using https://mydomain/ and the "Jenkins URL" in the settings is also set to "https://mydomain/".

          One place where the problem occurs is enabling/disabling a project - clicking the Enable/Disable button loads https://mydomain:80/job/JOBNAME/. The project does get enabled/disabled, but the redirect to that URL (including ":80") is wrong.

          Richard Fearn added a comment - - edited Same issue here. 2.260 was fine; 2.261 has this problem. I have Jenkins running on port 8443 with an iptables redirect from 443 to 8443. So I always access the instance using https://mydomain/ and the "Jenkins URL" in the settings is also set to "https://mydomain/". One place where the problem occurs is enabling/disabling a project - clicking the Enable/Disable button loads https://mydomain:80/job/JOBNAME/ . The project does get enabled/disabled, but the redirect to that URL (including ":80") is wrong.

          Can confirm this too. Running Jenkins on Docker with port redirect 443->8080, the correct FQDN is configured and HTTPS works perfect. The error occurs on various places, primarily after the login page (login works btw) and when starting a job manually (POST works, but the redirect after POST is wrong).

          Fabian Grutschus added a comment - Can confirm this too. Running Jenkins on Docker with port redirect 443->8080, the correct FQDN is configured and HTTPS works perfect. The error occurs on various places, primarily after the login page (login works btw) and when starting a job manually (POST works, but the redirect after POST is wrong).

          Bruce Coveny added a comment -

          oleg_nenashev - Yes we are using the embedded Jetty Server for running Jenkins.

          As stated by zbynek we are also starting the instance the same way setting only port for HTTPS and HTTP but only allow users to use HTTP as normal 443 port.  

          Also as stated by others it is random spots where the port is used and not consistent. 

          Bruce Coveny added a comment - oleg_nenashev - Yes we are using the embedded Jetty Server for running Jenkins. As stated by zbynek we are also starting the instance the same way setting only port for HTTPS and HTTP but only allow users to use HTTP as normal 443 port.   Also as stated by others it is random spots where the port is used and not consistent. 

          Zbynek Konecny added a comment - - edited

          olamy no proxy, on port 80 there is a simple NginX instance redirecting traffic to https://. But Jenkins should not be aware of it, since that's a redirect to a different URL (http -> https) rather than proxy. Could this be related to https://github.com/eclipse/jetty.project/pull/5419 (seems to be the only recent PR that has "port" in title)?

          Zbynek Konecny added a comment - - edited olamy no proxy, on port 80 there is a simple NginX instance redirecting traffic to https:// . But Jenkins should not be aware of it, since that's a redirect to a different URL (http -> https) rather than proxy. Could this be related to https://github.com/eclipse/jetty.project/pull/5419 (seems to be the only recent PR that has "port" in title)?

          Mike Litwak added a comment -

          This is happening on my Jenkins server, too:

          • Some Jenkins web UI links (or just logging in) result in "Unable to connect" errors in the web browser, and the URL's have had :80 appended to the base URL (regardless whether http is enabled)
          • The Reverse proxy set up is broken message under Manage Jenkins

          Started happening immediately after doing (1) plugin updates, followed by (2) an "automatic" upgrade from 2.242 to 2.261 (initiated via the web UI).

          My Jenkins server runs Jenkins as a service on Windows Server 2012 R2.  The base URL is https://jenkins (running on TCP port 443).  No proxy is configured or running.  Neither IIS nor Apache Web Server are apparently running – just the Jenkins service.  Computer's hostname (NetBIOS name) is just JENKINS, accessible over my local LAN only.  Using a self-signed SSL certificate (valid and present in the Java JRE cacerts file as a trusted cert).

          This Jenkins installation has been running fine for years and went through several previous upgrades without issues.

          Mike Litwak added a comment - This is happening on my Jenkins server, too: Some Jenkins web UI links (or just logging in) result in "Unable to connect" errors in the web browser, and the URL's have had :80 appended to the base URL (regardless whether http is enabled) The Reverse proxy set up is broken message under Manage Jenkins .  Started happening immediately after doing (1) plugin updates, followed by (2) an "automatic" upgrade from 2.242 to 2.261 (initiated via the web UI). My Jenkins server runs Jenkins as a service on Windows Server 2012 R2.  The base URL is https://jenkins (running on TCP port 443).  No proxy is configured or running.  Neither IIS nor Apache Web Server are apparently running – just the Jenkins service.  Computer's hostname (NetBIOS name) is just JENKINS, accessible over my local LAN only.  Using a self-signed SSL certificate (valid and present in the Java JRE cacerts file as a trusted cert). This Jenkins installation has been running fine for years and went through several previous upgrades without issues.

          Olivier Lamy added a comment -

          definitely need some investigation especially to understand why it;'s failing in some places but not others.....

          oleg_nenashev maybe a winstone release with downgrading jetty? wdyt?

          Olivier Lamy added a comment - definitely need some investigation especially to understand why it;'s failing in some places but not others..... oleg_nenashev maybe a winstone release with downgrading jetty? wdyt?

          Oleg Nenashev added a comment -

          Works for me olamy. It means that the Jetty upgrade will be postponed until the next LTS, but it looks feasible anyway

          Oleg Nenashev added a comment - Works for me olamy . It means that the Jetty upgrade will be postponed until the next LTS, but it looks feasible anyway

          Olivier Lamy added a comment -

          i may have an idea about what the problem is. Might be due to a bad configuration in Winstone which has been highlighted by a fix in Jetty.

          I will need to more testing next week (setup ngnix etc...)

          bcoveny how do you start Jenkins?

          Olivier Lamy added a comment - i may have an idea about what the problem is. Might be due to a bad configuration in Winstone which has been highlighted by a fix in Jetty. I will need to more testing next week (setup ngnix etc...) bcoveny how do you start Jenkins?

          Richard Fearn added a comment - - edited

          This only seems to occur when Jenkins is being accessed (a) over HTTPS and (b) on port 443.

          This can be reproduced quite easily without nginx. I use iptables to forward 443 to 8443, but even that isn't required if Jenkins is allowed to bind to 443, e.g. by doing:

          sudo sysctl -w net.ipv4.ip_unprivileged_port_start=443

          Then:

          java -jar jenkins.war --httpPort=-1 --httpsPort=443

          will start it up listening on port 443 (no nginx, no iptables redirect).

          The problem seems to originate in org.eclipse.jetty.server.ForwardedRequestCustomizer.customize. It's trying to figure out what the real protocol/host/port are. proto starts off set to "http" and never changes to "https". port is temporarily set to 0 (the HttpURI object contains 0, since it isn't specified in a URL like "https://machinename/"). Finally with port set to 0 and proto still "http" it decides to use the default port for HTTP - 80. The port number in the request is set to this value.

          This doesn't seem to cause problems except if Jenkins needs to send a redirect - this is why "Manage Plugins" is broken. The URL requested when you click that is /pluginManager (no trailing slash). Jenkins redirects to /pluginManager/ (with a trailing slash), but the URL sent back to the browser will include the now-incorrect port number from the request.

          Richard Fearn added a comment - - edited This only seems to occur when Jenkins is being accessed (a) over HTTPS and (b) on port 443. This can be reproduced quite easily without nginx. I use iptables to forward 443 to 8443, but even that isn't required if Jenkins is allowed to bind to 443, e.g. by doing: sudo sysctl -w net.ipv4.ip_unprivileged_port_start=443 Then: java -jar jenkins.war --httpPort=-1 --httpsPort=443 will start it up listening on port 443 (no nginx, no iptables redirect). The problem seems to originate in org.eclipse.jetty.server.ForwardedRequestCustomizer.customize . It's trying to figure out what the real protocol/host/port are. proto starts off set to "http" and never changes to "https". port is temporarily set to 0 (the HttpURI object contains 0, since it isn't specified in a URL like "https://machinename/"). Finally with port set to 0 and proto still "http" it decides to use the default port for HTTP - 80. The port number in the request is set to this value. This doesn't seem to cause problems except if Jenkins needs to send a redirect - this is why "Manage Plugins" is broken. The URL requested when you click that is /pluginManager (no trailing slash). Jenkins redirects to /pluginManager/ (with a trailing slash), but the URL sent back to the browser will include the now-incorrect port number from the request.

          Richard Fearn added a comment -

          As zbynek pointed out yesterday, https://github.com/eclipse/jetty.project/pull/5419 may be related to this. https://github.com/eclipse/jetty.project/issues/5443 looks more relevant, however, because after that PR ForwardedRequestCustomizer doesn't do anything if there aren't any Forwarded / X-Forwarded-* / etc. headers.

          I've tried running Jenkins 2.262 using the latest version of ForwardedRequestCustomizer from Jetty's 9.4.x branch, and it does seem to fix this problem.

          With any luck there will be a 9.4.33 release soon...

          Richard Fearn added a comment - As zbynek pointed out yesterday, https://github.com/eclipse/jetty.project/pull/5419 may be related to this. https://github.com/eclipse/jetty.project/issues/5443 looks more relevant, however, because after that PR ForwardedRequestCustomizer doesn't do anything if there aren't any Forwarded / X-Forwarded-* / etc. headers. I've tried running Jenkins 2.262 using the latest version of ForwardedRequestCustomizer from Jetty's 9.4.x branch, and it does seem to fix this problem. With any luck there will be a 9.4.33 release soon...

          Olivier Lamy added a comment -

           new release might be very soon

          Olivier Lamy added a comment -  new release might be very soon

          Bruce Coveny added a comment -

          olamy

          We have the instance installed on Windows and using the service to start Jenkins.  Only changes is in the jenkins.xml in the home folder for the arguments to be as follows:

          <arguments>-Xrs -Xmx2048m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=80 --httpsPort=443 --webroot="%BASE%\war"</arguments>
          

          Bruce Coveny added a comment - olamy We have the instance installed on Windows and using the service to start Jenkins.  Only changes is in the jenkins.xml in the home folder for the arguments to be as follows: <arguments>-Xrs -Xmx2048m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=80 --httpsPort=443 --webroot= "%BASE%\war" </arguments>

          Oleg Nenashev added a comment -

           We plan to either release a fix or rollback Jetty in the tomorrow's weekly

          Oleg Nenashev added a comment -  We plan to either release a fix or rollback Jetty in the tomorrow's weekly

          Oleg Nenashev added a comment -

          https://github.com/jenkinsci/jenkins/pull/5017 as a last resort rollback fix. Taking the confirmation by richardfearn, I believe our best option is to pick up the Jetty release which is expected soon. If it does not happen tomorrow until the weekly release, we might cut another out of order weekly release once new Jetty is available.

           

          It worth mentioning that yet another Jetty upgrade would increase risk of additional reressions in the weekly. Also, it might be impediment for the LTS baseline selection which is expected to happen this week.

           

          Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/5017  as a last resort rollback fix. Taking the confirmation by richardfearn , I believe our best option is to pick up the Jetty release which is expected soon. If it does not happen tomorrow until the weekly release, we might cut another out of order weekly release once new Jetty is available.   It worth mentioning that yet another Jetty upgrade would increase risk of additional reressions in the weekly. Also, it might be impediment for the LTS baseline selection which is expected to happen this week.  

          Richard Fearn added a comment -

          Everything seems to be working fine again with 2.263. Thanks to everyone who has worked on this

          Richard Fearn added a comment - Everything seems to be working fine again with 2.263. Thanks to everyone who has worked on this

          Can confirm it's working again.

          Fabian Grutschus added a comment - Can confirm it's working again.

          Bruce Coveny added a comment -

          Also confirming that 2.263 is back to functioning as expected. Thanks oleg_nenashev

          Bruce Coveny added a comment - Also confirming that 2.263 is back to functioning as expected. Thanks oleg_nenashev ! 

          Oleg Nenashev added a comment -

          Marking it as resolved based on the feedback. Thanks all!

          Oleg Nenashev added a comment - Marking it as resolved based on the feedback. Thanks all!

            Unassigned Unassigned
            bcoveny Bruce Coveny
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: