Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73760

Updates fail due to invalid JSON from HTTP updates.jenkins.io (HTTPS is ok)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core

      Updates currently fail due to the corresponding endpoint at http://updates.jenkins-ci.org/update-center.json (redirected to https://westeurope.cloudflare.jenkins.io/dynamic-2.476/update-center.json) returning invalid JSON (first characters shown here):

      updateCenter.post(
      {"connectionCheckUrl":"https://www.go
      

      Stacktrace on startup:

      java.io.IOException: Could not find JSON in http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476
              at hudson.model.DownloadService.loadJSON(DownloadService.java:129)
              at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:217)
              at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:212)
              at hudson.model.UpdateCenter.updateAllSitesNow(UpdateCenter.java:2866)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
              at java.base/java.lang.Thread.run(Thread.java:840)
      
      

      Firefox complains about this as well:

      SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
      

          [JENKINS-73760] Updates fail due to invalid JSON from HTTP updates.jenkins.io (HTTPS is ok)

          Stefan created issue -
          Damien Duportal made changes -
          Assignee New: Damien Duportal [ dduportal ]

          Hi! Most probably an unexpected consequence of https://github.com/jenkins-infra/helpdesk/issues/2649. We are working on it

          Damien Duportal added a comment - Hi! Most probably an unexpected consequence of https://github.com/jenkins-infra/helpdesk/issues/2649 . We are working on it

          Hi stefan6419846 could you retry to see if our fixes did correct the problem?

          Damien Duportal added a comment - Hi stefan6419846 could you retry to see if our fixes did correct the problem?

          On the Jenkins infra side: we do not see any error when retrieving the UC from our 2.476 Jenkins controllers. New files are published.

          Keeping the issue open until stefan6419846can confirm whether or not it works for them

          Damien Duportal added a comment - On the Jenkins infra side: we do not see any error when retrieving the UC from our 2.476 Jenkins controllers. New files are published. Keeping the issue open until stefan6419846 can confirm whether or not it works for them

          Stefan added a comment -

          I just logged into the system/machine again and I am still seeing this issue. Is there any caching going on or something which would fix itself after a server restart? (I will probably try again tomorrow when I would be able to reboot the instance as well - at the moment the build queue is full and thus I do not want to reboot it directly.)

          Stefan added a comment - I just logged into the system/machine again and I am still seeing this issue. Is there any caching going on or something which would fix itself after a server restart? (I will probably try again tomorrow when I would be able to reboot the instance as well - at the moment the build queue is full and thus I do not want to reboot it directly.)

          stefan6419846Jenkins caches the file for 24 hours, unless you restart OR you tell it to check: Manage Jenkins -> Plugins -> click on the grey button with the rounded arrow on top right, next to "Update" blue button.

          Damien Duportal added a comment - stefan6419846 Jenkins caches the file for 24 hours, unless you restart OR you tell it to check: Manage Jenkins -> Plugins -> click on the grey button with the rounded arrow on top right, next to "Update" blue button.

          Stefan added a comment -

          Then I used the correct button already for retrying. I just restarted the server and it still fails.

          Nevertheless, I have been able to further debug this using the script console and a copy of https://github.com/jenkinsci/jenkins/blob/65f374c35a3caf9b3a5aacff442edb552a0f4141/core/src/main/java/hudson/model/DownloadService.java#L116-L132 and apparently somehow the redirect does not work correctly.

          import org.apache.commons.io.IOUtils;
          import java.nio.charset.StandardCharsets;
          URLConnection con = ProxyConfiguration.open(new URL("http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476"));
          if (con instanceof HttpURLConnection) {
              // prevent problems from misbehaving plugins disabling redirects by default
              ((HttpURLConnection) con).setInstanceFollowRedirects(true);
          }
          InputStream is = con.getInputStream()
          String result = IOUtils.toString(is, StandardCharsets.UTF_8);
          println result
          
          <html>
          <head><title>308 Permanent Redirect</title></head>
          <body>
          <center><h1>308 Permanent Redirect</h1></center>
          <hr><center>nginx</center>
          </body>
          </html> 
          

          Stefan added a comment - Then I used the correct button already for retrying. I just restarted the server and it still fails. Nevertheless, I have been able to further debug this using the script console and a copy of https://github.com/jenkinsci/jenkins/blob/65f374c35a3caf9b3a5aacff442edb552a0f4141/core/src/main/java/hudson/model/DownloadService.java#L116-L132 and apparently somehow the redirect does not work correctly. import org.apache.commons.io.IOUtils; import java.nio.charset.StandardCharsets; URLConnection con = ProxyConfiguration.open( new URL( "http: //updates.jenkins-ci.org/update-center.json?id= default &version=2.476" )); if (con instanceof HttpURLConnection) {     // prevent problems from misbehaving plugins disabling redirects by default     ((HttpURLConnection) con).setInstanceFollowRedirects( true ); } InputStream is = con.getInputStream() String result = IOUtils.toString(is, StandardCharsets.UTF_8); println result <html> <head> <title> 308 Permanent Redirect </title> </head> <body> <center> <h1> 308 Permanent Redirect </h1> </center> <hr> <center> nginx </center> </body> </html>  

          Stefan added a comment - - edited

          I most likely have found the issue: HttpUrlConnection will not upgrade the connection from HTTP to HTTPS and thus is being stuck. I have been able to verify this using curl:

          stefan@localhost:~> curl -I "http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476"
          HTTP/1.1 308 Permanent Redirect
          Date: Thu, 12 Sep 2024 10:29:33 GMT
          Content-Type: text/html
          Content-Length: 164
          Connection: keep-alive
          Location: https://updates.jenkins-ci.org/update-center.json?id=default&version=2.476
          

          Corresponding JDK issue: https://bugs.openjdk.org/browse/JDK-4620571?focusedId=12159233&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-12159233

          I understand that it is desirable to do everything over HTTPS, but apparently this has not been enforced with the previous update site and some older Jenkins configurations break now. Judging from doing some git blames, everyone using the same Jenkins (base) configuration since version < 2.77 (released September 3, 2017) is affected. (References: https://github.com/jenkinsci/jenkins/commit/c709b1932c4a207db2463c147502fffe53e99018 https://www.jenkins.io/changelog/2.77/)

          Stefan added a comment - - edited I most likely have found the issue: HttpUrlConnection will not upgrade the connection from HTTP to HTTPS and thus is being stuck. I have been able to verify this using curl : stefan@localhost:~> curl -I "http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476" HTTP/1.1 308 Permanent Redirect Date: Thu, 12 Sep 2024 10:29:33 GMT Content-Type: text/html Content-Length: 164 Connection: keep-alive Location: https://updates.jenkins-ci.org/update-center.json?id=default&version=2.476 Corresponding JDK issue: https://bugs.openjdk.org/browse/JDK-4620571?focusedId=12159233&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-12159233 I understand that it is desirable to do everything over HTTPS, but apparently this has not been enforced with the previous update site and some older Jenkins configurations break now. Judging from doing some git blames , everyone using the same Jenkins (base) configuration since version < 2.77 (released September 3, 2017) is affected. (References: https://github.com/jenkinsci/jenkins/commit/c709b1932c4a207db2463c147502fffe53e99018 https://www.jenkins.io/changelog/2.77/ )

          Hi stefan6419846thanks for the analysis and feedbacks!

          May I ask where does the `http` URL comes from? I've started a fresh Jenkins 2.476 (with the Docker image) and it's set up by default to use `https://updates.jenkins.io` which works as expected.

          I guess you're using a custom UC configuration? Or an old or specific setup?

          I'm asking because Jenkins default to HTTPS since a few years so I'm not sure to catch the problem in your setup.

          In any case: switch to using HTTPS and the domain updates.jenkins.io.

          For reference: some users also had the same issue in https://groups.google.com/g/jenkinsci-dev/c/1XsxkwELTBI

          Damien Duportal added a comment - Hi stefan6419846 thanks for the analysis and feedbacks! May I ask where does the `http` URL comes from? I've started a fresh Jenkins 2.476 (with the Docker image) and it's set up by default to use ` https://updates.jenkins.io ` which works as expected. I guess you're using a custom UC configuration? Or an old or specific setup? I'm asking because Jenkins default to HTTPS since a few years so I'm not sure to catch the problem in your setup. In any case: switch to using HTTPS and the domain updates.jenkins.io. For reference: some users also had the same issue in https://groups.google.com/g/jenkinsci-dev/c/1XsxkwELTBI

            basil Basil Crow
            stefan6419846 Stefan
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: