Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73760

Updates fail due to invalid JSON from HTTP updates.jenkins.io (HTTPS is ok)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core

      Updates currently fail due to the corresponding endpoint at http://updates.jenkins-ci.org/update-center.json (redirected to https://westeurope.cloudflare.jenkins.io/dynamic-2.476/update-center.json) returning invalid JSON (first characters shown here):

      updateCenter.post(
      {"connectionCheckUrl":"https://www.go
      

      Stacktrace on startup:

      java.io.IOException: Could not find JSON in http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476
              at hudson.model.DownloadService.loadJSON(DownloadService.java:129)
              at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:217)
              at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:212)
              at hudson.model.UpdateCenter.updateAllSitesNow(UpdateCenter.java:2866)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
              at java.base/java.lang.Thread.run(Thread.java:840)
      
      

      Firefox complains about this as well:

      SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
      

          [JENKINS-73760] Updates fail due to invalid JSON from HTTP updates.jenkins.io (HTTPS is ok)

          Pinned comments

          Pinned by Damien Duportal

          Stefan added a comment -

          Going to "Manage Jenkins" > "Plugins" > "Advanced settings" and clicking the "Reset" link in the "Update site" section successfully fixed this for me by automatically updating the outdated URL.

          Stefan added a comment - Going to "Manage Jenkins" > "Plugins" > "Advanced settings" and clicking the "Reset" link in the "Update site" section successfully fixed this for me by automatically updating the outdated URL.

          All comments

          Hi! Most probably an unexpected consequence of https://github.com/jenkins-infra/helpdesk/issues/2649. We are working on it

          Damien Duportal added a comment - Hi! Most probably an unexpected consequence of https://github.com/jenkins-infra/helpdesk/issues/2649 . We are working on it

          Hi stefan6419846 could you retry to see if our fixes did correct the problem?

          Damien Duportal added a comment - Hi stefan6419846 could you retry to see if our fixes did correct the problem?

          On the Jenkins infra side: we do not see any error when retrieving the UC from our 2.476 Jenkins controllers. New files are published.

          Keeping the issue open until stefan6419846can confirm whether or not it works for them

          Damien Duportal added a comment - On the Jenkins infra side: we do not see any error when retrieving the UC from our 2.476 Jenkins controllers. New files are published. Keeping the issue open until stefan6419846 can confirm whether or not it works for them

          Stefan added a comment -

          I just logged into the system/machine again and I am still seeing this issue. Is there any caching going on or something which would fix itself after a server restart? (I will probably try again tomorrow when I would be able to reboot the instance as well - at the moment the build queue is full and thus I do not want to reboot it directly.)

          Stefan added a comment - I just logged into the system/machine again and I am still seeing this issue. Is there any caching going on or something which would fix itself after a server restart? (I will probably try again tomorrow when I would be able to reboot the instance as well - at the moment the build queue is full and thus I do not want to reboot it directly.)

          stefan6419846Jenkins caches the file for 24 hours, unless you restart OR you tell it to check: Manage Jenkins -> Plugins -> click on the grey button with the rounded arrow on top right, next to "Update" blue button.

          Damien Duportal added a comment - stefan6419846 Jenkins caches the file for 24 hours, unless you restart OR you tell it to check: Manage Jenkins -> Plugins -> click on the grey button with the rounded arrow on top right, next to "Update" blue button.

          Stefan added a comment -

          Then I used the correct button already for retrying. I just restarted the server and it still fails.

          Nevertheless, I have been able to further debug this using the script console and a copy of https://github.com/jenkinsci/jenkins/blob/65f374c35a3caf9b3a5aacff442edb552a0f4141/core/src/main/java/hudson/model/DownloadService.java#L116-L132 and apparently somehow the redirect does not work correctly.

          import org.apache.commons.io.IOUtils;
          import java.nio.charset.StandardCharsets;
          URLConnection con = ProxyConfiguration.open(new URL("http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476"));
          if (con instanceof HttpURLConnection) {
              // prevent problems from misbehaving plugins disabling redirects by default
              ((HttpURLConnection) con).setInstanceFollowRedirects(true);
          }
          InputStream is = con.getInputStream()
          String result = IOUtils.toString(is, StandardCharsets.UTF_8);
          println result
          
          <html>
          <head><title>308 Permanent Redirect</title></head>
          <body>
          <center><h1>308 Permanent Redirect</h1></center>
          <hr><center>nginx</center>
          </body>
          </html> 
          

          Stefan added a comment - Then I used the correct button already for retrying. I just restarted the server and it still fails. Nevertheless, I have been able to further debug this using the script console and a copy of https://github.com/jenkinsci/jenkins/blob/65f374c35a3caf9b3a5aacff442edb552a0f4141/core/src/main/java/hudson/model/DownloadService.java#L116-L132 and apparently somehow the redirect does not work correctly. import org.apache.commons.io.IOUtils; import java.nio.charset.StandardCharsets; URLConnection con = ProxyConfiguration.open( new URL( "http: //updates.jenkins-ci.org/update-center.json?id= default &version=2.476" )); if (con instanceof HttpURLConnection) {     // prevent problems from misbehaving plugins disabling redirects by default     ((HttpURLConnection) con).setInstanceFollowRedirects( true ); } InputStream is = con.getInputStream() String result = IOUtils.toString(is, StandardCharsets.UTF_8); println result <html> <head> <title> 308 Permanent Redirect </title> </head> <body> <center> <h1> 308 Permanent Redirect </h1> </center> <hr> <center> nginx </center> </body> </html>  

          Stefan added a comment - - edited

          I most likely have found the issue: HttpUrlConnection will not upgrade the connection from HTTP to HTTPS and thus is being stuck. I have been able to verify this using curl:

          stefan@localhost:~> curl -I "http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476"
          HTTP/1.1 308 Permanent Redirect
          Date: Thu, 12 Sep 2024 10:29:33 GMT
          Content-Type: text/html
          Content-Length: 164
          Connection: keep-alive
          Location: https://updates.jenkins-ci.org/update-center.json?id=default&version=2.476
          

          Corresponding JDK issue: https://bugs.openjdk.org/browse/JDK-4620571?focusedId=12159233&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-12159233

          I understand that it is desirable to do everything over HTTPS, but apparently this has not been enforced with the previous update site and some older Jenkins configurations break now. Judging from doing some git blames, everyone using the same Jenkins (base) configuration since version < 2.77 (released September 3, 2017) is affected. (References: https://github.com/jenkinsci/jenkins/commit/c709b1932c4a207db2463c147502fffe53e99018 https://www.jenkins.io/changelog/2.77/)

          Stefan added a comment - - edited I most likely have found the issue: HttpUrlConnection will not upgrade the connection from HTTP to HTTPS and thus is being stuck. I have been able to verify this using curl : stefan@localhost:~> curl -I "http://updates.jenkins-ci.org/update-center.json?id=default&version=2.476" HTTP/1.1 308 Permanent Redirect Date: Thu, 12 Sep 2024 10:29:33 GMT Content-Type: text/html Content-Length: 164 Connection: keep-alive Location: https://updates.jenkins-ci.org/update-center.json?id=default&version=2.476 Corresponding JDK issue: https://bugs.openjdk.org/browse/JDK-4620571?focusedId=12159233&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-12159233 I understand that it is desirable to do everything over HTTPS, but apparently this has not been enforced with the previous update site and some older Jenkins configurations break now. Judging from doing some git blames , everyone using the same Jenkins (base) configuration since version < 2.77 (released September 3, 2017) is affected. (References: https://github.com/jenkinsci/jenkins/commit/c709b1932c4a207db2463c147502fffe53e99018 https://www.jenkins.io/changelog/2.77/ )

          Hi stefan6419846thanks for the analysis and feedbacks!

          May I ask where does the `http` URL comes from? I've started a fresh Jenkins 2.476 (with the Docker image) and it's set up by default to use `https://updates.jenkins.io` which works as expected.

          I guess you're using a custom UC configuration? Or an old or specific setup?

          I'm asking because Jenkins default to HTTPS since a few years so I'm not sure to catch the problem in your setup.

          In any case: switch to using HTTPS and the domain updates.jenkins.io.

          For reference: some users also had the same issue in https://groups.google.com/g/jenkinsci-dev/c/1XsxkwELTBI

          Damien Duportal added a comment - Hi stefan6419846 thanks for the analysis and feedbacks! May I ask where does the `http` URL comes from? I've started a fresh Jenkins 2.476 (with the Docker image) and it's set up by default to use ` https://updates.jenkins.io ` which works as expected. I guess you're using a custom UC configuration? Or an old or specific setup? I'm asking because Jenkins default to HTTPS since a few years so I'm not sure to catch the problem in your setup. In any case: switch to using HTTPS and the domain updates.jenkins.io. For reference: some users also had the same issue in https://groups.google.com/g/jenkinsci-dev/c/1XsxkwELTBI

          Stefan added a comment -

          This seems to be the old default setup for Jenkins versions up to 2.76 - see the commit linked in my previous comment. In my specific case, we have been using the same (base) configuration for over ten years now and apparently this has never been updated by us or migrated automatically as there has been no need to do so until now.

          Stefan added a comment - This seems to be the old default setup for Jenkins versions up to 2.76 - see the commit linked in my previous comment. In my specific case, we have been using the same (base) configuration for over ten years now and apparently this has never been updated by us or migrated automatically as there has been no need to do so until now.

          Oooh good catch! Looks like we did not took in account the scenario "inherited from old Jenkins setups and not migrated by the controller upgrade". Many thanks for your help!

          Damien Duportal added a comment - Oooh good catch! Looks like we did not took in account the scenario "inherited from old Jenkins setups and not migrated by the controller upgrade". Many thanks for your help!

          Edoardo added a comment -

          This is what I see when I open the update page, on 2.462.2.

          Edoardo added a comment - This is what I see when I open the update page, on 2.462.2.

          Pinned by Damien Duportal

          Stefan added a comment -

          Going to "Manage Jenkins" > "Plugins" > "Advanced settings" and clicking the "Reset" link in the "Update site" section successfully fixed this for me by automatically updating the outdated URL.

          Stefan added a comment - Going to "Manage Jenkins" > "Plugins" > "Advanced settings" and clicking the "Reset" link in the "Update site" section successfully fixed this for me by automatically updating the outdated URL.

          Hi lppedd thanks, we are aware of this issue. As described by Stefan below, your Jenkins controller did carry over a really old configuration. If you set it up to https://updates.jenkins.io, it will work as expected.

          Damien Duportal added a comment - Hi lppedd thanks, we are aware of this issue. As described by Stefan below, your Jenkins controller did carry over a really old configuration. If you set it up to https://updates.jenkins.io , it will work as expected.

          Edoardo added a comment -

          stefan6419846 my URL was set to http://updates.jenkins-ci.org/update-center.json, and resetting it didn't change it. Still see the issue.

          Edoardo added a comment - stefan6419846 my URL was set to http://updates.jenkins-ci.org/update-center.json, and resetting it didn't change it. Still see the issue.

          Edoardo added a comment - - edited

          Thanks dduportal! What I did was manually change it to https://updates.jenkins.io/update-center.json, and then click on the Update/Refresh button under Updates.

          After refreshing it, issue's gone!

          Edoardo added a comment - - edited Thanks dduportal ! What I did was manually change it to https://updates.jenkins.io/update-center.json, and then click on the Update/Refresh button under Updates. After refreshing it, issue's gone!

          Basil Crow added a comment -

          we have been using the same (base) configuration for over ten years now and apparently this has never been updated by us or migrated automatically as there has been no need to do so until now

          https://github.com/jenkinsci/jenkins/pull/9760

          Basil Crow added a comment - we have been using the same (base) configuration for over ten years now and apparently this has never been updated by us or migrated automatically as there has been no need to do so until now ⇨ https://github.com/jenkinsci/jenkins/pull/9760

          Many thanks basil !

          For the record, the infra team is adding an HTTP component to the new Update Center system which will avoid the error to happen when we'll migrate.
          It should NOT restrain users to update (reset - see pinned comment) their default Update Center URL.
          But we do not want to create bad surprises such as this one.

          See https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-2364040032 for follows up. Main take away is that we'll run an additional brownout of 24 hours to validate that HTTP works as expected

          Damien Duportal added a comment - Many thanks basil ! For the record, the infra team is adding an HTTP component to the new Update Center system which will avoid the error to happen when we'll migrate. It should NOT restrain users to update (reset - see pinned comment) their default Update Center URL. But we do not want to create bad surprises such as this one. See https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-2364040032 for follows up. Main take away is that we'll run an additional brownout of 24 hours to validate that HTTP works as expected

          Hi stefan6419846lppedd,

          We decided to add a full HTTP-only support to the new Update Center system to avoid bad surprises for users.

          We just started another "brownout" (e.g. using the new system for ~24h) to validate everything works as expected: please let us know if you have any errors.

          Of course we've tried using the `http://updates.jenkins-ci.org/update-center.json` URL and it works very well right now!

          Damien Duportal added a comment - Hi stefan6419846 lppedd , We decided to add a full HTTP-only support to the new Update Center system to avoid bad surprises for users. We just started another "brownout" (e.g. using the new system for ~24h) to validate everything works as expected: please let us know if you have any errors. Of course we've tried using the ` http://updates.jenkins-ci.org/update-center.json ` URL and it works very well right now!

          Stefan added a comment -

          I have switched to HTTPS in the meantime, but just tested with the code from https://issues.jenkins.io/browse/JENKINS-73760?focusedId=449257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-449257 and it worked without issues this time.

          Stefan added a comment - I have switched to HTTPS in the meantime, but just tested with the code from https://issues.jenkins.io/browse/JENKINS-73760?focusedId=449257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-449257 and it worked without issues this time.

          Mark Waite added a comment -

          Marking it as resolved before the 2.480 release so that it appears in my lts-candidate queries

          Mark Waite added a comment - Marking it as resolved before the 2.480 release so that it appears in my lts-candidate queries

            basil Basil Crow
            stefan6419846 Stefan
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: