Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17677

Digest mismatch in update center for LTS 1.480.3

      I encountered an issue that seems that should be fixed since i saw it appear in JIRA a couple of times before.

      When installing a fresh Jenkins LTS 1.480.3 release on Windows Server 2008 R2 x64 I can't find any plugins in "Available" and "Updates" tabs.

      I receive the following error:
      Apr 19, 2013 1:34:59 AM hudson.model.UpdateSite doPostBack
      SEVERE: <div class=error><img src='/static/9545af49/images/none.gif' height=16 width=1>Digest mismatch: 1Woxyt3oLgioZax7SGlbwYITc98= vs +JLZbigPye2eA9Fsv6sa3DW3q2s= in update center 'default'</div>

      I have tried the latest Jenkins version and I don't receive this issue. The thing is - i want to go with the LTS version since i'm installing it on a PROD environment.

          [JENKINS-17677] Digest mismatch in update center for LTS 1.480.3

          Hello guys, i don't wont to be ungrateful, but is this issue at least been noticed by someone?
          Because a simple message - like "will be investigated in some time" at least would be helpful.
          Maybe I haven't created the issue in the correct manner - please advice?

          Potroshitel Jack added a comment - Hello guys, i don't wont to be ungrateful, but is this issue at least been noticed by someone? Because a simple message - like "will be investigated in some time" at least would be helpful. Maybe I haven't created the issue in the correct manner - please advice?

          Mark Waite added a comment -

          I can see the same error on Jenkins 1.480.3 on a Windows 7 machine. The SEVERE message is on the console window where I started Jenkins, but is not visible anywhere in the Jenkins user interface.

          Oddly enough, my Jenkins 1.480.3 on Debian Linux does not report the digest mismatch and has not reported that digest mismatch in any of its log files from months before.

          Mark Waite added a comment - I can see the same error on Jenkins 1.480.3 on a Windows 7 machine. The SEVERE message is on the console window where I started Jenkins, but is not visible anywhere in the Jenkins user interface. Oddly enough, my Jenkins 1.480.3 on Debian Linux does not report the digest mismatch and has not reported that digest mismatch in any of its log files from months before.

          Mark Waite added a comment - - edited

          Generally, opening a new bug report for the same issue will not improve responsiveness. The new JENKINS-17711 won't help people investigate it any sooner and makes work for someone to resolve it as a duplicate.

          Mark Waite added a comment - - edited Generally, opening a new bug report for the same issue will not improve responsiveness. The new JENKINS-17711 won't help people investigate it any sooner and makes work for someone to resolve it as a duplicate.

          I have this issue as well with the debian 1.480.3 LTS package on ubuntu. In my opinion this is an absolute must-fix for a build branded "LTS"! Are there any workarounds?

          Preston Jennings added a comment - I have this issue as well with the debian 1.480.3 LTS package on ubuntu. In my opinion this is an absolute must-fix for a build branded "LTS"! Are there any workarounds?

          Mark Waite added a comment -

          I doubt the problem is with the LTS package itself. I assume something is incorrect on the plugins site, or on one of the plugins sites (if they are mirrored somehow).

          Mark Waite added a comment - I doubt the problem is with the LTS package itself. I assume something is incorrect on the plugins site, or on one of the plugins sites (if they are mirrored somehow).

          The digest is computed from the submitted JSON payload, and except the unlikely situation where this is broken for everyone, there must be something going on that we don't fully understand. (But how could anyone in the middle would tamper with JSON!?)

          I think we need to improve the diagnostic code so that when we issue this error, we set aside the failed JSON payload and instruct the admin to attach that in the bug report.

          Kohsuke Kawaguchi added a comment - The digest is computed from the submitted JSON payload, and except the unlikely situation where this is broken for everyone, there must be something going on that we don't fully understand. (But how could anyone in the middle would tamper with JSON!?) I think we need to improve the diagnostic code so that when we issue this error, we set aside the failed JSON payload and instruct the admin to attach that in the bug report.

          Some more data points... we have the exact same issue with an rpm install of 1.480.3 on SLES 11 SP2 (rpm from http://pkg.jenkins-ci.org/opensuse-stable/). Uninstalled, installed 1.466.2 same issue. Uninstalled, installed 1.512 and the issue is gone. Really hate going with non-LTS release, so just to be sure, uninstalled 1.512 and reinstalled 1.480.3 again and the problem was back. Uninstalled and reinstalled 1.512 again and all is working. Guess I'll have to stick with 1.512 and hope that we get to a 1.5x LTS soon.

          Lance Johnston added a comment - Some more data points... we have the exact same issue with an rpm install of 1.480.3 on SLES 11 SP2 (rpm from http://pkg.jenkins-ci.org/opensuse-stable/ ). Uninstalled, installed 1.466.2 same issue. Uninstalled, installed 1.512 and the issue is gone. Really hate going with non-LTS release, so just to be sure, uninstalled 1.512 and reinstalled 1.480.3 again and the problem was back. Uninstalled and reinstalled 1.512 again and all is working. Guess I'll have to stick with 1.512 and hope that we get to a 1.5x LTS soon.

          Thank you for the comments, again i have opened a duplicate because I wasn't sure the issue was observer by someone from the Jenkins team, or was processed.
          On the topic: i can only speculate that it can be a plug-ins certificate issue that was happening a year ago - maybe it has something to do with the issue at hand.
          At any rate - it's a major issue for our organization and any fix or workaround would be welcome.

          Potroshitel Jack added a comment - Thank you for the comments, again i have opened a duplicate because I wasn't sure the issue was observer by someone from the Jenkins team, or was processed. On the topic: i can only speculate that it can be a plug-ins certificate issue that was happening a year ago - maybe it has something to do with the issue at hand. At any rate - it's a major issue for our organization and any fix or workaround would be welcome.

          Mark Waite added a comment -

          When I perform a "wget http://updates.jenkins-ci.org/update-center.json", it connects to updates.jenkins-ci.org, then follows the link to mirrors.jenkins-ci.org, then resolves that to either mirror.xmission.com or ftp-chi.osuosl.org or ftp-nyc.osuosl.org.

          I ran that wget 45 times and found 15 of the resulting 45 files differed from the original file, though the 15 files which differed from the original were all the same. In other words, the three servers were providing slightly different information in 1 of 3 cases (on average), though the server providing different information was consistent with itself.

          Mark Waite added a comment - When I perform a "wget http://updates.jenkins-ci.org/update-center.json ", it connects to updates.jenkins-ci.org, then follows the link to mirrors.jenkins-ci.org, then resolves that to either mirror.xmission.com or ftp-chi.osuosl.org or ftp-nyc.osuosl.org. I ran that wget 45 times and found 15 of the resulting 45 files differed from the original file, though the 15 files which differed from the original were all the same. In other words, the three servers were providing slightly different information in 1 of 3 cases (on average), though the server providing different information was consistent with itself.

          Mark Waite added a comment - - edited

          The JSON file delivered by http://mirror.xmission.com/jenkins/updates/update-center.json reports many differences compared to the JSON file delivered by http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json.

          For example, the xmission.com location reports the build-failure-analyzer buildDate as "Mar 14, 2013" while the ftp-nyc.osuosl.org location reports the build-failure-analyzer buildDare as "Apr 24, 2013". I assume that means the xmission.com site is out of date compared to the other two sites.

          The output from ftp-chi.osuosl.org matches the output from ftp-nyc.osuosl.org.

          Mark Waite added a comment - - edited The JSON file delivered by http://mirror.xmission.com/jenkins/updates/update-center.json reports many differences compared to the JSON file delivered by http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json . For example, the xmission.com location reports the build-failure-analyzer buildDate as "Mar 14, 2013" while the ftp-nyc.osuosl.org location reports the build-failure-analyzer buildDare as "Apr 24, 2013". I assume that means the xmission.com site is out of date compared to the other two sites. The output from ftp-chi.osuosl.org matches the output from ftp-nyc.osuosl.org.

          Mark Waite added a comment -

          A possible work around seems to be to bypass the updates.jenkins-ci.org site and request the update center directly from one of the mirrors. That's not a long term solution, but it seems to have worked in at least the two cases I tried.

          When I request updates from any of the three update centers which are backing the updates.jenkins-ci.org location, I seem to always be able to successfully update the list of plugins and do not see any message about a digest error on the JSON.

          Mark Waite added a comment - A possible work around seems to be to bypass the updates.jenkins-ci.org site and request the update center directly from one of the mirrors. That's not a long term solution, but it seems to have worked in at least the two cases I tried. When I request updates from any of the three update centers which are backing the updates.jenkins-ci.org location, I seem to always be able to successfully update the list of plugins and do not see any message about a digest error on the JSON.

          Potroshitel Jack added a comment - Have tried to update plugins from any of the 3 mirrors: http://mirror.xmission.com/jenkins/updates/update-center.json http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json http://ftp-chi.osuosl.org/pub/jenkins/updates/update-center.json No success.

          Mark Waite added a comment -

          That's surprising, since that change was the "magic" for me. When you changed the update center, did you press the "Submit" button to assure the site change was applied?

          I was surprised that I could not modify the update center field and then press "Update Now", because the "Update Now" button seemed to ignore the new value I entered into the text field.

          Mark Waite added a comment - That's surprising, since that change was the "magic" for me. When you changed the update center, did you press the "Submit" button to assure the site change was applied? I was surprised that I could not modify the update center field and then press "Update Now", because the "Update Now" button seemed to ignore the new value I entered into the text field.

          Joe Knudsen added a comment - - edited

          I am seeing the same on a Windows Server 2008 R2 x64. I installed this same war on a Windows Server 2008 R2 x64 a couple of weeks ago without an issue. Now I am getting this error on that system also. I have tried using the mirror sites directly with no success either.

          Joe Knudsen added a comment - - edited I am seeing the same on a Windows Server 2008 R2 x64. I installed this same war on a Windows Server 2008 R2 x64 a couple of weeks ago without an issue. Now I am getting this error on that system also. I have tried using the mirror sites directly with no success either.

          Mark Waite added a comment - - edited

          Joe, have you attempted the work around of using one of the 3 mirrors directly rather than using the default?

          The three mirror sites I tried included:

          http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json
          http://ftp-chi.osuosl.org/pub/jenkins/updates/update-center.json
          http://mirror.xmission.com/jenkins/updates/update-center.json

          I'm most suspicious of the xmission.com site, since it seems to be out of date compared to the two sites at osuosl.org.

          Mark Waite added a comment - - edited Joe, have you attempted the work around of using one of the 3 mirrors directly rather than using the default? The three mirror sites I tried included: http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json http://ftp-chi.osuosl.org/pub/jenkins/updates/update-center.json http://mirror.xmission.com/jenkins/updates/update-center.json I'm most suspicious of the xmission.com site, since it seems to be out of date compared to the two sites at osuosl.org.

          Joe Knudsen added a comment -

          Mark,
          I tried these 3 sites directly and still no updates and the same error message in jenkins.err file.
          Thanks,
          Joe

          Joe Knudsen added a comment - Mark, I tried these 3 sites directly and still no updates and the same error message in jenkins.err file. Thanks, Joe

          I tried http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json

          No dice.

          (1.480.3 LTS ubuntu debian package)

          Preston Jennings added a comment - I tried http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json No dice. (1.480.3 LTS ubuntu debian package)

          Mark Waite added a comment -

          And now the "alleged work around" that I proposed does not work on my freshly installed 1.480.3 on a Debian machine either. I still assume this is an infrastructure problem due to the three servers returning inconsistent information, but if that were really the case, it seems like targeting exactly one of the servers would prevent the problem.

          Mark Waite added a comment - And now the "alleged work around" that I proposed does not work on my freshly installed 1.480.3 on a Debian machine either. I still assume this is an infrastructure problem due to the three servers returning inconsistent information, but if that were really the case, it seems like targeting exactly one of the servers would prevent the problem.

          Joe Knudsen added a comment -

          Well having the problem on Linux and Windows at least rules out the OS patches. I also see the error on an existing machine if you look in the jenkins.err file. The DB must retain the updates from past successes and since no error is shown on the UI this could be having a larger impact then just new installs.

          Joe Knudsen added a comment - Well having the problem on Linux and Windows at least rules out the OS patches. I also see the error on an existing machine if you look in the jenkins.err file. The DB must retain the updates from past successes and since no error is shown on the UI this could be having a larger impact then just new installs.

          Jesse Glick added a comment -

          Reproducible in 1.505 but works in 1.506, so probably https://github.com/jenkinsci/jenkins/commit/eab4bb6a30563a21321e5ab62afaa348247169d6 (new json-lib) is relevant.

          Jesse Glick added a comment - Reproducible in 1.505 but works in 1.506, so probably https://github.com/jenkinsci/jenkins/commit/eab4bb6a30563a21321e5ab62afaa348247169d6 (new json-lib) is relevant.

          Jesse Glick added a comment - - edited

          My guess is that whatever produces the updates.jenkins-ci.org content was changed to assume that the digest would be computed from JSON canonicalized by the new JSON library, whereas older versions of Jenkins have slightly different content.

          Emergency workaround:

          1. Shut down Jenkins.
          2. Download http://jenkins-updates.cloudbees.com/update-center.json?version=1.480.3 and save it somewhere.
          3. Edit the file, stripping off the first and last lines: updateCenter.post( and );
          4. Save as $JENKINS_HOME/updates/default.json.
          5. Restart Jenkins.
          6. /pluginManager/available should now list the right plugins. Ignore those marked Jenkins Enterprise by CloudBees Plugin since they would require a Jenkins Enterprise license to run.
          7. Repeat 1–5 to check for new updates.

          I would have thought 1–5 could be done simply by saving a new URL in /pluginManager/advanced but it seems this does not work; even if you click Check now Jenkins does not even try to download the specified update center content. Not sure why not.

          Jesse Glick added a comment - - edited My guess is that whatever produces the updates.jenkins-ci.org content was changed to assume that the digest would be computed from JSON canonicalized by the new JSON library, whereas older versions of Jenkins have slightly different content. Emergency workaround: Shut down Jenkins. Download http://jenkins-updates.cloudbees.com/update-center.json?version=1.480.3 and save it somewhere. Edit the file, stripping off the first and last lines: updateCenter.post( and ); Save as $JENKINS_HOME/updates/default.json . Restart Jenkins. /pluginManager/available should now list the right plugins. Ignore those marked Jenkins Enterprise by CloudBees Plugin since they would require a Jenkins Enterprise license to run. Repeat 1–5 to check for new updates. I would have thought 1–5 could be done simply by saving a new URL in /pluginManager/advanced but it seems this does not work; even if you click Check now Jenkins does not even try to download the specified update center content. Not sure why not.

          Jesse Glick added a comment -

          Found out why setting this URL in the GUI does not work: the JSON reports "id":"jenkins-enterprise" rather than "id":"default". This is better reported as of 1.482 (285a508); in 1.480.3 you just get a cryptic Uncaught TypeError: Cannot read property 'postBack' of undefined.

          Jesse Glick added a comment - Found out why setting this URL in the GUI does not work: the JSON reports "id":"jenkins-enterprise" rather than "id":"default" . This is better reported as of 1.482 ( 285a508 ); in 1.480.3 you just get a cryptic Uncaught TypeError: Cannot read property 'postBack' of undefined .

          Joe Knudsen added a comment -

          Thanks. I copied the default.json from one of my other jenkins servers and now I am getting a lists of plugins again. Thanks for the workaround. Hoping for a better long term resolution to follow.

          Joe Knudsen added a comment - Thanks. I copied the default.json from one of my other jenkins servers and now I am getting a lists of plugins again. Thanks for the workaround. Hoping for a better long term resolution to follow.

          Jesse Glick added a comment -

          I wonder if https://github.com/jenkinsci/backend-update-center2/commit/88deadded6fd8bca4f524f0fba5d5d94e0464cb6 is related? But that was two months ago, and complaints about the update center just started appearing.

          Jesse Glick added a comment - I wonder if https://github.com/jenkinsci/backend-update-center2/commit/88deadded6fd8bca4f524f0fba5d5d94e0464cb6 is related? But that was two months ago, and complaints about the update center just started appearing.

          Jesse Glick added a comment -

          Using /script in 1.506:

          text = new java.io.File('…/updates/default.json').text;
          json = net.sf.json.JSONObject.fromObject(text);
          sig = json.remove('signature');
          sha1 = java.security.MessageDigest.getInstance("SHA1");
          baos = new ByteArrayOutputStream();
          w = new OutputStreamWriter(baos, "UTF-8");
          json.writeCanonical(w);
          w.close();
          sha1.update(baos.toByteArray())
          computedDigest = hudson.remoting.Base64.encode(sha1.digest());
          providedDigest = sig.optString("correct_digest");
          println('computed: ' + computedDigest);
          println('provided: ' + providedDigest);
          

          computed: A76w9trilqKAm1tpJ+SXC/VOu48=
          provided: A76w9trilqKAm1tpJ+SXC/VOu48=
          

          And in 1.505:

          computed: WxtpFxOUSOkkuCHxUidvdFExQWA=
          provided: A76w9trilqKAm1tpJ+SXC/VOu48=
          

          So this does indeed look like a json-lib incompatibility.

          Jesse Glick added a comment - Using /script in 1.506: text = new java.io.File( '…/updates/ default .json' ).text; json = net.sf.json.JSONObject.fromObject(text); sig = json.remove( 'signature' ); sha1 = java.security.MessageDigest.getInstance( "SHA1" ); baos = new ByteArrayOutputStream(); w = new OutputStreamWriter(baos, "UTF-8" ); json.writeCanonical(w); w.close(); sha1.update(baos.toByteArray()) computedDigest = hudson.remoting.Base64.encode(sha1.digest()); providedDigest = sig.optString( "correct_digest" ); println( 'computed: ' + computedDigest); println( 'provided: ' + providedDigest); → computed: A76w9trilqKAm1tpJ+SXC/VOu48= provided: A76w9trilqKAm1tpJ+SXC/VOu48= And in 1.505: computed: WxtpFxOUSOkkuCHxUidvdFExQWA= provided: A76w9trilqKAm1tpJ+SXC/VOu48= So this does indeed look like a json-lib incompatibility.

          Jesse Glick added a comment -

          There is a json-lib incompatibility, but it would not matter except for https://wiki.jenkins-ci.org/display/JENKINS/JobRequeue-Plugin which has weird formatting that tickles a bug in the older version of json-lib. That was released on Apr 17, causing the UC to break.

          https://wiki.jenkins-ci.org/display/JENKINS/drmemory+plugin is weird too but does not seem to be triggering the bug.

          Jesse Glick added a comment - There is a json-lib incompatibility, but it would not matter except for https://wiki.jenkins-ci.org/display/JENKINS/JobRequeue-Plugin which has weird formatting that tickles a bug in the older version of json-lib. That was released on Apr 17, causing the UC to break. https://wiki.jenkins-ci.org/display/JENKINS/drmemory+plugin is weird too but does not seem to be triggering the bug.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jvnet/hudson/update_center/Plugin.java
          http://jenkins-ci.org/commit/backend-update-center2/ea405aaa6d1aef116ce3966695a1c431eb1d161c
          Log:
          [FIXED JENKINS-17677] Avoiding excerpts likely to tickle a bug in the old json-lib.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jvnet/hudson/update_center/Plugin.java http://jenkins-ci.org/commit/backend-update-center2/ea405aaa6d1aef116ce3966695a1c431eb1d161c Log: [FIXED JENKINS-17677] Avoiding excerpts likely to tickle a bug in the old json-lib.

          Jesse Glick added a comment -

          My changes seem to have worked in that if you set the update center URL to https://ci.jenkins-ci.org/job/infra_update_center/ws/www2/update-center.json everything works. This does not seem to have propagated to mirrors yet; I am not sure when that happens.

          Jesse Glick added a comment - My changes seem to have worked in that if you set the update center URL to https://ci.jenkins-ci.org/job/infra_update_center/ws/www2/update-center.json everything works. This does not seem to have propagated to mirrors yet; I am not sure when that happens.

          Jesse Glick added a comment -

          I guess the system is “smart” and offers you the stable update center if you are running an LTS; since these are built only every four hours, my fix has not appeared yet:

          $ for version in 1.480 1.480.3; do if curl -sL "http://updates.jenkins-ci.org/update-center.json?version=$version" | fgrep -q '"{'; then echo $version bad; else echo $version OK; fi; done
          1.480 OK
          1.480.3 bad
          

          Jesse Glick added a comment - I guess the system is “smart” and offers you the stable update center if you are running an LTS; since these are built only every four hours, my fix has not appeared yet: $ for version in 1.480 1.480.3; do if curl -sL "http://updates.jenkins-ci.org/update-center.json?version=$version" | fgrep -q '"{'; then echo $version bad; else echo $version OK; fi; done 1.480 OK 1.480.3 bad

          Jesse Glick added a comment -

          Now it is there.

          Jesse Glick added a comment - Now it is there.

            jglick Jesse Glick
            bzlom Potroshitel Jack
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: