-
Bug
-
Resolution: Fixed
-
Blocker
-
Windows Server 2008 R2 x64 with latest updates (or without them)
-
Powered by SuggestiMate
I encountered an issue that seems that should be fixed since i saw it appear in JIRA a couple of times before.
When installing a fresh Jenkins LTS 1.480.3 release on Windows Server 2008 R2 x64 I can't find any plugins in "Available" and "Updates" tabs.
I receive the following error:
Apr 19, 2013 1:34:59 AM hudson.model.UpdateSite doPostBack
SEVERE: <div class=error><img src='/static/9545af49/images/none.gif' height=16 width=1>Digest mismatch: 1Woxyt3oLgioZax7SGlbwYITc98= vs +JLZbigPye2eA9Fsv6sa3DW3q2s= in update center 'default'</div>
I have tried the latest Jenkins version and I don't receive this issue. The thing is - i want to go with the LTS version since i'm installing it on a PROD environment.
- is duplicated by
-
JENKINS-17029 update-center.json seems to be broken
-
- Resolved
-
-
JENKINS-17711 Digest mismatch in update center for LTS 1.480.3
-
- Resolved
-
[JENKINS-17677] Digest mismatch in update center for LTS 1.480.3
I can see the same error on Jenkins 1.480.3 on a Windows 7 machine. The SEVERE message is on the console window where I started Jenkins, but is not visible anywhere in the Jenkins user interface.
Oddly enough, my Jenkins 1.480.3 on Debian Linux does not report the digest mismatch and has not reported that digest mismatch in any of its log files from months before.
Generally, opening a new bug report for the same issue will not improve responsiveness. The new JENKINS-17711 won't help people investigate it any sooner and makes work for someone to resolve it as a duplicate.
I have this issue as well with the debian 1.480.3 LTS package on ubuntu. In my opinion this is an absolute must-fix for a build branded "LTS"! Are there any workarounds?
I doubt the problem is with the LTS package itself. I assume something is incorrect on the plugins site, or on one of the plugins sites (if they are mirrored somehow).
The digest is computed from the submitted JSON payload, and except the unlikely situation where this is broken for everyone, there must be something going on that we don't fully understand. (But how could anyone in the middle would tamper with JSON!?)
I think we need to improve the diagnostic code so that when we issue this error, we set aside the failed JSON payload and instruct the admin to attach that in the bug report.
Some more data points... we have the exact same issue with an rpm install of 1.480.3 on SLES 11 SP2 (rpm from http://pkg.jenkins-ci.org/opensuse-stable/). Uninstalled, installed 1.466.2 same issue. Uninstalled, installed 1.512 and the issue is gone. Really hate going with non-LTS release, so just to be sure, uninstalled 1.512 and reinstalled 1.480.3 again and the problem was back. Uninstalled and reinstalled 1.512 again and all is working. Guess I'll have to stick with 1.512 and hope that we get to a 1.5x LTS soon.
Thank you for the comments, again i have opened a duplicate because I wasn't sure the issue was observer by someone from the Jenkins team, or was processed.
On the topic: i can only speculate that it can be a plug-ins certificate issue that was happening a year ago - maybe it has something to do with the issue at hand.
At any rate - it's a major issue for our organization and any fix or workaround would be welcome.
When I perform a "wget http://updates.jenkins-ci.org/update-center.json", it connects to updates.jenkins-ci.org, then follows the link to mirrors.jenkins-ci.org, then resolves that to either mirror.xmission.com or ftp-chi.osuosl.org or ftp-nyc.osuosl.org.
I ran that wget 45 times and found 15 of the resulting 45 files differed from the original file, though the 15 files which differed from the original were all the same. In other words, the three servers were providing slightly different information in 1 of 3 cases (on average), though the server providing different information was consistent with itself.
The JSON file delivered by http://mirror.xmission.com/jenkins/updates/update-center.json reports many differences compared to the JSON file delivered by http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json.
For example, the xmission.com location reports the build-failure-analyzer buildDate as "Mar 14, 2013" while the ftp-nyc.osuosl.org location reports the build-failure-analyzer buildDare as "Apr 24, 2013". I assume that means the xmission.com site is out of date compared to the other two sites.
The output from ftp-chi.osuosl.org matches the output from ftp-nyc.osuosl.org.
A possible work around seems to be to bypass the updates.jenkins-ci.org site and request the update center directly from one of the mirrors. That's not a long term solution, but it seems to have worked in at least the two cases I tried.
When I request updates from any of the three update centers which are backing the updates.jenkins-ci.org location, I seem to always be able to successfully update the list of plugins and do not see any message about a digest error on the JSON.
Have tried to update plugins from any of the 3 mirrors:
http://mirror.xmission.com/jenkins/updates/update-center.json
http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json
http://ftp-chi.osuosl.org/pub/jenkins/updates/update-center.json
No success.
That's surprising, since that change was the "magic" for me. When you changed the update center, did you press the "Submit" button to assure the site change was applied?
I was surprised that I could not modify the update center field and then press "Update Now", because the "Update Now" button seemed to ignore the new value I entered into the text field.
I am seeing the same on a Windows Server 2008 R2 x64. I installed this same war on a Windows Server 2008 R2 x64 a couple of weeks ago without an issue. Now I am getting this error on that system also. I have tried using the mirror sites directly with no success either.
Joe, have you attempted the work around of using one of the 3 mirrors directly rather than using the default?
The three mirror sites I tried included:
http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json
http://ftp-chi.osuosl.org/pub/jenkins/updates/update-center.json
http://mirror.xmission.com/jenkins/updates/update-center.json
I'm most suspicious of the xmission.com site, since it seems to be out of date compared to the two sites at osuosl.org.
Mark,
I tried these 3 sites directly and still no updates and the same error message in jenkins.err file.
Thanks,
Joe
I tried http://ftp-nyc.osuosl.org/pub/jenkins/updates/update-center.json
No dice.
(1.480.3 LTS ubuntu debian package)
And now the "alleged work around" that I proposed does not work on my freshly installed 1.480.3 on a Debian machine either. I still assume this is an infrastructure problem due to the three servers returning inconsistent information, but if that were really the case, it seems like targeting exactly one of the servers would prevent the problem.
Well having the problem on Linux and Windows at least rules out the OS patches. I also see the error on an existing machine if you look in the jenkins.err file. The DB must retain the updates from past successes and since no error is shown on the UI this could be having a larger impact then just new installs.
Reproducible in 1.505 but works in 1.506, so probably https://github.com/jenkinsci/jenkins/commit/eab4bb6a30563a21321e5ab62afaa348247169d6 (new json-lib) is relevant.
My guess is that whatever produces the updates.jenkins-ci.org content was changed to assume that the digest would be computed from JSON canonicalized by the new JSON library, whereas older versions of Jenkins have slightly different content.
Emergency workaround:
- Shut down Jenkins.
- Download http://jenkins-updates.cloudbees.com/update-center.json?version=1.480.3 and save it somewhere.
- Edit the file, stripping off the first and last lines: updateCenter.post( and );
- Save as $JENKINS_HOME/updates/default.json.
- Restart Jenkins.
- /pluginManager/available should now list the right plugins. Ignore those marked Jenkins Enterprise by CloudBees Plugin since they would require a Jenkins Enterprise license to run.
- Repeat 1–5 to check for new updates.
I would have thought 1–5 could be done simply by saving a new URL in /pluginManager/advanced but it seems this does not work; even if you click Check now Jenkins does not even try to download the specified update center content. Not sure why not.
Found out why setting this URL in the GUI does not work: the JSON reports "id":"jenkins-enterprise" rather than "id":"default". This is better reported as of 1.482 (285a508); in 1.480.3 you just get a cryptic Uncaught TypeError: Cannot read property 'postBack' of undefined.
Thanks. I copied the default.json from one of my other jenkins servers and now I am getting a lists of plugins again. Thanks for the workaround. Hoping for a better long term resolution to follow.
I wonder if https://github.com/jenkinsci/backend-update-center2/commit/88deadded6fd8bca4f524f0fba5d5d94e0464cb6 is related? But that was two months ago, and complaints about the update center just started appearing.
Using /script in 1.506:
text = new java.io.File('…/updates/default.json').text; json = net.sf.json.JSONObject.fromObject(text); sig = json.remove('signature'); sha1 = java.security.MessageDigest.getInstance("SHA1"); baos = new ByteArrayOutputStream(); w = new OutputStreamWriter(baos, "UTF-8"); json.writeCanonical(w); w.close(); sha1.update(baos.toByteArray()) computedDigest = hudson.remoting.Base64.encode(sha1.digest()); providedDigest = sig.optString("correct_digest"); println('computed: ' + computedDigest); println('provided: ' + providedDigest);
→
computed: A76w9trilqKAm1tpJ+SXC/VOu48= provided: A76w9trilqKAm1tpJ+SXC/VOu48=
And in 1.505:
computed: WxtpFxOUSOkkuCHxUidvdFExQWA= provided: A76w9trilqKAm1tpJ+SXC/VOu48=
So this does indeed look like a json-lib incompatibility.
There is a json-lib incompatibility, but it would not matter except for https://wiki.jenkins-ci.org/display/JENKINS/JobRequeue-Plugin which has weird formatting that tickles a bug in the older version of json-lib. That was released on Apr 17, causing the UC to break.
https://wiki.jenkins-ci.org/display/JENKINS/drmemory+plugin is weird too but does not seem to be triggering the bug.
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jvnet/hudson/update_center/Plugin.java
http://jenkins-ci.org/commit/backend-update-center2/ea405aaa6d1aef116ce3966695a1c431eb1d161c
Log:
[FIXED JENKINS-17677] Avoiding excerpts likely to tickle a bug in the old json-lib.
My changes seem to have worked in that if you set the update center URL to https://ci.jenkins-ci.org/job/infra_update_center/ws/www2/update-center.json everything works. This does not seem to have propagated to mirrors yet; I am not sure when that happens.
I guess the system is “smart” and offers you the stable update center if you are running an LTS; since these are built only every four hours, my fix has not appeared yet:
$ for version in 1.480 1.480.3; do if curl -sL "http://updates.jenkins-ci.org/update-center.json?version=$version" | fgrep -q '"{'; then echo $version bad; else echo $version OK; fi; done 1.480 OK 1.480.3 bad
Hello guys, i don't wont to be ungrateful, but is this issue at least been noticed by someone?
Because a simple message - like "will be investigated in some time" at least would be helpful.
Maybe I haven't created the issue in the correct manner - please advice?