-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Powered by SuggestiMate
Every time we restart jenkins we lose our configured Global build stats graphs. It seems to me that these should persist between restart and I'm wondering if we've set something up wrong or if its a bug.
Running jenkins v1.484, global build stats plugin v1.3
- is related to
-
JENKINS-24951 NPE in at hudson.plugins.global_build_stats.model.JobBuildResultSharder.load(JobBuildResultSharder.java:140)
-
- Open
-
[JENKINS-16475] Build stats graph configuration lost after Jenkins Restart
Here is the message I get:
Nov 29, 2016 10:56:02 AM hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver reloadPlugin
SEVERE: Unable to read /var/lib/jenkins/global-build-stats.xml
java.io.IOException: Unable to read /var/lib/jenkins/global-build-stats.xml
at hudson.XmlFile.unmarshal(XmlFile.java:165)
at hudson.Plugin.load(Plugin.java:264)
at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin.load(GlobalBuildStatsPlugin.java:97)
at hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver.reloadPlugin(GlobalBuildStatsPluginSaver.java:99)
at hudson.plugins.global_build_stats.business.GlobalBuildStatsBusiness.reloadPlugin(GlobalBuildStatsBusiness.java:395)
at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin$GlobalBuildStatsItemListener.onLoaded(GlobalBuildStatsPlugin.java:152)
at jenkins.model.Jenkins.<init>(Jenkins.java:994)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:231)
Caused by: com.thoughtworks.xstream.converters.ConversionException: null : null
---- Debugging information ----
cause-exception : java.lang.NullPointerException
cause-message : null
class : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
required-type : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
converter-type : hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter
path : /hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
line number : 2
version : not available
-------------------------------
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
Same here with Jenkins 2.34 & Hudson global-build-stats plugin 1.4
Nov 30, 2016 3:50:17 PM hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver reloadPlugin SEVERE: Unable to read /var/lib/jenkins/global-build-stats.xml java.io.IOException: Unable to read /var/lib/jenkins/global-build-stats.xml at hudson.XmlFile.unmarshal(XmlFile.java:161) at hudson.Plugin.load(Plugin.java:264) at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin.load(GlobalBuildStatsPlugin.java:97) at hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver.reloadPlugin(GlobalBuildStatsPluginSaver.java:99) at hudson.plugins.global_build_stats.business.GlobalBuildStatsBusiness.reloadPlugin(GlobalBuildStatsBusiness.java:395) at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin$GlobalBuildStatsItemListener.onLoaded(GlobalBuildStatsPlugin.java:152) at jenkins.model.Jenkins.<init>(Jenkins.java:995) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:231) Caused by: com.thoughtworks.xstream.converters.ConversionException: null : null ---- Debugging information ---- cause-exception : java.lang.NullPointerException cause-message : null class : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin required-type : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin converter-type : hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter path : /hudson.plugins.global_build_stats.GlobalBuildStatsPlugin line number : 2 version : not available ------------------------------- at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:114) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:159) ... 9 more Caused by: java.lang.NullPointerException at java.util.ArrayList.addAll(ArrayList.java:577) at hudson.plugins.global_build_stats.model.JobBuildResultSharder.load(JobBuildResultSharder.java:140) at hudson.plugins.global_build_stats.xstream.migration.AbstractMigrator.readGlobalBuildStatsPOJO(AbstractMigrator.java:35) at hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter.unmarshal(GlobalBuildStatsXStreamConverter.java:159) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) ... 18 more
Removing a job results file containing "<null/>" at ../jenkins/global-build-stats/jobresults/jobResults-2014-07.xml fixed this issue for me too.
I've encountered similar issue. I'm gathering data from 9 Jenkins instances (will be more soon).
Charts are dissapearing on one of them after safeRestart. Please see some details below.
root@*******:/home/jenkins/jenkins# ls -al global-build-stats.xml
rw-rr- 1 jenkins bm 22064 Jan 7 19:20 global-build-stats.xml
root@*******:/home/jenkins/jenkins# ls al global-build-stats.xml #<-- after (
/safeRestart)
rw-rr- 1 jenkins bm 220 Jan 7 19:27 global-build-stats.xml
I've found temporary workaround by restarting jenkins server (and tomcat):
service jenkins stop/start
Right now global-build-stats.xml file is not cleared but I'm wondering what was the root cause.
I've also noted that file is frequently accessed (each few minutes):
rw-rr- 1 jenkins bm 22064 Jan 9 14:06 global-build-stats.xml
(...)
rw-rr- 1 jenkins bm 22064 Jan 9 14:24 global-build-stats.xml
Would it be possible (at least) to add button to upload/reload global-build-stats.xml file to easy restore charts (I keep xml files in a separate repository) when such an issue come back again?
Jenkins ver: 2.7.4
Global build stats plugin: 1.4
I have the problem on 2 (out of 9) jenkinses and I was able to fix the issue on one of them with described: "removing a job results file containing "<null/>" at"
C:\jenkins\global-build-stats\jobresults\jobResults-2011-06.xml (the was also jobResults-2011-07.xml with "null" to remove)
Jenkins ver. 1.509.2 (installed as a service on Windows Server 2008 R2 Standard, with SP1)
Global build stats plugin: 1.3
Hello !
I could reproduce the issue with a manually corrupted jobResults.xml as well. The extra-button could be an option, but i would rather find out how the xml contained <null/> in the first place . Do you have any additional hints what could have caused the null in the xml-file? I will dig into the code soon, but since you have tested with so many instances maybe you might have some additional idea.
Hi David, I'm really glad seeing your comment!
What I've discovered so far is:
The plugin in my opinion is trying to let's say "reindex" its files (when the data about builds are gathered) during jenkins startup. The problem is when any job ends during that time. As far as I seen - pluging collects data catching "jenkins events". So when such event comes (with information that some build is done) during the "reindexing" period of time (which can take in our case even half of an hour) plugin drops charts definitions.
I was investigating it a few months ago I may have some gaps in my mind but I belive I can give you more details (console outputs, files, error logs) just tell me if you need them.
Anyway this startup problem is very serious for us and we have to look for another solution for counting our builds across all products (and Jenkinses). The new solution is still not perfect/ready so - we still have a time for improvement in the Global build stats plugin.
One more thing - a cosmetic one I'd say and not related with this issue.
We use a large regexe's to collect project names. It will be useful to extend the width of the "edit box" where the regexes are defined.
Regards, Piotr
"The problem is when any job ends during that time."
What do you mean by that? When you restart Jenkins while builds are running? Or how can a job end while Jenkins is starting?
Well that is a good question.
Startup of the Jenkins was delayed (with plugin installed). There is an open issue for it: https://issues.jenkins-ci.org/browse/JENKINS-24945. We experienced starting up Jenkins for a half of an hour. It usually takes 2-3 minutes without the plugin.
Jenkins/plugin is doing something in ".../jenkins/global-build-stats/jobresults" files because those files were touched after startup. I called it "indexing" in my previous comment.
I may be wrong about the root cause but after my tests I crated such an error scenario some time ago:
- Plugin is checking or reindexing its data files during Jenkins startup which are located here: /[...]/jenkins/global-build-stats/jobresults
- The checking/reindexing can take 30-60 minutes – especially after long period of time when there were no charts defined (empty global-build-stats.xml file for a few weeks)
- Jenkins itself is able to startup in 2-3 minutes but it will not start until above reindexing is not finished
- We have a frequently launched job called *** which is executed every 5 minutes
- After each executed build the Global Build Stats plugin is receiving the data from Jenkins
So when any of the job is finished (and it is often the *** job due to its frequent usage) the GBS plugin it trying to update its data. The update is impossible during long checking/reindexing of its data files during Jenkins startup. That leads to an exception and results in clearing charts.
I've found this log in an old e-mail:
May 21, 2017 11:33:16 AM FINER hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver updatePlugin
Global build stats state update queued !
May 21, 2017 11:33:16 AM FINER hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver$1 run
Changes applied and file saved !
May 21, 2017 11:34:05 AM SEVERE hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver reloadPlugin
Unable to read /[...]/jenkins/global-build-stats.xml
java.io.IOException: Unable to read /[...]/jenkins/global-build-stats.xml{{ }}
at hudson.XmlFile.unmarshal(XmlFile.java:165){{ }}
at hudson.Plugin.load(Plugin.java:264){{ }}
at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin.load(GlobalBuildStatsPlugin.java:97) at hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver.reloadPlugin(GlobalBuildStatsPluginSaver.java:99) at hudson.plugins.global_build_stats.business.GlobalBuildStatsBusiness.reloadPlugin(GlobalBuildStatsBusiness.java:395) at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin$GlobalBuildStatsItemListener.onLoaded(GlobalBuildStatsPlugin.java:152) at jenkins.model.Jenkins.<init>(Jenkins.java:920){{ }}
at hudson.model.Hudson.<init>(Hudson.java:85){{ }}
at hudson.model.Hudson.<init>(Hudson.java:81){{ }}
at hudson.WebAppMain$3.run(WebAppMain.java:227)
Caused by: com.thoughtworks.xstream.converters.ConversionException: : input contained no data : : input contained no data
---- Debugging information ----
message : : input contained no data
cause-exception : com.thoughtworks.xstream.io.StreamException
cause-message : : input contained no data
class : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
required-type : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
converter-type : hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter
path : /hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
line number : 2
version : not available-------------------------------
This PR is open:
https://github.com/jenkinsci/global-build-stats-plugin/pull/13
Could you please validate it?
I can confirm this is an issue with our RHEL5 Linux server with Jenkins ver. 2.23, and plugin 1.4 revison.
{{112 Sep 19, 2016 3:37:00 PM hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver reloadPlugin
113 SEVERE: Unable to read /data/jenkins/global-build-stats.xml
114 java.io.IOException: Unable to read /data/jenkins/global-build-stats.xml
115 at hudson.XmlFile.unmarshal(XmlFile.java:165)
116 at hudson.Plugin.load(Plugin.java:264)
117 at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin.load(GlobalBuildStatsPlugin.java:97)
118 at hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver.reloadPlugin(GlobalBuildStatsPluginSaver.java:99)
119 at hudson.plugins.global_build_stats.business.GlobalBuildStatsBusiness.reloadPlugin(GlobalBuildStatsBusiness.java:395)
120 at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin$GlobalBuildStatsItemListener.onLoaded(GlobalBuildStatsPlugin.java:152)
121 at jenkins.model.Jenkins.<init>(Jenkins.java:960)
122 at hudson.model.Hudson.<init>(Hudson.java:85)
123 at hudson.model.Hudson.<init>(Hudson.java:81)
124 at hudson.WebAppMain$3.run(WebAppMain.java:227)
125 Caused by: com.thoughtworks.xstream.converters.ConversionException: null : null
126 ---- Debugging information ----
127 cause-exception : java.lang.NullPointerException
128 cause-message : null
129 class : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
130 required-type : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
131 converter-type : hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter
132 path : /hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
133 line number : 2
134 version : not available
135 -------------------------------
136 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
137 at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
138 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
139 at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
140 at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
141 at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
142 at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
143 at hudson.util.XStream2.unmarshal(XStream2.java:114)
144 at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
145 at hudson.XmlFile.unmarshal(XmlFile.java:163)
146 ... 9 more
147 Caused by: java.lang.NullPointerException
148 at java.util.ArrayList.addAll(ArrayList.java:577)
149 at hudson.plugins.global_build_stats.model.JobBuildResultSharder.load(JobBuildResultSharder.java:140)
150 at hudson.plugins.global_build_stats.xstream.migration.AbstractMigrator.readGlobalBuildStatsPOJO(AbstractMigrator.java:35)
151 at hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter.unmarshal(GlobalBuildStatsXStreamConverter.java:159)
152 at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
153 ... 18 more}}
Solution:
I confirmed the .../jenkins/global-build-stats/jobresults/ directory contained two jobResults... files, one of which, jobResults-2016-04.xml, contained only "<null/>". The other file appears to contain the build results as you would expect. I removed the jobResults-2016-04.xml file and did a safeRestart on the machine. The graph survived the restart process so I'd say this was the cause of our issue. Hope this helps others.