Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16475

Build stats graph configuration lost after Jenkins Restart

    XMLWordPrintable

Details

    Description

      Every time we restart jenkins we lose our configured Global build stats graphs. It seems to me that these should persist between restart and I'm wondering if we've set something up wrong or if its a bug.

      Running jenkins v1.484, global build stats plugin v1.3

      Attachments

        Issue Links

          Activity

            Hello !

            I could reproduce the issue with a manually corrupted jobResults.xml as well. The extra-button could be an option, but i would rather find out how the xml contained <null/> in the first place . Do you have any additional hints what could have caused the null in the xml-file? I will dig into the code soon, but since you have tested with so many instances maybe you might have some additional idea.

            dhinske David Krischke added a comment - Hello ! I could reproduce the issue with a manually corrupted jobResults.xml as well. The extra-button could be an option, but i would rather find out how the xml contained <null/> in the first place . Do you have any additional hints what could have caused the null in the xml-file? I will dig into the code soon, but since you have tested with so many instances maybe you might have some additional idea.

            Hi David, I'm really glad seeing your comment!

            What I've discovered so far is:
            The plugin in my opinion is trying to let's say "reindex" its files (when the data about builds are gathered) during jenkins startup. The problem is when any job ends during that time. As far as I seen - pluging collects data catching "jenkins events". So when such event comes (with information that some build is done) during the "reindexing" period of time (which can take in our case even half of an hour) plugin drops charts definitions.
            I was investigating it a few months ago I may have some gaps in my mind but I belive I can give you more details (console outputs, files, error logs) just tell me if you need them.

            Anyway this startup problem is very serious for us and we have to look for another solution for counting our builds across all products (and Jenkinses). The new solution is still not perfect/ready so - we still have a time for improvement in the Global build stats plugin.

            One more thing - a cosmetic one I'd say and not related with this issue.
            We use a large regexe's to collect project names. It will be useful to extend the width of the "edit box" where the regexes are defined.
            Regards, Piotr

            itrop Piotr Ossowski added a comment - Hi David, I'm really glad seeing your comment! What I've discovered so far is: The plugin in my opinion is trying to let's say "reindex" its files (when the data about builds are gathered) during jenkins startup. The problem is when any job ends during that time. As far as I seen - pluging collects data catching "jenkins events". So when such event comes (with information that some build is done) during the "reindexing" period of time (which can take in our case even half of an hour) plugin drops charts definitions. I was investigating it a few months ago I may have some gaps in my mind but I belive I can give you more details (console outputs, files, error logs) just tell me if you need them. Anyway this startup problem is very serious for us and we have to look for another solution for counting our builds across all products (and Jenkinses). The new solution is still not perfect/ready so - we still have a time for improvement in the Global build stats plugin. One more thing - a cosmetic one I'd say and not related with this issue. We use a large regexe's to collect project names. It will be useful to extend the width of the "edit box" where the regexes are defined. Regards, Piotr

            "The problem is when any job ends during that time."
            What do you mean by that? When you restart Jenkins while builds are running? Or how can a job end while Jenkins is starting?

            dhinske David Krischke added a comment - "The problem is when any job ends during that time." What do you mean by that? When you restart Jenkins while builds are running? Or how can a job end while Jenkins is starting?

            Well that is a good question.

            Startup of the Jenkins was delayed (with plugin installed). There is an open issue for it: https://issues.jenkins-ci.org/browse/JENKINS-24945. We experienced starting up Jenkins for a half of an hour. It usually takes 2-3 minutes without the plugin.

            Jenkins/plugin is doing something in ".../jenkins/global-build-stats/jobresults" files because those files were touched after startup. I called it "indexing" in my previous comment.

            I may be wrong about the root cause but after my tests I crated such an error scenario some time ago:

            1. Plugin is checking or reindexing its data files during Jenkins startup which are located here: /[...]/jenkins/global-build-stats/jobresults
            2. The checking/reindexing can take 30-60 minutes – especially after long period of time when there were no charts defined (empty global-build-stats.xml file for a few weeks)
            3. Jenkins itself is able to startup in 2-3 minutes but it will not start until above reindexing is not finished
            4. We have a frequently launched job called *** which is executed every 5 minutes
            5. After each executed build the Global Build Stats plugin is receiving the data from Jenkins

            So when any of the job is finished (and it is often the *** job due to its frequent usage) the GBS plugin it trying to update its data. The update is impossible during long checking/reindexing of its data files during Jenkins startup. That leads to an exception and results in clearing charts.

            I've found this log in an old e-mail:

            May 21, 2017 11:33:16 AM FINER hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver updatePlugin 

            Global build stats state update queued !
            May 21, 2017 11:33:16 AM FINER hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver$1 run 

            Changes applied and file saved !
            May 21, 2017 11:34:05 AM SEVERE hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver reloadPlugin 

            Unable to read /[...]/jenkins/global-build-stats.xml
            java.io.IOException: Unable to read /[...]/jenkins/global-build-stats.xml{{        }}
            at hudson.XmlFile.unmarshal(XmlFile.java:165){{        }}
            at hudson.Plugin.load(Plugin.java:264){{        }}
            at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin.load(GlobalBuildStatsPlugin.java:97)        at hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver.reloadPlugin(GlobalBuildStatsPluginSaver.java:99)        at hudson.plugins.global_build_stats.business.GlobalBuildStatsBusiness.reloadPlugin(GlobalBuildStatsBusiness.java:395)        at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin$GlobalBuildStatsItemListener.onLoaded(GlobalBuildStatsPlugin.java:152)        at jenkins.model.Jenkins.<init>(Jenkins.java:920){{        }}
            at hudson.model.Hudson.<init>(Hudson.java:85){{        }}
            at hudson.model.Hudson.<init>(Hudson.java:81){{        }}
            at hudson.WebAppMain$3.run(WebAppMain.java:227)
            Caused by: com.thoughtworks.xstream.converters.ConversionException:  : input contained no data :  : input contained no data

            ---- Debugging information ----

            message             :  : input contained no data
            cause-exception     : com.thoughtworks.xstream.io.StreamException
            cause-message       :  : input contained no data
            class               : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
            required-type       : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
            converter-type      : hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter
            path                : /hudson.plugins.global_build_stats.GlobalBuildStatsPlugin
            line number         : 2
            version             : not available-------------------------------

             

            itrop Piotr Ossowski added a comment - Well that is a good question. Startup of the Jenkins was delayed (with plugin installed). There is an open issue for it:  https://issues.jenkins-ci.org/browse/JENKINS-24945 . We experienced starting up Jenkins for a half of an hour. It usually takes 2-3 minutes without the plugin. Jenkins/plugin is doing something in ".../jenkins/global-build-stats/jobresults" files because those files were touched after startup. I called it "indexing" in my previous comment. I may be wrong about the root cause but after my tests I crated such an error scenario some time ago: Plugin is checking or reindexing its data files during Jenkins startup which are located here: / [...] /jenkins/global-build-stats/jobresults The checking/reindexing can take 30-60 minutes – especially after long period of time when there were no charts defined (empty global-build-stats.xml file for a few weeks) Jenkins itself is able to startup in 2-3 minutes but it will not start until above reindexing is not finished We have a frequently launched job called *** which is executed every 5 minutes After each executed build the Global Build Stats plugin is receiving the data from Jenkins So when any of the job is finished (and it is often the *** job due to its frequent usage) the GBS plugin it trying to update its data. The update is impossible during long checking/reindexing of its data files during Jenkins startup. That leads to an exception and results in clearing charts. I've found this log in an old e-mail: May 21, 2017 11:33:16 AM FINER hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver updatePlugin   Global build stats state update queued ! May 21, 2017 11:33:16 AM FINER hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver$1 run   Changes applied and file saved ! May 21, 2017 11:34:05 AM SEVERE hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver reloadPlugin   Unable to read / [...] /jenkins/global-build-stats.xml java.io.IOException: Unable to read / [...] /jenkins/global-build-stats.xml {{        }} at hudson.XmlFile.unmarshal(XmlFile.java:165) {{        }} at hudson.Plugin.load(Plugin.java:264) {{        }} at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin.load(GlobalBuildStatsPlugin.java:97)         at hudson.plugins.global_build_stats.business.GlobalBuildStatsPluginSaver.reloadPlugin(GlobalBuildStatsPluginSaver.java:99)         at hudson.plugins.global_build_stats.business.GlobalBuildStatsBusiness.reloadPlugin(GlobalBuildStatsBusiness.java:395)         at hudson.plugins.global_build_stats.GlobalBuildStatsPlugin$GlobalBuildStatsItemListener.onLoaded(GlobalBuildStatsPlugin.java:152)         at jenkins.model.Jenkins.<init>(Jenkins.java:920) {{        }} at hudson.model.Hudson.<init>(Hudson.java:85) {{        }} at hudson.model.Hudson.<init>(Hudson.java:81) {{        }} at hudson.WebAppMain$3.run(WebAppMain.java:227) Caused by: com.thoughtworks.xstream.converters.ConversionException:  : input contained no data :  : input contained no data ---- Debugging information ---- message             :  : input contained no data cause-exception     : com.thoughtworks.xstream.io.StreamException cause-message       :  : input contained no data class               : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin required-type       : hudson.plugins.global_build_stats.GlobalBuildStatsPlugin converter-type      : hudson.plugins.global_build_stats.xstream.GlobalBuildStatsXStreamConverter path                : /hudson.plugins.global_build_stats.GlobalBuildStatsPlugin line number         : 2 version             : not available -------------------------------  

            This PR is open:
            https://github.com/jenkinsci/global-build-stats-plugin/pull/13

            Could you please validate it?

            itrop Piotr Ossowski added a comment - This PR is open: https://github.com/jenkinsci/global-build-stats-plugin/pull/13 Could you please validate it?

            People

              dhinske David Krischke
              maxfields2000 Maxfield Stewart
              Votes:
              8 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated: