Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17265

Builds disappearing from history

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Incomplete
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 1.506
    • Similar Issues:

      Description

      Revisiting https://issues.jenkins-ci.org/browse/JENKINS-15156 once again.

      The issue still persists. Builds are disappearing from history, even though they are present on the disk. They reappear after Jenkins restart.

        Attachments

        1. check_invalid_builds.sh
          3 kB
        2. check_invalid_builds.sh
          3 kB
        3. check_missing_builds.sh
          1 kB
        4. check_missing_builds.sh
          1 kB
        5. plugin1.bmp
          plugin1.bmp
          3.65 MB
        6. plugin2.bmp
          plugin2.bmp
          1.92 MB
        7. plugin3.bmp
          plugin3.bmp
          3.65 MB

          Issue Links

            Activity

            Hide
            hnain Harpreet Nain added a comment -

            We have the same issue on version 1.494, 1.499 and 1.502. We dont have any master/slave configuration in place. The job history seems to come back after we do a "Reload Configuration from Disk"

            Show
            hnain Harpreet Nain added a comment - We have the same issue on version 1.494, 1.499 and 1.502. We dont have any master/slave configuration in place. The job history seems to come back after we do a "Reload Configuration from Disk"
            Hide
            mddubs Matt Williamson added a comment -

            Same issue on 1.506, job history for some jobs disappears. Selecting "Reload Configuration from Disk" brings the history back.

            Show
            mddubs Matt Williamson added a comment - Same issue on 1.506, job history for some jobs disappears. Selecting "Reload Configuration from Disk" brings the history back.
            Hide
            kflorian kflorian added a comment -

            Confirming this on Jenkins 1.505, running on Linux with Linux slaves.
            Build results seem to disappear from some jobs, while they show up as usual on others.
            Seems to happen even more frequently in 1.505 than in the version(s) before!
            Also confirming that "Reload Configuration from Disk" brings back the history.

            Show
            kflorian kflorian added a comment - Confirming this on Jenkins 1.505, running on Linux with Linux slaves. Build results seem to disappear from some jobs, while they show up as usual on others. Seems to happen even more frequently in 1.505 than in the version(s) before! Also confirming that "Reload Configuration from Disk" brings back the history.
            Hide
            josesa Jose Sa added a comment -

            Please notice that "Reload from disk" may be unsafe if you have actively running jobs as indicated in JENKINS-3265

            Show
            josesa Jose Sa added a comment - Please notice that "Reload from disk" may be unsafe if you have actively running jobs as indicated in JENKINS-3265
            Hide
            wbauer wbauer added a comment -

            Same issue for me, had it with the 1.4xxx builds and now still with 1.505 - really annoying!

            Show
            wbauer wbauer added a comment - Same issue for me, had it with the 1.4xxx builds and now still with 1.505 - really annoying!
            Hide
            oldelvet Richard Mortimer added a comment -

            We need to try and identify what may/may not be causing the problems. Quite a few fixes went into 1.505 and earlier but there are obviously still a few gremlins in there.

            Here's a few questions that those of you seeing the problem can try to answer:

            • are there any relevant messages in any of the Jenkins logs. Any exceptions, notices about failure to load are useful.
            • what job types do problem occur on? maven, freestyle, other.
            • are you running Jenkins direct from the war file or within tomcat/similar environment.
            • what OS are your masters/slaves running?

            Answers to those will help to point towards the right person to fix/investigate further.

            Show
            oldelvet Richard Mortimer added a comment - We need to try and identify what may/may not be causing the problems. Quite a few fixes went into 1.505 and earlier but there are obviously still a few gremlins in there. Here's a few questions that those of you seeing the problem can try to answer: are there any relevant messages in any of the Jenkins logs. Any exceptions, notices about failure to load are useful. what job types do problem occur on? maven, freestyle, other. are you running Jenkins direct from the war file or within tomcat/similar environment. what OS are your masters/slaves running? Answers to those will help to point towards the right person to fix/investigate further.
            Hide
            hnain Harpreet Nain added a comment -

            Richard,

            Here are the answers as per our environment:

            1. There is nothing that we can see coming through in the logs
            2. We are using freestyle jobs
            3. The problem occurs on both i.e running direct from war file and from tomcat.
            4. We dont have a master / slave setup. We have 2 instances one on linux and and one on windows. It happens on both.

            Show
            hnain Harpreet Nain added a comment - Richard, Here are the answers as per our environment: 1. There is nothing that we can see coming through in the logs 2. We are using freestyle jobs 3. The problem occurs on both i.e running direct from war file and from tomcat. 4. We dont have a master / slave setup. We have 2 instances one on linux and and one on windows. It happens on both.
            Hide
            nickolay_martinov Nikolay Martynov added a comment -

            1. Havent spotted exceptions. The only thing reporting error is Copy artifact plugin that cant find a build
            2. Freestyle jobs. Before 1.505 in matrix jobs builds disappeared veeery often . Matrix jobs dont seem to disappear now or eat least not so often
            3. Running using winstone embedded in jenkins
            4. HP-UX 11.21 Itanium Java 7 master. HP-UX 11.31 Itanium Java 7 and RHEL 5.4/5.6/5.8 Java 7 slaves.

            Show
            nickolay_martinov Nikolay Martynov added a comment - 1. Havent spotted exceptions. The only thing reporting error is Copy artifact plugin that cant find a build 2. Freestyle jobs. Before 1.505 in matrix jobs builds disappeared veeery often . Matrix jobs dont seem to disappear now or eat least not so often 3. Running using winstone embedded in jenkins 4. HP-UX 11.21 Itanium Java 7 master. HP-UX 11.31 Itanium Java 7 and RHEL 5.4/5.6/5.8 Java 7 slaves.
            Hide
            wbauer wbauer added a comment -

            1. Nothing visible in the logs.
            2. We use freestyle jobs across the board.
            3. Running from war file.
            4. master jenkins 1.505 on Mac OS X 10.8.2 / Java 1.7 latest, slaves Win XP/7/w2k8s + RedHat 6.2 with Java 1.6 + Mac OS X 10.6/7/8 with Java 1.6

            Show
            wbauer wbauer added a comment - 1. Nothing visible in the logs. 2. We use freestyle jobs across the board. 3. Running from war file. 4. master jenkins 1.505 on Mac OS X 10.8.2 / Java 1.7 latest, slaves Win XP/7/w2k8s + RedHat 6.2 with Java 1.6 + Mac OS X 10.6/7/8 with Java 1.6
            Hide
            mddubs Matt Williamson added a comment -

            FYI - This issue hasn't occurred again for us since "Reload from Disk" and updating all plugins about a week ago. That's progress, this use to happen all the time.

            1. Can't find anything in the logs.
            2. All freestyle projects.
            3. Running from war file I assume.
            4. Amazon Linux (amzn-ami-pv-2012.03.3.x86_64-ebs (ami-aecd60c7)), installed Jenkins through yum.

            Show
            mddubs Matt Williamson added a comment - FYI - This issue hasn't occurred again for us since "Reload from Disk" and updating all plugins about a week ago. That's progress, this use to happen all the time. 1. Can't find anything in the logs. 2. All freestyle projects. 3. Running from war file I assume. 4. Amazon Linux (amzn-ami-pv-2012.03.3.x86_64-ebs (ami-aecd60c7)), installed Jenkins through yum.
            Hide
            alex01ves Alex Vesely added a comment -

            I'm not sure, but at one time I was under the impression that this issue is seen more often with newly-created jobs - i.e., jobs created during the current Jenkins session.
            I'm using the Jenkins CLI to create chains of jobs (A runs B, B runs C, etc). After several runs of the chain, some or all builds disappear.

            1. Haven't seen anything. Where should I look?
            2. "Freestyle -> matrix -> matrix -> freestyle" chain
            3. From the .war
            4. SLES 10 for the master, SLES 10, Win2003, Win2008 slaves.

            Show
            alex01ves Alex Vesely added a comment - I'm not sure, but at one time I was under the impression that this issue is seen more often with newly-created jobs - i.e., jobs created during the current Jenkins session. I'm using the Jenkins CLI to create chains of jobs (A runs B, B runs C, etc). After several runs of the chain, some or all builds disappear. 1. Haven't seen anything. Where should I look? 2. "Freestyle -> matrix -> matrix -> freestyle" chain 3. From the .war 4. SLES 10 for the master, SLES 10, Win2003, Win2008 slaves.
            Hide
            oldelvet Richard Mortimer added a comment -

            @Alex Vesely you can generally find the logs at
            http://<myjenkins.url>/log/all
            (replace <myjenkins.uk> with the base URL of your instance).

            Show
            oldelvet Richard Mortimer added a comment - @Alex Vesely you can generally find the logs at http://<myjenkins.url>/log/all (replace <myjenkins.uk> with the base URL of your instance).
            Hide
            oldelvet Richard Mortimer added a comment -

            All, thanks for you quick feedback. It certainly looks like freestyle jobs are affected and maybe other types of jobs too. Certainly there no obvious common denominator in what has been reported.

            Ok. So I guess its time to look for a few more possibilities.

            • does it happen on any specific revision control system? SVN, GIT etc.
            • what plugins are loaded in your jenkins and are there any common build steps added to failing jobs.
            • are there any jobs on your system that do not seem to go missing?
            Show
            oldelvet Richard Mortimer added a comment - All, thanks for you quick feedback. It certainly looks like freestyle jobs are affected and maybe other types of jobs too. Certainly there no obvious common denominator in what has been reported. Ok. So I guess its time to look for a few more possibilities. does it happen on any specific revision control system? SVN, GIT etc. what plugins are loaded in your jenkins and are there any common build steps added to failing jobs. are there any jobs on your system that do not seem to go missing?
            Hide
            josesa Jose Sa added a comment -

            I was trying to create some script to crosscheck locally available builds with the ones reported on web page using REST api appending to job "/api/json?tree=builds[number]" and noticed something curious, not all builds are reported in that list when comparing with the ones actually visible in build history, only the most recent ones on variable ammount.

            Not sure if there is any limitation/restriction to the api, but maybe this is a symptom of this problem because if it already thinks it has less builds that it actually has, on some cleanup procedure it might only keep the ones reported and discard the rest.

            Show
            josesa Jose Sa added a comment - I was trying to create some script to crosscheck locally available builds with the ones reported on web page using REST api appending to job "/api/json?tree=builds [number] " and noticed something curious, not all builds are reported in that list when comparing with the ones actually visible in build history, only the most recent ones on variable ammount. Not sure if there is any limitation/restriction to the api, but maybe this is a symptom of this problem because if it already thinks it has less builds that it actually has, on some cleanup procedure it might only keep the ones reported and discard the rest.
            Hide
            hnain Harpreet Nain added a comment -

            Richard,

            We are using Git. Builds disappear randomly. I have not seen any pattern. On one occasion a build that had been run about 200 times without issues, lost all the build history which obviously reappeared when we did a reload.

            I am attaching screenshots of the plugins installed and their versions

            Show
            hnain Harpreet Nain added a comment - Richard, We are using Git. Builds disappear randomly. I have not seen any pattern. On one occasion a build that had been run about 200 times without issues, lost all the build history which obviously reappeared when we did a reload. I am attaching screenshots of the plugins installed and their versions
            Hide
            hnain Harpreet Nain added a comment -

            HN

            Show
            hnain Harpreet Nain added a comment - HN
            Hide
            nickolay_martinov Nikolay Martynov added a comment -

            Using SVN, standard plugin. Using only matrix and freestyle jobs. Previously both were disappearing like crazy. Recently matrix jobs seems to become more stable. Freestyle jobs still have problems. Perhaps makes to note that jobs are not standalone: both matrix and freestyle jobs trigger each other (build other builds) heavily.

            Installed plugins (env injector is disabled):
            Jenkins Mailer Plugin 1.4
            External Monitor Job Type Plugin 1.1
            LDAP Plugin 1.2
            pam-auth 1.0
            Javadoc Plugin 1.1
            Token Macro Plugin
            This plug-in adds reusable macro expansion capability for other plug-ins to use. 1.6
            Maven 2 Project Plugin 1.505
            Jenkins Subversion Plug-in 1.45
            Ant Plugin 1.2
            Jenkins CVS Plug-in
            Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient. 2.8
            Dump Info BuildWrapper Plugin
            The Dump Info BuildWrapper Plugin allows jobs to automatically dump some important Hudson-specific information into the job log. 1.1
            Hudson platformlabeler plugin
            Assigns labels on nodes automatically based on the computer operating system. 1.1
            Tracking SVN Plugin 1.1
            Priority Sorter
            This plugin allows for the build queue to be sorted based on pre-assigned priorities for each job. 1.3
            Jenkins Translation Assistance plugin 1.10
            Static Analysis Utilities
            This plug-in provides utilities for the static code analysis plug-ins.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 1.49
            Task Scanner Plugin
            This plug-in scans for open tasks in a specified set of files in the project modules and visualizes the results.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.35
            Disk Usage Plugin
            This plugin counts disk usage. 0.18
            SSH Slaves plugin 0.22
            FindBugs Plug-in
            This plug-in collects the FindBugs analysis results of the project modules and visualizes the found warnings.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.48
            Checkstyle Plug-in
            This plug-in collects the Checkstyle analysis results of the project modules and visualizes the found warnings.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 3.35
            Git Client Plugin 1.0.3
            Git Plugin
            This plugin integrates GIT with Jenkins. 1.2.0
            View Job Filters
            Create smart views with exactly the jobs you want. Your smart views can automatically include or exclude jobs by using things like the SCM path or type, the job type, build statuses or trends or triggers, relevance to the logged-in user, email recipients, Maven configuration, job parameterization, and user permissions. Mix and match filters to narrow down to exactly what you want. 1.22
            Warnings Plug-in
            This plug-in collects the compiler warnings of the project modules and visualizes the results.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.23
            PMD Plug-in
            This plug-in collects the PMD analysis results of the project modules and visualizes the found warnings.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 3.34
            Copy Artifact Plugin
            Adds a build step to copy artifacts from another project. 1.25
            Downstream build view
            This plug-in allows you to view the full status all the downstream builds so that we can graphically see that everything for this build has been completed successfully. 1.8
            Hudson Blame Subversion Plug-in 1.200
            Groovy Postbuild
            Groovy postbuild task. 1.8
            groovyaxis
            This plugin is a sample to explain how to write a Jenkins plugin. 0.3
            Duplicate Code Scanner Plug-in
            This plug-in collects the duplicate code analysis results of the project modules and visualizes the found warnings. Supported duplicate code analysis tools:
            • PMD's Copy Paste Detector (CPD)
            • Simian
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 2.34
            Static Analysis Collector Plug-in
            This plug-in is an add-on for the plug-ins Checkstyle, Dry, FindBugs, PMD, Tasks, and Warnings: the plug-in collects the different analysis results and shows the results in a combined trend graph. Additionally, the plug-in provides health reporting and build stability based on these combined results.
            If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 1.35
            Slave Prerequisites plugin
            Run a script on slave node before starting job on it to check some prerequisites. 1.0
            Matrix Tie Parent plugin 1.1
            Jenkins Maven Invoker plugin 1.2
            M2 Repository Cleanup Plugin
            This plugin allows you to configure a maven2 job to clean some or all of the artifacts from the repository before it runs. 1.0
            Jenkins Multijob plugin
            This plugin is a MultiJob plugin. 1.8
            Jenkins Maven Release Plug-in Plug-in
            A plug-in that enables you to perform releases using the maven-release-plugin from Hudson. 0.9.1
            Config File Provider Plugin
            Ability to provide configuration files (e.g. settings.xml for maven, XML, groovy, custom files,...) loaded trough the UI which will be copy to the job workspace. 2.4
            Run Condition Plugin 0.10
            conditional-buildstep
            A buildstep wrapping any number of other buildsteps, controlling there execution based on a defined condition (e.g. BuildParameter). 1.2.1
            Copy data to workspace plugin
            This plugin copies data to workspace directory before each build. After build data will be deleted. 1.0
            Tool Environment plugin
            Lets you use "tools" in unusual ways, such as from shell scripts. 1.0
            Copy To Slave Plugin
            This plugin allows copying files located somewhere on the master node into the jobs' workspaces, whether their builds take place on the master node or on slave nodes. 1.4
            Async Http Client
            This plugin provides a shared dependency on the async-http-client library so that other plugins can co-operate when using this library. 1.7.8
            Maven Deployment Linker 1.5
            Build Pipeline Plugin
            This plugin renders upstream and downstream connected jobs that typically form a build pipeline. In addition, it offers the ability to define manual triggers for jobs that require intervention prior to execution, e.g. an approval process outside of Jenkins. 1.3.3
            Downstream-Ext
            This plugin provides extended options for triggering downstream projects. 1.8
            Matrix Reloaded Plugin
            The Matrix Reloaded plugin. Used for rebuilding a subset of a matrix' configurations, reusing the rest. 1.1.0
            Environment Injector Plugin
            This plugin makes it possible to set an environment for the builds. 1.83
            Jenkins Maven Repository Server Plugin 0.11
            Hudson Groovy builder
            This plugin executes Groovy code. 1.13
            All changes plugin
            This plugin shows all changes (also from dependent projects, sub-projects, ...) for a project 1.3
            Pending Changes plugin
            This plugin adds a view of all changes in the scm after the last build 0.3.0
            Changes since last successfull build Plugin
            Generate changelog for a range of jenkins builds. 0.5

            Show
            nickolay_martinov Nikolay Martynov added a comment - Using SVN, standard plugin. Using only matrix and freestyle jobs. Previously both were disappearing like crazy. Recently matrix jobs seems to become more stable. Freestyle jobs still have problems. Perhaps makes to note that jobs are not standalone: both matrix and freestyle jobs trigger each other (build other builds) heavily. Installed plugins (env injector is disabled): Jenkins Mailer Plugin 1.4 External Monitor Job Type Plugin 1.1 LDAP Plugin 1.2 pam-auth 1.0 Javadoc Plugin 1.1 Token Macro Plugin This plug-in adds reusable macro expansion capability for other plug-ins to use. 1.6 Maven 2 Project Plugin 1.505 Jenkins Subversion Plug-in 1.45 Ant Plugin 1.2 Jenkins CVS Plug-in Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient. 2.8 Dump Info BuildWrapper Plugin The Dump Info BuildWrapper Plugin allows jobs to automatically dump some important Hudson-specific information into the job log. 1.1 Hudson platformlabeler plugin Assigns labels on nodes automatically based on the computer operating system. 1.1 Tracking SVN Plugin 1.1 Priority Sorter This plugin allows for the build queue to be sorted based on pre-assigned priorities for each job. 1.3 Jenkins Translation Assistance plugin 1.10 Static Analysis Utilities This plug-in provides utilities for the static code analysis plug-ins. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 1.49 Task Scanner Plugin This plug-in scans for open tasks in a specified set of files in the project modules and visualizes the results. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.35 Disk Usage Plugin This plugin counts disk usage. 0.18 SSH Slaves plugin 0.22 FindBugs Plug-in This plug-in collects the FindBugs analysis results of the project modules and visualizes the found warnings. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.48 Checkstyle Plug-in This plug-in collects the Checkstyle analysis results of the project modules and visualizes the found warnings. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 3.35 Git Client Plugin 1.0.3 Git Plugin This plugin integrates GIT with Jenkins. 1.2.0 View Job Filters Create smart views with exactly the jobs you want. Your smart views can automatically include or exclude jobs by using things like the SCM path or type, the job type, build statuses or trends or triggers, relevance to the logged-in user, email recipients, Maven configuration, job parameterization, and user permissions. Mix and match filters to narrow down to exactly what you want. 1.22 Warnings Plug-in This plug-in collects the compiler warnings of the project modules and visualizes the results. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 4.23 PMD Plug-in This plug-in collects the PMD analysis results of the project modules and visualizes the found warnings. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 3.34 Copy Artifact Plugin Adds a build step to copy artifacts from another project. 1.25 Downstream build view This plug-in allows you to view the full status all the downstream builds so that we can graphically see that everything for this build has been completed successfully. 1.8 Hudson Blame Subversion Plug-in 1.200 Groovy Postbuild Groovy postbuild task. 1.8 groovyaxis This plugin is a sample to explain how to write a Jenkins plugin. 0.3 Duplicate Code Scanner Plug-in This plug-in collects the duplicate code analysis results of the project modules and visualizes the found warnings. Supported duplicate code analysis tools: • PMD's Copy Paste Detector (CPD) • Simian If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 2.34 Static Analysis Collector Plug-in This plug-in is an add-on for the plug-ins Checkstyle, Dry, FindBugs, PMD, Tasks, and Warnings: the plug-in collects the different analysis results and shows the results in a combined trend graph. Additionally, the plug-in provides health reporting and build stability based on these combined results. If you like this open source plug-in please consider supporting my work by buying my Android game Inca Trails. 1.35 Slave Prerequisites plugin Run a script on slave node before starting job on it to check some prerequisites. 1.0 Matrix Tie Parent plugin 1.1 Jenkins Maven Invoker plugin 1.2 M2 Repository Cleanup Plugin This plugin allows you to configure a maven2 job to clean some or all of the artifacts from the repository before it runs. 1.0 Jenkins Multijob plugin This plugin is a MultiJob plugin. 1.8 Jenkins Maven Release Plug-in Plug-in A plug-in that enables you to perform releases using the maven-release-plugin from Hudson. 0.9.1 Config File Provider Plugin Ability to provide configuration files (e.g. settings.xml for maven, XML, groovy, custom files,...) loaded trough the UI which will be copy to the job workspace. 2.4 Run Condition Plugin 0.10 conditional-buildstep A buildstep wrapping any number of other buildsteps, controlling there execution based on a defined condition (e.g. BuildParameter). 1.2.1 Copy data to workspace plugin This plugin copies data to workspace directory before each build. After build data will be deleted. 1.0 Tool Environment plugin Lets you use "tools" in unusual ways, such as from shell scripts. 1.0 Copy To Slave Plugin This plugin allows copying files located somewhere on the master node into the jobs' workspaces, whether their builds take place on the master node or on slave nodes. 1.4 Async Http Client This plugin provides a shared dependency on the async-http-client library so that other plugins can co-operate when using this library. 1.7.8 Maven Deployment Linker 1.5 Build Pipeline Plugin This plugin renders upstream and downstream connected jobs that typically form a build pipeline. In addition, it offers the ability to define manual triggers for jobs that require intervention prior to execution, e.g. an approval process outside of Jenkins. 1.3.3 Downstream-Ext This plugin provides extended options for triggering downstream projects. 1.8 Matrix Reloaded Plugin The Matrix Reloaded plugin. Used for rebuilding a subset of a matrix' configurations, reusing the rest. 1.1.0 Environment Injector Plugin This plugin makes it possible to set an environment for the builds. 1.83 Jenkins Maven Repository Server Plugin 0.11 Hudson Groovy builder This plugin executes Groovy code. 1.13 All changes plugin This plugin shows all changes (also from dependent projects, sub-projects, ...) for a project 1.3 Pending Changes plugin This plugin adds a view of all changes in the scm after the last build 0.3.0 Changes since last successfull build Plugin Generate changelog for a range of jenkins builds. 0.5
            Hide
            nickolay_martinov Nikolay Martynov added a comment -

            Jose has made interesting note. I recall that sometimes we can see build listed in UI but Copy Artifacts Plugin seems to fail to find the build.

            Show
            nickolay_martinov Nikolay Martynov added a comment - Jose has made interesting note. I recall that sometimes we can see build listed in UI but Copy Artifacts Plugin seems to fail to find the build.
            Hide
            ivves Ivan Veselkov added a comment - - edited

            We have a matrix job with two configurations (os = hpux or linux ). There is a keep-forever build which appears in parent history but it is absent in histories of configurations. From parent build one can still access both configurations.
            The copy plugin almost always cannot copy artifacts from this build: "Unable to find a build for artifact copy from" but at least once it succeeded for os=hpux.

            Show
            ivves Ivan Veselkov added a comment - - edited We have a matrix job with two configurations (os = hpux or linux ). There is a keep-forever build which appears in parent history but it is absent in histories of configurations. From parent build one can still access both configurations. The copy plugin almost always cannot copy artifacts from this build: "Unable to find a build for artifact copy from" but at least once it succeeded for os=hpux.
            Hide
            fmerrow Frank Merrow added a comment -

            Just chiming in with "me too" on 1.502 . . . it is new installation and no matter what I do it only seems to show jobs from "today". (Have not been around at midnight to look . . . but in morning onward I only see jobs that ran today).

            FWIW: If I look in the jobs directory, the correct number of history directories are there . . . but web view just doesn't show them. (???)

            Windows7, no slaves, nothing odd about the installation that isn't in 10 others I have that are working on Jenkins from 1.488 onward.

            Also, I have a second 1.502 installation that was brought on-line around the same time . . . everything shows there. I cannot say the plugin mix on the two systems is identical, but it is very similar.

            Show
            fmerrow Frank Merrow added a comment - Just chiming in with "me too" on 1.502 . . . it is new installation and no matter what I do it only seems to show jobs from "today". (Have not been around at midnight to look . . . but in morning onward I only see jobs that ran today). FWIW: If I look in the jobs directory, the correct number of history directories are there . . . but web view just doesn't show them. (???) Windows7, no slaves, nothing odd about the installation that isn't in 10 others I have that are working on Jenkins from 1.488 onward. Also, I have a second 1.502 installation that was brought on-line around the same time . . . everything shows there. I cannot say the plugin mix on the two systems is identical, but it is very similar.
            Hide
            josesa Jose Sa added a comment - - edited

            Both "/api/json?tree=builds[number]" and "/rssAll" job urls seem to be returning only latest builds (not all as expected), like I wrote above this could be a symptom for this case.

            Luckily "/buildHistory/all" does still provide all visible builds, so I was able to create created a bash script to help automatically to check for this kind of problems and provide a text report.

            # Assuming some host specific variables
            jenkins_url=https://localhost:8080
            jobs_dir=/work/jenkins/jobs
            
            # This should work for any unix environment using bash
            for job_dir in ${jobs_dir}/*; do
                job=$(basename ${job_dir})
                echo "Checking $job..." >&2
                job_url=${jenkins_url}/job/${job}
                wget -q --no-check-certificate -O all ${job_url}/buildHistory/all
                gawk -F'#' '/ #/{print $2}' all > online_builds
                find ${jobs_dir}/${job}/builds/* -type l ! -name "last*" -prune | sed "s,${jobs_dir}/${job}/builds/,,g" > local_builds
                missing=$(gawk '
                BEGIN {first_file = ""}
                {
                    if (first_file == "") first_file = FILENAME
                    if (first_file == FILENAME) {build_list[$1]=1} else {delete build_list[$1]}
                }
                END {for (build in build_list) print build}
                ' local_builds online_builds)
                [ ! -z "${missing}" ] && echo Job $job missing builds: $missing
            done | tee missing_builds.txt
            

            I found out that after 3 days online, 85 (out of 1406) jobs are showing missing builds varying from a few to all.

            I have FreeStyle and MultiJob types and they both show this problem.
            We are using mixed ClearCase and SVN (both showing same problems)
            Server is Solaris 10 Sparc, having slaves on HP-UX, Windows (XP, 2k3, 7), RHEL (5, 6), Solaris.
            Running Jenkins from war file, currently with 1.505 (but noticing this problem since 1.485)

            Show
            josesa Jose Sa added a comment - - edited Both "/api/json?tree=builds [number] " and "/rssAll" job urls seem to be returning only latest builds (not all as expected), like I wrote above this could be a symptom for this case. Luckily "/buildHistory/all" does still provide all visible builds, so I was able to create created a bash script to help automatically to check for this kind of problems and provide a text report. # Assuming some host specific variables jenkins_url=https: //localhost:8080 jobs_dir=/work/jenkins/jobs # This should work for any unix environment using bash for job_dir in ${jobs_dir}/*; do job=$(basename ${job_dir}) echo "Checking $job..." >&2 job_url=${jenkins_url}/job/${job} wget -q --no-check-certificate -O all ${job_url}/buildHistory/all gawk -F '#' '/ #/{print $2}' all > online_builds find ${jobs_dir}/${job}/builds/* -type l ! -name "last*" -prune | sed "s,${jobs_dir}/${job}/builds/,,g" > local_builds missing=$(gawk ' BEGIN {first_file = ""} { if (first_file == "") first_file = FILENAME if (first_file == FILENAME) {build_list[$1]=1} else {delete build_list[$1]} } END { for (build in build_list) print build} ' local_builds online_builds) [ ! -z "${missing}" ] && echo Job $job missing builds: $missing done | tee missing_builds.txt I found out that after 3 days online, 85 (out of 1406) jobs are showing missing builds varying from a few to all. I have FreeStyle and MultiJob types and they both show this problem. We are using mixed ClearCase and SVN (both showing same problems) Server is Solaris 10 Sparc, having slaves on HP-UX, Windows (XP, 2k3, 7), RHEL (5, 6), Solaris. Running Jenkins from war file, currently with 1.505 (but noticing this problem since 1.485)
            Hide
            nickolay_martinov Nikolay Martynov added a comment -

            I confirm that script is perfect reproducer:

            Checking Infrastructure_update_sys...
            Checking Infrastructure_update_valgrind...
            Checking NOM_6_build...
            Job NOM_6_build missing builds: 887 618
            Checking NOM_6_kit...
            Job NOM_6_kit missing builds: 204
            Checking NOM_6_kit_smoke...
            Job NOM_6_kit_smoke missing builds: 139
            Checking NOM_6_test_f...
            Checking NOM_6_test_f_m...
            Job NOM_6_test_f_m missing builds: 88
            Checking NOM_6_test_l...
            Checking NOM_6_test_l_m...
            Job NOM_6_test_l_m missing builds: 204
            Checking NOM_6_test_sdk...
            Job NOM_6_test_sdk missing builds: 85
            Checking NOM_6_test_sdk_m...
            Checking NOM_V600_build...
            Checking NOM_V600_kit...
            Checking NOM_V600_kit_smoke...

            Show
            nickolay_martinov Nikolay Martynov added a comment - I confirm that script is perfect reproducer: Checking Infrastructure_update_sys... Checking Infrastructure_update_valgrind... Checking NOM_6_build... Job NOM_6_build missing builds: 887 618 Checking NOM_6_kit... Job NOM_6_kit missing builds: 204 Checking NOM_6_kit_smoke... Job NOM_6_kit_smoke missing builds: 139 Checking NOM_6_test_f... Checking NOM_6_test_f_m... Job NOM_6_test_f_m missing builds: 88 Checking NOM_6_test_l... Checking NOM_6_test_l_m... Job NOM_6_test_l_m missing builds: 204 Checking NOM_6_test_sdk... Job NOM_6_test_sdk missing builds: 85 Checking NOM_6_test_sdk_m... Checking NOM_V600_build... Checking NOM_V600_kit... Checking NOM_V600_kit_smoke...
            Hide
            josesa Jose Sa added a comment -

            Got this exception on some jobs when doing the wget for <job>/buildHistory/all a few minutes after jenkins was restarted.

            Mar 29, 2013 9:26:04 PM hudson.ExpressionFactory2$JexlExpression evaluate
            WARNING: Caught exception evaluating: it.renderList in /hudson/job/PM_OSS5_EBSS_LV2_FUNCTIONAL_GA12/buildHistory/all. Reason: java.lan
            g.reflect.InvocationTargetException
            java.lang.reflect.InvocationTargetException
                    at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:601)
                    at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
                    at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
                    at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
                    at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
                    at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
                    at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
                    at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
                    at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
                    at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
                    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
                    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
                    at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
                    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
                    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
                    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
                    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
                    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                    at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
                    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
                    at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
                    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
                    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
                    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
                    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
                    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
                    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
                    at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
                    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
                    at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
                    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
                    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
                    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
                    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
                    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487)
                    at org.kohsuke.stapler.Stapler.service(Stapler.java:164)
                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
                    at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
                    at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
                    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
                    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
                    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
                    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
                    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
                    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
                    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
                    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
                    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
                    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
                    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
                    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
                    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
                    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
                    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
                    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
                    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
                    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
                    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
                    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
                    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
                    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
                    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
                    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
                    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
                    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
                    at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
                    at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
                    at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                    at java.lang.Thread.run(Thread.java:722)
            Caused by: java.lang.NullPointerException
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214)
                    at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349)
                    at hudson.model.Run.onLoad(Run.java:319)
                    at hudson.model.RunMap.retrieve(RunMap.java:226)
                    at hudson.model.RunMap.retrieve(RunMap.java:59)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358)
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214)
                    at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349)
                    at hudson.model.Run.onLoad(Run.java:319)
                    at hudson.model.RunMap.retrieve(RunMap.java:226)
                    at hudson.model.RunMap.retrieve(RunMap.java:59)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358)
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214)
                    at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349)
                    at hudson.model.Run.onLoad(Run.java:319)
                    at hudson.model.RunMap.retrieve(RunMap.java:226)
                    at hudson.model.RunMap.retrieve(RunMap.java:59)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358)
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214)
                    at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349)
                    at hudson.model.Run.onLoad(Run.java:319)
                    at hudson.model.RunMap.retrieve(RunMap.java:226)
                    at hudson.model.RunMap.retrieve(RunMap.java:59)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358)
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214)
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:101)
                    at hudson.model.RunMap$1.next(RunMap.java:107)
                    at hudson.model.RunMap$1.next(RunMap.java:96)
                    at hudson.util.Iterators$2$1.next(Iterators.java:204)
                    at hudson.widgets.HistoryWidget.updateFirstTransientBuildKey(HistoryWidget.java:110)
                    at hudson.widgets.HistoryWidget.getRenderList(HistoryWidget.java:139)
                    ... 126 more
            

            Which seems related to another issue JENKINS-16845 regarding the specific NPE on AbstractBuild.getPreviousBuild.

            Show
            josesa Jose Sa added a comment - Got this exception on some jobs when doing the wget for <job>/buildHistory/all a few minutes after jenkins was restarted. Mar 29, 2013 9:26:04 PM hudson.ExpressionFactory2$JexlExpression evaluate WARNING: Caught exception evaluating: it.renderList in /hudson/job/PM_OSS5_EBSS_LV2_FUNCTIONAL_GA12/buildHistory/all. Reason: java.lan g.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487) at org.kohsuke.stapler.Stapler.service(Stapler.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang. Thread .run( Thread .java:722) Caused by: java.lang.NullPointerException at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:619) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:358) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:101) at hudson.model.RunMap$1.next(RunMap.java:107) at hudson.model.RunMap$1.next(RunMap.java:96) at hudson.util.Iterators$2$1.next(Iterators.java:204) at hudson.widgets.HistoryWidget.updateFirstTransientBuildKey(HistoryWidget.java:110) at hudson.widgets.HistoryWidget.getRenderList(HistoryWidget.java:139) ... 126 more Which seems related to another issue JENKINS-16845 regarding the specific NPE on AbstractBuild.getPreviousBuild.
            Hide
            oldelvet Richard Mortimer added a comment -

            Good detective work all!

            Caused by: java.lang.NullPointerException
                    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214)
                    at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349)
                    at hudson.model.Run.onLoad(Run.java:319)
                    at hudson.model.RunMap.retrieve(RunMap.java:226)
                    at hudson.model.RunMap.retrieve(RunMap.java:59)
                    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657)
            

            I think this relates to the following in AbstractBuild

                            R pb = getParent().builds.search(number-1, Direction.DESC);
            

            This could be builds not being initialised as per the suggested JENKINS-16845 or it could be getParent() returning null.

            getParent() returns the transient project value from Run.java

            protected transient final JobT project;
            

            @Jose Sa can you confirm what version you saw this error on.

            Show
            oldelvet Richard Mortimer added a comment - Good detective work all! Caused by: java.lang.NullPointerException at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:214) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:349) at hudson.model.Run.onLoad(Run.java:319) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) I think this relates to the following in AbstractBuild R pb = getParent().builds.search(number-1, Direction.DESC); This could be builds not being initialised as per the suggested JENKINS-16845 or it could be getParent() returning null. getParent() returns the transient project value from Run.java protected transient final JobT project; @Jose Sa can you confirm what version you saw this error on.
            Hide
            josesa Jose Sa added a comment - - edited

            I'm with core 1.505 now. Tried more recent versions like 1.506 and 1.508, but had to revert back due worse problems caused by NPE not being able to load jobs like the previous issue, but now extended to Promote plugin JENKINS-17381 along with weird changed in SSH Slaves plugin credentials definitions that got all messed up.

            Show
            josesa Jose Sa added a comment - - edited I'm with core 1.505 now. Tried more recent versions like 1.506 and 1.508, but had to revert back due worse problems caused by NPE not being able to load jobs like the previous issue, but now extended to Promote plugin JENKINS-17381 along with weird changed in SSH Slaves plugin credentials definitions that got all messed up.
            Hide
            josesa Jose Sa added a comment - - edited

            After some analysis of my previous script I realized it contained a lot of false positives possibly due to years of accumulated history of broken builds, java crashes, wrong rsync of jobs between servers, etc, that lead to the following most common problems:

            • numbered links pointing to removed build directories (solution: remove links)
            • build directories without build.xml file (solution: remove directories)
            • build directories without numbered links (solution: create links)
            • build directories not in date format (solution: delete directories)

            Since these happen often I created some scripts to:

            • Detect those common problems with optional cleanup (passing argument '-c') - check_invalid_builds.sh
            • Checking the missing builds as given as excerpt above - check_missing_builds.sh

            NOTE: to effectively apply the cleanups first run the script without '-c' to run in preview mode and review the suggestions, when ready to apply them shutdown Jenkins first so that there are no files pending in memory to be written to file system.

            Tested in RHEL6 and Solaris 10 (with gnu utils)

            Show
            josesa Jose Sa added a comment - - edited After some analysis of my previous script I realized it contained a lot of false positives possibly due to years of accumulated history of broken builds, java crashes, wrong rsync of jobs between servers, etc, that lead to the following most common problems: numbered links pointing to removed build directories (solution: remove links) build directories without build.xml file (solution: remove directories) build directories without numbered links (solution: create links) build directories not in date format (solution: delete directories) Since these happen often I created some scripts to: Detect those common problems with optional cleanup (passing argument '-c') - check_invalid_builds.sh Checking the missing builds as given as excerpt above - check_missing_builds.sh NOTE: to effectively apply the cleanups first run the script without '-c' to run in preview mode and review the suggestions, when ready to apply them shutdown Jenkins first so that there are no files pending in memory to be written to file system. Tested in RHEL6 and Solaris 10 (with gnu utils)
            Hide
            jglick Jesse Glick added a comment -

            Seems to be the same as JENKINS-16845.

            Show
            jglick Jesse Glick added a comment - Seems to be the same as JENKINS-16845 .
            Hide
            westhusing Adam Westhusing added a comment - - edited

            The shell scripts don't work when the jobs have spaces in the names. What modifications need to be made to the scripts to adapt it to be able to work when there are spaces in the job names?

            Additionally, running this script in Cygwin on Windows is generating output that indicates EVERY build has a missing build link.

            Show
            westhusing Adam Westhusing added a comment - - edited The shell scripts don't work when the jobs have spaces in the names. What modifications need to be made to the scripts to adapt it to be able to work when there are spaces in the job names? Additionally, running this script in Cygwin on Windows is generating output that indicates EVERY build has a missing build link.
            Hide
            josesa Jose Sa added a comment -

            Allowing jobs with spaces in names does complicate things, making it harder (but not impossible) to handle via script. I prefer not to allow any spaces using "Create Job Advanced" plugin to enforce some common rules.

            To allow spaces, basically any path that contains the job name should be enclosed with double quotes either on variable assignment as well in usage. Since command "find" doesn't like to have wildcard enclosed in double quotes, that can be split into first a change directory and then the find command with wildcard. I uploaded new versions of scripts but it didn't seem to have replaced the old ones when uploaded, so refer to the most recent ones.

            Jenkins on Windows also seems to have a different behavior since it doesn't seem to use any symbolic links (not visible on cygwin anyway with 1.484), so the the link related checks probably are not applicable on Windows.

            Show
            josesa Jose Sa added a comment - Allowing jobs with spaces in names does complicate things, making it harder (but not impossible) to handle via script. I prefer not to allow any spaces using "Create Job Advanced" plugin to enforce some common rules. To allow spaces, basically any path that contains the job name should be enclosed with double quotes either on variable assignment as well in usage. Since command "find" doesn't like to have wildcard enclosed in double quotes, that can be split into first a change directory and then the find command with wildcard. I uploaded new versions of scripts but it didn't seem to have replaced the old ones when uploaded, so refer to the most recent ones. Jenkins on Windows also seems to have a different behavior since it doesn't seem to use any symbolic links (not visible on cygwin anyway with 1.484), so the the link related checks probably are not applicable on Windows.
            Hide
            westhusing Adam Westhusing added a comment -

            Thanks for updating the script, but I suppose the behavior is different on Windows. Jenkins has never created symbolic links on Windows until upgrading from 1.506 to 1.509. In the logs, Jenkins is complaining all over the place that it can't find directories but the build directories are actually there and nothing appears to be wrong when bringing up the specific build being complained about in the Jenkins logs.

            Show
            westhusing Adam Westhusing added a comment - Thanks for updating the script, but I suppose the behavior is different on Windows. Jenkins has never created symbolic links on Windows until upgrading from 1.506 to 1.509. In the logs, Jenkins is complaining all over the place that it can't find directories but the build directories are actually there and nothing appears to be wrong when bringing up the specific build being complained about in the Jenkins logs.
            Hide
            josesa Jose Sa added a comment -

            You can remove from the script the link sections in Windows environment. But probably in some past upgrade it may have been possible that this same behaviour was accidentally induced in Unix environments which could explain why so many situations similar to that.

            Still after running the cleanup and running for a few days I don't the problem happening so often now. Actually i saw it happening right in the the first day, but somehow it "recovered" without doing a reload from disk or restart.

            Show
            josesa Jose Sa added a comment - You can remove from the script the link sections in Windows environment. But probably in some past upgrade it may have been possible that this same behaviour was accidentally induced in Unix environments which could explain why so many situations similar to that. Still after running the cleanup and running for a few days I don't the problem happening so often now. Actually i saw it happening right in the the first day, but somehow it "recovered" without doing a reload from disk or restart.
            Hide
            egsejenkins Reece Johnston added a comment -

            I just saw the same issue on version 1.511. It occurred after renaming a job and running that job with a scheduled build. Oddly, it did not occur with the first few manual builds.

            Show
            egsejenkins Reece Johnston added a comment - I just saw the same issue on version 1.511. It occurred after renaming a job and running that job with a scheduled build. Oddly, it did not occur with the first few manual builds.
            Hide
            deepchip Martin d'Anjou added a comment -

            Seeing this problem coming back on LTS 1.509.2. I have no idea how to debug...

            Show
            deepchip Martin d'Anjou added a comment - Seeing this problem coming back on LTS 1.509.2. I have no idea how to debug...
            Hide
            seder Stefan Prietl added a comment -

            Same problem with 1.509.1 LTS.

            We are using two Jenkins instances with the same version right now, and this issue seems to affect only the one where the builds are executed on slaves only. Additionally only builds which where renamed and are triggered automatically by croned upstream projects are having this issue.

            The only thing I noticed in the logs was the following:

            Aug 01, 2013 12:44:28 AM hudson.model.Run execute
            INFO: some-build #15 main build action completed: SUCCESS
            Aug 01, 2013 12:21:20 AM hudson.model.listeners.RunListener report
            WARNING: RunListener failed
            java.lang.NullPointerException
            

            This happened right before/during every build which "lost" builds. However, this does not occur on the Jenkins instance which builds on master only.
            Unfortunately there was no trace attached to the NullPointerException in the log, but I'll try to get one with a higher log output setting. Maybe it's connected to issue JENKINS-18823.

            Show
            seder Stefan Prietl added a comment - Same problem with 1.509.1 LTS. We are using two Jenkins instances with the same version right now, and this issue seems to affect only the one where the builds are executed on slaves only . Additionally only builds which where renamed and are triggered automatically by croned upstream projects are having this issue. The only thing I noticed in the logs was the following: Aug 01, 2013 12:44:28 AM hudson.model.Run execute INFO: some-build #15 main build action completed: SUCCESS Aug 01, 2013 12:21:20 AM hudson.model.listeners.RunListener report WARNING: RunListener failed java.lang.NullPointerException This happened right before/during every build which "lost" builds. However, this does not occur on the Jenkins instance which builds on master only. Unfortunately there was no trace attached to the NullPointerException in the log, but I'll try to get one with a higher log output setting. Maybe it's connected to issue JENKINS-18823 .
            Hide
            binary binary added a comment -

            Status: Resolved
            Resolution: Duplicate


            If it is resolved, why is it still appearing?

            We have it for multiple jobs, including jobs that are triggered manually. Core 1.526, master on Ubuntu 10.04, slaves on different platforms (linux, windows, os x; all are affected).

            Show
            binary binary added a comment - Status: Resolved Resolution: Duplicate If it is resolved, why is it still appearing? We have it for multiple jobs, including jobs that are triggered manually. Core 1.526, master on Ubuntu 10.04, slaves on different platforms (linux, windows, os x; all are affected).
            Hide
            jglick Jesse Glick added a comment -

            @binary: this was marked as a duplicate of JENKINS-16845, a confirmed and fixed bug, based on a detailed match of symptoms with that bug. If you have a general issue with builds disappearing from history, it is probably a different cause and should be opened as a separate bug report with any steps to reproduce; or, failing that, exhaustive diagnostics such as warnings in the log, analysis of the contents of the builds directory, anomalies in build.xml files, etc.

            Show
            jglick Jesse Glick added a comment - @binary: this was marked as a duplicate of JENKINS-16845 , a confirmed and fixed bug, based on a detailed match of symptoms with that bug. If you have a general issue with builds disappearing from history, it is probably a different cause and should be opened as a separate bug report with any steps to reproduce; or, failing that, exhaustive diagnostics such as warnings in the log, analysis of the contents of the builds directory, anomalies in build.xml files, etc.
            Hide
            bavoiculescu Voiculescu Bogdan Alexandru added a comment -

            This is more than funny! I suggest you to open a new section for bugs: "Builds disappearing from history".
            Let's hope the 'reload configuration workarround' will work for next years.

            Show
            bavoiculescu Voiculescu Bogdan Alexandru added a comment - This is more than funny! I suggest you to open a new section for bugs: "Builds disappearing from history". Let's hope the 'reload configuration workarround' will work for next years.
            Hide
            eguess74 eguess74 added a comment -

            @Jesse Glick
            I tend to agree with others that it seems unreasonable to mark this issue as a resolved and duplicate, because obviously the fix of the JENKINS-16845 did not fix this issue. So, by asking users to resubmit another one we are adding unnecessary burden on users + "loosing all the information" that we have already gathered.
            Clearly the issue persist and by some reports even to the most current release (529) - so this is hardly a dup.

            I for one see this issue on the 513 - two releases after 16845 was pushed.
            But we have two instances and one is currently on LTS 509.1 and it doesn't seem to have such problem at all. The major difference between those - promoted builds plugin. We don't have it on LTS instance and we do have it on the 513.

            It would be nice if other reporters could confirm if they use promoted builds - if so may be this is the culprit?
            Just throwing an idea...

            Show
            eguess74 eguess74 added a comment - @Jesse Glick I tend to agree with others that it seems unreasonable to mark this issue as a resolved and duplicate, because obviously the fix of the JENKINS-16845 did not fix this issue. So, by asking users to resubmit another one we are adding unnecessary burden on users + "loosing all the information" that we have already gathered. Clearly the issue persist and by some reports even to the most current release (529) - so this is hardly a dup. I for one see this issue on the 513 - two releases after 16845 was pushed. But we have two instances and one is currently on LTS 509.1 and it doesn't seem to have such problem at all. The major difference between those - promoted builds plugin. We don't have it on LTS instance and we do have it on the 513. It would be nice if other reporters could confirm if they use promoted builds - if so may be this is the culprit? Just throwing an idea...
            Hide
            deepchip Martin d'Anjou added a comment -

            I do not use promoted builds and I do see the issue.

            Show
            deepchip Martin d'Anjou added a comment - I do not use promoted builds and I do see the issue.
            Hide
            wbauer wbauer added a comment -

            I see the issue on 1.525 still and don't use promoted builds.

            Show
            wbauer wbauer added a comment - I see the issue on 1.525 still and don't use promoted builds.
            Hide
            eguess74 eguess74 added a comment -

            ok, seems like the promoted builds is not the reason... something else then...

            Show
            eguess74 eguess74 added a comment - ok, seems like the promoted builds is not the reason... something else then...
            Hide
            jglick Jesse Glick added a comment -

            There may be (in fact probably are) multiple issues with similar-seeming symptoms; without a way to reproduce it may not be possible to tell. JENKINS-19418 is the one issue in this area with a diagnosis known to me.

            Show
            jglick Jesse Glick added a comment - There may be (in fact probably are) multiple issues with similar-seeming symptoms; without a way to reproduce it may not be possible to tell. JENKINS-19418 is the one issue in this area with a diagnosis known to me.
            Hide
            eguess74 eguess74 added a comment -

            The simplest workaround to get the full list of the builds that i found is to use the "trend" link on the top right corner of the build history. All the builds are listed there correctly even if they are not shown in the main history table. May be that fact can help pinpoint faulty code?

            Show
            eguess74 eguess74 added a comment - The simplest workaround to get the full list of the builds that i found is to use the "trend" link on the top right corner of the build history. All the builds are listed there correctly even if they are not shown in the main history table. May be that fact can help pinpoint faulty code?
            Hide
            jglick Jesse Glick added a comment -

            @eguess74: interesting. Means RunList.iterator() is working, though that is what the main history list (HistoryWidget) should be showing too. Might not actually matter which widget is being used to search through the list—could simply be that the first iteration triggers loading of something which does not get returned until the second time.

            Really we need better diagnostics in this area. JENKINS-19400 might help.

            Show
            jglick Jesse Glick added a comment - @eguess74: interesting. Means RunList.iterator() is working, though that is what the main history list ( HistoryWidget ) should be showing too. Might not actually matter which widget is being used to search through the list—could simply be that the first iteration triggers loading of something which does not get returned until the second time. Really we need better diagnostics in this area. JENKINS-19400 might help.
            Hide
            nickolay_martinov Nikolay Martynov added a comment -

            @Jesse
            It's not about first time load as reloading history page several times after displaying the trend page with full list still produces partial result for history widget. See it here: http://s13.postimg.org/kaccjxc07/Clipboard_1.gif
            The interesting thing, however, is that history widget shows exactly "Max # of builds to keep" builds. There are more than this numbers of builds due to keep forever and trend finds them. Setting "Max # of builds to keep" to bigger value, however, doesn't make the rest of builds appear in history widget. Might be something in xml that is corrupted because of "Max # of builds to keep" coupled with lazy loading. Just a guess...

            Show
            nickolay_martinov Nikolay Martynov added a comment - @Jesse It's not about first time load as reloading history page several times after displaying the trend page with full list still produces partial result for history widget. See it here: http://s13.postimg.org/kaccjxc07/Clipboard_1.gif The interesting thing, however, is that history widget shows exactly "Max # of builds to keep" builds. There are more than this numbers of builds due to keep forever and trend finds them. Setting "Max # of builds to keep" to bigger value, however, doesn't make the rest of builds appear in history widget. Might be something in xml that is corrupted because of "Max # of builds to keep" coupled with lazy loading. Just a guess...
            Hide
            iknd iikka eklund added a comment -

            We are having exactly the same problem with Jenkins version 1.547
            This error is closed but in which Jenkins version this is fixed?

            Show
            iknd iikka eklund added a comment - We are having exactly the same problem with Jenkins version 1.547 This error is closed but in which Jenkins version this is fixed?
            Hide
            alex_ouzounis Alex Ouzounis added a comment -

            Hi all,

            I am seeing the same problem occasionally and either restarting or reloading from disk fixes the problem.

            I have a feeling that its related with:

            "WARNING: Multiple builds have the same number: "

            because every time I delete conflicting builds it goes away.

            Show
            alex_ouzounis Alex Ouzounis added a comment - Hi all, I am seeing the same problem occasionally and either restarting or reloading from disk fixes the problem. I have a feeling that its related with: "WARNING: Multiple builds have the same number: " because every time I delete conflicting builds it goes away.
            Hide
            alex_ouzounis Alex Ouzounis added a comment -

            Problem no longer present in the latest versions ie 1.570

            Show
            alex_ouzounis Alex Ouzounis added a comment - Problem no longer present in the latest versions ie 1.570
            Hide
            eguess74 eguess74 added a comment -

            I'm seeing it happening right now on Jenkins 1.578
            I had a build that i triggered using gerrit trigger (query and trigger gerrit patches)
            It appeared in the GUI, I saw it running in the the build queue, it sent an email about build being fixed, registered the result with gerrit, but no matter how i'm trying to access it the link to the build number leads to 404. Build is present on the file system as well.

            Show
            eguess74 eguess74 added a comment - I'm seeing it happening right now on Jenkins 1.578 I had a build that i triggered using gerrit trigger (query and trigger gerrit patches) It appeared in the GUI, I saw it running in the the build queue, it sent an email about build being fixed, registered the result with gerrit, but no matter how i'm trying to access it the link to the build number leads to 404. Build is present on the file system as well.
            Hide
            jglick Jesse Glick added a comment -

            eguess74 if you are using the Gerrit Trigger plugin, you are probably seeing JENKINS-23152.

            Show
            jglick Jesse Glick added a comment - eguess74 if you are using the Gerrit Trigger plugin, you are probably seeing JENKINS-23152 .
            Hide
            eguess74 eguess74 added a comment -

            Jesse,

            Thanks for pointing me to the right direction!

            Show
            eguess74 eguess74 added a comment - Jesse, Thanks for pointing me to the right direction!
            Hide
            seaware Frank Canova added a comment -

            I'm seeing this same issue when changing between v1.590 and v1.610. Is there a work-around or way to recreate the build.xml files? The history directories are intact but history has disappeared from the web view.

            Show
            seaware Frank Canova added a comment - I'm seeing this same issue when changing between v1.590 and v1.610. Is there a work-around or way to recreate the build.xml files? The history directories are intact but history has disappeared from the web view.
            Hide
            danielbeck Daniel Beck added a comment -

            I'm seeing this same issue when changing between v1.590 and v1.610. Is there a work-around or way to recreate the build.xml files? The history directories are intact but history has disappeared from the web view.

            Unrelated issue, almost sure this is the storage layout change from http://jenkins-ci.org/changelog#v1.597 – Jenkins provides information how to downgrade the storage format in its logs, and on the UI of 1.597+. As there are 50+ watchers here, and your issue has nothing to do with this one, please move any follow-up discussion to the users mailing list or IRC.

            Show
            danielbeck Daniel Beck added a comment - I'm seeing this same issue when changing between v1.590 and v1.610. Is there a work-around or way to recreate the build.xml files? The history directories are intact but history has disappeared from the web view. Unrelated issue, almost sure this is the storage layout change from http://jenkins-ci.org/changelog#v1.597 – Jenkins provides information how to downgrade the storage format in its logs, and on the UI of 1.597+. As there are 50+ watchers here, and your issue has nothing to do with this one, please move any follow-up discussion to the users mailing list or IRC.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              alex01ves Alex Vesely
              Votes:
              22 Vote for this issue
              Watchers:
              58 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: