I have a matrix job with two axis, one is our platform name, the other is the postgres version (which only has one entry and which we use to make sure the builds happen on machines that have postgres installed).

      We've been having sporadic failures in this job (that's not a bug in jenkins ) but when I went to look at two failed builds this morning, I got nothing underneath the "Console Output" heading on the:

      /job/<job-name>/environment=<x>,label=pg848/201

      The build artifacts were also missing.

      Clicking around different builds, I found this to be the case on all builds.
      The blue balls showed in:

      /job/<job-name>/201/

      ...but clicking on any of them gave the result I described above.

      I went off to google, found some similar issues but they all seemed old.
      I went to nose around the file system of the master.

      The really alarming this is that:

      jenkins@<host>:~/jobs/<job-name>/configurations/axis-environment

      ...didn't have subfolders for all the environment axis options, and those it did have were missing contents for the subfolders for the label axis:

      jenkins@<host>:~/jobs/<job-name>/configurations/axis-environment/<x>/axis-label> ls

      ...gave nothing.

      I had a bit more of a google, and by now the job had kicked off again this morning.

      I returned to the web ui. Now, going to:

      /job/<job-name>/environment=<x>,label=pg848/201

      ...gives grey balls for each of the environenments and if I go to:

      /job/<job-name>/environment=<x>,label=pg848/201

      ...I now get a 404, whereas before I got the empty configuration page.

      This is pretty scary.

          [JENKINS-16638] child builds going missing from matrix jobs

          Chris Withers added a comment -

          As follow up, after a restart, new builds of this job now appear to be correctly recording their artifacts and logs.
          Looking on disks, folder now exist for the correct axes, but they only contain builds done since the restart.

          Chris Withers added a comment - As follow up, after a restart, new builds of this job now appear to be correctly recording their artifacts and logs. Looking on disks, folder now exist for the correct axes, but they only contain builds done since the restart.

          Chris Withers added a comment -

          By way of debugging:

          What logging can I configure such that whenever a file gets deleted on the master in the build history, a line gets logged?

          Chris Withers added a comment - By way of debugging: What logging can I configure such that whenever a file gets deleted on the master in the build history, a line gets logged?

          Arne Nordmann added a comment -

          I can confirm this behavior. On our machine (Jenkins ver. 1.499) builds and artifacts of multi-configuration jobs vanish every once in a while. My impression is, that this happens foremost to newer jobs resp. newer configurations in multi-configuration jobs (e. g. if we add more build nodes to a job configuration).

          Arne Nordmann added a comment - I can confirm this behavior. On our machine (Jenkins ver. 1.499) builds and artifacts of multi-configuration jobs vanish every once in a while. My impression is, that this happens foremost to newer jobs resp. newer configurations in multi-configuration jobs (e. g. if we add more build nodes to a job configuration).

          Chris Withers added a comment -

          Interesting... I think some nodes were added to and removed from our pool around the time this happened...

          Chris Withers added a comment - Interesting... I think some nodes were added to and removed from our pool around the time this happened...

          Kevin R. added a comment -

          I'm on 1.504, I've been having the same issue and I only have 1 node running. Status updates?

          Kevin R. added a comment - I'm on 1.504, I've been having the same issue and I only have 1 node running. Status updates?

          We are still unable to reproduce this problem locally, which is making the fix difficult.

          To better understand what's going on, we need help from those who are seeing this problem.

          Our current leading hypothesis is that the log rotator is going rogue and deleting build records that it's not supposed to delete. While we don't know how this happens, if this is the case, then we can monitor its behavior by installing additional RunListener into the running system.

          To do so, please go to the Groovy script console (from "Manage Jenkins" link) and run the following command:

          import java.util.logging.*;
          
          class DeletionListener extends hudson.model.listeners.RunListener<Run>  {
            def l = Logger.getLogger("zd-13124");
          
            public void onDeleted(Run r) {
              l.log(Level.WARNING, "Deleting "+r.fullDisplayName, new Exception());
            }
          }
          
          new DeletionListener().register();
          

          With this installed, you should start seeing the stack trace reported on every time Jenkins deletes a build record. Note that this is only effective until you restart Jenkins, so if you restart Jenkins you'll have to run this script again.

          Do a new build, and if you observe that the records are gone, please check the Jenkins log and see if you can spot logging output from this listener. Attach the stack trace to this ticket as a comment so that we can udnerstand what's going on.

          Kohsuke Kawaguchi added a comment - We are still unable to reproduce this problem locally, which is making the fix difficult. To better understand what's going on, we need help from those who are seeing this problem. Our current leading hypothesis is that the log rotator is going rogue and deleting build records that it's not supposed to delete. While we don't know how this happens, if this is the case, then we can monitor its behavior by installing additional RunListener into the running system. To do so, please go to the Groovy script console (from "Manage Jenkins" link) and run the following command: import java.util.logging.*; class DeletionListener extends hudson.model.listeners.RunListener<Run> { def l = Logger.getLogger("zd-13124"); public void onDeleted(Run r) { l.log(Level.WARNING, "Deleting "+r.fullDisplayName, new Exception()); } } new DeletionListener().register(); With this installed, you should start seeing the stack trace reported on every time Jenkins deletes a build record. Note that this is only effective until you restart Jenkins, so if you restart Jenkins you'll have to run this script again. Do a new build, and if you observe that the records are gone, please check the Jenkins log and see if you can spot logging output from this listener. Attach the stack trace to this ticket as a comment so that we can udnerstand what's going on.

          Richard Adams added a comment -

          Added a stack trace for this:
          loosing matrix sub builds

          Richard Adams added a comment - Added a stack trace for this: loosing matrix sub builds

          Upgraded to Jenkins version 1.539 several days (or months ago).
          LogRotation is set to X days and Y builds and another job is set to Y builds only.
          Problem does not occur anymore on both machines.

          Please can anyone confirm this?

          Then also bug 15799 can be set to "fixed"..

          Sven Appenrodt added a comment - Upgraded to Jenkins version 1.539 several days (or months ago). LogRotation is set to X days and Y builds and another job is set to Y builds only. Problem does not occur anymore on both machines. Please can anyone confirm this? Then also bug 15799 can be set to "fixed"..

            Unassigned Unassigned
            cjw296 Chris Withers
            Votes:
            7 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: