Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10197

Matrix jobs default to "locked" (and are difficult to delete)

      With a matrix job now when it completes it defaults to being a "keep forever" type build. Only by deleting the current job can a previous job be deleted, in some cases.

      (09:36:12 AM) Tartarus: kohsuke: With matrix jobs and 1.418 I'm seeing all of my jobs default to being locked, didn't see this with 1.409, any ideas?
      (09:36:32 AM) kohsuke: probably related to the partial rebuild support. Any ticket for this?
      (09:36:54 AM) Tartarus: Hadn't filed one yet
      (09:37:18 AM) kohsuke: Would you please? I don't think I can get it right now, but I don't want to forget
      

          [JENKINS-10197] Matrix jobs default to "locked" (and are difficult to delete)

          Tom Rini added a comment -

          Just now when I had one build spinning up and I tried to delete the previous one it says that can't delete build #182 because #183 depends on it (roughly, forgot to copy in time).

          Tom Rini added a comment - Just now when I had one build spinning up and I tried to delete the previous one it says that can't delete build #182 because #183 depends on it (roughly, forgot to copy in time).

          Andrew Bayer added a comment -

          I ran into this a while back - the problem is that you've changed your axes, and now there's a chain of builds depending on each other back to the last one with the old axes. I got around this by deleting the actual directory from $JENKINS_HOME/jobs/$JOB_NAME/configurations/axis-label. That said, this is definitely still a real bug.

          Andrew Bayer added a comment - I ran into this a while back - the problem is that you've changed your axes, and now there's a chain of builds depending on each other back to the last one with the old axes. I got around this by deleting the actual directory from $JENKINS_HOME/jobs/$JOB_NAME/configurations/axis-label. That said, this is definitely still a real bug.

          Gary Yund added a comment -

          We have this issue as well. Because every build is kept and the logs are rather large, in approximately 2 months time, our logs are over 5GB for one job.

          Gary Yund added a comment - We have this issue as well. Because every build is kept and the logs are rather large, in approximately 2 months time, our logs are over 5GB for one job.

          Barry Wark added a comment -

          Also experiencing the same issue, and we're quickly filling up disk space on our master. Since we've saved release builds in the artifacts of jobs, I can't just delete the entire jobs/$JOB_NAME/configurations/axis-label folder (though I'll start working through and copying the artifacts out so that we can). Has anyone figured out how to delete particular builds when they're in this "locked" state?

          Barry Wark added a comment - Also experiencing the same issue, and we're quickly filling up disk space on our master. Since we've saved release builds in the artifacts of jobs, I can't just delete the entire jobs/$JOB_NAME/configurations/axis-label folder (though I'll start working through and copying the artifacts out so that we can). Has anyone figured out how to delete particular builds when they're in this "locked" state?

          I have this problem too....

          I'm wondering if it's related to JENKINS-10131 - Git Plugin kicks off builds for executors.

          All the builds that are having this issue are triggered by git looking for workspaces. I don't know if that's the case with the other builds.

          Christian Höltje added a comment - I have this problem too.... I'm wondering if it's related to JENKINS-10131 - Git Plugin kicks off builds for executors. All the builds that are having this issue are triggered by git looking for workspaces. I don't know if that's the case with the other builds.

          Barry Wark added a comment -

          Although we have the git plugin installed, all of the relevant builds are from SVN.

          Barry Wark added a comment - Although we have the git plugin installed, all of the relevant builds are from SVN.

          Tom Rini added a comment -

          I've ran a git bisect on this problem and it's:

          51c85a9fa030013553299e59649251b4f09560d5 is the first bad commit
          commit 51c85a9fa030013553299e59649251b4f09560d5
          Author: Kohsuke Kawaguchi <kk@kohsuke.org>
          Date: Thu May 12 10:50:18 2011 -0700

          control GC so that records are always full

          Tom Rini added a comment - I've ran a git bisect on this problem and it's: 51c85a9fa030013553299e59649251b4f09560d5 is the first bad commit commit 51c85a9fa030013553299e59649251b4f09560d5 Author: Kohsuke Kawaguchi <kk@kohsuke.org> Date: Thu May 12 10:50:18 2011 -0700 control GC so that records are always full

          Yeah, that isPartial() function looks wrong....

          Christian Höltje added a comment - Yeah, that isPartial() function looks wrong....

          Christian Höltje added a comment - Link to the code in question: https://github.com/jenkinsci/jenkins/commit/51c85a9fa030013553299e59649251b4f09560d5

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          changelog.html
          core/src/main/java/hudson/matrix/MatrixBuild.java
          http://jenkins-ci.org/commit/jenkins/eea67c761e78fa1b26f9d2ae82826e3e74b877ee
          Log:
          [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: changelog.html core/src/main/java/hudson/matrix/MatrixBuild.java http://jenkins-ci.org/commit/jenkins/eea67c761e78fa1b26f9d2ae82826e3e74b877ee Log: [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          Andrew Bayer added a comment -

          Fix will be in 1.425.

          Andrew Bayer added a comment - Fix will be in 1.425.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1002
          [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          Andrew Bayer : eea67c761e78fa1b26f9d2ae82826e3e74b877ee
          Files :

          • changelog.html
          • core/src/main/java/hudson/matrix/MatrixBuild.java

          dogfood added a comment - Integrated in jenkins_main_trunk #1002 [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations. Andrew Bayer : eea67c761e78fa1b26f9d2ae82826e3e74b877ee Files : changelog.html core/src/main/java/hudson/matrix/MatrixBuild.java

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          changelog.html
          core/src/main/java/hudson/matrix/MatrixBuild.java
          http://jenkins-ci.org/commit/jenkins/eea67c761e78fa1b26f9d2ae82826e3e74b877ee
          Log:
          [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: changelog.html core/src/main/java/hudson/matrix/MatrixBuild.java http://jenkins-ci.org/commit/jenkins/eea67c761e78fa1b26f9d2ae82826e3e74b877ee Log: [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          ebann added a comment -

          I'm seeing this problem in Jenkins 1.430.

          Isn't it "resolved" ?

          ebann added a comment - I'm seeing this problem in Jenkins 1.430. Isn't it "resolved" ?

          Trevor Baker added a comment -

          Please re-open. I am seeing this behavior in 1.434

          Trevor Baker added a comment - Please re-open. I am seeing this behavior in 1.434

          Eric Long added a comment -

          I am still seeing this problem in 1.424

          Please re-open.

          Eric Long added a comment - I am still seeing this problem in 1.424 Please re-open.

          Trevor Baker added a comment -

          The "don't keep this build forever" button doesn't work either.

          Trevor Baker added a comment - The "don't keep this build forever" button doesn't work either.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          changelog.html
          core/src/main/java/hudson/matrix/MatrixBuild.java
          http://jenkins-ci.org/commit/jenkins/eea67c761e78fa1b26f9d2ae82826e3e74b877ee
          Log:
          [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: changelog.html core/src/main/java/hudson/matrix/MatrixBuild.java http://jenkins-ci.org/commit/jenkins/eea67c761e78fa1b26f9d2ae82826e3e74b877ee Log: [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          Eric Long added a comment -

          <newbie>
          What would be the best way to get this build?

          I have checked http://mirrors.jenkins-ci.org/ and http://ci.jenkins-ci.org/ for nightly builds without success.

          Thanks.
          </newbie>

          Eric Long added a comment - <newbie> What would be the best way to get this build? I have checked http://mirrors.jenkins-ci.org/ and http://ci.jenkins-ci.org/ for nightly builds without success. Thanks. </newbie>

          Reopend, as pushing to "guice" branch unfortunately has resolved this issue.

          OHTAKE Tomohiro added a comment - Reopend, as pushing to "guice" branch unfortunately has resolved this issue.

          Andrew Bayer added a comment -

          Is anyone still seeing this?

          Andrew Bayer added a comment - Is anyone still seeing this?

          Uwe Kubosch added a comment - - edited

          We are still seeing this at http://ci.jruby.org/view/Ruboto/

          It may be that the error is not still there for new builds, but we are unable to delete the oldest builds.

          Uwe Kubosch added a comment - - edited We are still seeing this at http://ci.jruby.org/view/Ruboto/ It may be that the error is not still there for new builds, but we are unable to delete the oldest builds.

          Spencer Oliver added a comment - still seeing it here aswell: http://openocd.zylin.com/jenkins/job/openocd-gerrit-build/

          I have been looking into this issue further.
          It seems to be caused because the job has a touchstone build enabled.

          If the touchstone build fails (so no other matrix build run) then the previous build will be locked, eg.
          When build 11 fails due to touchstone then build 10 will be locked.

          Spencer Oliver added a comment - I have been looking into this issue further. It seems to be caused because the job has a touchstone build enabled. If the touchstone build fails (so no other matrix build run) then the previous build will be locked, eg. When build 11 fails due to touchstone then build 10 will be locked.

          Andrew Bayer added a comment -

          ah-ha! Awesome discovery! I should be able to figure it out from there.

          Andrew Bayer added a comment - ah-ha! Awesome discovery! I should be able to figure it out from there.

          Jason Stanley added a comment -

          This problem also occurs on Jenkins version 1.445.

          Jason Stanley added a comment - This problem also occurs on Jenkins version 1.445.

          I am seeing this with Jenkins ver. 1.454, I have jobs that when I click "don't keep this build forever", I don't get the option to delete it. Is there any progress on a fix for this? (or a work-around)

          Gerald Bringhurst added a comment - I am seeing this with Jenkins ver. 1.454, I have jobs that when I click "don't keep this build forever", I don't get the option to delete it. Is there any progress on a fix for this? (or a work-around)

          Eric Long added a comment -

          Does anyone know a safe way to manually delete the old matrix jobs?

          We are having issues where our server is running out of space.
          We are hesitant to delete files manually as it may break the matrix job.

          Eric Long added a comment - Does anyone know a safe way to manually delete the old matrix jobs? We are having issues where our server is running out of space. We are hesitant to delete files manually as it may break the matrix job.

          This seems to be fixed in 1.456

          Spencer Oliver added a comment - This seems to be fixed in 1.456

          Brad Leonard added a comment -

          I'm seeing this with Jenkins 1.462. I have multiple matrix projects and they all exhibit the same behavior. When a project that was passing changes to failing, builds prior to the failure are marked Keep Forever and I cannot revert this. I can see the button "Don't keep this build forever" on each axis within a matrix project (not at the top level project itself). Nothing happens when I click the button so I have no way of deleting these builds (short of the doing it via the file system) or stopping additional builds from being marked as "Keep Forever".

          Brad Leonard added a comment - I'm seeing this with Jenkins 1.462. I have multiple matrix projects and they all exhibit the same behavior. When a project that was passing changes to failing, builds prior to the failure are marked Keep Forever and I cannot revert this. I can see the button "Don't keep this build forever" on each axis within a matrix project (not at the top level project itself). Nothing happens when I click the button so I have no way of deleting these builds (short of the doing it via the file system) or stopping additional builds from being marked as "Keep Forever".

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          core/src/main/java/hudson/matrix/MatrixBuild.java
          http://jenkins-ci.org/commit/matrix-project-plugin/b1ff7352170cf5777e5f11dc4e79961e5db02b81
          Log:
          [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations.

          Originally-Committed-As: eea67c761e78fa1b26f9d2ae82826e3e74b877ee

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: core/src/main/java/hudson/matrix/MatrixBuild.java http://jenkins-ci.org/commit/matrix-project-plugin/b1ff7352170cf5777e5f11dc4e79961e5db02b81 Log: [FIXED JENKINS-10197] isPartial() check for matrix builds now only reference active configurations. Originally-Committed-As: eea67c761e78fa1b26f9d2ae82826e3e74b877ee

          Not sure what is going on here, thought this was fixed.

          However i have just checked and this issue is still happening - http://openocd.zylin.com/jenkins/job/openocd-gerrit-build/
          just as a recap it is a matrix build using a touchstone that is failing, when the touchstone build fails the job before gets locked.

          Spencer Oliver added a comment - Not sure what is going on here, thought this was fixed. However i have just checked and this issue is still happening - http://openocd.zylin.com/jenkins/job/openocd-gerrit-build/ just as a recap it is a matrix build using a touchstone that is failing, when the touchstone build fails the job before gets locked.

          Liam Reimers added a comment -

          Seeing this in Jenkins 1.466... Not fixed.

          Liam Reimers added a comment - Seeing this in Jenkins 1.466... Not fixed.

          Guy Rozendorn added a comment -

          Looks like these are related, if not duplicated

          Guy Rozendorn added a comment - Looks like these are related, if not duplicated

          Guy Rozendorn added a comment -

          Still happens in 1.500

          Guy Rozendorn added a comment - Still happens in 1.500

          Jesse Glick added a comment - - edited

          https://github.com/jenkinsci/jenkins/commit/7bbe2c8 (in 1.481) improved the behavior. A matrix build may still be locked, but only if there is a subsequent build which built fewer configurations than this one, so that this one is the source of the “last known complete” builds of certain configurations.

          I have also improved the UI in the trunk for 1.510 with several fixes in https://github.com/jenkinsci/jenkins/compare/57900a235514...5b6be91d9237 to properly display the reason that a build is locked, when the locking is implicit rather than due to an explicit toggle.

          Jesse Glick added a comment - - edited https://github.com/jenkinsci/jenkins/commit/7bbe2c8 (in 1.481) improved the behavior. A matrix build may still be locked, but only if there is a subsequent build which built fewer configurations than this one, so that this one is the source of the “last known complete” builds of certain configurations. I have also improved the UI in the trunk for 1.510 with several fixes in https://github.com/jenkinsci/jenkins/compare/57900a235514...5b6be91d9237 to properly display the reason that a build is locked, when the locking is implicit rather than due to an explicit toggle.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/1c64ad15b15717c8fde1028a50aba668bb17e6a9
          Log:
          JENKINS-10197 Noting UI fixes.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html http://jenkins-ci.org/commit/jenkins/1c64ad15b15717c8fde1028a50aba668bb17e6a9 Log: JENKINS-10197 Noting UI fixes.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2415
          JENKINS-10197 Noting UI fixes. (Revision 1c64ad15b15717c8fde1028a50aba668bb17e6a9)

          Result = SUCCESS
          Jesse Glick : 1c64ad15b15717c8fde1028a50aba668bb17e6a9
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2415 JENKINS-10197 Noting UI fixes. (Revision 1c64ad15b15717c8fde1028a50aba668bb17e6a9) Result = SUCCESS Jesse Glick : 1c64ad15b15717c8fde1028a50aba668bb17e6a9 Files : changelog.html

          I reproduce this issue on 1.546 version.

          Alexander Artemov added a comment - I reproduce this issue on 1.546 version.

          Oleg Nenashev added a comment - - edited

          The issue appears in 1.509.4 as well.
          It also leads to the infinite disk usage if a job has timeouts or disabled configs.

          It makes sense to add an option, which disables such linking.

          Oleg Nenashev added a comment - - edited The issue appears in 1.509.4 as well. It also leads to the infinite disk usage if a job has timeouts or disabled configs. It makes sense to add an option, which disables such linking.

          Courtney Sims added a comment -

          This is still an issue in version 1.550. We have a matrix job where often only one column of configurations is built in a single build. The next time the job is run, a different column might be built, and then the previous build gets locked, I'm assuming because it's being referenced (as "skipped") in the new build. But what this means for our matrix job is that almost every single build ends up locked.

          I would like to be able to unlock these builds so that I can delete them. Rather than forcing a build to be implicitly locked because it's being referenced in a later build, why can't the reference be removed/updated if the build is deleted?

          Courtney Sims added a comment - This is still an issue in version 1.550. We have a matrix job where often only one column of configurations is built in a single build. The next time the job is run, a different column might be built, and then the previous build gets locked, I'm assuming because it's being referenced (as "skipped") in the new build. But what this means for our matrix job is that almost every single build ends up locked. I would like to be able to unlock these builds so that I can delete them. Rather than forcing a build to be implicitly locked because it's being referenced in a later build, why can't the reference be removed/updated if the build is deleted?

          Oleg Nenashev added a comment -

          Assigned the issue to "matrix-project"

          Oleg Nenashev added a comment - Assigned the issue to "matrix-project"

          Daniel Beck added a comment -

          Is there an actual bug left to resolve here (rather than an RFE)?

          I mean, it's quite obvious from e.g. Jesse's comment on 27/Mar/13 that the behavior csims describes is intended.

          aartemov unfortunately provided no further details what exactly he's seeing for this to be reopened, but it may well be the same.

          Daniel Beck added a comment - Is there an actual bug left to resolve here (rather than an RFE)? I mean, it's quite obvious from e.g. Jesse's comment on 27/Mar/13 that the behavior csims describes is intended. aartemov unfortunately provided no further details what exactly he's seeing for this to be reopened, but it may well be the same.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/resources/hudson/model/Run/KeepLogBuildBadge/badge.jelly
          http://jenkins-ci.org/commit/jenkins/0f0c2a7d00c22dc9946c18b20e7f576a20cf8a86
          Log:
          JENKINS-10197 Show keep log reason in tool tip

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/resources/hudson/model/Run/KeepLogBuildBadge/badge.jelly http://jenkins-ci.org/commit/jenkins/0f0c2a7d00c22dc9946c18b20e7f576a20cf8a86 Log: JENKINS-10197 Show keep log reason in tool tip

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/resources/hudson/model/Run/KeepLogBuildBadge/badge.jelly
          http://jenkins-ci.org/commit/jenkins/c4df05b652b25c7bb96511e3f6e7f1467dc34a33
          Log:
          Merge pull request #1422 from daniel-beck/JENKINS-10197

          JENKINS-10197 Show keep log reason in tool tip

          Compare: https://github.com/jenkinsci/jenkins/compare/e518f009c515...c4df05b652b2

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/resources/hudson/model/Run/KeepLogBuildBadge/badge.jelly http://jenkins-ci.org/commit/jenkins/c4df05b652b25c7bb96511e3f6e7f1467dc34a33 Log: Merge pull request #1422 from daniel-beck/ JENKINS-10197 JENKINS-10197 Show keep log reason in tool tip Compare: https://github.com/jenkinsci/jenkins/compare/e518f009c515...c4df05b652b2

          Oleg Nenashev added a comment -

          @Daniel
          I suppose the issue could be converted to "Improvement" or closed as "Not a defect".
          The behavior is quite painful, but it works as designed.

          Oleg Nenashev added a comment - @Daniel I suppose the issue could be converted to "Improvement" or closed as "Not a defect". The behavior is quite painful, but it works as designed.

          Daniel Beck added a comment -

          Resolving this issue as fixed again, as the concerns for which it was reopened appear to relate to current behavior that is by design.

          Starting from 1.585 or so, the reason for a build to be kept forever is also shown on the tool tip of the lock icon, so it's at least easier to determine why a build is kept.

          Please file a new improvement or feature request against the matrix-project component if you think the current behavior should be changed, or made configurable. You can create a bidirectional link to this issue ("More Actions" button) to establish these are related.

          (Oleg: Given that this was already fixed and backported, a clean slate for the improvement seems preferable.)

          Daniel Beck added a comment - Resolving this issue as fixed again, as the concerns for which it was reopened appear to relate to current behavior that is by design. Starting from 1.585 or so, the reason for a build to be kept forever is also shown on the tool tip of the lock icon, so it's at least easier to determine why a build is kept. Please file a new improvement or feature request against the matrix-project component if you think the current behavior should be changed, or made configurable. You can create a bidirectional link to this issue ("More Actions" button) to establish these are related. (Oleg: Given that this was already fixed and backported, a clean slate for the improvement seems preferable.)

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3741
          JENKINS-10197 Show keep log reason in tool tip (Revision 0f0c2a7d00c22dc9946c18b20e7f576a20cf8a86)

          Result = SUCCESS
          daniel-beck : 0f0c2a7d00c22dc9946c18b20e7f576a20cf8a86
          Files :

          • core/src/main/resources/hudson/model/Run/KeepLogBuildBadge/badge.jelly

          dogfood added a comment - Integrated in jenkins_main_trunk #3741 JENKINS-10197 Show keep log reason in tool tip (Revision 0f0c2a7d00c22dc9946c18b20e7f576a20cf8a86) Result = SUCCESS daniel-beck : 0f0c2a7d00c22dc9946c18b20e7f576a20cf8a86 Files : core/src/main/resources/hudson/model/Run/KeepLogBuildBadge/badge.jelly

          Gino Atlas added a comment - - edited

          Hello, this issue came up with version 1.7 on Jenkins 2.29 again.
          I've got a plenty of matrix jobs, which I can't delete. I don't see any configuration options the handle this problem.

          Thank you!
          Gino

          Gino Atlas added a comment - - edited Hello, this issue came up with version 1.7 on Jenkins 2.29 again. I've got a plenty of matrix jobs, which I can't delete. I don't see any configuration options the handle this problem. Thank you! Gino

          Daniel Beck added a comment -

          giat7133 Please see the preceding comment. Thank you.

          Daniel Beck added a comment - giat7133 Please see the preceding comment. Thank you.

          Gino Atlas added a comment -

          So I will need to create an improvement request. Thank you!

          Gino Atlas added a comment - So I will need to create an improvement request. Thank you!

          Lukasova M added a comment -

          HI, I also have a multi-configuration job and i cannot delete any job, because it shows it depends on itself or another job. But that another job also depends on some. And it doesn't give me an option to unmark the build to be kept forever. Is there some solution? (Jenkins v. 2.27)

          Thanks

          Lukasova M added a comment - HI, I also have a multi-configuration job and i cannot delete any job, because it shows it depends on itself or another job. But that another job also depends on some. And it doesn't give me an option to unmark the build to be kept forever. Is there some solution? (Jenkins v. 2.27) Thanks

            vjuranek vjuranek
            tomrini Tom Rini
            Votes:
            24 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated:
              Resolved: