Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
None
Description
Hello
I'm reporting an important issue on Jenkins with matrix build. The 'Discard Old Builds' setting doesn't remove builds in configuration, leading to increase disk usage without limit.
Here is the behavior :
- the master job is correctly cleaned, for instance, after a month if configured so, the artifacts are dropped and the build are correctly deleted
- however this is not the case for the configuration builds, if i go into the page of one I can see a very long list of builds not having been deleted (so this is not synchronized with the 'master').
The files in
~/jenkins/jobs/MyJobName/configurations/axis-FOO/foo/axis-BAR/bar/builds
are no deleted while
~/jenkins/jobs/MyJobName/builds/
are correctly deleted.
Also, disk usage is not correctly computed in the disk usage space : the size of the configuration are not taken in account.
This leads on my build machine a complete saturation of the disks after 1 week of work. I think this is a very critical issue.
Thanks,
Gaetan
Attachments
Issue Links
- is duplicated by
-
JENKINS-25990 Discard Old Builds doesn't work on Matrix Job
-
- Resolved
-
Activity
I think this has been introduced in 1.451 when the new delete button has been added in matrix job ("Delete this Build and all configuration in this build").
This is very annoying, since I have to manually check for disk space a remove the configuration build every 2 days..
I'm pretty sure the discard old build perform the same action as "Delete this build" while it should do the same action as "Delete this build and all configuration in this build".
I'll check it, but IIRC addition of "Delete this Build and all configuration in this build" only added new function to delete matrix job with all it's sub-jobs as "Delete this build" deletes only matrix jobs parent itself.
Here are information about my install:
Jenkins 1.460 under linux 64b
System Properties
executable-war /usr/share/jenkins/jenkins.war
file.encoding UTF-8
file.encoding.pkg sun.io
file.separator /
hudson.diyChunking true
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.headless true
java.awt.printerjob sun.print.PSPrinterJob
java.class.path /usr/share/jenkins/jenkins.war
java.class.version 50.0
java.endorsed.dirs /usr/lib/jvm/java-6-openjdk/jre/lib/endorsed
java.ext.dirs /usr/lib/jvm/java-6-openjdk/jre/lib/ext:/usr/java/packages/lib/ext
java.home /usr/lib/jvm/java-6-openjdk/jre
java.io.tmpdir /tmp
java.library.path /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.6.0_23-b23
java.specification.name Java Platform API Specification
java.specification.vendor Sun Microsystems Inc.
java.specification.version 1.6
java.vendor Sun Microsystems Inc.
java.vendor.url http://java.sun.com/
java.vendor.url.bug http://java.sun.com/cgi-bin/bugreport.cgi
java.version 1.6.0_23
java.vm.info mixed mode
java.vm.name OpenJDK 64-Bit Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Sun Microsystems Inc.
java.vm.specification.version 1.0
java.vm.vendor Sun Microsystems Inc.
java.vm.version 20.0-b11
javamelody.analytics-id UA-1335263-7
javamelody.http-transform-pattern /\d+/|/site/.|avadoc/.|/ws/.|obertura/.|estReport/.|iolations/file/.|/user/.|/static/\w/
javamelody.no-database true
javamelody.storage-directory //home/jenkins/jenkins/monitoring
javamelody.system-actions-enabled true
javax.accessibility.assistive_technologies org.GNOME.Accessibility.JavaBridge
jna.platform.library.path /usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib64:/usr/lib:/lib
line.separator
mail.smtp.sendpartial true
mail.smtps.sendpartial true
os.arch amd64
os.name Linux
os.version 3.0.0-15-generic
path.separator :
pid 24094
sun.arch.data.model 64
sun.boot.class.path /usr/lib/jvm/java-6-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/netx.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/plugin.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/rhino.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/modules/jdk.boot.jar:/usr/lib/jvm/java-6-openjdk/jre/classes
sun.boot.library.path /usr/lib/jvm/java-6-openjdk/jre/lib/amd64
sun.cpu.endian little
sun.cpu.isalist
sun.io.unicode.encoding UnicodeLittle
sun.java.command /usr/share/jenkins/jenkins.war --webroot=/var/run/jenkins/war --httpPort=8080 --ajp13Port=-1
sun.java.launcher SUN_STANDARD
sun.jnu.encoding UTF-8
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.os.patch.level unknown
svnkit.http.methods Digest,Basic,NTLM,Negotiate
svnkit.ssh2.persistent false
svnkit.symlinks false
user.country US
user.dir /
user.home /home/jenkins
user.language en
user.name jenkins
user.timezone Europe/Paris
Name Version Enabled Pinned
javadoc 1.0 true false
ant 1.1 true false
token-macro 1.5.1 true false
maven-plugin 1.460 true true
cvs 2.3 true true
doclinks 0.5 true false
dashboard-view 2.2 true false
analysis-core 1.39 true false
tasks 4.28 true false
warnings 4.0 true false
ci-game 1.18 true false
subversion 1.39 true true
viewVC 1.5 true false
jquery 1.0.2 true false
URLSCM 1.6 true false
project-stats-plugin 0.3 true false
slave-status 1.6 true false
mercurial 1.38 true false
klaros-testmanagement 1.4 true false
build-timeout 1.8 true false
ssh 1.3 true false
description-setter 1.8 true false
scm-sync-configuration 0.0.4 true false
cccc 0.6 true false
scis-ad 1.2 true false
htmlpublisher 0.7 true false
build-pipeline-plugin 1.2.3 false false
saferestart 0.2 true false
progress-bar-column-plugin 1.0 true false
depgraph-view 0.2 true false
svn-tag 1.16 true false
validating-string-parameter 2.2 true false
xfpanel 1.0.11 true false
sidebar-link 1.6 true false
exclusive-execution 0.7 true false
cvs-tag 1.7 true false
promoted-builds 2.5 true false
release 2.2 true false
xvnc 1.10 true false
statusmonitor 1.3 false false
svn-release-mgr 1.2 true false
measurement-plots 0.1 true false
timestamper 1.2.2 true false
jquery-ui 1.0.2 true false
filesystem_scm 1.20 true false
screenshot 1.1 true false
jobtype-column 1.2 true false
keepSlaveOffline 1.0 true false
show-build-parameters 1.0 true false
tmpcleaner 1.1 true false
nested-view 1.8 true false
copyartifact 1.22 true false
promoted-builds-simple 1.9 true false
cmakebuilder 1.9 true false
emotional-jenkins-plugin 1.1 true false
versionnumber 1.4.1 true false
vmware 0.8 true false
tracking-svn 1.1 true false
girls 1.0.0 false false
lastsuccessversioncolumn 1.1 true false
description-column-plugin 1.3 true false
backup 1.6.1 true false
plot 1.5 true false
scp 1.8 true false
systemloadaverage-monitor 1.2 true false
copy-to-slave 1.4 true false
postbuild-task 1.8 true false
sectioned-view 1.16 true false
hudson-wsclean-plugin 1.0.4 true false
locale 1.2 true false
favorite 1.6 true false
terminal 1.3 true false
radiatorviewplugin 1.13 false false
emotional-hudson 1.3 false false
copyarchiver 0.5.1 true false
role-strategy 1.1.2 true false
parameterized-trigger 2.13 true false
email-ext 2.20 true false
git 1.1.17 true false
all-changes 1.3 true false
ftppublisher 1.2 true false
svnpublisher 0.1 true false
template-project 1.3 true false
built-on-column 1.1 true false
matrixtieparent 1.1 true false
greenballs 1.11 true false
sloccount 1.6 true false
dropdown-viewstabbar-plugin 1.4 true false
xunit 1.43 true false
cppunit 1.10 true false
next-executions 1.0.4 true false
doxygen 0.10 true false
rebuild 1.10 true false
disk-usage 0.16 true false
virtualbox 0.4.6 true false
jobConfigHistory 1.13 true false
extra-columns 1.3 true false
clone-workspace-scm 0.4 true false
ssh-slaves 0.21 true true
multiple-scms 0.2 true false
groovy-postbuild 1.6 true false
ui-samples-plugin 1.460 false false
jenkinswalldisplay 0.6.9 true false
run-condition 0.8 true false
cppcheck 1.7 true false
scripttrigger 0.16 true false
downstream-ext 1.7 true false
join 1.14 true false
ws-cleanup 0.8 true false
groovy 1.12 true false
downstream-buildview 1.7 true false
job-exporter 0.4 true false
cobertura 1.3 true false
testlink 3.1.2 true false
analysis-collector 1.25 true false
monitoring 1.36.0 true false
conditional-buildstep 1.0 true false
testng-plugin 0.32 true false
thinBackup 1.5 true false
batch-task 1.15 true false
ansicolor 0.2.1 true false
translation 1.9 true true
trac 1.13 true false
view-job-filters
Just another data point: we disabled the Promoted Builds plugin, restarted, and are still having the issue.
After disabling the promoted builds plugin, and cleaning up all the abandoned builds, it seems we no longer have this issue.
I'm also seeing this issue... very annoying. I'm spending hours every week cleaning up old build artefacts from jenkins/jobs/EveryJobName/modules/EveryModuleName/builds/*
Jenkins 1.505 on Windows 2008R2
Plugins:
Jenkins Mailer Plugin 1.4
External Monitor Job Type Plugin 1.1
LDAP Plugin 1.2
pam-auth 1.0
Ant Plugin 1.2
Javadoc Plugin 1.1
Jenkins CVS Plug-in 2.8
Hudson Support Subscription Notification Plugin 1.2
Maven 2 Project Plugin 1.505
Jenkins Artifact Deployer Plug-in 0.26
Jenkins Subversion Plug-in 1.45
Config AutoRefresh Plugin 1.0
Jenkins Workspace Cleanup Plugin 0.12
Configuration Slicing plugin 1.36
Jenkins DocLinks plugin 0.5
xUnit plugin 1.52
Jenkins Gallio plugin 1.6
Jenkins GIT client plugin 1.0.4
Jenkins Parameterized Trigger plugin 2.17
Jenkins GIT plugin 1.3.0
Green Balls 1.12
Nested View Plugin 1.9
Jenkins SSH Slaves plugin 0.22
Startup Trigger 2.3
ThinBackup 1.7.2
Jenkins Translation Assistance plugin 1.10
I am seeing the same issue with LTS version 1.509.2. We have the following plugins.
mailer 1.4 true false
pam-auth 1.0 true false
external-monitor-job 1.1 true false
ldap 1.1 true false
ant 1.1 true false
javadoc 1.0 true false
cvs 1.6 true false
translation 1.10 true false
credentials 1.3.1 true false
ssh-credentials 0.2 true false
ssh-slaves 0.25 true false
maven-plugin 1.509.2 true false
subversion 1.39 true false
shelve-project-plugin 1.4.4
Does this issue still occur on recent Jenkins versions, where Matrix Project is its own plugin?
Issue confirmed on Jenkins 1.580 with Matrix Project Plugin 1.3.
Of course, the problem is that the Matrix Project type distinguishes between regular "Delete Build" and "Delete this build and all configurations in this build", so it's not straightforward to say what the correct behavior should be.
The behavior we were experiencing was that artifacts were not properly deleted when jobs were "expired."
Note that artifacts get deleted properly if you explicitly set that option in the build discarder. You may need to keep one more top-level build than has artifacts.
I tested build discarding with keeping 3 builds with artifacts, and 5 builds in general. After build 6 finished, only 4-6 had artifacts, top level builds 2-6 existed, and per-configuration builds 1-6 existed.
Proposed fix in https://github.com/jenkinsci/matrix-project-plugin/pull/13.
Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/hudson/matrix/MatrixBuild.java
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll.jelly
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_de.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_es.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_ja.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_pt.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_ru.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_zh_TW.properties
src/main/resources/hudson/matrix/MatrixBuild/delete.jelly
src/main/resources/hudson/matrix/MatrixBuild/delete_ca.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_cs.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_de.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_es.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_ja.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_ko.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_pl.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_pt.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_pt_BR.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_ru.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_zh_CN.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_zh_TW.properties
src/test/groovy/hudson/matrix/MatrixProjectTest.groovy
http://jenkins-ci.org/commit/matrix-project-plugin/4e22a02287c358260fde84f6b3c5d33f480601e0
Log:
[FIXED JENKINS-13554] Delete configurations from MatrixBuild#delete
Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/hudson/matrix/MatrixBuild.java
src/test/groovy/hudson/matrix/MatrixProjectTest.groovy
http://jenkins-ci.org/commit/matrix-project-plugin/815b8d08c4b11ec3315ef8745b2554d663c0b5ac
Log:
JENKINS-13554 Do not keep locked builds
Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/hudson/matrix/MatrixBuild.java
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll.jelly
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_de.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_es.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_ja.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_pt.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_ru.properties
src/main/resources/hudson/matrix/MatrixBuild/confirmDeleteAll_zh_TW.properties
src/main/resources/hudson/matrix/MatrixBuild/delete.jelly
src/main/resources/hudson/matrix/MatrixBuild/delete_ca.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_cs.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_de.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_es.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_ja.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_ko.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_pl.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_pt.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_pt_BR.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_ru.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_zh_CN.properties
src/main/resources/hudson/matrix/MatrixBuild/delete_zh_TW.properties
src/test/groovy/hudson/matrix/MatrixProjectTest.groovy
http://jenkins-ci.org/commit/matrix-project-plugin/cfdf9e33bdfda54f29f2871ac9dd49c60310c298
Log:
Merge pull request #13 from olivergondza/JENKINS-13554
[FIXED JENKINS-13554] Delete configurations from MatrixBuild#delete
Compare: https://github.com/jenkinsci/matrix-project-plugin/compare/5b4dedbc7ff0...cfdf9e33bdfd
Version 1.4 (Oct 14, 2014) Automatic deletion of inactive configurations from the disk:
if "set # builds to keep" axis-builds is not synchronized with the 'master'
if "days to keep builds" it ok.
tch, do you mean that parent and child jobs have different number of builds? Which of them is not correct? How to reproduce?
Oliver, not so.
Before updating the plugin I had a lot of old builds in directories ~ / jenkins / jobs / MyJobName / configurations / axis-FOO /.../ builds and
~ / jenkins / jobs / MyJobName / builds /
After updating the plugin i set "discard old builds" = true,
"# Builds to keep" = 3. In MyJobName / builds / is 3 builds, in MyJobName / axis are all (> 3). (should be and here and there 3)
changed to "days to keep builds" = 1. In MyJobName / builds / is 2 builds (by today), in MyJobName / axis are also 2. (everything is OK)
Code changed in jenkins
User: Oliver Gondža
Path:
src/test/java/hudson/matrix/MatrixProjectTest.java
http://jenkins-ci.org/commit/matrix-project-plugin/d3e68d0d83ce1b6f7a4d49d802666059c823f59f
Log:
Fix tests for JENKINS-13554
This is affecting us too. jenkins-1.457-1.1.noarch RPM.
Plugins:
ant - 1.1
javadoc - 1.0
subversion - 1.33
parameterized-trigger - 2.13
git - 1.1.12
gerrit-trigger - 2.3.1
hudson-pview-plugin - 1.8
build-timeout - 1.8
maven-plugin - 1.457
analysis-core - 1.31
warnings - 3.22
doxygen - 0.9
plot - 1.5
copyarchiver - 0.5.1
confluence-publisher - 1.3
jira - 1.29
cvs - 1.6
global-build-stats - 1.2
scm-sync-configuration - 0.0.4
platformlabeler - 1.1
monitoring - 1.32.1
checkstyle - 3.21
disk-usage - 0.14
build-pipeline-plugin - 1.2.3
cppcheck - 1.0.2
libvirt-slave - 1.6
matrix-reloaded - 1.0.1
github - 0.7
analysis-collector - 1.18
iphoneview - 0.2
twitter - 0.6
promoted-builds - 2.4
copyartifact - 1.21
ssh-slaves - 0.21
translation - 1.8
join - 1.14
publish-over-ssh - 1.6
We also had promoted builds 2.4 activated, but I disabled that. I will check to see if that made any difference.