-
Bug
-
Resolution: Fixed
-
Major
-
Fedora 18 x86_64
-
Powered by SuggestiMate
Activating the 'Discard Old Builds' checkbox on a job allows an Advanced option to specify 'Max # of builds to keep with artifacts'.
This does not seem to clean up the artifacts for my builds anymore.
I have tested 1.502, 1.503, 1.504 up to 1.510. It works fine in 1.502 but in none of the later versions.
The artifacts I refer to are archived with the default archiving mechanism (not the archiving that can be configured as a post-step).
- duplicates
-
JENKINS-17355 LogRotator does not delete Maven Artifacts
-
- Resolved
-
- is duplicated by
-
JENKINS-17553 Archived artifacts of Maven Modules are no longer cleaned up
-
- Resolved
-
- is related to
-
JENKINS-22395 Run.delete (from LogRotator) failing with "...looks to have already been deleted"
-
- Resolved
-
[JENKINS-17508] The 'Discard Old Builds' advanced option - removal of only artifacts - does not work for me after 1.503.
Wondering if it's related to this change:
What's new in 1.503 (2013/02/26)
- "Discard old build records" behavior is now pluggable, allowing plugins to define custom logic.
Yes, it is related to that change.
Unfortunately I don't have the insight to fix the problem, because it is causing us full disks...
It seems to me that each of the modules have their own configuration file. But the tag e.g.
<logRotator class="hudson.tasks.LogRotator">
<daysToKeep>-1</daysToKeep>
<numToKeep>50</numToKeep>
<artifactDaysToKeep>-1</artifactDaysToKeep>
<artifactNumToKeep>2</artifactNumToKeep>
</logRotator>
that enables the cleanup is not propagated to these build modules configuration files. It is in the master configuration file of the job though. I think if you add logRotator to the modules config.xml they will be cleaned. I'll add it to some jobs on our Jenkins on Monday to see if that is the case.
LogRotator delegates the delete to the actual Run, for just the log files the delete is delegated all the way to the MavenModuleSetBuild, however for the deleteArtifacts this is not done.
My knowledge of the Jenkins code base is very small at the moment. Just starting to get a grasp of it. But here is what I mean, https://github.com/redlab/jenkins/commit/d4459d9d0f99b951974f6ab1bedf6a580a some lines of code can say more then words. I've ran all unit tests but didn't add one yet. ( if this could fix it I can create a pull request if you want )
I have tested your code change in my test environment and it seems to be working!
The LogRotator plugin now deletes the artifacts and keeps the logs as it should.
Great, I had some merge issues with the pull request yesterday night. I'll try send a new one to jenkins-ci github
I've send a pull request, https://github.com/jenkinsci/jenkins/pull/759 AFIAK there no merge issues on my side. Maybe there's an issue with the workspace at https://jenkins.ci.cloudbees.com/job/core/job/jenkins_main_trunk/
Code changed in jenkins
User: redlab
Path:
maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
http://jenkins-ci.org/commit/jenkins/d4459d9d0f99b951974f6ab1bedf6a580aad9248
Log:
As this is my first look inside Jenkins codebase I not sure it's correct
but perhaps artifacts for modulebuilds should be deleted just as logs
are deleted. see https://issues.jenkins-ci.org/browse/JENKINS-17508
I did not fully test it yet, all unit tests ran fine, but didn't succeed
to run jobs inside Jenkins, due to classloader issues while running
Jenkins in Jetty, to verify deletion
Code changed in jenkins
User: redlab
Path:
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifacts.java
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
http://jenkins-ci.org/commit/jenkins/aa1d28b04014c18838e87b3a8b1707cc5eb28b14
Log:
JENKINS-17508 updated the test to use @Rule and no longer extend from the old HudsonTestCase (shouldn't that one be marked deprecated?), add @ Bug and @ For as jesse suggested
Code changed in jenkins
User: redlab
Path:
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
http://jenkins-ci.org/commit/jenkins/f9ce6558c93d7f7ff0f8df12fb80f9ecc868ff99
Log:
JENKINS-17508 removed usage of deprecated elements in test
Code changed in jenkins
User: Jesse Glick
Path:
maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
http://jenkins-ci.org/commit/jenkins/153c9a65ca1393e97a76588d7076883ec39710c5
Log:
Merge pull request #759 from redlab/JENKINS-17508
[FIXED JENKINS-17508] Artifacts not being deleted by LogRotator
Compare: https://github.com/jenkinsci/jenkins/compare/31eeb753aadc...153c9a65ca13
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/09ae9c00d39dbe824e5f0fdcd8861ec245ed057f
Log:
JENKINS-17508 Noting.
Integrated in jenkins_main_trunk #2507
JENKINS-17508 updated the test to use @Rule and no longer extend from the old HudsonTestCase (shouldn't that one be marked deprecated?), add @ Bug and @ For as jesse suggested (Revision aa1d28b04014c18838e87b3a8b1707cc5eb28b14)
JENKINS-17508 removed usage of deprecated elements in test (Revision f9ce6558c93d7f7ff0f8df12fb80f9ecc868ff99)
JENKINS-17508 Noting. (Revision 09ae9c00d39dbe824e5f0fdcd8861ec245ed057f)
Result = SUCCESS
balder : aa1d28b04014c18838e87b3a8b1707cc5eb28b14
Files :
- test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
- test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifacts.java
balder : f9ce6558c93d7f7ff0f8df12fb80f9ecc868ff99
Files :
- test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
Jesse Glick : 09ae9c00d39dbe824e5f0fdcd8861ec245ed057f
Files :
- changelog.html
I've upgraded to Jenkins 1.515 from 1.506 exactly because I wanted this functionality to work but it still does not remove old modules from SOME_JOB/modules/SOME_MODULE/builds
I have tested 1.515 in my environment and it works as I expect.
I use the following config:
------------------------------------------
'Days to keep builds': empty
'Max # of builds to keep': 500
'Days to keep artifacts': empty
'Max # of builds to keep with artifacts': 2
-------------------------------------------
It clears all my artifacts except the 2 last, as I would expect.
I have not tried other combinations of the 4 parameters above though.
Upgraded from 1.509 and does not work either, have configured
Days to keep builds': empty
'Max # of builds to keep': 15
'Days to keep artifacts': empty
'Max # of builds to keep with artifacts': 1
I have tried various options but it still does not work - after each build I can see the number of artifacts in modules/SOME_MODULE/builds growing
Wierd
'Days to keep builds': empty
'Max # of builds to keep': 2000
'Days to keep artifacts': empty
'Max # of builds to keep with artifacts': 1
And already reclaimed 50GB of diskspace (note: the cleaning job only runs after a build has run, if no build has run for that job it will not be cleaned)
This fix has allowed me to reclaim 100GB of disk space - thanks!!
Note to Tomek:
As far as I know, artifacts refers to files in the followin directories:
${jenkins_home}/jobs/${jobname}/modules/${moduleName}/builds/${build_number}/archive
and not modules/SOME_MODULE/builds
- this may be the # of builds to keep option
Upgraded to 1.516 and the problem seems to be still there.
Artifacts located in ${jenkins_home}/jobs/${jobname}/modules/${moduleName}/builds/${build_number}/archive are not deleted after a build has been executed.
Just want to comment that in my environment it works fine. Got 250GB disk space back.
In my env. the path to the archive is:
${jenkins_home}/jobs/${jobname}/modules/${moduleName}/builds/${build_timestamp}/archive
Do you really have a 'build_number' or 'build_timestamp' in the path before '/archive' ?
you are correct it is a "build_timestamp" not a "build_number"
Code changed in jenkins
User: redlab
Path:
src/main/java/hudson/maven/MavenModuleSetBuild.java
http://jenkins-ci.org/commit/maven-plugin/62637aad472803c7954bc54120dc4f5121146568
Log:
As this is my first look inside Jenkins codebase I not sure it's correct
but perhaps artifacts for modulebuilds should be deleted just as logs
are deleted. see https://issues.jenkins-ci.org/browse/JENKINS-17508
I did not fully test it yet, all unit tests ran fine, but didn't succeed
to run jobs inside Jenkins, due to classloader issues while running
Jenkins in Jetty, to verify deletion
Originally-Committed-As: d4459d9d0f99b951974f6ab1bedf6a580aad9248
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/hudson/maven/MavenModuleSetBuild.java
http://jenkins-ci.org/commit/maven-plugin/6f769759743d3765b27e0eff1c9b07b7b82f0878
Log:
Merge pull request #759 from redlab/JENKINS-17508
[FIXED JENKINS-17508] Artifacts not being deleted by LogRotator
Originally-Committed-As: 153c9a65ca1393e97a76588d7076883ec39710c5
Code changed in jenkins
User: redlab
Path:
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifacts.java
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
http://jenkins-ci.org/commit/maven-plugin/40bdc86e98e38dee1a536e8151ac45b53f447be4
Log:
JENKINS-17508 updated the test to use @Rule and no longer extend from the old HudsonTestCase (shouldn't that one be marked deprecated?), add @ Bug and @ For as jesse suggested
Code changed in jenkins
User: redlab
Path:
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
http://jenkins-ci.org/commit/maven-plugin/8fbb38d171470fa339a277eb38f9564f272f9148
Log:
JENKINS-17508 removed usage of deprecated elements in test
Hi,
We are on Jenkins version 1.528, and we are still having an issue where module directories are not cleaned up. We came across this problem when we received a "low free disk space" alert from our build server, and discovered that some projects had tens of orphaned modules.
At that point (18 Oct 2013), I changed the settings on all our Jenkins projects to the below, and removed all module directories from the file system. I checked it again today (01 Nov 2013), and there are new orphaned modules there again. If you look below, there is a module for each build, and then there are 8 orphaned modules for earlier builds.
I have a theory that the modules are not properly cleaned up if the build fails, but I have not got a conclusive proof for that.
Can you please confirm, which version of Jenkins or Maven plugin contains a bug fix for this issue?
<logRotator class="hudson.tasks.LogRotator"> <daysToKeep>7</daysToKeep> <numToKeep>5</numToKeep> <artifactDaysToKeep>7</artifactDaysToKeep> <artifactNumToKeep>5</artifactNumToKeep> </logRotator>
./mvn-xxx/builds/2013-10-28_16-55-29 ./mvn-xxx/builds/2013-10-29_17-57-18 ./mvn-xxx/builds/2013-10-30_14-03-54 ./mvn-xxx/builds/2013-10-30_17-23-24 ./mvn-xxx/builds/2013-10-30_17-29-20 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-18_18-14-42 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-21_17-09-59 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-23_17-08-49 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-23_17-11-45 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-25_17-45-23 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-28_14-03-14 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-28_14-56-33 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-28_16-33-42 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-28_16-55-31 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-29_17-57-20 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-30_14-03-55 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-30_17-23-25 ./mvn-xxx/modules/com.xxx.xxx$xxx/builds/2013-10-30_17-29-22
Hello,
We're running Jenkins 1.533 and we still notice the same behavior reported by Sergei Ivanov.
Even though we have old builds properly deleted, the clean is not propagated to the underlying modules for which we keep having a bunch of orphans.
Best regards.
Got an
Jan 28, 2014 9:03:44 AM hudson.model.Run execute
SEVERE: Failed to rotate log
java.io.IOException: e:\jenkins\jobs\<.....>\modules\com.<...>$<....>\builds\2013-10-23_11-57-46 looks to have already been deleted
{{ at hudson.model.Run.delete(Run.java:1432)}}
{{ at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:430)}}
{{ at hudson.tasks.LogRotator.perform(LogRotator.java:124)}}
{{ at hudson.model.Job.logRotate(Job.java:437)}}
{{ at hudson.maven.MavenModuleSet.logRotate(MavenModuleSet.java:840)}}
{{ at hudson.model.Run.execute(Run.java:1728)}}
{{ at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:519)}}
{{ at hudson.model.ResourceController.execute(ResourceController.java:88)}}
{{ at hudson.model.Executor.run(Executor.java:246)}}
with Jenkins ver. 1.532.1.
(Mentioned directory 2013-10-23_11-57-46 is not existing on file system).
LogRotator should continue deleting all other matching jobs.
@baboulama, @sc_rsc: the originally reported bug here is fixed; do not reopen.
Filed JENKINS-22395 for a problem actually deleted old builds under unknown conditions.
I have investigated this problem further and found the following:
I have added some log to the LogRotator.java in both 1.502 and 1.510 - in the 'perform' method:
Added print-out of artifactsDir in the loop below:
if(artifactNumToKeep!=null && artifactNumToKeep!=-1) {
{0}List<? extends Run<?,?>> builds = job.getBuilds();
for (Run r : copy(builds.subList(Math.min(builds.size(), artifactNumToKeep), builds.size()))) {
LOGGER.log(INFO, "artifactsDir =
", r.getArtifactsDir());
The log seen in version 1.502 is e.g:
INFO: artifactsDir = /var/lib/jenkins/jobs/PJLTEST/modules/com.corena.core$repository-management/builds/2013-04-08_20-28-13/archive
while in 1.510 it is e.g.:
INFO: artifactsDir = /var/lib/jenkins/jobs/PJLTEST/builds/2013-04-08_20-29-49/archive
This shows that it is a completely different behaviour between the two version and now artifacts will be removed as the directory seen in 1.510 does not even exist.