[JENKINS-66566] Job DSL / Job Configuration History plugin creates tons of xml files

Type: Bug
Resolution: Unresolved
Priority: Minor
Component/s: job-dsl-plugin, jobconfighistory-plugin
Labels:
None
Environment:
jenkins 2.184
job-dsl 1.74.1
jobConfigHistory 2.22

Similar Issues:
Powered by SuggestiMate

Show

We have a number of jobs described via DSL files and a seed job that turns them into actual Jenkins items. This job is basically triggered on each commit or PR that changes .groovy files (90+ in two repos), and goes like

findAllDslFiles().each { jobDsl(scriptText: readFile(it)) }

Recently, we noticed that there is a tremendous amount of obscure XML files in `${JENKINS_HOME}/config-history/javaposse.jobdsl.plugin.ExecuteDslScripts`:

# ls 2021-09-06_15-42*
2021-09-06_15-42-06:
 history.xml javaposse.jobdsl.plugin.ExecuteDslScripts.xml
2021-09-06_15-42-09:
 history.xml javaposse.jobdsl.plugin.ExecuteDslScripts.xml
...
# ls -1 | wc -l
160316

These piled over little more than a year and currently take about 85GB of space. The folders correlate to number of DSL files: f.x. today (2021.09.06) the seed job triggered at 10.38, there are 94 folders with names from `2021-09-06_10-31-01` to `2021-09-06_10-33-56`, and we have 93 DSL files in repo. The files are almost identical:

--- 2021-09-06_10-31-01/javaposse.jobdsl.plugin.ExecuteDslScripts.xml	2021-09-06 10:31:01.118703955 +0300
+++ 2021-09-06_10-31-02/javaposse.jobdsl.plugin.ExecuteDslScripts.xml	2021-09-06 10:31:02.615671917 +0300
@@ -457,6 +457,13 @@
       </javaposse.jobdsl.plugin.SeedReference>
     </entry>
     <entry>
+      <string>staging/D10811/test</string>
+      <javaposse.jobdsl.plugin.SeedReference>
+        <seedJobName>maintenance/dsl-deployer</seedJobName>
+        <digest>cf5ff13ea2e9fa70733bdaf08281fd14</digest>
+      </javaposse.jobdsl.plugin.SeedReference>
+    </entry>
+    <entry>
       <string>staging/D9320_dsl_check/maintenance/sync-ubase</string>
       <javaposse.jobdsl.plugin.SeedReference>
         <seedJobName>staging/D9320/maintenance/dsl-deployer</seedJobName>
@@ -2585,13 +2592,6 @@
       </javaposse.jobdsl.plugin.SeedReference>
     </entry>
     <entry>
-      <string>staging/D10811</string>
-      <javaposse.jobdsl.plugin.SeedReference>
-        <seedJobName>maintenance/dsl-deployer</seedJobName>
-        <digest>d267a35b960390d61d110ab098c875cf</digest>
-      </javaposse.jobdsl.plugin.SeedReference>
-    </entry>
-    <entry>
       <string>staging/D10506/build/custom/lnvr/diff</string>
       <javaposse.jobdsl.plugin.SeedReference>
         <seedJobName>maintenance/dsl-deployer</seedJobName>

, and it seems very weird to have so much of them.

Since our seed job recreates the whole folder+job tree on each trigger, is it safe to assume that we can simply remove everything older than N days to preserve disk space?

is related to

JENKINS-66533 Exception: "Could not create rootDir"

Open

Artalus S. created issue - 2021-09-06 14:09

Artalus S. made changes - 2021-09-06 14:12

Description

Original: We have a number of jobs described via DSL files and a seed job that turns them into actual Jenkins items. This job is basically triggered on each commit or PR that changes .groovy files (90+ in two repos), and goes like `findAllDslFiles().each \{ jobDsl(scriptText: readFile(it)) }`.

Recently, we noticed that there is a tremendous amount of seemingly useless XML files in `${JENKINS_HOME}/config-history/javaposse.jobdsl.plugin.ExecuteDslScripts`:

{{# ls 2021-09-06_15-42*}}
{{2021-09-06_15-42-06:}}
{{history.xml javaposse.jobdsl.plugin.ExecuteDslScripts.xml}}
{{2021-09-06_15-42-09:}}
{{history.xml javaposse.jobdsl.plugin.ExecuteDslScripts.xml}}
{{...}}
{{# ls -1 | wc -l}}
{{160316}}

These piled over little more than a year and currently take about 85GB of space. The folders correlate to number of DSL files: f.x. today (2021.09.06) the seed job triggered at 10.38, there are 94 folders with names from `2021-09-06_10-31-01` to `2021-09-06_10-33-56`, and we have 93 DSL files in repo. The files are almost identical:

{{--- 2021-09-06_10-31-01/javaposse.jobdsl.plugin.ExecuteDslScripts.xml 2021-09-06 10:31:01.118703955 +0300}}
{{+++ 2021-09-06_10-31-02/javaposse.jobdsl.plugin.ExecuteDslScripts.xml 2021-09-06 10:31:02.615671917 +0300}}
{{@@ -457,6 +457,13 @@}}
{{ </javaposse.jobdsl.plugin.SeedReference>}}
{{ </entry>}}
{{ <entry>}}
{{+ <string>staging/D10811/test</string>}}
{{+ <javaposse.jobdsl.plugin.SeedReference>}}
{{+ <seedJobName>maintenance/dsl-deployer</seedJobName>}}
{{+ <digest>cf5ff13ea2e9fa70733bdaf08281fd14</digest>}}
{{+ </javaposse.jobdsl.plugin.SeedReference>}}
{{+ </entry>}}
{{+ <entry>}}
{{ <string>staging/D9320_dsl_check/maintenance/sync-ubase</string>}}
{{ <javaposse.jobdsl.plugin.SeedReference>}}
{{ <seedJobName>staging/D9320/maintenance/dsl-deployer</seedJobName>}}
{{@@ -2585,13 +2592,6 @@}}
{{ </javaposse.jobdsl.plugin.SeedReference>}}
{{ </entry>}}
{{ <entry>}}
{{- <string>staging/D10811</string>}}
{{- <javaposse.jobdsl.plugin.SeedReference>}}
{{- <seedJobName>maintenance/dsl-deployer</seedJobName>}}
{{- <digest>d267a35b960390d61d110ab098c875cf</digest>}}
{{- </javaposse.jobdsl.plugin.SeedReference>}}
{{- </entry>}}
{{- <entry>}}
{{ <string>staging/D10506/build/custom/lnvr/diff</string>}}
{{ <javaposse.jobdsl.plugin.SeedReference>}}
{{ <seedJobName>maintenance/dsl-deployer</seedJobName>}}

, and it seems very weird to have so much of them.

Since our seed job recreates the whole folder+job tree on each trigger, is it safe to assume that we can simply remove everything older than N days to preserve disk space?

New: We have a number of jobs described via DSL files and a seed job that turns them into actual Jenkins items. This job is basically triggered on each commit or PR that changes .groovy files (90+ in two repos), and goes like
{code:java}
findAllDslFiles().each { jobDsl(scriptText: readFile(it)) }
{code}
Recently, we noticed that there is a tremendous amount of obscure XML files in `${JENKINS_HOME}/config-history/javaposse.jobdsl.plugin.ExecuteDslScripts`:
{code:java}
# ls 2021-09-06_15-42*
2021-09-06_15-42-06:
history.xml javaposse.jobdsl.plugin.ExecuteDslScripts.xml
2021-09-06_15-42-09:
history.xml javaposse.jobdsl.plugin.ExecuteDslScripts.xml
...
# ls -1 | wc -l
160316{code}
These piled over little more than a year and currently take about 85GB of space. The folders correlate to number of DSL files: f.x. today (2021.09.06) the seed job triggered at 10.38, there are 94 folders with names from `2021-09-06_10-31-01` to `2021-09-06_10-33-56`, and we have 93 DSL files in repo. The files are almost identical:
{code:java}
--- 2021-09-06_10-31-01/javaposse.jobdsl.plugin.ExecuteDslScripts.xml 2021-09-06 10:31:01.118703955 +0300
+++ 2021-09-06_10-31-02/javaposse.jobdsl.plugin.ExecuteDslScripts.xml 2021-09-06 10:31:02.615671917 +0300
@@ -457,6 +457,13 @@
       </javaposse.jobdsl.plugin.SeedReference>
     </entry>
     <entry>
+ <string>staging/D10811/test</string>
+ <javaposse.jobdsl.plugin.SeedReference>
+ <seedJobName>maintenance/dsl-deployer</seedJobName>
+ <digest>cf5ff13ea2e9fa70733bdaf08281fd14</digest>
+ </javaposse.jobdsl.plugin.SeedReference>
+ </entry>
+ <entry>
       <string>staging/D9320_dsl_check/maintenance/sync-ubase</string>
       <javaposse.jobdsl.plugin.SeedReference>
         <seedJobName>staging/D9320/maintenance/dsl-deployer</seedJobName>
@@ -2585,13 +2592,6 @@
       </javaposse.jobdsl.plugin.SeedReference>
     </entry>
     <entry>
- <string>staging/D10811</string>
- <javaposse.jobdsl.plugin.SeedReference>
- <seedJobName>maintenance/dsl-deployer</seedJobName>
- <digest>d267a35b960390d61d110ab098c875cf</digest>
- </javaposse.jobdsl.plugin.SeedReference>
- </entry>
- <entry>
       <string>staging/D10506/build/custom/lnvr/diff</string>
       <javaposse.jobdsl.plugin.SeedReference>
         <seedJobName>maintenance/dsl-deployer</seedJobName>
{code}
, and it seems very weird to have so much of them.

Since our seed job recreates the whole folder+job tree on each trigger, is it safe to assume that we can simply remove everything older than N days to preserve disk space?

Ian Williams made changes - 2021-09-09 08:48

Link

New: This issue is related to JENKINS-66533 [ JENKINS-66533 ]

Jamie Tanna made changes - 2021-12-21 11:34

Assignee

Original: Daniel Spilker [ daspilker ]

New: Jamie Tanna [ jamietanna ]

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates