-
Bug
-
Resolution: Unresolved
-
Minor
-
Jenkins LTS 2.46.3
thinBackup plugin 1.9
Jenkins hosted on Linux (CentOS Linux release 7.4.1708 (Core))
We are using the thinBackup plugin to backup our Jenkins instance.
Our backup folder is hosted on Amazon S3.
We're using an "Amazon Storage Gateway" to present the bucket as an NFS share.
A copy of our configuration is attached to the issue.
Around 1200 BST, our Jenkins server became slower and slower.
We found a lot of SEVERE error like below in the Jenkins log file:
Jun 26, 2018 11:46:24 AM org.jvnet.hudson.plugins.thinbackup.backup.DirectoriesZipper handleFileJun 26, 2018 11:46:24 AM org.jvnet.hudson.plugins.thinbackup.backup.DirectoriesZipper handleFile SEVERE: Could not create ZIP entryjava.io.FileNotFoundException: /forgerock/jenkins-backup/FULL-2018-06-17_00-00/jobs/Agents/jobs/Push extlibs/builds/294/14.log (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.jvnet.hudson.plugins.thinbackup.backup.DirectoriesZipper.handleFile(DirectoriesZipper.java:45) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:367) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:364) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:364) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:364) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:364) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:364) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:364) at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335) at org.jvnet.hudson.plugins.thinbackup.backup.DirectoriesZipper.addToZip(DirectoriesZipper.java:34) at org.jvnet.hudson.plugins.thinbackup.backup.BackupSet.zipTo(BackupSet.java:366) at org.jvnet.hudson.plugins.thinbackup.utils.Utils.moveOldBackupsToZipFile(Utils.java:394) at org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup$ZipperThread.run(HudsonBackup.java:588)
We think that due to the latency with Amazon S3, the plugin wasn't able to zip the oldest folder. Every minute, we got a lot of errors in the Jenkins log file.
We took a look to the GC log files (I've attached three screen copies). The JVM was using all the heap available (we are using the following JVM settings: -Xms2G -Xmx6G).
The JVM wasn't able to manage the GC process and we had to restart our Jenkins instance.
Can you help to fix this issue with the thinBackup plugin? (maybe a memory leak)
Thanks
