Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10077

archiving large files takes extrordinarily long time

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Critical Critical
    • None
    • Mac OS 10.6.7, Xserve Dual 2.8 Ghz Intel Xeon, 2 TB HDD RAID-1

      We have a project that creates a 3.2 GB file that needs to be archived. We are currently running Jenkins 1.417. The building of the project works just fine but when the archival step begins the process takes forever. I allowed the job to run for more than 12 hours and the archive never completed.

      I can manually copy the files in a fraction of the time (only a few minutes for a build directory of over 15 GB). Increasing the amount of memory available to Jenkins does not improve the copy performance. While the files are being archived the java process takes 100% of the processor cycles on a single processor core. Memory usage does not increase significantly regardless of the number or size of the files to be archived. And disk utilization during the archival process is unbelievably low as well.

          [JENKINS-10077] archiving large files takes extrordinarily long time

          This may be related to JENKIN-7013 in that the underlying problem may be the implementation of copy by Apache ant. As the performance of ant appears to be extremely low it would seem a better alternative should be utilized.

          Jarom Loveridge added a comment - This may be related to JENKIN-7013 in that the underlying problem may be the implementation of copy by Apache ant. As the performance of ant appears to be extremely low it would seem a better alternative should be utilized.

          While this is happening, can you follow https://wiki.jenkins-ci.org/display/JENKINS/Build+is+hanging and get us the stack trace?

          I'd like to see if this is the same thing as what's reported against core or if this is something else.

          Kohsuke Kawaguchi added a comment - While this is happening, can you follow https://wiki.jenkins-ci.org/display/JENKINS/Build+is+hanging and get us the stack trace? I'd like to see if this is the same thing as what's reported against core or if this is something else.

          I set the number of executors down to 1. This build has been tying to archive for several hours at this point. Please see the attached threads.log

          Jarom Loveridge added a comment - I set the number of executors down to 1. This build has been tying to archive for several hours at this point. Please see the attached threads.log

          Coleman added a comment -

          I'm having a similar issue. Any files over 3 GB for archiving seem to hang Jenkins. I'm seeing the same issue. Low disk utilization and a pegged cpu core. I let one job in the archive phase for 12 hours.

          Here is the relevant threadDump information:

          {{Executor #0 for master : executing REDACTED #24

          "Executor #0 for master : executing REDACTED #24" Id=42 Group=main RUNNABLE
          at sun.nio.ch.FileChannelImpl.unmap0(Native Method)
          at sun.nio.ch.FileChannelImpl.access$100(FileChannelImpl.java:32)
          at sun.nio.ch.FileChannelImpl$Unmapper.run(FileChannelImpl.java:667)
          at sun.misc.Cleaner.clean(Cleaner.java:125)
          at sun.nio.ch.FileChannelImpl.unmap(FileChannelImpl.java:676)
          at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:442)
          at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:510)
          at org.apache.tools.ant.util.ResourceUtils.copyResource(ResourceUtils.java:468)
          at org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:519)
          at org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:481)
          at org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:310)
          at org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:827)
          at hudson.FilePath$31$1CopyImpl.doFileOperations(FilePath.java:1509)
          at org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:508)
          at hudson.FilePath$31.invoke(FilePath.java:1523)
          at hudson.FilePath$31.invoke(FilePath.java:1493)
          at hudson.FilePath.act(FilePath.java:758)
          at hudson.FilePath.act(FilePath.java:740)
          at hudson.FilePath.copyRecursiveTo(FilePath.java:1493)
          at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
          at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
          at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
          at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
          at hudson.model.Build$RunnerImpl.post2(Build.java:161)
          at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
          at hudson.model.Run.run(Run.java:1400)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:175)}}

          Coleman added a comment - I'm having a similar issue. Any files over 3 GB for archiving seem to hang Jenkins. I'm seeing the same issue. Low disk utilization and a pegged cpu core. I let one job in the archive phase for 12 hours. Here is the relevant threadDump information: {{Executor #0 for master : executing REDACTED #24 "Executor #0 for master : executing REDACTED #24" Id=42 Group=main RUNNABLE at sun.nio.ch.FileChannelImpl.unmap0(Native Method) at sun.nio.ch.FileChannelImpl.access$100(FileChannelImpl.java:32) at sun.nio.ch.FileChannelImpl$Unmapper.run(FileChannelImpl.java:667) at sun.misc.Cleaner.clean(Cleaner.java:125) at sun.nio.ch.FileChannelImpl.unmap(FileChannelImpl.java:676) at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:442) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:510) at org.apache.tools.ant.util.ResourceUtils.copyResource(ResourceUtils.java:468) at org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:519) at org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:481) at org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:310) at org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:827) at hudson.FilePath$31$1CopyImpl.doFileOperations(FilePath.java:1509) at org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:508) at hudson.FilePath$31.invoke(FilePath.java:1523) at hudson.FilePath$31.invoke(FilePath.java:1493) at hudson.FilePath.act(FilePath.java:758) at hudson.FilePath.act(FilePath.java:740) at hudson.FilePath.copyRecursiveTo(FilePath.java:1493) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635) at hudson.model.Build$RunnerImpl.post2(Build.java:161) at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604) at hudson.model.Run.run(Run.java:1400) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:175)}}

          Does the issue still exist with the last version of Jenkins core?
          Do you have tested?

          Gregory Boissinot added a comment - Does the issue still exist with the last version of Jenkins core? Do you have tested?

          Guy Mahieu added a comment - - edited

          We still observe similar problems on Jenkins 1.436.

          Do you expect the problem to be fixed in a more recent version?

          Guy Mahieu added a comment - - edited We still observe similar problems on Jenkins 1.436. Do you expect the problem to be fixed in a more recent version?

          Regarding the ArtifactDeployer plugin, Ant is used to copy artifacts from a location to an other target location. Ant enables the plugin implement to give a great fexibility.
          Therefore if it is an Ant problem, there are no solutions.

          Gregory Boissinot added a comment - Regarding the ArtifactDeployer plugin, Ant is used to copy artifacts from a location to an other target location. Ant enables the plugin implement to give a great fexibility. Therefore if it is an Ant problem, there are no solutions.

          Unfortunately, I can't fix this issue.
          If in your context it is slow, I suggest you should switch to an another alternative (other plugin maybe, a simple script called for example with the post-build-script plugin and so on).

          Gregory Boissinot added a comment - Unfortunately, I can't fix this issue. If in your context it is slow, I suggest you should switch to an another alternative (other plugin maybe, a simple script called for example with the post-build-script plugin and so on).

            gbois Gregory Boissinot
            jloveridge Jarom Loveridge
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: