Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63840

Extreme CPU utilization leading to bad performance and failing jobs

    XMLWordPrintable

Details

    Description

      Jenkins consumes vast amounts of CPU resources. E.g. on one machine during 103 days uptime htop reports 7387 CPU hours, i.e. in the mean 3 out of 8 cores are 100% utilzed during uptime. Actually it seems to be worse:

        1  [||||||||||||||||||||||||||||||||||||||||||||           71.7%]   5  [|||||||||||||||||||||||||||||||||||||||||||||||||||    84.0%]
        2  [|||||||||||||||||||||||||||||||||||||||||||||||||      79.6%]   6  [|||||||||||||||||||||||||||||||||||||||||||            69.4%]
        3  [|||||||||||||||||||||||||||||||||||||||||||||||        76.6%]   7  [|||||||||||||||||||||||||||||||||||||||||||||||||||||| 88.7%]
        4  [||||||||||||||||||||||||||||||||||||||||||||||||||||   85.2%]   8  [|||||||||||||||||||||||||||||||||||||||||||||||||||    82.4%]
        Mem[|||||||||||||||||||||||||||||||||||||||||||||||||4.03G/7.79G]   Tasks: 48; 6 running
        Swp[|                                                 253M/16.0G]   Load average: 8.30 8.02 8.28
                                                                            Uptime: 103 days(!), 20:54:02
      
      603 jenkins    20   0 7779M 2524M  5564 S 421. 31.6    7387h ├─ /java/jdk8/bin/java -XX:ErrorFile=/.mntpts/sdb1/jenkins/hs_err_pidjenkins...
      

      A possible cause of this might be {{AsyncResourceDisposer.worker}}s that run into errors due to insufficient file permissions for the user (jenkins) that runs the Jenkins daemon. It seems that these workers are unable to recover and skip the offending files and directories.

      In our specific case the files have been created by Docker containers, but it may have other causes as well, so it's probably a quite common source of problems.

      As a sidenote, I did not find the error in Jenkin's log files, which made it difficult to track down possible reasons for the bad Jenkins performance and Jobs that failed for no good reasons.

      Name: AsyncResourceDisposer.worker [#9]
      State: RUNNABLE
      Total blocked: 33  Total waited: 11.060
      
      Stack trace: 
      java.lang.Throwable.fillInStackTrace(Native Method)
      java.lang.Throwable.fillInStackTrace(Throwable.java:783)
         - locked java.nio.file.AccessDeniedException@45ff02ae
      java.lang.Throwable.<init>(Throwable.java:265)
      java.lang.Exception.<init>(Exception.java:66)
      java.io.IOException.<init>(IOException.java:58)
      java.nio.file.FileSystemException.<init>(FileSystemException.java:73)
      java.nio.file.AccessDeniedException.<init>(AccessDeniedException.java:66)
      sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
      sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
      sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
      java.nio.file.Files.deleteIfExists(Files.java:1165)
      jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:237)
      jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:205)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:216)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:226)
      jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:215)
      jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96)
      hudson.Util.deleteRecursive(Util.java:296)
      hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1310)
      hudson.FilePath$DeleteRecursive.invoke(FilePath.java:1306)
      hudson.FilePath.act(FilePath.java:1076)
      hudson.FilePath.act(FilePath.java:1059)
      hudson.FilePath.deleteRecursive(FilePath.java:1304)
      hudson.plugins.ws_cleanup.Wipeout.performDelete(Wipeout.java:82)
      hudson.plugins.ws_cleanup.Wipeout$DisposableImpl.dispose(Wipeout.java:114)
      org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:283)
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      java.util.concurrent.FutureTask.run(FutureTask.java:266)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.Thread.run(Thread.java:748)
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            alexanderv Alexander Veit
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: