Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38334

Jenkins (master) hangs; high cpu-load w/o running jobs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: core
    • Labels:
    • Environment:
      Windows Server 2008 R2 x64,
      Java Server JRE 1.8.0_101
    • Similar Issues:

      Description

      Hi,

      Since upgrade to Jenkins 2 (LTS 2.7.2, 2.7.3 and also with 2.22) there's a performance-issue at the Jenkins-master after about 2-4 days: The UI hangs (loading needs 5 or more minutes) and the Java-process needs about 50-99 percent CPU. It's necessary to kill the affected Java-process and to restart the Jenkins-service to re-establish our Jenkins (-farm).

      By "Debugging" the Java-process (with Visual Studio native remote-debugging) I have seen, that there are 1-2 *) WIN32-Threads with highest priority, which are running without noticeable yield/sleep (and occupy almost one core - of two).
      *) Sometimes there is only one of this running thread and the jenkins-master will response with respond with a delay of minutes; In other cases, there are two this (CPU-occupying) threads and jenkins is unusable ...

      Currently, we are using Jenkins 2.22 with following installed plugins:

      • active-directory 1.48
      • all-changes 1.4
      • analysis-core 1.79
      • antisamy-markup-formatter 1.5
      • artifactdeployer 0.33
      • audit-trail 2.2
      • bouncycastle-api 2.16.0
      • build-failure-analyzer 1.17.1
      • build-monitor-plugin 1.10+build.201608030223
      • build-pipeline-plugin 1.5.4
      • build-timeout 1.17.1
      • build-token-root 1.4
      • buildresult-trigger 0.17
      • claim 2.8
      • clone-workspace-scm 0.6
      • cloudbees-folder 5.12
      • conditional-buildstep 1.3.5
      • copy-to-slave 1.4.4
      • copyartifact 1.38.1
      • credentials 2.1.4
      • cron_column 1.4
      • dashboard-view 2.9.10
      • delivery-pipeline-plugin 0.9.12
      • display-upstream-changes 0.1
      • display-url-api 0.3
      • downstream-buildview 1.9
      • downstream-ext 1.8
      • email-ext 2.47
      • emailext-template 1.0
      • envinject 1.92.1
      • extra-columns 1.17
      • flexible-publish 0.15.2
      • groovy-postbuild 2.3.1
      • htmlpublisher 1.11
      • icon-shim 2.0.3
      • javadoc 1.4
      • jenkinswalldisplay 0.6.30
      • job-parameter-summary 0.3
      • job-poll-action-plugin 1.0
      • jobConfigHistory 2.15
      • jquery 1.11.2-0
      • junit 1.18
      • locale 1.2
      • lockable-resources 1.8-SNAPSHOT (private-08/24/2015 10:54-jenkins)
      • log-parser 2.0
      • mailer 1.18
      • mapdb-api 1.0.9.0
      • matrix-auth 1.4
      • matrix-groovy-execution-strategy 1.0.4
      • matrix-project 1.6
      • matrix-reloaded 1.1.3
      • Matrix-sorter-plugin 1.1
      • maven-plugin 2.13
      • modernstatus 1.2
      • nodelabelparameter 1.7.2
      • ownership 0.8
      • parameterized-trigger 2.32
      • plot 1.9
      • postbuildscript 0.17
      • powershell 1.3
      • preSCMbuildstep 0.3
      • promoted-builds 2.27
      • publish-over-ftp 1.12
      • rebuild 1.25
      • role-strategy 2.3.2
      • run-condition 1.0
      • scm-api 1.3
      • script-security 1.22
      • show-build-parameters 1.0
      • ssh-credentials 1.12
      • structs 1.5
      • subversion 2.5.7
      • template-project 1.5.2
      • throttle-concurrents 1.9.0
      • timestamper 1.8.5
      • token-macro 1.12.1
      • validating-string-parameter 2.3
      • warnings 4.56
      • windows-slaves 1.2
      • workflow-step-api 2.3
      • ws-cleanup 0.29
      • xunit 1.102
      • zentimestamp 4.2

      Today, I was able to create a thread-dump (see attachment); at this time, the UI was "only" extremely slow has shown no running jobs, but the java-process has consumed 50-70% CPU.

      Best regards from Salzburg,
      Markus

        Attachments

          Issue Links

            Activity

            Hide
            gmc_devel GMC Software Development B&R Corporate added a comment -

            After some further investigations, I think that this issue may relates to (or caused by) JENKINS-33358 (Groovy and PermGen memory leak);
            I.e., after setting Java-property -Dgroovy.use.classvalue=true our origin problem (CPU-load, unusable UI after a few days) seems to be fixed. Additionally the Java-process does not occupy as much memory (over a few days).

            Show
            gmc_devel GMC Software Development B&R Corporate added a comment - After some further investigations, I think that this issue may relates to (or caused by) JENKINS-33358 (Groovy and PermGen memory leak); I.e., after setting Java-property -Dgroovy.use.classvalue=true our origin problem (CPU-load, unusable UI after a few days) seems to be fixed. Additionally the Java-process does not occupy as much memory (over a few days).
            Hide
            ademdc Adem Dinarevic added a comment -

            GMC Software Development B&R Corporate

            Can you please tell me how to set the Java-property -Dgroovy.use.classvalue=true. 

             

            Thank you very much.

            Show
            ademdc Adem Dinarevic added a comment - GMC Software Development B&R Corporate Can you please tell me how to set the Java-property -Dgroovy.use.classvalue=true.    Thank you very much.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              gmc_devel GMC Software Development B&R Corporate
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: