Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19346

100% on one CPU

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Duplicate
    • core
    • None

    Description

      We're running Jenkins on an Ubuntu-12.10 virtual machine, with two quad-core CPUs and 16GB RAM.

      This instance has 66 slaves, and around 80 jobs.
      Our instance is pretty much on 100% on one CPU, and the thread count dump keeps going up.

      Attaching the graphs from the monitoring plug-in, and a thread dump
      Any help on debugging this would be appreciated

      Attachments

        1. cpu.png
          cpu.png
          22 kB
        2. cpu (1).png
          cpu (1).png
          25 kB
        3. threadCount.png
          threadCount.png
          25 kB
        4. threadCount (1).png
          threadCount (1).png
          28 kB
        5. Thread Dump
          1.04 MB

        Issue Links

          Activity

            evernat evernat added a comment -

            It's perhaps not the real issue or not the only issue, but you have 4 threads running "deflateBytes", for org.kohsuke.stapler.compression.CompressionFilter.reportException.

            1) Do you still have threads running "deflateBytes" in a new thread dump?

            2) What version of Jenkins are you using? (Is it Jenkins v1.520 or later?)

            See JENKINS-14362 and JENKINS-17349

            evernat evernat added a comment - It's perhaps not the real issue or not the only issue, but you have 4 threads running "deflateBytes", for org.kohsuke.stapler.compression.CompressionFilter.reportException. 1) Do you still have threads running "deflateBytes" in a new thread dump? 2) What version of Jenkins are you using? (Is it Jenkins v1.520 or later?) See JENKINS-14362 and JENKINS-17349
            guyr Guy Rozendorn added a comment - - edited

            We're running Jenkins 1.519; we upgraded a few weeks ago to some later version (don't remember which) but it made things worse so we reverted back.

            and yes, I still see threads in deflateBytes

            guyr Guy Rozendorn added a comment - - edited We're running Jenkins 1.519; we upgraded a few weeks ago to some later version (don't remember which) but it made things worse so we reverted back. and yes, I still see threads in deflateBytes
            guyr Guy Rozendorn added a comment -

            we'll upgrade to the latest now and report back tomorrow

            guyr Guy Rozendorn added a comment - we'll upgrade to the latest now and report back tomorrow
            guyr Guy Rozendorn added a comment -

            we had to downgrade again, as we started hitting on other issues in the newer builds :-\

            guyr Guy Rozendorn added a comment - we had to downgrade again, as we started hitting on other issues in the newer builds :-\
            guyr Guy Rozendorn added a comment -

            I applied the workaround mentioned in JENKINS-14362, will report back tomorrow

            guyr Guy Rozendorn added a comment - I applied the workaround mentioned in JENKINS-14362 , will report back tomorrow
            guyr Guy Rozendorn added a comment -

            The workaround seems to help with the CPU usage, but not with the thread count, uploading new graphs

            guyr Guy Rozendorn added a comment - The workaround seems to help with the CPU usage, but not with the thread count, uploading new graphs
            oleg_nenashev Oleg Nenashev added a comment -

            Marking as resolved/duplicated. The submitter confirmed that the fix for JENKINS-14362 resolves his issue

            oleg_nenashev Oleg Nenashev added a comment - Marking as resolved/duplicated. The submitter confirmed that the fix for JENKINS-14362 resolves his issue

            People

              Unassigned Unassigned
              guyr Guy Rozendorn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: