Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20620

Memory Leak on Jenkins LTS 1.509.4/1.532.1

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Critical Critical
    • core
    • Linux Server with 24 CPUs and 64GB RAM
      Jenkins version LTS 1.509.4/1.532.1 on Jetty
      Memory allocated for Jenkins/Jetty process: 42GB
      Environment: Jenkins working with 600 jobs with high activities + 40 slave machines (Linux and Windows)

      After Jenkins upgraded to LTS 1.509.4 from LTS 1.509.3 I notice that over time (24 hours) Jenkins becomes very slowly.
      It turns out that Jenkins (under Jetty service) slowly "eats" the server memory. It's takes about 24 hours to take all the memory allocated to Jenkins (42GB). See the snapshots with examples...

      TEST #1 on LTS 1.509.4:
      1. Machine with Jetty up after restart
      2. Jenkins Used - After one Hour: 22GB
      3. Jenkins Used - After 12 Hours: 27GB
      4. Jenkins Used - After 20 Hours: 35GB -> Memory leaks between 10:00-10:20 as you can see after GC it's still think Java in-use and fail to cleanup the all memory as it should be.
      5. Jenkins Used - After 23 Hours: 39GB -> Very slow response and Heap is almost 100%

      TEST #2 on LTS 1.509.4:
      I tried to do manual GC, Doesn't help!
      (see attach file: "Monitor_Memory_Over_Time_Manual_GC")

      TEST #3 on LTS 1.509.3:
      Unfortunately I downgrade to LTS 1.509.3 because of the memory leak, for me it's a blocker issue!

      Please note that on version LTS 1.509.3 Jenkins works stable even on high environment without any memory leak... (See attach files: "Good_GC_A1.509.3" and "Good_GC_B1.509.3") but unfortunately there is a BIG unsolved problem/bug in this version, I can't rename jobs (Deadlock! which solved on the next version LTS 1.509.4/1.532.1 that I can't use because of the memory leak).

      TEST #4 with LTS 1.532.1:
      Same issue! Jenkins stuck with 100% memory usage after only 12 hours!

      Thank You,
      Ronen.

        1. gc.log
          15 kB
        2. Good_GC_A1.509.3.JPG
          Good_GC_A1.509.3.JPG
          268 kB
        3. Good_GC_B1.509.3.jpg
          Good_GC_B1.509.3.jpg
          322 kB
        4. Monitor_After_1_Hour.jpg
          Monitor_After_1_Hour.jpg
          238 kB
        5. Monitor_After_12_Hours.jpg
          Monitor_After_12_Hours.jpg
          224 kB
        6. Monitor_After_20_Hours.jpg
          Monitor_After_20_Hours.jpg
          321 kB
        7. Monitor_After_23_Hours.JPG
          Monitor_After_23_Hours.JPG
          253 kB
        8. Monitor_Memory_Over_Time_Manual_GC.jpg
          Monitor_Memory_Over_Time_Manual_GC.jpg
          572 kB

            oleg_nenashev Oleg Nenashev
            ronenpg Ronen Peleg
            Votes:
            21 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: