Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37570

Jenkins starts to response super slow after creating 400 branch jobs from Multi-branch-project

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • branch-api-plugin, core
    • None
    • Ubuntu 14.04 x86_64
      OpenJDK-7u101
      Jenkins 2.17
      Multi-branch-project 0.5.1

      8-core CPU | 8GB RAM |
      JVM: -Xms2048m -Xmx 2048m

      Right after doing branch-indexing to create 400 branch jobs,
      Jenkins starts to response super slow when open any pages (job page, manage jenkins page, etc).

      And after delete the Multi-branch job of 400 branch jobs,
      everything is back to normal.

      From the log,
      I don't see any out of memory error.
      But I do see the Java process has high %CPU (400~500) with about 1.9G RES memory.

          [JENKINS-37570] Jenkins starts to response super slow after creating 400 branch jobs from Multi-branch-project

          Rick Liu created issue -

          Daniel Beck added a comment -

          Daniel Beck added a comment - Would probably help if you did this: https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump
          Rick Liu made changes -
          Attachment New: threddump-01.log [ 33774 ]

          Rick Liu added a comment -

          This is the threddump I did using method1 (from Jenkins Web UI)
          when I click the multi-branch project that contains 300+ branch jobs.

          Rick Liu added a comment - This is the threddump I did using method1 (from Jenkins Web UI) when I click the multi-branch project that contains 300+ branch jobs.
          Rick Liu made changes -
          Attachment New: threaddump-02 [ 33798 ]

          Hi,

          I'm facing also a similar problem. We have around 60 multibranch pipelines jobs (not so many) and all 'branch indexing' are run at the same time. At this moment, jenkins becomes really slow and indexing might last 3 or 4 minutes instead of a few seconds, taken 1 by 1.

          There is no specific hardware limitation on the jenkins master and GIT server is also well sized. The configuration "Periodically if not otherwise run", if set to '5 minutes' for example has not the expected impact. Indeed the load is still the same because it starts all indexing operations at the same time, every 5 minutes. it seems that this configuration is based on absolute time so there is no repartition of load using this.

          Thanks for your help.

           

           

          Alexandre Aubert added a comment - Hi, I'm facing also a similar problem. We have around 60 multibranch pipelines jobs (not so many) and all 'branch indexing' are run at the same time. At this moment, jenkins becomes really slow and indexing might last 3 or 4 minutes instead of a few seconds, taken 1 by 1. There is no specific hardware limitation on the jenkins master and GIT server is also well sized. The configuration "Periodically if not otherwise run", if set to '5 minutes' for example has not the expected impact. Indeed the load is still the same because it starts all indexing operations at the same time, every 5 minutes. it seems that this configuration is based on absolute time so there is no repartition of load using this. Thanks for your help.    

          i increase the criticity because our Jenkins is unusable when too many branch indexing operations are triggered, and this is not configurable so it directly depends on multibranch jobs count.

           

          It would be really nice to find a solution for this.

          Alexandre Aubert added a comment - i increase the criticity because our Jenkins is unusable when too many branch indexing operations are triggered, and this is not configurable so it directly depends on multibranch jobs count.   It would be really nice to find a solution for this.
          Alexandre Aubert made changes -
          Priority Original: Minor [ 4 ] New: Major [ 3 ]
          Matthew DeTullio made changes -
          Component/s New: branch-api-plugin [ 18621 ]
          Component/s Original: multi-branch-project-plugin [ 21127 ]
          Matthew DeTullio made changes -
          Assignee Original: Matthew DeTullio [ mjdetullio ] New: Stephen Connolly [ stephenconnolly ]

            Unassigned Unassigned
            totoroliu Rick Liu
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: