Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68116

Slow processing of multi branch events

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • None

    Description

      I'm seeing delays of about an hour from when a multi branch event is created in GitHub to when Jenkins actually processes it.

      If I run the in the script console:

      jenkins.scm.api.SCMEvent.executorService
      

      I see:

      java.util.concurrent.ScheduledThreadPoolExecutor@8b0291b[Running, pool size = 10, active threads = 10, queued tasks = 2401, completed tasks = 12838]
      

      Relevant code appears to be around:
      https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1179-L1198
      and

      https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1385

      Running:

      Jenkins.get().getAllItems(jenkins.branch.MultiBranchProject.class).size()
      

      Gives:
      579 multi branch projects.

      From what I can see in the log in
      /var/jenkins_home/jenkins.branch.* logs

      Events seem to be process in 0-1 seconds but based on Maths comparing events processed over an hour we were only processing 22 a minute, so some must be quite slow.

      We only get to keep 15 minutes worth of logs as the file max size is set to 33 kilobytes for some reason:
      https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1138-L1142

      Context:
      Large organistion with 1.7K repositories, and an organisation webhook is pointing at Jenkins

      Few ideas so far:
      1. Throw more threads at it

      https://github.com/jenkinsci/scm-api-plugin/blob/master/src/main/java/jenkins/scm/api/SCMEvent.java#L215

      Is hardcoded to 10 threads at a time

      2. Keep logs for longer

      3. See if there's something specific that could be holding this up?

      4. Global configuration to filter projects out of webhook processing? We have lots that don't need to be processed and will never be matched

      Attachments

        1. events-trace.txt
          53 kB
        2. events-trace-90s.txt
          53 kB
        3. github-api-get-repos-sorted.txt
          0.4 kB
        4. over-10-seconds.txt
          7 kB
        5. over-10-seconds-29-3.txt
          19 kB
        6. over-1-second.txt
          21 kB
        7. threads.6.20220325093536.txt
          667 kB
        8. threads.6.20220325093550.txt
          670 kB
        9. threads.6.20220325093602.txt
          642 kB
        10. threads.6.20220325093615.txt
          642 kB
        11. threads.6.20220325093629.txt
          678 kB
        12. threads.6.20220325093641.txt
          689 kB
        13. threads.6.20220325093655.txt
          716 kB
        14. threads.6.20220325093706.txt
          696 kB
        15. threads.6.20220325093720.txt
          704 kB
        16. threads.6.20220325093732.txt
          698 kB
        17. threads.6.20220325093745.txt
          662 kB
        18. threads.6.20220325093759.txt
          671 kB

        Activity

          People

            Unassigned Unassigned
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: