Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73208

"Scan Repository" on a big one leads to huge amount of threads and kills our Jenkins

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • branch-api-plugin
    • None

      One of our repos under Github Organization has 4100 branches and 1100 pull requests.
      The scanning is performed nightly, once it's running the amount of threads goes through the roof.

      This happens right before "Finished branch indexing":

      [Tue May 21 21:31:06 UTC 2024] Starting branch indexing...
...
[Tue May 21 22:09:16 UTC 2024] Finished branch indexing. Indexing took 38 min
Evaluating orphaned items in Pipeline Builder ยป

      It finished scanning at 22:09, here is the threads graph for that timeframe

      Another multi-branch repo had 4330 branches and 3600 PRs.
      I ran a manual scan, here's the result:

      [Wed May 22 14:48:57 UTC 2024] Finished branch indexing. Indexing took 16 min

      And the threads graph:


      The Jenkins restarted at 14:50.

      From docker logs:

      2024-05-22 14:49:13.931+0000 [id=2668668]	WARNING	hudson.slaves.SlaveComputer#doJenkinsAgentJnlp: Agent "............." is connecting with the "-jnlpUrl" argument, which is deprecated. Use "-url" and "-name" instead, potentially also passing in "-webSocket", "-tunnel", and/or work directory options as needed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69800000, 16384, 0) failed; error='Not enough space' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69700000, 16384, 0) failed; error='Not enough space' (errno=12)
[thread 301033 also had an error]#

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
[thread 301034 also had an error]
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69600000, 16384, 0) failed; error='Not enough space' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69500000, 16384, 0) failed; error='Not enough space' (errno=12)

[lots of similar messages]

      This got to more than 21000 threads:

      CheckMK
APP  17:49
Service PROBLEM notification
Host: jenkins.mycompany.domain (IP: jenkins.mycompany.domain)
Service: Number of threads
State: CRITICAL
Additional Info
CRIT - Count: 21608 threads (warn/crit at 3500 threads/4500 threads)(!!), Usage: 3.85%
Please take a look: @cmkadmin
Check_MK notification: Wed May 22 14:49:15 GMT 2024

      Please let me know if any other info needed.

        1. screenshot-1.png
          screenshot-1.png
          77 kB
        2. image (9).png
          image (9).png
          25 kB
        3. image (11).png
          image (11).png
          67 kB
        4. image (10).png
          image (10).png
          11 kB

            Unassigned Unassigned
            eplodn1 efo plo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: