-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins: 2.321
OS: Linux - 5.4.0-1073-azure
---
ace-editor:1.1
ansicolor:1.0.1
antisamy-markup-formatter:2.5
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
azure-ad:185.v3b416408dcb1
azure-artifact-manager:97.v074e1332e88d
azure-credentials:198.vf9c2fdfde55c
azure-keyvault:131.v867845ef6ae9
azure-sdk:70.v63f6a95999a7
azure-vm-agents:799.va4c741108611
basic-branch-build-strategies:1.3.2
blueocean:1.25.2
blueocean-autofavorite:1.2.4
blueocean-bitbucket-pipeline:1.25.2
blueocean-commons:1.25.2
blueocean-config:1.25.2
blueocean-core-js:1.25.2
blueocean-dashboard:1.25.2
blueocean-display-url:2.4.1
blueocean-events:1.25.2
blueocean-git-pipeline:1.25.2
blueocean-github-pipeline:1.25.2
blueocean-i18n:1.25.2
blueocean-jwt:1.25.2
blueocean-personalization:1.25.2
blueocean-pipeline-api-impl:1.25.2
blueocean-pipeline-editor:1.25.2
blueocean-pipeline-scm-api:1.25.2
blueocean-rest:1.25.2
blueocean-rest-impl:1.25.2
blueocean-web:1.25.2
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-3
bouncycastle-api:2.25
branch-api:2.7.0
build-monitor-plugin:1.13+build.202111220004
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
cloud-stats:0.27
cloudbees-bitbucket-branch-source:726.vb0c1ea6c9336
cloudbees-disk-usage-simple:0.10
cloudbees-folder:6.16
cloudbees-jenkins-advisor:3.3.2
command-launcher:1.6
configuration-as-code:1.54
copyartifact:1.46.2
credentials:2.6.2
credentials-binding:1.27
dark-theme:155.v497c78bbdbb3
display-url-api:2.3.5
docker-commons:1.17
docker-workflow:1.26
durable-task:493.v195aefbb0ff2
echarts-api:5.2.2-1
extended-read-permission:3.2
favorite:2.3.3
font-awesome-api:5.15.4-3
gatling:1.3.0
git:4.10.0
git-client:3.10.0
git-server:1.10
github:1.34.1
github-api:1.133
github-branch-source:2.11.3
github-checks:1.0.13
github-scm-trait-notification-context:1.1
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-1.0
htmlpublisher:1.28
http_request:1.12
jackson2-api:2.13.0-230.v59243c64b0a5
jacoco:3.3.0
javadoc:1.6
jaxb:2.3.0.1
jdk-tool:1.5
jenkins-design-language:1.25.2
jjwt-api:0.11.2-9.c8b45b8bb173
job-dsl:1.78.1
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.53
kubernetes:1.30.10
kubernetes-client-api:5.4.1
kubernetes-credentials:0.9.0
lockable-resources:2.12
mailer:1.34
matrix-auth:2.6.8
matrix-project:1.19
maven-plugin:3.15.1
metrics:4.0.2.8
momentjs:1.1.1
monitoring:1.88.0
okhttp-api:4.9.2-20211102
pipeline-build-step:2.15
pipeline-github-lib:1.0
pipeline-graph-analysis:1.11
pipeline-graph-view:46.vedbf23da719f
pipeline-input-step:2.12
pipeline-milestone-step:1.3.2
pipeline-model-api:1.9.3
pipeline-model-definition:1.9.3
pipeline-model-extensions:1.9.3
pipeline-rest-api:2.19
pipeline-stage-step:2.5
pipeline-stage-tags-metadata:1.9.3
pipeline-stage-view:2.19
pipeline-utility-steps:2.10.0
plain-credentials:1.7
plugin-util-api:2.5.1
popper-api:1.16.1-2
popper2-api:2.10.2-1
prometheus:2.0.10
pubsub-light:1.16
run-condition:1.5
saml:2.0.9
sauce-ondemand:1.197
scm-api:2.6.5
script-security:1.78
slack:2.48
snakeyaml-api:1.29.1
sonar:2.14
sse-gateway:1.24
ssh-credentials:1.19
sshd:3.1.0
structs:1.23
support-core:2.79
theme-manager:0.6
timestamper:1.14
token-macro:267.vcdaea6462991
trilead-api:1.0.13
variant:1.4
windows-azure-storage:365.vf41653c43b01
workflow-aggregator:2.6
workflow-api:2.47
workflow-basic-steps:2.24
workflow-cps:2633.v6baeedc13805
workflow-cps-global-lib:548.v9085a486966a
workflow-durable-task-step:1097.veac1aacfbda8
workflow-job:2.42
workflow-multibranch:2.26
workflow-scm-step:2.13
workflow-step-api:2.24
workflow-support:3.8Jenkins: 2.321 OS: Linux - 5.4.0-1073-azure --- ace-editor:1.1 ansicolor:1.0.1 antisamy-markup-formatter:2.5 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 azure-ad:185.v3b416408dcb1 azure-artifact-manager:97.v074e1332e88d azure-credentials:198.vf9c2fdfde55c azure-keyvault:131.v867845ef6ae9 azure-sdk:70.v63f6a95999a7 azure-vm-agents:799.va4c741108611 basic-branch-build-strategies:1.3.2 blueocean:1.25.2 blueocean-autofavorite:1.2.4 blueocean-bitbucket-pipeline:1.25.2 blueocean-commons:1.25.2 blueocean-config:1.25.2 blueocean-core-js:1.25.2 blueocean-dashboard:1.25.2 blueocean-display-url:2.4.1 blueocean-events:1.25.2 blueocean-git-pipeline:1.25.2 blueocean-github-pipeline:1.25.2 blueocean-i18n:1.25.2 blueocean-jwt:1.25.2 blueocean-personalization:1.25.2 blueocean-pipeline-api-impl:1.25.2 blueocean-pipeline-editor:1.25.2 blueocean-pipeline-scm-api:1.25.2 blueocean-rest:1.25.2 blueocean-rest-impl:1.25.2 blueocean-web:1.25.2 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-3 bouncycastle-api:2.25 branch-api:2.7.0 build-monitor-plugin:1.13+build.202111220004 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloud-stats:0.27 cloudbees-bitbucket-branch-source:726.vb0c1ea6c9336 cloudbees-disk-usage-simple:0.10 cloudbees-folder:6.16 cloudbees-jenkins-advisor:3.3.2 command-launcher:1.6 configuration-as-code:1.54 copyartifact:1.46.2 credentials:2.6.2 credentials-binding:1.27 dark-theme:155.v497c78bbdbb3 display-url-api:2.3.5 docker-commons:1.17 docker-workflow:1.26 durable-task:493.v195aefbb0ff2 echarts-api:5.2.2-1 extended-read-permission:3.2 favorite:2.3.3 font-awesome-api:5.15.4-3 gatling:1.3.0 git:4.10.0 git-client:3.10.0 git-server:1.10 github:1.34.1 github-api:1.133 github-branch-source:2.11.3 github-checks:1.0.13 github-scm-trait-notification-context:1.1 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.28 http_request:1.12 jackson2-api:2.13.0-230.v59243c64b0a5 jacoco:3.3.0 javadoc:1.6 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.2 jjwt-api:0.11.2-9.c8b45b8bb173 job-dsl:1.78.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.53 kubernetes:1.30.10 kubernetes-client-api:5.4.1 kubernetes-credentials:0.9.0 lockable-resources:2.12 mailer:1.34 matrix-auth:2.6.8 matrix-project:1.19 maven-plugin:3.15.1 metrics:4.0.2.8 momentjs:1.1.1 monitoring:1.88.0 okhttp-api:4.9.2-20211102 pipeline-build-step:2.15 pipeline-github-lib:1.0 pipeline-graph-analysis:1.11 pipeline-graph-view:46.vedbf23da719f pipeline-input-step:2.12 pipeline-milestone-step:1.3.2 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.19 pipeline-stage-step:2.5 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.19 pipeline-utility-steps:2.10.0 plain-credentials:1.7 plugin-util-api:2.5.1 popper-api:1.16.1-2 popper2-api:2.10.2-1 prometheus:2.0.10 pubsub-light:1.16 run-condition:1.5 saml:2.0.9 sauce-ondemand:1.197 scm-api:2.6.5 script-security:1.78 slack:2.48 snakeyaml-api:1.29.1 sonar:2.14 sse-gateway:1.24 ssh-credentials:1.19 sshd:3.1.0 structs:1.23 support-core:2.79 theme-manager:0.6 timestamper:1.14 token-macro:267.vcdaea6462991 trilead-api:1.0.13 variant:1.4 windows-azure-storage:365.vf41653c43b01 workflow-aggregator:2.6 workflow-api:2.47 workflow-basic-steps:2.24 workflow-cps:2633.v6baeedc13805 workflow-cps-global-lib:548.v9085a486966a workflow-durable-task-step:1097.veac1aacfbda8 workflow-job:2.42 workflow-multibranch:2.26 workflow-scm-step:2.13 workflow-step-api:2.24 workflow-support:3.8
I'm seeing delays of about an hour from when a multi branch event is created in GitHub to when Jenkins actually processes it.
If I run the in the script console:
jenkins.scm.api.SCMEvent.executorService
I see:
java.util.concurrent.ScheduledThreadPoolExecutor@8b0291b[Running, pool size = 10, active threads = 10, queued tasks = 2401, completed tasks = 12838]
Relevant code appears to be around:
https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1179-L1198
and
Running:
Jenkins.get().getAllItems(jenkins.branch.MultiBranchProject.class).size()
Gives:
579 multi branch projects.
From what I can see in the log in
/var/jenkins_home/jenkins.branch.* logs
Events seem to be process in 0-1 seconds but based on Maths comparing events processed over an hour we were only processing 22 a minute, so some must be quite slow.
We only get to keep 15 minutes worth of logs as the file max size is set to 33 kilobytes for some reason:
https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1138-L1142
Context:
Large organistion with 1.7K repositories, and an organisation webhook is pointing at Jenkins
Few ideas so far:
1. Throw more threads at it
Is hardcoded to 10 threads at a time
2. Keep logs for longer
3. See if there's something specific that could be holding this up?
4. Global configuration to filter projects out of webhook processing? We have lots that don't need to be processed and will never be matched