-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Powered by SuggestiMate
Hello, I`ve recently started using delivery pipeline plugin. I`ve configured ~
15 jobs across 2 components. But, while trying to open the view with my pipeline like /api/json, it takes 12+ sec, while other pages are loading in less than a second.
json?page=1&component=1&fullscreen=false&_=1483005743196 200 xhr jquery.full.js:9659 2.5 KB 10.78 s
Jenkins version: 2.38
Pipeline version: 0.10.1
Are there any ways to determines any bottlenecks?
[JENKINS-40714] Getting json is too long
Our Jenkins also encountered similar problem. Pipeline does not work when few threads on the server just make `futex` and `sched_yield` and that's all (the only I know about that threads is that they are for displaying delivery pipeline). My guess is that there are many historical builds and displaying pipeline that have about 10 jobs is quite expensive. If few developers attempt to open that pipeline then javascript in browser show information about timeout and re-try request and thread on server side is not interrupted but still working. Requests seems to be queued on server side (few threads spinning).
Such situation cause that it's hard to use Jenkins (very slow response) and delivery pipeline is not working at all.
Jenkins restart cause everything is working again but build are stopped. Better workaround i "Reload configuration from disk"...
happen again
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1386 jenkins 20 0 4833M 1599M 9408 S 391. 40.5 24h14:21 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23256 jenkins 20 0 4833M 1599M 9408 R 86.2 40.5 31:23.93 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23257 jenkins 20 0 4833M 1599M 9408 R 86.2 40.5 31:22.28 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23258 jenkins 20 0 4833M 1599M 9408 R 85.5 40.5 31:23.51 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23255 jenkins 20 0 4833M 1599M 9408 R 84.9 40.5 30:33.95 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 8532 jenkins 20 0 4833M 1599M 9408 S 7.9 40.5 0:14.51 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 16105 jenkins 20 0 4833M 1599M 9408 S 4.6 40.5 0:00.70 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN```
sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 futex(0x7fdadc022e54, FUTEX_WAIT_PRIVATE, 531871, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0x7fdadc022e28, FUTEX_WAKE_PRIVATE, 1) = 0 futex(0x7fdadc01f454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fdadc01f450, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fdadc01f428, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7fdadc022e54, FUTEX_WAIT_PRIVATE, 531873, NULL) = 0 futex(0x7fdadc022e28, FUTEX_WAKE_PRIVATE, 1) = 0
"Handling GET /job/XXX/view/Delivery%20pipeline/api/json from 192.168.254.11 : RequestHandlerThread[#55]" Id=46219 Group=main RUNNABLE at java.util.AbstractList$Itr.hasNext(AbstractList.java:351) at java.util.AbstractCollection.toArray(AbstractCollection.java:139) at java.util.ArrayList.<init>(ArrayList.java:177) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:339) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:418) at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:89) at se.diabol.jenkins.pipeline.domain.task.Task.getLatestTask(Task.java:187) at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:219) at se.diabol.jenkins.pipeline.domain.Pipeline.getPipelines(Pipeline.java:329) at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:299) at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:542) at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:494) at java.lang.invoke.LambdaForm$DMH/1117519786.invokeVirtual_L_L(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/998156947.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/1851807626.invoke_MT(LambdaForm$MH) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:72) at org.kohsuke.stapler.export.Property.safeGetValue(Property.java:152) at org.kohsuke.stapler.export.Property.writeTo(Property.java:133) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:227) at org.kohsuke.stapler.export.Model.writeTo(Model.java:198) at org.kohsuke.stapler.ResponseImpl.writeOne(ResponseImpl.java:285) at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:276) at hudson.model.Api.doJson(Api.java:208) ...
generating delivery pipeline was failing with timeout until I've removed old builds and I've found one strange thing:
- removed old builds found by find */builds/ -maxdepth 1 -mindepth 1 -type d -mtime +90
- used Reload Configuration from Disk
- Displaying first time delivery pipeline timed out (strange)
- ...but right now it's already working.
I assume that first time delivery pipeline failed due to some caches. Anyway right now it is working. Creating delivery pipeline page is taking about 3 secs.
Sounds like a workaround would be to limit the amount of historical builds preserved but that doesn't solve the problem. Currently, the pipelines are evaluated one by one per request which in cases like this is not ideal.
Is that plugin touching all historical builds?
Maybe it would be enough if delivery pipeline gets only builds that are not older then the last one build from Initial Job?
dhavryl I have not encountered this myself. Does it persist with the latest version of the Delivery Pipeline plugin? If so, could you please attach jobs and pipeline configurations on how to reproduce this?