Hello, I`ve recently started using delivery pipeline plugin. I`ve configured ~
      15 jobs across 2 components. But, while trying to open the view with my pipeline like /api/json, it takes 12+ sec, while other pages are loading in less than a second.

      json?page=1&component=1&fullscreen=false&_=1483005743196 200 xhr jquery.full.js:9659 2.5 KB 10.78 s

      Jenkins version: 2.38
      Pipeline version: 0.10.1

      Are there any ways to determines any bottlenecks?

          [JENKINS-40714] Getting json is too long

          Tommy Tynjä added a comment -

          dhavryl I have not encountered this myself. Does it persist with the latest version of the Delivery Pipeline plugin? If so, could you please attach jobs and pipeline configurations on how to reproduce this?

          Tommy Tynjä added a comment - dhavryl I have not encountered this myself. Does it persist with the latest version of the Delivery Pipeline plugin? If so, could you please attach jobs and pipeline configurations on how to reproduce this?

          Our Jenkins also encountered similar problem. Pipeline does not work when few threads on the server just make `futex` and `sched_yield` and that's all (the only I know about that threads is that they are for displaying delivery pipeline). My guess is that there are many historical builds and displaying pipeline that have about 10 jobs is quite expensive. If few developers attempt to open that pipeline then javascript in browser show information about timeout and re-try request and thread on server side is not interrupted but still working. Requests seems to be queued on server side (few threads spinning).

          Such situation cause that it's hard to use Jenkins (very slow response) and delivery pipeline is not working at all.

          Jenkins restart cause everything is working again but build are stopped. Better workaround i  "Reload configuration from disk"...

           

          Darek Działak added a comment - Our Jenkins also encountered similar problem. Pipeline does not work when few threads on the server just make `futex` and `sched_yield` and that's all (the only I know about that threads is that they are for displaying delivery pipeline). My guess is that there are many historical builds and displaying pipeline that have about 10 jobs is quite expensive. If few developers attempt to open that pipeline then javascript in browser show information about timeout and re-try request and thread on server side is not interrupted but still working. Requests seems to be queued on server side (few threads spinning). Such situation cause that it's hard to use Jenkins (very slow response) and delivery pipeline is not working at all. Jenkins restart cause everything is working again but build are stopped. Better workaround i  "Reload configuration from disk"...  

          Darek Działak added a comment - - edited

          happen again

          htop
          PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
          1386 jenkins 20 0 4833M 1599M 9408 S 391. 40.5 24h14:21 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN
          23256 jenkins 20 0 4833M 1599M 9408 R 86.2 40.5 31:23.93 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN
          23257 jenkins 20 0 4833M 1599M 9408 R 86.2 40.5 31:22.28 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN
          23258 jenkins 20 0 4833M 1599M 9408 R 85.5 40.5 31:23.51 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN
          23255 jenkins 20 0 4833M 1599M 9408 R 84.9 40.5 30:33.95 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN
          8532 jenkins 20 0 4833M 1599M 9408 S 7.9 40.5 0:14.51 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN
          16105 jenkins 20 0 4833M 1599M 9408 S 4.6 40.5 0:00.70 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN```
           

           

          strace
          sched_yield() = 0
          sched_yield() = 0
          sched_yield() = 0
          sched_yield() = 0
          sched_yield() = 0
          sched_yield() = 0
          sched_yield() = 0
          sched_yield() = 0
          futex(0x7fdadc022e54, FUTEX_WAIT_PRIVATE, 531871, NULL) = -1 EAGAIN (Resource temporarily unavailable)
          futex(0x7fdadc022e28, FUTEX_WAKE_PRIVATE, 1) = 0
          futex(0x7fdadc01f454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fdadc01f450, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
          futex(0x7fdadc01f428, FUTEX_WAKE_PRIVATE, 1) = 1
          futex(0x7fdadc022e54, FUTEX_WAIT_PRIVATE, 531873, NULL) = 0
          futex(0x7fdadc022e28, FUTEX_WAKE_PRIVATE, 1) = 0
          

           

           

          threadDump
          "Handling GET /job/XXX/view/Delivery%20pipeline/api/json from 192.168.254.11 : RequestHandlerThread[#55]" Id=46219 Group=main RUNNABLE
          	at java.util.AbstractList$Itr.hasNext(AbstractList.java:351)
          	at java.util.AbstractCollection.toArray(AbstractCollection.java:139)
          	at java.util.ArrayList.<init>(ArrayList.java:177)
          	at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:339)
          	at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74)
          	at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
          	at java.util.AbstractMap$2$1.next(AbstractMap.java:418)
          	at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:89)
          	at se.diabol.jenkins.pipeline.domain.task.Task.getLatestTask(Task.java:187)
          	at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:219)
          	at se.diabol.jenkins.pipeline.domain.Pipeline.getPipelines(Pipeline.java:329)
          	at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:299)
          	at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:542)
          	at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:494)
          	at java.lang.invoke.LambdaForm$DMH/1117519786.invokeVirtual_L_L(LambdaForm$DMH)
          	at java.lang.invoke.LambdaForm$BMH/998156947.reinvoke(LambdaForm$BMH)
          	at java.lang.invoke.LambdaForm$MH/1851807626.invoke_MT(LambdaForm$MH)
          	at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:72)
          	at org.kohsuke.stapler.export.Property.safeGetValue(Property.java:152)
          	at org.kohsuke.stapler.export.Property.writeTo(Property.java:133)
          	at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:227)
          	at org.kohsuke.stapler.export.Model.writeTo(Model.java:198)
          	at org.kohsuke.stapler.ResponseImpl.writeOne(ResponseImpl.java:285)
          	at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:276)
          	at hudson.model.Api.doJson(Api.java:208)
          ...
          

          Darek Działak added a comment - - edited happen again htop PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1386 jenkins 20 0 4833M 1599M 9408 S 391. 40.5 24h14:21 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23256 jenkins 20 0 4833M 1599M 9408 R 86.2 40.5 31:23.93 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23257 jenkins 20 0 4833M 1599M 9408 R 86.2 40.5 31:22.28 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23258 jenkins 20 0 4833M 1599M 9408 R 85.5 40.5 31:23.51 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 23255 jenkins 20 0 4833M 1599M 9408 R 84.9 40.5 30:33.95 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 8532 jenkins 20 0 4833M 1599M 9408 S 7.9 40.5 0:14.51 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN 16105 jenkins 20 0 4833M 1599M 9408 S 4.6 40.5 0:00.70 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -DJENKIN```     strace sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 futex(0x7fdadc022e54, FUTEX_WAIT_PRIVATE, 531871, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0x7fdadc022e28, FUTEX_WAKE_PRIVATE, 1) = 0 futex(0x7fdadc01f454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fdadc01f450, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 futex(0x7fdadc01f428, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7fdadc022e54, FUTEX_WAIT_PRIVATE, 531873, NULL) = 0 futex(0x7fdadc022e28, FUTEX_WAKE_PRIVATE, 1) = 0     threadDump "Handling GET /job/XXX/view/Delivery%20pipeline/api/json from 192.168.254.11 : RequestHandlerThread[#55]" Id=46219 Group=main RUNNABLE at java.util.AbstractList$Itr.hasNext(AbstractList.java:351) at java.util.AbstractCollection.toArray(AbstractCollection.java:139) at java.util.ArrayList.<init>(ArrayList.java:177) at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:339) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:418) at se.diabol.jenkins.pipeline.util.BuildUtil.match(BuildUtil.java:89) at se.diabol.jenkins.pipeline.domain.task.Task.getLatestTask(Task.java:187) at se.diabol.jenkins.pipeline.domain.Stage.createLatestStage(Stage.java:219) at se.diabol.jenkins.pipeline.domain.Pipeline.getPipelines(Pipeline.java:329) at se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineLatest(Pipeline.java:299) at se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(DeliveryPipelineView.java:542) at se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines(DeliveryPipelineView.java:494) at java.lang.invoke.LambdaForm$DMH/1117519786.invokeVirtual_L_L(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/998156947.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/1851807626.invoke_MT(LambdaForm$MH) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:72) at org.kohsuke.stapler.export.Property.safeGetValue(Property.java:152) at org.kohsuke.stapler.export.Property.writeTo(Property.java:133) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:227) at org.kohsuke.stapler.export.Model.writeTo(Model.java:198) at org.kohsuke.stapler.ResponseImpl.writeOne(ResponseImpl.java:285) at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:276) at hudson.model.Api.doJson(Api.java:208) ...

          generating delivery pipeline was failing with timeout until I've removed old builds and I've found one strange thing:

          • removed old builds found by find */builds/ -maxdepth 1 -mindepth 1 -type d -mtime +90
          • used Reload Configuration from Disk
          • Displaying first time delivery pipeline timed out (strange)
          • ...but right now it's already working.

          I assume that first time delivery pipeline failed due to some caches. Anyway right now it is working. Creating delivery pipeline page is taking about 3 secs.

          Darek Działak added a comment - generating delivery pipeline was failing with timeout until I've removed old builds and I've found one strange thing: removed old builds found by find */builds/ -maxdepth 1 -mindepth 1 -type d -mtime +90 used Reload Configuration from Disk Displaying first time delivery pipeline timed out (strange) ...but right now it's already working. I assume that first time delivery pipeline failed due to some caches. Anyway right now it is working. Creating delivery pipeline page is taking about 3 secs.

          Tommy Tynjä added a comment -

          Sounds like a workaround would be to limit the amount of historical builds preserved but that doesn't solve the problem. Currently, the pipelines are evaluated one by one per request which in cases like this is not ideal.

          Tommy Tynjä added a comment - Sounds like a workaround would be to limit the amount of historical builds preserved but that doesn't solve the problem. Currently, the pipelines are evaluated one by one per request which in cases like this is not ideal.

          Is that plugin touching all historical builds?
          Maybe it would be enough if delivery pipeline gets only builds that are not older then the last one build from Initial Job?

          Darek Działak added a comment - Is that plugin touching all historical builds? Maybe it would be enough if delivery pipeline gets only builds that are not older then the last one build from Initial Job ?

            patbos Patrik Boström
            dhavryl Dmytro Havrylevych
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: