Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39770

Pipeline visualization not rendered when there is more that 100 nodes

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • blueocean-plugin
    • None
    • Blue Ocean 1.0-rc3

      Using the Blue Ocean plugin, I see the following traceback:

      FIREFOX:

      error rendering PipelineRunGraph TypeError: child is undefined
      Stack trace:
      render/selectedStage</childMatches<@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:10705:25
      render/selectedStage<@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:10704:40
      render@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:10701:33
      [394]</ReactCompositeComponentMixin._renderValidatedComponentWithoutOwnerOrContext@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:68260:27
      [394]</ReactCompositeComponentMixin._renderValidatedComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:68283:29
      [394]</ReactCompositeComponentMixin.performInitialMount@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67825:25
      [394]</ReactCompositeComponentMixin.mountComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67721:16
      [445]</ReactReconciler.mountComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:74734:18
      [394]</ReactCompositeComponentMixin.performInitialMount@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67834:18
      [394]</ReactCompositeComponentMixin.mountComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67721:16
      [445]</ReactReconciler.mountComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:74734:18
      [394]</ReactCompositeComponentMixin.performInitialMount@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67834:18
      [394]</ReactCompositeComponentMixin.mountComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67721:16
      [445]</ReactReconciler.mountComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:74734:18
      mountComponentIntoNode@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:72953:16
      [471]</Mixin.perform@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:77932:13
      batchedMountComponentIntoNode@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:72975:3
      [416]</ReactDefaultBatchingStrategy.batchedUpdates@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:71613:7
      batchedUpdates@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:75831:3
      [433]</ReactMount._renderNewRootComponent@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:73169:5
      [433]</ReactMount._renderSubtreeIntoContainer@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:73250:21
      [433]</ReactMount.render@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:73271:12
      _renderExtension@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:7547:17
      _renderAllExtensions@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:7529:25
      componentDidUpdate@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:7447:13
      [361]</<.notifyAll@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:62506:9
      [444]</ON_DOM_READY_QUEUEING.close@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:74600:5
      [471]</Mixin.closeAll@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:77998:11
      [471]</Mixin.perform@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:77945:11
      [471]</Mixin.perform@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:77932:13
      [453]</<.perform@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:75823:12
      [453]</flushBatchedUpdates@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:75906:7
      [471]</Mixin.closeAll@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:77998:11
      [471]</Mixin.perform@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:77945:11
      [416]</ReactDefaultBatchingStrategy.batchedUpdates@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:71615:7
      enqueueUpdate@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:75934:5
      enqueueUpdate@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:75542:3
      [452]</ReactUpdateQueue.enqueueSetState@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:75727:5
      [390]</ReactComponent.prototype.setState@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67033:3
      _onScrollHandler@http://dev018.ant.nmhn.lab:8080/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:12224:17
      

      CHROMIUM:

      error rendering PipelineRunGraph TypeError: Cannot read property 'id' of undefined
          at http://127.0.0.1:15000/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:10705:37
          at Array.filter (native)
          at http://127.0.0.1:15000/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:10704:54
          at Array.filter (native)
          at PipelineRunGraph.render (http://127.0.0.1:15000/adjuncts/5b1d2830/org/jenkins/ui/jsmodules/blueocean-dashboard/jenkins-js-extension.js:10701:44)
          at ReactCompositeComponentWrapper._renderValidatedComponentWithoutOwnerOrContext (http://127.0.0.1:15000/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:68260:32)
          at ReactCompositeComponentWrapper._renderValidatedComponent (http://127.0.0.1:15000/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:68283:34)
          at ReactCompositeComponentWrapper.performInitialMount (http://127.0.0.1:15000/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67825:30)
          at ReactCompositeComponentWrapper.mountComponent (http://127.0.0.1:15000/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:67721:21)
          at Object.mountComponent (http://127.0.0.1:15000/adjuncts/5b1d2830/io/jenkins/blueocean/blueocean.js:74734:35)
      

      Using Jenkins 2.30 ; Blue Ocean beta 12.

        1. undef6.png
          undef6.png
          70 kB
        2. undef7.png
          undef7.png
          58 kB
        3. undef3.png
          undef3.png
          43 kB
        4. undef.png
          undef.png
          139 kB

          [JENKINS-39770] Pipeline visualization not rendered when there is more that 100 nodes

          Suggestion is that pipeline does not work when there are more than 100 steps because /run/x/nodes only returns 100 entries.

          Julien Pivotto added a comment - Suggestion is that pipeline does not work when there are more than 100 steps because /run/x/nodes only returns 100 entries.

          Julien Pivotto added a comment - - edited

          HOW TO REPREDUCE

          Pipeline:

          def max=100
          stage('test'){
              node {
                  checkouts = [:]
                  for (i=0; i<max; i++) {
                      def x = i
                      checkouts["foo${x}"] = {
                          checkout([$class: 'GitSCM', branches: [[name: '*/master']],
                          doGenerateSubmoduleConfigurations: false, extensions:
                          [[$class: 'RelativeTargetDirectory', relativeTargetDir: "foo${x}"],
                          [$class: 'ScmName', name: "foo${x}"]],
                          submoduleCfg: [], userRemoteConfigs: [[url: "/tmp/foo${x}"]]])
                     }
                  }
                  parallel checkouts 
              }
          }
          

          Shell:

          cd /tmp; git init foo;(cd foo; touch a; git add a; git commit -m a); for i in $(seq 0 150); do cp -r foo foo$i;(cd foo$i; git commit --amend -m a); done
          

          Works with def max=98

          Julien Pivotto added a comment - - edited HOW TO REPREDUCE Pipeline: def max=100 stage( 'test' ){ node { checkouts = [:] for (i=0; i<max; i++) { def x = i checkouts[ "foo${x}" ] = { checkout([$class: 'GitSCM' , branches: [[name: '*/master' ]], doGenerateSubmoduleConfigurations: false , extensions: [[$class: 'RelativeTargetDirectory' , relativeTargetDir: "foo${x}" ], [$class: 'ScmName' , name: "foo${x}" ]], submoduleCfg: [], userRemoteConfigs: [[url: "/tmp/foo${x}" ]]]) } } parallel checkouts } } Shell: cd /tmp; git init foo;(cd foo; touch a; git add a; git commit -m a); for i in $(seq 0 150); do cp -r foo foo$i;(cd foo$i; git commit --amend -m a); done Works with def max=98

          Works without parallel.

          Julien Pivotto added a comment - Works without parallel.

          I think the scope of this ticket should be: raise a correct error messages if there are more than 100 nodes to display

          Julien Pivotto added a comment - I think the scope of this ticket should be: raise a correct error messages if there are more than 100 nodes to display

          James Dumay added a comment -

          Thanks for the report Julien - ill have someone take a peek at this early next week

          James Dumay added a comment - Thanks for the report Julien - ill have someone take a peek at this early next week

          Michael Neale added a comment -

          So it works without parallel - ie lots of stages?

          Not sure how that many parallel can practically be displayed either

          Michael Neale added a comment - So it works without parallel - ie lots of stages? Not sure how that many parallel can practically be displayed either

          Michael Neale added a comment -

          is 100 the exact limit?

          Michael Neale added a comment - is 100 the exact limit?

          James Dumay added a comment -

          Thats 100 parallels inside a stage?

          James Dumay added a comment - Thats 100 parallels inside a stage?

          Julien Pivotto added a comment - - edited

          We have 33 repositories

          We run three operations on each of them:

          checkout – fetch tags – push new tag

          Plus other stages (build, publish, doc, release).

          The total is greater that 100.

          I think this is because the API /run/x/nodes is paginated ( https://github.com/jenkinsci/blueocean-plugin/blob/f1475e883adbfa8d77777aeafc6a4e6aff216cfc/blueocean-rest/src/main/java/io/jenkins/blueocean/rest/pageable/PagedResponse.java#L31 )

          I do not CARE about SEEING the 100 bullets. Actually that would be horrible. But having that parallelism decreases the duration from 60s to 5s.

          Julien Pivotto added a comment - - edited We have 33 repositories We run three operations on each of them: checkout – fetch tags – push new tag Plus other stages (build, publish, doc, release). The total is greater that 100. I think this is because the API /run/x/nodes is paginated ( https://github.com/jenkinsci/blueocean-plugin/blob/f1475e883adbfa8d77777aeafc6a4e6aff216cfc/blueocean-rest/src/main/java/io/jenkins/blueocean/rest/pageable/PagedResponse.java#L31 ) I do not CARE about SEEING the 100 bullets. Actually that would be horrible. But having that parallelism decreases the duration from 60s to 5s.

          lock('checkout') {
              stage('checkout') {
                  checkout([$class: 'GitSCM',
                          branches: [[name: branch]],
                          doGenerateSubmoduleConfigurations: false,
                          extensions: [[$class: 'CleanCheckout'], [$class: 'ScmName', name: 'super']],
                          submoduleCfg: [],
                          userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}"]]])
          
                  def repos = readFile('.gitslave')
                  def reposLines = repos.readLines()
                  def checkouts = [:]
                  for (line in reposLines) {
                      def repoInfo = line.split(' ')
                      def repoUrl = repoInfo[0]
                      def repoPath = repoInfo[1]
                      def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1)
                      def curatedRepoPath = repoPath.substring(1, repoPath.length()-1)
                      def thisCheckout = {
                          retry(3) {
                              checkout([$class: 'GitSCM',
                                      branches: [[name: branch]],
                                      doGenerateSubmoduleConfigurations: false,
                                      extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: curatedRepoPath], [$class: 'CleanCheckout'], [$class: 'ScmName', name: curatedRepoPath]],
                                      submoduleCfg: [],
                                      userRemoteConfigs: [[url: "${gitBaseUrl}/${curatedRepoUrl}"]]])
                          }
                      
                      checkouts[curatedRepoPath] = thisCheckout
                  }
                  parallel checkouts
              }
          }
          

          That piece of code does a SCM checkout of multiple repos in parallel. That creates a lot of nodes in blue ocean but those nodes are not necessary. Maybe a noNodes {} step around them could do the trick.

          Julien Pivotto added a comment - lock( 'checkout' ) { stage( 'checkout' ) { checkout([$class: 'GitSCM' , branches: [[name: branch]], doGenerateSubmoduleConfigurations: false , extensions: [[$class: 'CleanCheckout' ], [$class: 'ScmName' , name: ' super ' ]], submoduleCfg: [], userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}" ]]]) def repos = readFile( '.gitslave' ) def reposLines = repos.readLines() def checkouts = [:] for (line in reposLines) { def repoInfo = line.split( ' ' ) def repoUrl = repoInfo[0] def repoPath = repoInfo[1] def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1) def curatedRepoPath = repoPath.substring(1, repoPath.length()-1) def thisCheckout = { retry(3) { checkout([$class: 'GitSCM' , branches: [[name: branch]], doGenerateSubmoduleConfigurations: false , extensions: [[$class: 'RelativeTargetDirectory' , relativeTargetDir: curatedRepoPath], [$class: 'CleanCheckout' ], [$class: 'ScmName' , name: curatedRepoPath]], submoduleCfg: [], userRemoteConfigs: [[url: "${gitBaseUrl}/${curatedRepoUrl}" ]]]) } checkouts[curatedRepoPath] = thisCheckout } parallel checkouts } } That piece of code does a SCM checkout of multiple repos in parallel. That creates a lot of nodes in blue ocean but those nodes are not necessary. Maybe a noNodes {} step around them could do the trick.

          Michael Neale added a comment -

          hrm... so the problem is that we actively do not want to show that many things in parallel. there isn't a way to do that now that I can see though... so "fixing" it to show 100+ in parallel is not right though...

          Michael Neale added a comment - hrm... so the problem is that we actively do not want to show that many things in parallel. there isn't a way to do that now that I can see though... so "fixing" it to show 100+ in parallel is not right though...

          michaelneale We should at least handle that case and print a message that we can not display more than X nodes (and X can even be smaller, e.g 42!)

          Julien Pivotto added a comment - michaelneale We should at least handle that case and print a message that we can not display more than X nodes (and X can even be smaller, e.g 42!)

          Michael Neale added a comment -

          Have started some discussion here: https://groups.google.com/forum/#!topic/jenkinsci-ux/5wHIChTF2VQ which may be better than in this ticket... work out what consensus is.

          Michael Neale added a comment - Have started some discussion here: https://groups.google.com/forum/#!topic/jenkinsci-ux/5wHIChTF2VQ which may be better than in this ticket... work out what consensus is.

          Arron Mabrey added a comment -

          I just ran into this 100 node limit as well. We have one mono-repo with >30 projects inside it.

          We have 4 stages `deps`, `pre-test`, `test`, `post-test`. Within each stage we run each project in parallel. Bringing the node count to > 120, thus causing the error.

          I'm trying to understand if this is a hard technical limit, or something else... like an undesirable user experience?

          Thanks,
          – Arron

          Arron Mabrey added a comment - I just ran into this 100 node limit as well. We have one mono-repo with >30 projects inside it. We have 4 stages `deps`, `pre-test`, `test`, `post-test`. Within each stage we run each project in parallel. Bringing the node count to > 120, thus causing the error. I'm trying to understand if this is a hard technical limit, or something else... like an undesirable user experience? Thanks, – Arron

          James Dumay added a comment -

          arronmabrey at the moment this is a hard technical limit but it does raise questions to how big a Pipeline can get before the UI becomes somewhat useless. This is something we want to solve in the near future.

          James Dumay added a comment - arronmabrey at the moment this is a hard technical limit but it does raise questions to how big a Pipeline can get before the UI becomes somewhat useless. This is something we want to solve in the near future.

          Arron Mabrey added a comment -

          jamesdumay Thanks for the feedback. I also agree that the ui could get a little crazy with many nodes. If possible can you expand somewhat on what the particular hard limit is? The roundness of the number 100 seems somewhat arbitrary (although I realize it's probably not), what hard limit does this number represent?

          Arron Mabrey added a comment - jamesdumay Thanks for the feedback. I also agree that the ui could get a little crazy with many nodes. If possible can you expand somewhat on what the particular hard limit is? The roundness of the number 100 seems somewhat arbitrary (although I realize it's probably not), what hard limit does this number represent?

          James Dumay added a comment -

          arronmabrey 100 is the default limit for pagination in our rest APIs

          James Dumay added a comment - arronmabrey 100 is the default limit for pagination in our rest APIs

          Arron Mabrey added a comment -

          Ahh okay.

          Arron Mabrey added a comment - Ahh okay.

          James Dumay added a comment -

          arronmabrey for context, when the visualization was written it we did not have pagination for our REST API enabled by default.

          James Dumay added a comment - arronmabrey for context, when the visualization was written it we did not have pagination for our REST API enabled by default.

          Vivek's comments here suggest that the limit can be altered via a parameter on the REST request:
          https://github.com/jenkinsci/blueocean-plugin/pull/486/files#r77956164

          Would it be possible to do that, rather than using the default of 100?

          Alastair D'Silva added a comment - Vivek's comments here suggest that the limit can be altered via a parameter on the REST request: https://github.com/jenkinsci/blueocean-plugin/pull/486/files#r77956164 Would it be possible to do that, rather than using the default of 100?

          James Dumay added a comment -

          A bit more complicated than that but effectively yes we are looking into it

          James Dumay added a comment - A bit more complicated than that but effectively yes we are looking into it

          James Dumay added a comment -

          This will be released in rc2 this week.

          James Dumay added a comment - This will be released in rc2 this week.

          Reopening as I still see only 100 steps in a Blue Ocean visualization here in May 2021.

          Brian J Murrell added a comment - Reopening as I still see only 100 steps in a Blue Ocean visualization here in May 2021.

            tscherler Thorsten Scherler
            roidelapluie Julien Pivotto
            Votes:
            6 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: