Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60479

Building a matrix with exclusions in a declarative pipeline using a pipeline library finds no branches to run

      I have run into an issue when attempting to build a matrix in a declarative pipeline using a pipeline library through p4.

      If the excludes section is used, the pipeline is run successfully, but it reports that there are no branches to run. Without the excludes section, all targets in the matrix are run.

       

      To reproduce, I created a Perforce instance with Jenkins in a docker network and installed the p4-plugin 1.10.7. I then created a multibranch pipeline with a source of //testDepot/... from perforce.

      I created two streams and a library stream.

       

      In the first stream, I created a Jenkinsfile:

      @Library('test-pipeline') _
      lib()
      

      Where the test-pipeline is a pipeline library at //testDepot/libstream/... with the pipeline code in //testDepot/libstream/vars/lib.groovy

      def call()
      {
        pipeline
        {
          agent any
          stages
          {
            stage('Build')
            {
              matrix
              {
                axes
                {
                  axis
                  {
                    name 'PLATFORM'
                    values 'Linux', 'Windows', 'OSX'
                  }
                  axis
                  {
                    name 'CONFIGURATION'
                    values 'Debug', 'DebugMT', 'DebugNoPCH', 'Release', 'ReleaseMT', 'Retail', 'RetailMT'
                  }
                }
                excludes
                {
                  exclude
                  {
                    axis
                    {
                      name 'PLATFORM'
                      notValues 'Windows'
                    }
                    axis
                    {
                      name 'CONFIGURATION'
                      values 'DebugMT', 'DebugNoPCH', 'ReleaseMT', 'RetailMT'
                    }
                  }
                }
                stages
                {
                  stage('Build')
                  {
                    steps
                    {
                      echo "Building ${PLATFORM} ${CONFIGURATION}"
                    }
                  }
                  stage('Test')
                  {
                    steps
                    {
                      echo "Testing ${PLATFORM} ${CONFIGURATION}"
                    }
                  }
                }
              }
            }
          }
        }
      }

      When run, this pipeline is successful, but nothing runs.

      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Build)
      [Pipeline] parallel
      No branches to run
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS

      If the excludes section is commented out, all targets in the matrix are run.

       

      In the second stream, I simply put the pipeline from the groovy into a Jenkinsfile. It runs correctly both with and without the excludes section.

       

      Our declarative pipeline is rather large and complex, and we have run into the "Method code too large" issue. We were hoping that we could get around it by building with a matrix in the declarative pipeline, but have run into this issue instead.

       

          [JENKINS-60479] Building a matrix with exclusions in a declarative pipeline using a pipeline library finds no branches to run

          Matthew Brunton created issue -
          Karl Wirth made changes -
          Labels New: P4_SUPPORT

          Karl Wirth added a comment - - edited

          Hi abayer,

          Thanks for reporting this. Before we start, from a pure Perforce streams point of view I'd recommend against having a library stream in the same stream path as code.

          The code expects that streams are divergent forms of the same basic code. For example a stream may contain: 

          //Project1Stream/Rel1/...
          //Project1Stream/Rel2/...
          //Project1Stream/Main/...
          
          //Project1Stream/Dev/...
          

           which could import

          //LibraryStream/main/...

          Alternatively if you have a deeper stream depth set you could have the two stream trees in the same depot:

          //StreamDepot/Project1/Rel1/...
          //StreamDepot/Project1/Rel2/...
          //StreamDepot/Project1/Main/...
          //StreamDepot/Project1/Dev/...
          
          //StreamDepot/Library/Main/...
          //StreamDepot/Library/Dev/...
          

          If you don't have this structure then there will always be outstanding changes that need to be branched between the library stream and it's parents.

          Now back to the problem. I tried setting this up and keep getting compilation errors:

           

          org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed:
          /var/lib/jenkins/jobs/JENKINS-60479_Exclusions_MultiBranch/branches/main/builds/2/libs/test-pipeline/vars/lib.groovy: 8: Unknown stage section "matrix". Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 8, column 7.
                   stage('Build')
          

          I am on 2.176.1 so this may be a version problem. Just in case can you please send me the screenshots of the job definition to make sure I'm doing this correctly.

           

           

          Karl Wirth added a comment - - edited Hi abayer , Thanks for reporting this. Before we start, from a pure Perforce streams point of view I'd recommend against having a library stream in the same stream path as code. The code expects that streams are divergent forms of the same basic code. For example a stream may contain:  //Project1Stream/Rel1/... //Project1Stream/Rel2/... //Project1Stream/Main/... //Project1Stream/Dev/...  which could import //LibraryStream/main/... Alternatively if you have a deeper stream depth set you could have the two stream trees in the same depot: //StreamDepot/Project1/Rel1/... //StreamDepot/Project1/Rel2/... //StreamDepot/Project1/Main/... //StreamDepot/Project1/Dev/... //StreamDepot/Library/Main/... //StreamDepot/Library/Dev/... If you don't have this structure then there will always be outstanding changes that need to be branched between the library stream and it's parents. Now back to the problem. I tried setting this up and keep getting compilation errors:   org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed: / var /lib/jenkins/jobs/JENKINS-60479_Exclusions_MultiBranch/branches/main/builds/2/libs/test-pipeline/vars/lib.groovy: 8: Unknown stage section "matrix" . Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 8, column 7. stage( 'Build' ) I am on 2.176.1 so this may be a version problem. Just in case can you please send me the screenshots of the job definition to make sure I'm doing this correctly.    
          Matthew Brunton made changes -
          Attachment New: image-2019-12-17-13-37-15-881.png [ 49798 ]
          Matthew Brunton made changes -
          Attachment New: image-2019-12-17-13-40-13-460.png [ 49799 ]
          Matthew Brunton made changes -
          Attachment New: image-2019-12-17-13-41-38-310.png [ 49800 ]
          Matthew Brunton made changes -
          Attachment New: image-2019-12-17-13-43-11-090.png [ 49801 ]
          Matthew Brunton made changes -
          Attachment New: image-2019-12-17-13-55-17-110.png [ 49803 ]

          Thanks for responding. I'll supply screenshots of the setup I'm using to reproduce. I'm also using the Pipeline: Declarative plugin 1.5, which was recently released to add matrix support to declarative pipelines.

          Matthew Brunton added a comment - Thanks for responding. I'll supply screenshots of the setup I'm using to reproduce. I'm also using the Pipeline: Declarative plugin 1.5, which was recently released to add matrix support to declarative pipelines.

          Karl Wirth added a comment -

          Hi mbrunton27 - Thanks. I'll try again and let you know how it goes.

          Karl Wirth added a comment - Hi mbrunton27 - Thanks. I'll try again and let you know how it goes.

            Unassigned Unassigned
            mbrunton27 Matthew Brunton
            Votes:
            11 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: