Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52066

Poll per change with multibranch pipeline jobs

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Feature request: Implement poll per change on multibranch pipeline jobs.

      With multibranch poll per change doesnt poll every change but does build only the next change that needs to be built.For example:

      (1) Add CL 626, 627 and 628.

      (2) The multi branch top level poll is configured for 1 min (see below). When it triggers it detects that 625 to 628 exist.

      (3) The job checks SyncID and correctly finds '625' as the previous last sync.   

      Found last change 625 on syncID jenkins-NODE_NAME-MultiBranch_PollPerChange-main-EXECUTOR_NUMBER

      (4) Due to filtering only the next changelist is synced:

                P4 Task: syncing files at change: 626

      (5) Jenkins records the last successful build for this branch correctly:

                branches/main/lastSuccessful/build.xml:          <change>626</change>

      (6) But (and I'm guessing this is the cause) sets '628' in the following file:

                branches/main/scm-revision-hash.xml

      The next time a poll runs it will only fire if a new changelist has been created but will build changelist '627' only (so will now be building 2 changelists behind current).
       
      My Jenkinsfile:

           1    pipeline {
           2      agent { label 'master' }
           3      options { skipDefaultCheckout() }
           4      stages {
           5        stage("Repro") {
           6          steps {
           7            script {
           8               checkout perforce(credential: 'MasterServer', filter: [incremental(true)], populate: autoClean(delete: true, modtime: false, parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], pin: '', quiet: true, replace: true, tidy: false), workspace: manualSpec(charset: 'none', name: 'jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}', pinHost: false, spec: clientSpec(allwrite: false, backup: false, clobber: true, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, serverID: '', streamName: '', type: 'WRITABLE', view: '//depot/multibranch_poll_per_change/main/... //jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...')))
           9               sh "pwd"
          10               sh "ls -l"
          11           sh "cat -n *"
          12           sh "echo ${env.BRANCH_NAME}"
          13            }
          14          }
          15        }
          16      }
          17    }

       

       

        Attachments

          Issue Links

            Activity

            Hide
            s3037560 Kevin Park added a comment - - edited

            It also occurs on Freestyle project.

            As per the attached screen shot, "Change 12264" submit is not run after "Change 12263" submit has been run. "Change 12264" submit will be run after "Change 12265" is submitted.

             

            $ p4 edit a.txt

            //test/a.txt#23 - opened for edit

            $ p4 submit -d "test"

            Submitting change 12263.

            Locking 1 files ...

            edit //test/a.txt#24

            Change 12263 submitted.

             

            $ p4 edit a.txt

            //test/a.txt#24 - opened for edit

            $ p4 submit -d "test"

            Submitting change 12264.

            Locking 1 files ...

            edit //test/a.txt#25

            Change 12264 submitted.

             

            $ p4 edit a.txt

            //test/a.txt#25 - opened for edit

            $ p4 submit -d "test"

            Submitting change 12265.

            Locking 1 files ...

            edit //test/a.txt#26

            Change 12265 submitted.

             

            When not using "Poll per change", concurrent submitted changelists are run together in one Jenkins build. 

             

            Hence, it would be good if one Jenkins build is run automatically per submit without remaining submitted changelists like "Poll per change".

             

            Hope it makes sense.

            Show
            s3037560 Kevin Park added a comment - - edited It also occurs on Freestyle project. As per the attached screen shot, "Change 12264" submit is not run after "Change 12263" submit has been run. "Change 12264" submit will be run after "Change 12265" is submitted.   $ p4 edit a.txt //test/a.txt#23 - opened for edit $ p4 submit -d "test" Submitting change 12263. Locking 1 files ... edit //test/a.txt#24 Change 12263 submitted.   $ p4 edit a.txt //test/a.txt#24 - opened for edit $ p4 submit -d "test" Submitting change 12264. Locking 1 files ... edit //test/a.txt#25 Change 12264 submitted.   $ p4 edit a.txt //test/a.txt#25 - opened for edit $ p4 submit -d "test" Submitting change 12265. Locking 1 files ... edit //test/a.txt#26 Change 12265 submitted.   When not using "Poll per change", concurrent submitted changelists are run together in one Jenkins build.    Hence, it would be good if one Jenkins build is run automatically per submit without remaining submitted changelists like "Poll per change".   Hope it makes sense.
            Hide
            danielresolutiongames Daniel Hagström added a comment -

            As JENKINS-51525 seems to be moving towards a close, I'm looking forward to hearing updates on this ticket too. These two would allow us to actually do incremental builds with explicit checkout, which would be very nice.

            Show
            danielresolutiongames Daniel Hagström added a comment - As JENKINS-51525  seems to be moving towards a close, I'm looking forward to hearing updates on this ticket too. These two would allow us to actually do incremental builds with explicit checkout, which would be very nice.
            Hide
            p4karl Karl Wirth added a comment -

            Hi Daniel Hagström. No change in behavior.

            Poll per change will cause only the next change to build, but subsequent pipeline scans will only trigger a build for a change submitted after the previous job ?completed?. Therefore you get:

             

            Submit 1,2,3.
            Pipeline scan builds 1.
            Next pipeline scan detects no new changes since 3.
            Submit 4.
            Next pipeline scan detects new changes (4) and builds 2.

            The fix for JENKINS-51525  has not implemented the poll per change against multibranch. I will therefore highlight this to the product managers.

            Show
            p4karl Karl Wirth added a comment - Hi Daniel Hagström . No change in behavior. Poll per change will cause only the next change to build, but subsequent pipeline scans will only trigger a build for a change submitted after the previous job ?completed?. Therefore you get:   Submit 1,2,3. Pipeline scan builds 1. Next pipeline scan detects no new changes since 3. Submit 4. Next pipeline scan detects new changes (4) and builds 2. The fix for JENKINS-51525   has not implemented the poll per change against multibranch. I will therefore highlight this to the product managers.
            Hide
            danielresolutiongames Daniel Hagström added a comment -

            Hi Karl Wirth. Thanks for the update. It's really unfortunate that this isn't working.

            Show
            danielresolutiongames Daniel Hagström added a comment - Hi Karl Wirth . Thanks for the update. It's really unfortunate that this isn't working.
            Hide
            p4paul Paul Allen added a comment -

            1.9.6 adds support for MultiBranch Scan per change for 'Helix Branches' implemented as a Polling Filter in the Advanced options.  If you disable implicit checkout and use 'p4sync' or 'checkout' steps then you need to add the poll per change filter to the step as well as enabling MultiBranch Scan per change. 

            Show
            p4paul Paul Allen added a comment - 1.9.6 adds support for MultiBranch Scan per change for 'Helix Branches' implemented as a Polling Filter in the Advanced options.  If you disable implicit checkout and use 'p4sync' or 'checkout' steps then you need to add the poll per change filter to the step as well as enabling MultiBranch Scan per change. 
            Hide
            p4paul Paul Allen added a comment -

            Released in 1.9.6

            Show
            p4paul Paul Allen added a comment - Released in 1.9.6
            Hide
            danielresolutiongames Daniel Hagström added a comment -

            I'm not seeing this working as intended in 1.9.6, but rather just as broken as before. I have several multibranch pipelines that poll e.g. once per minute, and are still lagging behind on their builds. The scanning log says something like this:

            {{
            ... p4 changes -m1 //project/master/...@now -
            p4 changes -m1 //project/master/...@now

            Change 41254 on 2019/01/22 by bjorn@bjorn_project_workstation 'RES-11336 Update localization w'
            ... p4 files -e //project/master/Jenkinsfile -
            p4 files -e //project/master/Jenkinsfile

            //project/master/Jenkinsfile#3 - edit change 39828 (text)
            ‘Jenkinsfile’ found
            No changes detected: master (still at 41254)
            }}

            Whereas the last build is only at @41189. When manually triggering a build, the build correctly contains the next changelist, in this example @41224. It does not trigger any further builds automatically, but if I keep pressing "Build now" I keep getting builds until I'm caught up again.

            In my eyes this inconsistency is exactly as it was before this bug was considered fixed.

            Show
            danielresolutiongames Daniel Hagström added a comment - I'm not seeing this working as intended in 1.9.6, but rather just as broken as before. I have several multibranch pipelines that poll e.g. once per minute, and are still lagging behind on their builds. The scanning log says something like this: {{ ... p4 changes -m1 //project/master/...@now - p4 changes -m1 //project/master/...@now Change 41254 on 2019/01/22 by bjorn@bjorn_project_workstation 'RES-11336 Update localization w' ... p4 files -e //project/master/Jenkinsfile - p4 files -e //project/master/Jenkinsfile //project/master/Jenkinsfile#3 - edit change 39828 (text) ‘Jenkinsfile’ found No changes detected: master (still at 41254) }} Whereas the last build is only at @41189. When manually triggering a build, the build correctly contains the next changelist, in this example @41224. It does not trigger any further builds automatically, but if I keep pressing "Build now" I keep getting builds until I'm caught up again. In my eyes this inconsistency is exactly as it was before this bug was considered fixed.

              People

              Assignee:
              p4paul Paul Allen
              Reporter:
              p4karl Karl Wirth
              Votes:
              4 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: