Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56286

Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: In Progress (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: p4-plugin
    • Labels:
    • Environment:
      Jenkins: 2.150.3 (official docker image)
      P4 Plugin: 1.9.6
    • Similar Issues:

      Description

      Hello

      We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

      P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

      The problem appears on all streams with different delay in time.

      All streams have the same Jenkinsfile and Jenkins Library.

      All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

      The problem appears after a while, and I am not able to reproduce STR.

      The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library.

      Workaround for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace

      Next actions sometimes trigger the problem:

      • Manual Job start
      • Jenkins restart

      Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop but it re-appears after some time.

       

      Update #1

      I have found one of possible steps to reproduce this bug:

      1. Create //playground/jenkins-main stream

      2. Submit //playground/jenkins-main/library/vars/foo1.groovy

      def call() {
          echo("foo1")
      }
      

       3. Submit //playground/jenkins-main/Jenkinsfile

      pipeline {
          agent(none)
      
          libraries {
              lib("foo")
          }
      
          options {
              buildDiscarder(logRotator(numToKeepStr: "10"))
              disableConcurrentBuilds()
              disableResume()
              quietPeriod(150)
              skipDefaultCheckout(true)
          }
      
          triggers {
              pollSCM("* * * * *")
          }
      
          stages {
              stage("Init") {
                  agent {
                      node {
                          label("master")
                      }
                  }
      
                  steps {
                      checkout scm: perforce(credential: "perforce-ticket-jenkins", populate: flushOnly(pin: "", quiet: false),
                              workspace: streamSpec(charset: "none", format: "test1--${NODE_NAME}", pinHost: true, streamName: "//playground/${BRANCH_NAME}"))
                      echo("hello")
                      foo1()
                      sleep(80)
                  }
              }
          }
      }
      

      4. Run Jenkins container:

      docker run --name=jenkins -p 8080:8080 jenkins/jenkins:2.150.3
      

      5. Click button "Install suggested plugins"

       

      6. Install p4 plugin (1.9.6)

      7. Restart Jenkins

      8. Add Perforce Credentials:
      Jenkins > Credentials > Global > Add Credentials > Perforce Ticker Credential:

      • ID: perforce-ticket-jenkins
      • P4Port: perforce:1666
      • Username: jenkins
      • Ticket: <censored>

      9. Add Jenkins Library:

      • Name: foo
      • Default version: now
      • Load implicitly: False
      • Allow default version to be overridden: False
      • Include Library changes in job recent changes: False
      • Retrieval method: Modern SCM
      • Source Code Management: Helix Library
      • Perforce Credentials: jenkins
      • Library Depot Path: //playground/jenkins-main/library/...

      10. Add Multibranch Pipeline Job:

      • Name: test1
      • Branch Sources: Helix Streams
      • Perforce Credentials: jenkins
      • Include streams: //playground/jenkins-main
      • Click "Save"

      Result: test1/jenkins-main job will be automatically started.

      11. Checkout and submit //playground/jenkins-main/library/vars/foo1.groovy
      as soon as possible while test1/jenkins-main is running.

      Result: infinite build loop.

      Build interval between jobs - 3 minutes.
      Polling starts each minute (* * * * *).

      So polling is executed three times between builds and polling results are quite interesting:

      1. first polling in loop: Changes found => new build is scheduled (after 2 minutes due to quietPeriod)
      2. seconds polling in loop: No changes
      3. third polling in loop: No changes
      4. Job starts (quietPeriod expired)

      And this loop repeats after each build (without any submits to perforce).

      Perforce Software Polling Log (first polling in loop)

      Started on Mar 1, 2019 3:36:00 PM
      P4: Polling on: master with:jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      P4Task: cleanup Client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4 Task: cleanup client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      ... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
      P4 Task: remove client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      ... p4 client -d jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      P4: Polling on: master with:test1--master
      Found last change 1183310 on syncID test1--master
      ... p4 client -o test1--master +
      ... p4 info +
      ... p4 info +
      ... p4 client -o test1--master +
      ... No change in client detected.
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,now
      ... p4 changes -m20 //test1--master/...@1183310,now +
      ... p4 repos -C +
      Done. Took 0.14 sec
      Changes found
      

      Perforce Software Polling Log (second polling in loop)

      Started on Mar 1, 2019 3:37:00 PM
      P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,1183310
      ... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
      ... p4 repos -C +
      P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
      P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      P4: Polling on: master with:test1--master
      Found last change 1183310 on syncID test1--master
      ... p4 client -o test1--master +
      ... p4 info +
      ... p4 info +
      ... p4 client -o test1--master +
      ... No change in client detected.
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,now
      ... p4 changes -m20 //test1--master/...@1183310,now +
      ... p4 repos -C +
      Done. Took 0.19 sec
      No changes
      

      Perforce Software Polling Log (third polling in loop)

      Started on Mar 1, 2019 3:38:00 PM
      P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,1183310
      ... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
      ... p4 repos -C +
      P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
      P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      P4: Polling on: master with:test1--master
      Found last change 1183310 on syncID test1--master
      ... p4 client -o test1--master +
      ... p4 info +
      ... p4 info +
      ... p4 client -o test1--master +
      ... No change in client detected.
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,now
      ... p4 changes -m20 //test1--master/...@1183310,now +
      ... p4 repos -C +
      Done. Took 0.2 sec
      No changes
      

       

      Also I have spent some hours in IDE and found an interesting case:

       

      If the problem occurs,
      on each poll iteration the method TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID) is always executed with the same syncID value
      but lastActions(run) returns the action[0] with the different syncID value on each poll iteration.

      Result: TagAction.getLastChange returns empty list because the following expression is always false:

      for (TagAction action : actions) {
         if (syncID.equals(action.getSyncID())) {
            ...

      Result: PerforceSCM.lookForChanges returns null because lastRefs is empty and a build triggers again:

      List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);
      
      if (lastRefs == null || lastRefs.isEmpty()) {
         // no previous build, return null.
         return null;
      }
      
      

      Please fix the bug.

      Thank you

        Attachments

          Issue Links

            Activity

            Hide
            p4karl Karl Wirth added a comment -

            Confirmed that the steps in the original description do cause infinite polling and do not cause multiple 'Started by an SCM change' however every invocation does start with a single 'Started by an SCM change'. Problem is reproducible with classic Perforce paths also.

            Decreasing the sleep to 10 (sleep(10) ) in the above example and setting 5 second quiet period ( quietPeriod(5) ) stops the infinite builds running. Therefore overlapping polls is a factor and nested SyncIDs may not be relevant.

            Passing to development.

            Show
            p4karl Karl Wirth added a comment - Confirmed that the steps in the original description do cause infinite polling and do not cause multiple 'Started by an SCM change' however every invocation does start with a single 'Started by an SCM change'. Problem is reproducible with classic Perforce paths also. Decreasing the sleep to 10 ( sleep(10) ) in the above example and setting 5 second quiet period ( quietPeriod(5) ) stops the infinite builds running. Therefore overlapping polls is a factor and nested SyncIDs may not be relevant. Passing to development.
            Hide
            p4karl Karl Wirth added a comment -

            FYI - David has highlighted the following changelist that may have had an impact here:

                        https://swarm.workshop.perforce.com/changes/24591

            Thoughts - Does this code also run for libraries? If not does the library code need a similar check?

            Show
            p4karl Karl Wirth added a comment - FYI - David has highlighted the following changelist that may have had an impact here:             https://swarm.workshop.perforce.com/changes/24591 Thoughts - Does this code also run for libraries? If not does the library code need a similar check?
            Hide
            lystor Mykola Ulianytskyi added a comment -

            If not does the library code need a similar check?

            Libraries should have the same workaround because they usually are part of the same stream.

            Show
            lystor Mykola Ulianytskyi added a comment - If not does the library code need a similar check? Libraries should have the same workaround because they usually are part of the same stream.
            Hide
            p4karl Karl Wirth added a comment -

            Hi Mykola Ulianytskyi - Thanks. That is not the normal way we see users using it. Usually the libraries are in a different location from the source code. This is because the libraries tend to be  cross project and streams are often just one part of a project at a certain stage of development (for example game-engine at dev, main, qa, rel).

            However this observation is just based on the support cases I have worked. There may be a large part of the user population that does put Jenkins libraries in the same stream as their source code.

            Show
            p4karl Karl Wirth added a comment - Hi Mykola Ulianytskyi - Thanks. That is not the normal way we see users using it. Usually the libraries are in a different location from the source code. This is because the libraries tend to be  cross project and streams are often just one part of a project at a certain stage of development (for example game-engine at dev, main, qa, rel). However this observation is just based on the support cases I have worked. There may be a large part of the user population that does put Jenkins libraries in the same stream as their source code.
            Hide
            lystor Mykola Ulianytskyi added a comment -

            Usually the libraries are in a different location from the source code.

            1) Libraries in different location are used to share code between projects.

            2) Libraries in the same stream are used to share common code between large pipelines in the same project. Pipelines can be different between streams so libraries belongs to streams for code consistency with pipelines.

            Show
            lystor Mykola Ulianytskyi added a comment - Usually the libraries are in a different location from the source code. 1) Libraries in different location are used to share code between projects. 2) Libraries in the same stream are used to share common code between large pipelines in the same project. Pipelines can be different between streams so libraries belongs to streams for code consistency with pipelines.

              People

              Assignee:
              p4karl Karl Wirth
              Reporter:
              lystor Mykola Ulianytskyi
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated: