Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56286

Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • Jenkins: 2.150.3 (official docker image)
      P4 Plugin: 1.9.6

      Hello

      We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

      P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

      The problem appears on all streams with different delay in time.

      All streams have the same Jenkinsfile and Jenkins Library.

      All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

      The problem appears after a while, and I am not able to reproduce STR.

      The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library.

      Workaround for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace

      Next actions sometimes trigger the problem:

      • Manual Job start
      • Jenkins restart

      Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop but it re-appears after some time.

       

      Update #1

      I have found one of possible steps to reproduce this bug:

      1. Create //playground/jenkins-main stream

      2. Submit //playground/jenkins-main/library/vars/foo1.groovy

      def call() {
          echo("foo1")
      }
      

       3. Submit //playground/jenkins-main/Jenkinsfile

      pipeline {
          agent(none)
      
          libraries {
              lib("foo")
          }
      
          options {
              buildDiscarder(logRotator(numToKeepStr: "10"))
              disableConcurrentBuilds()
              disableResume()
              quietPeriod(150)
              skipDefaultCheckout(true)
          }
      
          triggers {
              pollSCM("* * * * *")
          }
      
          stages {
              stage("Init") {
                  agent {
                      node {
                          label("master")
                      }
                  }
      
                  steps {
                      checkout scm: perforce(credential: "perforce-ticket-jenkins", populate: flushOnly(pin: "", quiet: false),
                              workspace: streamSpec(charset: "none", format: "test1--${NODE_NAME}", pinHost: true, streamName: "//playground/${BRANCH_NAME}"))
                      echo("hello")
                      foo1()
                      sleep(80)
                  }
              }
          }
      }
      

      4. Run Jenkins container:

      docker run --name=jenkins -p 8080:8080 jenkins/jenkins:2.150.3
      

      5. Click button "Install suggested plugins"

       

      6. Install p4 plugin (1.9.6)

      7. Restart Jenkins

      8. Add Perforce Credentials:
      Jenkins > Credentials > Global > Add Credentials > Perforce Ticker Credential:

      • ID: perforce-ticket-jenkins
      • P4Port: perforce:1666
      • Username: jenkins
      • Ticket: <censored>

      9. Add Jenkins Library:

      • Name: foo
      • Default version: now
      • Load implicitly: False
      • Allow default version to be overridden: False
      • Include Library changes in job recent changes: False
      • Retrieval method: Modern SCM
      • Source Code Management: Helix Library
      • Perforce Credentials: jenkins
      • Library Depot Path: //playground/jenkins-main/library/...

      10. Add Multibranch Pipeline Job:

      • Name: test1
      • Branch Sources: Helix Streams
      • Perforce Credentials: jenkins
      • Include streams: //playground/jenkins-main
      • Click "Save"

      Result: test1/jenkins-main job will be automatically started.

      11. Checkout and submit //playground/jenkins-main/library/vars/foo1.groovy
      as soon as possible while test1/jenkins-main is running.

      Result: infinite build loop.

      Build interval between jobs - 3 minutes.
      Polling starts each minute (* * * * *).

      So polling is executed three times between builds and polling results are quite interesting:

      1. first polling in loop: Changes found => new build is scheduled (after 2 minutes due to quietPeriod)
      2. seconds polling in loop: No changes
      3. third polling in loop: No changes
      4. Job starts (quietPeriod expired)

      And this loop repeats after each build (without any submits to perforce).

      Perforce Software Polling Log (first polling in loop)

      Started on Mar 1, 2019 3:36:00 PM
      P4: Polling on: master with:jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      P4Task: cleanup Client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4 Task: cleanup client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      ... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
      P4 Task: remove client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
      ... p4 client -d jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
      P4: Polling on: master with:test1--master
      Found last change 1183310 on syncID test1--master
      ... p4 client -o test1--master +
      ... p4 info +
      ... p4 info +
      ... p4 client -o test1--master +
      ... No change in client detected.
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,now
      ... p4 changes -m20 //test1--master/...@1183310,now +
      ... p4 repos -C +
      Done. Took 0.14 sec
      Changes found
      

      Perforce Software Polling Log (second polling in loop)

      Started on Mar 1, 2019 3:37:00 PM
      P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,1183310
      ... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
      ... p4 repos -C +
      P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
      P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      P4: Polling on: master with:test1--master
      Found last change 1183310 on syncID test1--master
      ... p4 client -o test1--master +
      ... p4 info +
      ... p4 info +
      ... p4 client -o test1--master +
      ... No change in client detected.
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,now
      ... p4 changes -m20 //test1--master/...@1183310,now +
      ... p4 repos -C +
      Done. Took 0.19 sec
      No changes
      

      Perforce Software Polling Log (third polling in loop)

      Started on Mar 1, 2019 3:38:00 PM
      P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,1183310
      ... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
      ... p4 repos -C +
      P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 client -i +
      ... View: +
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      ... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
      P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
      ... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
      P4: Polling on: master with:test1--master
      Found last change 1183310 on syncID test1--master
      ... p4 client -o test1--master +
      ... p4 info +
      ... p4 info +
      ... p4 client -o test1--master +
      ... No change in client detected.
      P4 Task: establishing connection.
      ... server: perforce:1666
      ... node: fd9e6ec9c042
      P4: Polling with range: 1183310,now
      ... p4 changes -m20 //test1--master/...@1183310,now +
      ... p4 repos -C +
      Done. Took 0.2 sec
      No changes
      

       

      Also I have spent some hours in IDE and found an interesting case:

       

      If the problem occurs,
      on each poll iteration the method TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID) is always executed with the same syncID value
      but lastActions(run) returns the action[0] with the different syncID value on each poll iteration.

      Result: TagAction.getLastChange returns empty list because the following expression is always false:

      for (TagAction action : actions) {
         if (syncID.equals(action.getSyncID())) {
            ...

      Result: PerforceSCM.lookForChanges returns null because lastRefs is empty and a build triggers again:

      List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);
      
      if (lastRefs == null || lastRefs.isEmpty()) {
         // no previous build, return null.
         return null;
      }
      
      

      Please fix the bug.

      Thank you

          [JENKINS-56286] Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

          Karl Wirth added a comment -

          Hi lystor - Again thank you for all your work on this. Is the library being under a branch a normal situation for you or did you just do it force the bug? The problem is that if the library is under the branch it becomes included in two separate SyncIDs. The way we had assumed libraries would be used is that the library is external to the branch in some global location.

          For example:

          //build-dependencies/jenkins-main/library/vars/foo1.groovy
          //playground/jenkins-main/Jenkinsfile
          //playground/jenkins-main/src/f1.c
          //playground/jenkins-main/src/f1.h
          

          Karl Wirth added a comment - Hi lystor - Again thank you for all your work on this. Is the library being under a branch a normal situation for you or did you just do it force the bug? The problem is that if the library is under the branch it becomes included in two separate SyncIDs. The way we had assumed libraries would be used is that the library is external to the branch in some global location. For example: //build-dependencies/jenkins-main/library/vars/foo1.groovy //playground/jenkins-main/Jenkinsfile //playground/jenkins-main/src/f1.c //playground/jenkins-main/src/f1.h

          Hi Karl,

          Is the library being under a branch a normal situation

          The library and all Jenkins Pipelines belong to the same stream. The library contains common functions used by pipelines.

          It is a normal practice and it works perfectly if used Legacy SCM > Perforce Software > Manual Workspace.

          Mykola Ulianytskyi added a comment - Hi Karl, Is the library being under a branch a normal situation The library and all Jenkins Pipelines belong to the same stream. The library contains common functions used by pipelines. It is a normal practice and it works perfectly if used Legacy SCM > Perforce Software > Manual Workspace.

          Dave Nichols added a comment -

          Hi Karl,

          our jenkins files are in the pipeline job in jenkins as text for most of our jobs.  Our pipeline code is in a different depot in Perforce from the code it builds.  Does this make sense?

          Thanks,

          Dave

          Dave Nichols added a comment - Hi Karl, our jenkins files are in the pipeline job in jenkins as text for most of our jobs.  Our pipeline code is in a different depot in Perforce from the code it builds.  Does this make sense? Thanks, Dave

          Karl Wirth added a comment -

          Hi lystor - I have asked David to look for this pattern already, but can you please check if your builds go crazy after a delay in getting an executor? One pattern I have seen in David's case is that at the top of the console output there is more than one 'Started by an SCM change'. For example:

          Started by an SCM change
          Started by an SCM change 
          Started by an SCM change
          

          This seems to coincide with multiple polls occurring and finding the same change that needs to be built but being unable to run the build. For example:

           Poll 1 - Change 123 found - No executor available
           Poll 2 - Same change 123 found - No executor available
           Poll 3 - Same change 123 found - No executor available
           Poll 4 - Same change 123 found - No executor available
           Poll 5 - Executor becomes available - Runs build triggered by Poll 1
           Poll 6 - Executor available - Runs build triggered by Poll 2
           Poll 7 - Executor available - Runs build triggered by Poll 3
           Poll 8 - Does nothing (poll 4 has not triggered a build).
          

          Do you see a similar pattern?

          Karl Wirth added a comment - Hi lystor - I have asked David to look for this pattern already, but can you please check if your builds go crazy after a delay in getting an executor? One pattern I have seen in David's case is that at the top of the console output there is more than one 'Started by an SCM change'. For example: Started by an SCM change Started by an SCM change Started by an SCM change This seems to coincide with multiple polls occurring and finding the same change that needs to be built but being unable to run the build. For example: Poll 1 - Change 123 found - No executor available Poll 2 - Same change 123 found - No executor available Poll 3 - Same change 123 found - No executor available Poll 4 - Same change 123 found - No executor available Poll 5 - Executor becomes available - Runs build triggered by Poll 1 Poll 6 - Executor available - Runs build triggered by Poll 2 Poll 7 - Executor available - Runs build triggered by Poll 3 Poll 8 - Does nothing (poll 4 has not triggered a build). Do you see a similar pattern?

          Do you see a similar pattern?

          No. I have a different one.

          You can repeat steps to reproduce from description of this bug and get required logs.

          Mykola Ulianytskyi added a comment - Do you see a similar pattern? No. I have a different one. You can repeat steps to reproduce from description of this bug and get required logs.

          Karl Wirth added a comment -

          Confirmed that the steps in the original description do cause infinite polling and do not cause multiple 'Started by an SCM change' however every invocation does start with a single 'Started by an SCM change'. Problem is reproducible with classic Perforce paths also.

          Decreasing the sleep to 10 (sleep(10) ) in the above example and setting 5 second quiet period ( quietPeriod(5) ) stops the infinite builds running. Therefore overlapping polls is a factor and nested SyncIDs may not be relevant.

          Passing to development.

          Karl Wirth added a comment - Confirmed that the steps in the original description do cause infinite polling and do not cause multiple 'Started by an SCM change' however every invocation does start with a single 'Started by an SCM change'. Problem is reproducible with classic Perforce paths also. Decreasing the sleep to 10 ( sleep(10) ) in the above example and setting 5 second quiet period ( quietPeriod(5) ) stops the infinite builds running. Therefore overlapping polls is a factor and nested SyncIDs may not be relevant. Passing to development.

          Karl Wirth added a comment -

          FYI - David has highlighted the following changelist that may have had an impact here:

                      https://swarm.workshop.perforce.com/changes/24591

          Thoughts - Does this code also run for libraries? If not does the library code need a similar check?

          Karl Wirth added a comment - FYI - David has highlighted the following changelist that may have had an impact here:             https://swarm.workshop.perforce.com/changes/24591 Thoughts - Does this code also run for libraries? If not does the library code need a similar check?

          If not does the library code need a similar check?

          Libraries should have the same workaround because they usually are part of the same stream.

          Mykola Ulianytskyi added a comment - If not does the library code need a similar check? Libraries should have the same workaround because they usually are part of the same stream.

          Karl Wirth added a comment -

          Hi lystor - Thanks. That is not the normal way we see users using it. Usually the libraries are in a different location from the source code. This is because the libraries tend to be  cross project and streams are often just one part of a project at a certain stage of development (for example game-engine at dev, main, qa, rel).

          However this observation is just based on the support cases I have worked. There may be a large part of the user population that does put Jenkins libraries in the same stream as their source code.

          Karl Wirth added a comment - Hi lystor - Thanks. That is not the normal way we see users using it. Usually the libraries are in a different location from the source code. This is because the libraries tend to be  cross project and streams are often just one part of a project at a certain stage of development (for example game-engine at dev, main, qa, rel). However this observation is just based on the support cases I have worked. There may be a large part of the user population that does put Jenkins libraries in the same stream as their source code.

          Usually the libraries are in a different location from the source code.

          1) Libraries in different location are used to share code between projects.

          2) Libraries in the same stream are used to share common code between large pipelines in the same project. Pipelines can be different between streams so libraries belongs to streams for code consistency with pipelines.

          Mykola Ulianytskyi added a comment - Usually the libraries are in a different location from the source code. 1) Libraries in different location are used to share code between projects. 2) Libraries in the same stream are used to share common code between large pipelines in the same project. Pipelines can be different between streams so libraries belongs to streams for code consistency with pipelines.

            p4karl Karl Wirth
            lystor Mykola Ulianytskyi
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: