Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56286

Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • Jenkins: 2.150.3 (official docker image)
      P4 Plugin: 1.9.6

      Hello

      We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

      P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

      The problem appears on all streams with different delay in time.

      All streams have the same Jenkinsfile and Jenkins Library.

      All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

      The problem appears after a while, and I am not able to reproduce STR.

      The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library.

      Workaround for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace

      Next actions sometimes trigger the problem:

      • Manual Job start
      • Jenkins restart

      Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop but it re-appears after some time.

       

      Update #1

      I have found one of possible steps to reproduce this bug:

      1. Create //playground/jenkins-main stream

      2. Submit //playground/jenkins-main/library/vars/foo1.groovy

      def call() {
    echo("foo1")
}

       3. Submit //playground/jenkins-main/Jenkinsfile

      pipeline {
    agent(none)

    libraries {
        lib("foo")
    }

    options {
        buildDiscarder(logRotator(numToKeepStr: "10"))
        disableConcurrentBuilds()
        disableResume()
        quietPeriod(150)
        skipDefaultCheckout(true)
    }

    triggers {
        pollSCM("* * * * *")
    }

    stages {
        stage("Init") {
            agent {
                node {
                    label("master")
                }
            }

            steps {
                checkout scm: perforce(credential: "perforce-ticket-jenkins", populate: flushOnly(pin: "", quiet: false),
                        workspace: streamSpec(charset: "none", format: "test1--${NODE_NAME}", pinHost: true, streamName: "//playground/${BRANCH_NAME}"))
                echo("hello")
                foo1()
                sleep(80)
            }
        }
    }
}

      4. Run Jenkins container:

      docker run --name=jenkins -p 8080:8080 jenkins/jenkins:2.150.3

      5. Click button "Install suggested plugins"

       

      6. Install p4 plugin (1.9.6)

      7. Restart Jenkins

      8. Add Perforce Credentials:
      Jenkins > Credentials > Global > Add Credentials > Perforce Ticker Credential:

      • ID: perforce-ticket-jenkins
      • P4Port: perforce:1666
      • Username: jenkins
      • Ticket: <censored>

      9. Add Jenkins Library:

      • Name: foo
      • Default version: now
      • Load implicitly: False
      • Allow default version to be overridden: False
      • Include Library changes in job recent changes: False
      • Retrieval method: Modern SCM
      • Source Code Management: Helix Library
      • Perforce Credentials: jenkins
      • Library Depot Path: //playground/jenkins-main/library/...

      10. Add Multibranch Pipeline Job:

      • Name: test1
      • Branch Sources: Helix Streams
      • Perforce Credentials: jenkins
      • Include streams: //playground/jenkins-main
      • Click "Save"

      Result: test1/jenkins-main job will be automatically started.

      11. Checkout and submit //playground/jenkins-main/library/vars/foo1.groovy
      as soon as possible while test1/jenkins-main is running.

      Result: infinite build loop.

      Build interval between jobs - 3 minutes.
      Polling starts each minute (* * * * *).

      So polling is executed three times between builds and polling results are quite interesting:

      1. first polling in loop: Changes found => new build is scheduled (after 2 minutes due to quietPeriod)
      2. seconds polling in loop: No changes
      3. third polling in loop: No changes
      4. Job starts (quietPeriod expired)

      And this loop repeats after each build (without any submits to perforce).

      Perforce Software Polling Log (first polling in loop)

      Started on Mar 1, 2019 3:36:00 PM
P4: Polling on: master with:jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
P4Task: cleanup Client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -d jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.14 sec
Changes found

      Perforce Software Polling Log (second polling in loop)

      Started on Mar 1, 2019 3:37:00 PM
P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,1183310
... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
... p4 repos -C +
P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.19 sec
No changes

      Perforce Software Polling Log (third polling in loop)

      Started on Mar 1, 2019 3:38:00 PM
P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,1183310
... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
... p4 repos -C +
P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.2 sec
No changes

       

      Also I have spent some hours in IDE and found an interesting case:

       

      If the problem occurs,
      on each poll iteration the method TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID) is always executed with the same syncID value
      but lastActions(run) returns the action[0] with the different syncID value on each poll iteration.

      Result: TagAction.getLastChange returns empty list because the following expression is always false:

      for (TagAction action : actions) {
   if (syncID.equals(action.getSyncID())) {
      ...

      Result: PerforceSCM.lookForChanges returns null because lastRefs is empty and a build triggers again:

      List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

      Please fix the bug.

      Thank you

        1. 01.png
          01.png
          290 kB

            p4karl Karl Wirth
            lystor Mykola Ulianytskyi
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: