Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56286

Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • Jenkins: 2.150.3 (official docker image)
      P4 Plugin: 1.9.6

      Hello

      We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

      P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

      The problem appears on all streams with different delay in time.

      All streams have the same Jenkinsfile and Jenkins Library.

      All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

      The problem appears after a while, and I am not able to reproduce STR.

      The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library.

      Workaround for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace

      Next actions sometimes trigger the problem:

      • Manual Job start
      • Jenkins restart

      Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop but it re-appears after some time.

       

      Update #1

      I have found one of possible steps to reproduce this bug:

      1. Create //playground/jenkins-main stream

      2. Submit //playground/jenkins-main/library/vars/foo1.groovy

      def call() {
    echo("foo1")
}

       3. Submit //playground/jenkins-main/Jenkinsfile

      pipeline {
    agent(none)

    libraries {
        lib("foo")
    }

    options {
        buildDiscarder(logRotator(numToKeepStr: "10"))
        disableConcurrentBuilds()
        disableResume()
        quietPeriod(150)
        skipDefaultCheckout(true)
    }

    triggers {
        pollSCM("* * * * *")
    }

    stages {
        stage("Init") {
            agent {
                node {
                    label("master")
                }
            }

            steps {
                checkout scm: perforce(credential: "perforce-ticket-jenkins", populate: flushOnly(pin: "", quiet: false),
                        workspace: streamSpec(charset: "none", format: "test1--${NODE_NAME}", pinHost: true, streamName: "//playground/${BRANCH_NAME}"))
                echo("hello")
                foo1()
                sleep(80)
            }
        }
    }
}

      4. Run Jenkins container:

      docker run --name=jenkins -p 8080:8080 jenkins/jenkins:2.150.3

      5. Click button "Install suggested plugins"

       

      6. Install p4 plugin (1.9.6)

      7. Restart Jenkins

      8. Add Perforce Credentials:
      Jenkins > Credentials > Global > Add Credentials > Perforce Ticker Credential:

      • ID: perforce-ticket-jenkins
      • P4Port: perforce:1666
      • Username: jenkins
      • Ticket: <censored>

      9. Add Jenkins Library:

      • Name: foo
      • Default version: now
      • Load implicitly: False
      • Allow default version to be overridden: False
      • Include Library changes in job recent changes: False
      • Retrieval method: Modern SCM
      • Source Code Management: Helix Library
      • Perforce Credentials: jenkins
      • Library Depot Path: //playground/jenkins-main/library/...

      10. Add Multibranch Pipeline Job:

      • Name: test1
      • Branch Sources: Helix Streams
      • Perforce Credentials: jenkins
      • Include streams: //playground/jenkins-main
      • Click "Save"

      Result: test1/jenkins-main job will be automatically started.

      11. Checkout and submit //playground/jenkins-main/library/vars/foo1.groovy
      as soon as possible while test1/jenkins-main is running.

      Result: infinite build loop.

      Build interval between jobs - 3 minutes.
      Polling starts each minute (* * * * *).

      So polling is executed three times between builds and polling results are quite interesting:

      1. first polling in loop: Changes found => new build is scheduled (after 2 minutes due to quietPeriod)
      2. seconds polling in loop: No changes
      3. third polling in loop: No changes
      4. Job starts (quietPeriod expired)

      And this loop repeats after each build (without any submits to perforce).

      Perforce Software Polling Log (first polling in loop)

      Started on Mar 1, 2019 3:36:00 PM
P4: Polling on: master with:jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
P4Task: cleanup Client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -d jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.14 sec
Changes found

      Perforce Software Polling Log (second polling in loop)

      Started on Mar 1, 2019 3:37:00 PM
P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,1183310
... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
... p4 repos -C +
P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.19 sec
No changes

      Perforce Software Polling Log (third polling in loop)

      Started on Mar 1, 2019 3:38:00 PM
P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,1183310
... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
... p4 repos -C +
P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.
P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.2 sec
No changes

       

      Also I have spent some hours in IDE and found an interesting case:

       

      If the problem occurs,
      on each poll iteration the method TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID) is always executed with the same syncID value
      but lastActions(run) returns the action[0] with the different syncID value on each poll iteration.

      Result: TagAction.getLastChange returns empty list because the following expression is always false:

      for (TagAction action : actions) {
   if (syncID.equals(action.getSyncID())) {
      ...

      Result: PerforceSCM.lookForChanges returns null because lastRefs is empty and a build triggers again:

      List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

      Please fix the bug.

      Thank you

        1. 01.png
          01.png
          290 kB

          [JENKINS-56286] Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

          Mykola Ulianytskyi created issue -

          Karl Wirth added a comment -

          Hi lystor, I'm going to try and contact you directly via an company email address we have for you.

          Karl Wirth added a comment - Hi lystor , I'm going to try and contact you directly via an company email address we have for you.

          Mykola Ulianytskyi added a comment -

          I have sent my company email and Perforce License information to kwirth@perforce.com

          Thank you

          Mykola Ulianytskyi added a comment - I have sent my company email and Perforce License information to  kwirth@perforce.com Thank you
          Karl Wirth made changes -
          Labels New: P4_SUPPORT
          Karl Wirth made changes -
          Assignee New: Karl Wirth [ p4karl ]
          Mykola Ulianytskyi made changes -
          Summary Original: Multibranch Pipeline + P4Plugin + Polling = Infinite Build Loop New: Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop
          Mykola Ulianytskyi made changes -
          Description Original: Hello

           

          We have problem with Multibranch Pipeline + P4Plugin + Polling.

          P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

          Some streams are affected, some are not.

          All streams have the same Jenkinsfile and Jenkins Library.

          All workspaces in Polling logs have unique syncID.

          Multibranch Pipeline was recreated several times.

           

          I am not able to reproduce STR and create minimal Jenkinsfile.

          The problem appears after after a while, and I don't see the cause of it.

           

          Next actions don't solve the problem:
           * Jenkins restart
           * Job cancellation
           * Manual Job start

          Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop.

           

          Multibranch Pipeline Configuration:
           * Branch Sources > Helix Streams
           * Perforce Credentials: jenkins
           * Script Path: Tools/Jenkins/jobs/build/Jenkinsfile
           * Include streams:

           
          {code:java}
             //k02/main
             //k02/dev-prototype
             //k02/dev-prototype-99
          {code}
           

          *scm-polling.log*
          {code:java}
          Started on Feb 26, 2019 10:29:00 AM
          P4: Polling on: master with:jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          P4Task: cleanup Client: jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          (p4):cmd:... p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579(p4):stop:4
          (p4):cmd:... p4 info
          p4 info(p4):stop:5
          (p4):cmd:... p4 info
          p4 info(p4):stop:6
          (p4):cmd:... p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579(p4):stop:7
          (p4):cmd:... p4 client -i
          p4 client -iClient jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579 saved.(p4):stop:8
          (p4):cmd:... View:
                //k02/dev-prototype/Tools/Jenkins/library/... //jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579/...
          (p4):stop:P4 Task: establishing connection.
          ... server: perforce:1666
          ... node: k02-jenkins
          P4 Task: cleanup client: jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          (p4):cmd:... p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579(p4):stop:9
          (p4):cmd:... p4 revert /var/jenkins_home/jobs/build/branches/dev-prototype/...
          p4 revert /var/jenkins_home/jobs/build/branches/dev-prototype/.../var/jenkins_home/jobs/build/branches/dev-prototype/... - file(s) not opened on this client.(p4):stop:10
          P4 Task: remove client: jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          (p4):cmd:... p4 client -d jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          p4 client -d jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579Client jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579 deleted.(p4):stop:11


          P4: Polling on: master with:k02-dev-prototype--poll--master
          Found last change 1181055 on syncID k02-dev-prototype--poll--master
          (p4):cmd:... p4 client -o k02-dev-prototype--poll--master
          p4 client -o k02-dev-prototype--poll--master(p4):stop:4
          (p4):cmd:... p4 info
          p4 info(p4):stop:5
          (p4):cmd:... p4 info
          p4 info(p4):stop:6
          (p4):cmd:... p4 client -o k02-dev-prototype--poll--master
          p4 client -o k02-dev-prototype--poll--master(p4):stop:7
          (p4):cmd:... p4 client -i
          p4 client -iClient k02-dev-prototype--poll--master not changed.(p4):stop:8
          (p4):cmd:... View:
                null null
          (p4):stop:P4 Task: establishing connection.
          ... server: perforce:1666
          ... node: k02-jenkins
          P4: Polling with range: 1181055,now
          (p4):cmd:... p4 changes -m20 //k02-dev-prototype--poll--master/...@1181055,now
          p4 changes -m20 //k02-dev-prototype--poll--master/...@1181055,nowChange 1181055 on 2019/02/25 by yuriyp@k02-dev-prototype-art--kiepc2056 '!U concept
          '
          (p4):stop:9
          (p4):cmd:... p4 repos -C
          p4 repos -C(p4):stop:10


          P4: Polling on: master with:k02-dev-prototype--build-win64-sync--k02-bld-win-01
          Found last change 1181055 on syncID k02-dev-prototype--build-win64-sync--k02-bld-win-01
          (p4):cmd:... p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01
          p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01(p4):stop:4
          (p4):cmd:... p4 info
          p4 info(p4):stop:5
          (p4):cmd:... p4 info
          p4 info(p4):stop:6
          (p4):cmd:... p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01
          p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01(p4):stop:7
          (p4):cmd:... p4 client -i
          p4 client -iClient k02-dev-prototype--build-win64-sync--k02-bld-win-01 not changed.(p4):stop:8
          (p4):cmd:... View:
                null null
          (p4):stop:P4 Task: establishing connection.
          ... server: perforce:1666
          ... node: k02-jenkins
          P4: Polling with range: 1181055,1181055
          (p4):cmd:... p4 changes -m20 //k02-dev-prototype--build-win64-sync--k02-bld-win-01/...@1181055,1___
          p4 changes -m20 //k02-dev-prototype--build-win64-sync--k02-bld-win-01/...@1181055,1181055Change 1181055 on 2019/02/25 by yuriyp@k02-dev-prototype-art--kiepc2056 '!U concept
          '
          (p4):stop:9
          (p4):cmd:... p4 repos -C
          p4 repos -CClient 'k02-dev-prototype--build-win64-sync--k02-bld-win-01' can only be used from host 'k02-bld-win-01'.(p4):stop:10


          P4: Polling on: master with:k02-dev-prototype--build-linux-sync--k02-bld-win-02
          Found last change 1181055 on syncID k02-dev-prototype--build-linux-sync--k02-bld-win-02
          (p4):cmd:... p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02
          p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02(p4):stop:4
          (p4):cmd:... p4 info
          p4 info(p4):stop:5
          (p4):cmd:... p4 info
          p4 info(p4):stop:6
          (p4):cmd:... p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02
          p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02(p4):stop:7
          (p4):cmd:... p4 client -i
          p4 client -iClient k02-dev-prototype--build-linux-sync--k02-bld-win-02 not changed.(p4):stop:8
          (p4):cmd:... View:
                null null
          (p4):stop:P4 Task: establishing connection.
          ... server: perforce:1666
          ... node: k02-jenkins
          P4: Polling with range: 1181055,1181055
          (p4):cmd:... p4 changes -m20 //k02-dev-prototype--build-linux-sync--k02-bld-win-02/...@1181055,1___
          p4 changes -m20 //k02-dev-prototype--build-linux-sync--k02-bld-win-02/...@1181055,1181055Change 1181055 on 2019/02/25 by yuriyp@k02-dev-prototype-art--kiepc2056 '!U concept
          '
          (p4):stop:9
          (p4):cmd:... p4 repos -C
          p4 repos -CClient 'k02-dev-prototype--build-linux-sync--k02-bld-win-02' can only be used from host 'k02-bld-win-02'.(p4):stop:10
          Done. Took 0.22 sec

          Changes found
          {code}

          config.xml is attached.


          Any help would be really appreciated.

          Thank you           		New: Hello

          We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

          P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

          The problem appears on all streams with different delay in time.

          All streams have the same Jenkinsfile and Jenkins Library.

          All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

          The problem appears after a while, and I am not able to reproduce STR.

           

          *The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library!*

          *Workaround to this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace*

           

          Next actions sometimes trigger the problem:
           * Manual Job start
           * Jenkins restart

          Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop.

          *Library Configuration:*
           * Name: <some name>
           * Default version: now
           * Load implicitly: False
           * Allow default version to be overridden: False
           * Include Library changes in job recent changes: False
           * Retrieval method: Modern SCM
           * Source Code Management: Helix Library
           * Perforce Credentials: jenkins
           * Library Depot Path: //playground/jenkins-main/library/...

          *Multibranch Pipeline Configuration:*
           * Branch Sources > Helix Streams
           * Perforce Credentials: jenkins
           * Script Path: Jenkinsfile
           * Add property:  (Deprecated) Suppress automatic SCM triggering
           * Disable: Discard old items
           * Include streams:

          {code:java}
          //playground/jenkins-main
          {code}
           *//playground/jenkins-main/library/vars/foo1.groovy*
          {code:java}
          def call() {
              echo("foo1")
          }
          {code}
          *//playground/jenkins-main/Jenkinsfile*
          {code:java}
          pipeline {
              agent {
                  node {
                      label("master")
                  }
              }

              libraries {
                  lib("foo")
              }

              options {
                  disableConcurrentBuilds()
                  skipDefaultCheckout(true)
              }

              triggers {
                  pollSCM("* * * * *")
              }

              stages {
                  stage("Init") {
                      steps {
                          checkout scm: perforce(credential: 'perforce-ticket-jenkins', populate: flushOnly(pin: '', quiet: false),
                                  workspace: streamSpec(charset: 'none', format: 'bar--kiepc2006', pinHost: true, streamName: '//playground/jenkins-main'))
                          echo("hello")
                          foo1()
                      }
                  }
              }
          }
          {code}
          *scm-polling.log*
          {code:java}
          P4: Polling on: master with:jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
          ...
          Changes found
          {code}
          Any help would be really appreciated.

          Thank you
          Mykola Ulianytskyi made changes -
          Attachment Original: multibranch-state.xml [ 46180 ]
          Mykola Ulianytskyi made changes -
          Attachment Original: multibranch-config.xml [ 46181 ]
          Mykola Ulianytskyi made changes -
          Attachment Original: config.xml [ 46183 ]

            p4karl Karl Wirth
            lystor Mykola Ulianytskyi
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: