Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60213

P4_CLIENT environment variable non-reentrant

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Version 1.10.6 of the p4-plugin causes issues when multiple jobs are ran in parallel.

      The result is that the p4 created environment variables of one job can corrupt those in another job that is running at the same time.

      This bug wasn't in 1.10.5

      Steps to reproduce issue:

      • Create a new Jenkins test job with the following settings
        • General:
          • Execute concurrent builds if necessary
          • Use custom workspace
            • Directory
              /tmp/${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
        • Source Code Management
          • Perforce Software:
            • Workspace behaviour: Template (view generated for each node)
              • Template workspace: Your test template
              • Workspace Name Format being
                 jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
      • Add a build step

      To see the issue:

      • Kick off two runs of the newly creates client

      Result:

      • Pre 1.10.5 both would have worked and both will set the expected P4 environment variables.
      • 1.10.6 gets confused. So in my case:

        Job1 Environment variables

        P4_CLIENT	_jenkins-master-ClientClashTest-0
        WORKSPACE	/tmp/master-ClientClashTest-0
        

        Job2 Environment variables (THIS IS THE ISSUE – Wrong P4_CLIENT value)

        P4_CLIENT	_jenkins-master-ClientClashTest-0
        WORKSPACE	/tmp/master-ClientClashTest-1
        

        If I reinstall the 1.10.5 (or earlier) plugin then for job 2 I would see:

        P4_CLIENT	_jenkins-master-ClientClashTest-1
        WORKSPACE	/tmp/master-ClientClashTest-1
        

        Attachments

          Issue Links

            Activity

            Hide
            p4karl Karl Wirth added a comment -

            Hi James Bateman - Thanks for letting us know.

            This looks like a freestyle job. Is that correct? Please also attach the code you are using to display the variables.

            Thanks in advance,

            Karl

            Show
            p4karl Karl Wirth added a comment - Hi James Bateman - Thanks for letting us know. This looks like a freestyle job. Is that correct? Please also attach the code you are using to display the variables. Thanks in advance, Karl
            Hide
            jbateman James Bateman added a comment - - edited

            More info.

            • We are running Jenkins 2.190.2 on our test Jenkins server
            • We have just upgraded all plugins to the latest versions
            • We have tested every verison of the P4 plugin from 1.9.3 --> 1.10.6 (Only 1.10.6 has this particular issue)
            • To see this particular bug, kick off two jobs so they are running in parallel
            • Console output of the two jobs (in this case #32 + #33) – You will see that the 'P4_CLIENT' environment variable for job #32 is wrong
            • Job configuration

            NOTE: If you have the InjectedEnvVars plugin installed then you can also inspect the environment variables of a previously ran job and again this will show the same issue. To be able to see the P4 variables you will need to add a 'Inject environment variables' build step (it can be blank). Looking at the environment variables both P4_CLIENT and P4_ROOT can become corrupted.

            Show
            jbateman James Bateman added a comment - - edited More info. We are running Jenkins 2.190.2 on our test Jenkins server We have just upgraded all plugins to the latest versions We have tested every verison of the P4 plugin from 1.9.3 --> 1.10.6 (Only 1.10.6 has this particular issue) To see this particular bug, kick off two jobs so they are running in parallel Console output of the two jobs (in this case #32 + #33) – You will see that the 'P4_CLIENT' environment variable for job #32 is wrong Job configuration NOTE: If you have the InjectedEnvVars plugin installed then you can also inspect the environment variables of a previously ran job and again this will show the same issue. To be able to see the P4 variables you will need to add a ' Inject environment variables ' build step (it can be blank). Looking at the environment variables both P4_CLIENT  and P4_ROOT can become corrupted.
            Hide
            p4karl Karl Wirth added a comment -

            Hi James Bateman - That's great thanks. I'll test here and get back to you.

            Regards,

            Karl

            Show
            p4karl Karl Wirth added a comment - Hi James Bateman - That's great thanks. I'll test here and get back to you. Regards, Karl
            Hide
            p4karl Karl Wirth added a comment -

            Hi James Bateman - Thanks again. I was easily able to reproduce this by spamming the 'Build Now' button 3 times on 1.10.6 and the following script:

            #!/bin/bash
            echo "=========================================="
            echo "WORKSPACE:            ${WORKSPACE}"
            echo "P4_CLIENT:            ${P4_CLIENT}"
            echo "EXECUTOR NUMBER:      ${EXECUTOR_NUMBER}"
            echo "=========================================="
            sleep 30
            

            1.10.6 behavior uses same workspace:

             

            [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins7949726486877689022.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-3
            EXECUTOR NUMBER:     2
            ==========================================
            

             

             

             

            [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins8997855946897277031.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-3
            EXECUTOR NUMBER:     6
            ==========================================
            

             

             

            On 1.10.4 I see different workspaces:

            Build 17:
            [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins5428347631299328773.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-3
            EXECUTOR NUMBER:     3
            ==========================================
            
            Build 18:
            [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins7094128855131134804.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-2
            EXECUTOR NUMBER:     2
            ==========================================
            
            Build 19:
            [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins387246113188966258.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-0
            EXECUTOR NUMBER:     0
            ==========================================
            

            On 1.10.6 I see different workspaces sometimes.

             

            Test 1 works as expected:

            Build 23:
            [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6724175724105556673.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-2
            EXECUTOR NUMBER:     2
            ==========================================
            
            Build 24:[JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins2594984598801278037.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-9
            EXECUTOR NUMBER:     9
            ==========================================
            
            Build 25:[JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins7864300692310335405.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-7
            EXECUTOR NUMBER:     7
            ==========================================
            

            Test 2 uses the same P4_CLIENT on 2 out of 3 concurrent executors:

            Build 26:
            [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins7858524505176489757.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-6
            EXECUTOR NUMBER:     8
            ==========================================
            
            Build 27:[JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins2825147022357472107.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-6
            EXECUTOR NUMBER:     6
            ==========================================
            
            Build 28:[JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins8052525907828215068.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-4
            EXECUTOR NUMBER:     4
            ==========================================

            IMPORTANT NOTE: The P4CLIENT used in the job for build 8 is 'jenkins-master-JENKINS-60213-ConcurrentVariables-8'. Therefore it correctly uses a different client but sets the env variable to the wrong value.

             

            Passing to developers.

             

             

             

            Show
            p4karl Karl Wirth added a comment - Hi James Bateman - Thanks again. I was easily able to reproduce this by spamming the 'Build Now' button 3 times on 1.10.6 and the following script: #!/bin/bash echo "==========================================" echo "WORKSPACE: ${WORKSPACE}" echo "P4_CLIENT: ${P4_CLIENT}" echo "EXECUTOR NUMBER: ${EXECUTOR_NUMBER}" echo "==========================================" sleep 30 1.10.6 behavior uses same workspace:   [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins7949726486877689022.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-3 EXECUTOR NUMBER: 2 ==========================================       [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins8997855946897277031.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-3 EXECUTOR NUMBER: 6 ==========================================     On 1.10.4 I see different workspaces: Build 17: [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins5428347631299328773.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-3 EXECUTOR NUMBER: 3 ========================================== Build 18: [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins7094128855131134804.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-2 EXECUTOR NUMBER: 2 ========================================== Build 19: [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins387246113188966258.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-0 EXECUTOR NUMBER: 0 ========================================== On 1.10.6 I see different workspaces sometimes.   Test 1 works as expected: Build 23: [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6724175724105556673.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-2 EXECUTOR NUMBER: 2 ========================================== Build 24:[JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins2594984598801278037.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-9 EXECUTOR NUMBER: 9 ========================================== Build 25:[JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins7864300692310335405.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-7 EXECUTOR NUMBER: 7 ========================================== Test 2 uses the same P4_CLIENT on 2 out of 3 concurrent executors: Build 26: [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins7858524505176489757.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-6 EXECUTOR NUMBER: 8 ========================================== Build 27:[JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins2825147022357472107.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-6 EXECUTOR NUMBER: 6 ========================================== Build 28:[JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins8052525907828215068.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-4 EXECUTOR NUMBER: 4 ========================================== IMPORTANT NOTE: The P4CLIENT used in the job for build 8 is 'jenkins-master- JENKINS-60213 -ConcurrentVariables-8'. Therefore it correctly uses a different client but sets the env variable to the wrong value.   Passing to developers.      
            Hide
            p4karl Karl Wirth added a comment - - edited

            After discussing with the developers I tried the nightly with a different variable fix but same behavior:

            See below for update.

             

            #30 [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-masterJENKINS-60213ConcurrentVariables-0 EXECUTOR NUMBER: 0 ========================================== #31 [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-masterJENKINS-60213ConcurrentVariables-0 EXECUTOR NUMBER: 0 ========================================== #32 [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-masterJENKINS-60213ConcurrentVariables-0 EXECUTOR NUMBER: 0 ==========================================

            Show
            p4karl Karl Wirth added a comment - - edited After discussing with the developers I tried the nightly with a different variable fix but same behavior: See below for update.   #30 [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: /var/lib/jenkins/workspace/ JENKINS-60213 -ConcurrentVariables P4_CLIENT: jenkins-master JENKINS-60213 ConcurrentVariables-0 EXECUTOR NUMBER: 0 ========================================== #31 [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: /var/lib/jenkins/workspace/ JENKINS-60213 -ConcurrentVariables P4_CLIENT: jenkins-master JENKINS-60213 ConcurrentVariables-0 EXECUTOR NUMBER: 0 ========================================== #32 [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: /var/lib/jenkins/workspace/ JENKINS-60213 -ConcurrentVariables P4_CLIENT: jenkins-master JENKINS-60213 ConcurrentVariables-0 EXECUTOR NUMBER: 0 ==========================================
            Hide
            jbateman James Bateman added a comment - - edited

            Looks like your copy-and-paste went wrong, all the example in your last message are exactly the same  (basically there is no way that each job can use the same bash script – i.e. /tmp/jenkins6285754148517933441.sh)

            Show
            jbateman James Bateman added a comment - - edited Looks like your copy-and-paste went wrong, all the example in your last message are exactly the same  (basically there is no way that each job can use the same bash script – i.e.  /tmp/jenkins6285754148517933441.sh )
            Hide
            p4karl Karl Wirth added a comment -

             

            Hi James Bateman - Urgh. Sorry. You are correct. My fault for manually editing the URL.

            30:
            [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-0
            EXECUTOR NUMBER:     0
            ==========================================31:
            [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins1481999870837323074.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-6
            EXECUTOR NUMBER:     6
            ==========================================32:
            [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins3645232859628494475.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-3
            EXECUTOR NUMBER:     3
            ==========================================
            

             

             

            Are you able to test the nightly at the following location to prove that it fixes the bug?

                https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fp4-plugin/detail/PR-93/38/artifacts

            Show
            p4karl Karl Wirth added a comment -   Hi James Bateman - Urgh. Sorry. You are correct. My fault for manually editing the URL. 30: [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-0 EXECUTOR NUMBER: 0 ==========================================31: [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins1481999870837323074.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-6 EXECUTOR NUMBER: 6 ==========================================32: [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins3645232859628494475.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-3 EXECUTOR NUMBER: 3 ==========================================     Are you able to test the nightly at the following location to prove that it fixes the bug?     https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fp4-plugin/detail/PR-93/38/artifacts
            Hide
            jbateman James Bateman added a comment -

            It is still broken

            I have installed the groovy plugin and before my bash script ran this script

            • import jenkins.model.*
              import hudson.PluginWrapper
              
              println '-' * 80
              println 'Jenkins Version = ' + Jenkins.instance.getVersion()
              
              PluginWrapper p4 = Jenkins.instance.pluginManager.getPlugin('p4')
              if(p4) {
                  println p4.getDisplayName() + ':'
                  println '    | Short name = ' + p4.getShortName()
                  println '    | Version = ' + p4.getVersion()
                  println '    | Plugin URL = ' + p4.getUrl()
                  println '    | Is actived = ' + (p4.isActive() ? 'Yes' : 'No')
                  println '    | Is enabled = ' + (p4.isEnabled() ? 'Yes' : 'No')
              }
              
              println '-' * 80
              

            The output from this script confirms that I'm using your patched version of the p4 plugin, the output being:

            • --------------------------------------------------------------------------------
              Jenkins Version = 2.190.2
              P4 Plugin:
                  | Short name = p4
                  | Version = 1.10.7-SNAPSHOT (private-a1cfcc02-jenkins)
                  | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
                  | Is actived = Yes
                  | Is enabled = Yes
              --------------------------------------------------------------------------------
              

            Kicking off two jobs one after another (i.e. clicking on 'Build Now' twice in short session), I get this output

            1st job:

            • [master-ClientClashTest-0] $ /bin/bash /tmp/jenkins1556703573399779365.sh
              =================================
              WORKSPACE:       /tmp/master-ClientClashTest-0
              P4_CLIENT:       jenkins-master-ClientClashTest-1
              EXECUTOR NUMBER: 0
              =================================
              
              • This job has been corrupted since the P4_CLIENT name should be:

                • jenkins-master-ClientClashTest-0

                and not

                • jenkins-master-ClientClashTest-1

            2nd job:

            • [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins10551339048100737039.sh
              =================================
              WORKSPACE:       /tmp/master-ClientClashTest-1
              P4_CLIENT:       jenkins-master-ClientClashTest-1
              EXECUTOR NUMBER: 1
              =================================
              
              • This job run is OK

            PS. If you wanted the bash build step to fail in cases where the items do not match up correctly (which makes it easier to find problematic runs), then you could change the bash script in your test job to look like this:

            • #!/bin/bash
              
              echo "================================="
              echo "WORKSPACE:       ${WORKSPACE}"
              echo "P4_CLIENT:       ${P4_CLIENT}"
              echo "EXECUTOR NUMBER: ${EXECUTOR_NUMBER}"
              echo "================================="
              
              EXITCODE=0
              if [ "${WORKSPACE##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad Workspace"; EXITCODE=1; fi
              if [ "${P4_CLIENT##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad P4 Client"; EXITCODE=1; fi
              
              exit $EXITCODE
              
              • Note: The above script assumes your 'Workspace Name Format' mapping looks like this:

                • jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
                  

                and your 'custom workspace' / 'Directory name' looks something like this:

                • /tmp/${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
                  
            Show
            jbateman James Bateman added a comment - It is still broken I have installed the groovy plugin and before my bash script ran this script import jenkins.model.* import hudson.PluginWrapper println '-' * 80 println 'Jenkins Version = ' + Jenkins.instance.getVersion() PluginWrapper p4 = Jenkins.instance.pluginManager.getPlugin( 'p4' ) if (p4) { println p4.getDisplayName() + ':' println ' | Short name = ' + p4.getShortName() println ' | Version = ' + p4.getVersion() println ' | Plugin URL = ' + p4.getUrl() println ' | Is actived = ' + (p4.isActive() ? 'Yes' : 'No' ) println ' | Is enabled = ' + (p4.isEnabled() ? 'Yes' : 'No' ) } println '-' * 80 The output from this script confirms that I'm using your patched version of the p4 plugin, the output being: -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7-SNAPSHOT ( private -a1cfcc02-jenkins) | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- Kicking off two jobs one after another (i.e. clicking on 'Build Now' twice in short session), I get this output 1st job: [master-ClientClashTest-0] $ /bin/bash /tmp/jenkins1556703573399779365.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-0 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 0 ================================= This job has been corrupted since the P4_CLIENT name should be: jenkins-master-ClientClashTest- 0 and not jenkins-master-ClientClashTest- 1 2nd job: [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins10551339048100737039.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-1 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 1 ================================= This job run is OK PS. If you wanted the bash build step to fail in cases where the items do not match up correctly (which makes it easier to find problematic runs) , then you could change the bash script in your test job to look like this: #!/bin/bash echo "=================================" echo "WORKSPACE: ${WORKSPACE}" echo "P4_CLIENT: ${P4_CLIENT}" echo "EXECUTOR NUMBER: ${EXECUTOR_NUMBER}" echo "=================================" EXITCODE=0 if [ "${WORKSPACE##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad Workspace" ; EXITCODE=1; fi if [ "${P4_CLIENT##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad P4 Client" ; EXITCODE=1; fi exit $EXITCODE Note: The above script assumes your ' Workspace Name Format ' mapping looks like this: jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER} and your ' custom workspace ' / ' Directory name' looks something like this: /tmp/${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
            Hide
            p4karl Karl Wirth added a comment -

            Hi James Bateman - I need to apologise. The nightly I gave you did not contain the fix. Are you able to test 1.10.7 that we shipped yesterday that does contain the correct bug fix?

            Show
            p4karl Karl Wirth added a comment - Hi James Bateman - I need to apologise. The nightly I gave you did not contain the fix. Are you able to test 1.10.7 that we shipped yesterday that does contain the correct bug fix?
            Hide
            jbateman James Bateman added a comment -

            Plugin 1.10.7 still has the same issues

            Job #78: (BAD)

            • [EnvInject] - Injecting environment variables from a build step.
              --------------------------------------------------------------------------------
              Jenkins Version = 2.190.2
              P4 Plugin:
                  | Short name = p4
                  | Version = 1.10.7
                  | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
                  | Is actived = Yes
                  | Is enabled = Yes
              --------------------------------------------------------------------------------
              [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins822701625673778739.sh
              =================================
              WORKSPACE:       /tmp/master-ClientClashTest-1
              P4_CLIENT:       jenkins-master-ClientClashTest-0
              EXECUTOR NUMBER: 1
              =================================
              *** ERROR - Bad P4 Client
              

            Job #79: (BAD)

            • [EnvInject] - Injecting environment variables from a build step.
              --------------------------------------------------------------------------------
              Jenkins Version = 2.190.2
              P4 Plugin:
                  | Short name = p4
                  | Version = 1.10.7
                  | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
                  | Is actived = Yes
                  | Is enabled = Yes
              --------------------------------------------------------------------------------
              [master-ClientClashTest-0] $ /bin/bash /tmp/jenkins825491192926150409.sh
              =================================
              WORKSPACE:       /tmp/master-ClientClashTest-0
              P4_CLIENT:       jenkins-master-ClientClashTest-1
              EXECUTOR NUMBER: 0
              =================================
              *** ERROR - Bad P4 Client
              

            Job #80: (GOOD)

            • [EnvInject] - Injecting environment variables from a build step.
              --------------------------------------------------------------------------------
              Jenkins Version = 2.190.2
              P4 Plugin:
                  | Short name = p4
                  | Version = 1.10.7
                  | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
                  | Is actived = Yes
                  | Is enabled = Yes
              --------------------------------------------------------------------------------
              [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins14602602394965857665.sh
              =================================
              WORKSPACE:       /tmp/master-ClientClashTest-1
              P4_CLIENT:       jenkins-master-ClientClashTest-1
              EXECUTOR NUMBER: 1
              =================================
              Finished: SUCCESS
              

            Summary:

            • So 2 of 3 of the above jobs on a server running the latest plug (1.10.7) ended up with corrupt P4_CLIENT environment variables. If I downgrade the plugin to version 1.10.5 then 100% of the jobs I kick off have the correct P4_CLIENT variables.
            Show
            jbateman James Bateman added a comment - Plugin 1.10.7 still has the same issues Job #78: (BAD) [EnvInject] - Injecting environment variables from a build step. -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7 | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins822701625673778739.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-1 P4_CLIENT: jenkins-master-ClientClashTest-0 EXECUTOR NUMBER: 1 ================================= *** ERROR - Bad P4 Client Job #79: (BAD) [EnvInject] - Injecting environment variables from a build step. -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7 | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- [master-ClientClashTest-0] $ /bin/bash /tmp/jenkins825491192926150409.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-0 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 0 ================================= *** ERROR - Bad P4 Client Job #80: (GOOD) [EnvInject] - Injecting environment variables from a build step. -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7 | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins14602602394965857665.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-1 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 1 ================================= Finished: SUCCESS Summary: So 2 of 3 of the above jobs on a server running the latest plug (1.10.7) ended up with corrupt P4_CLIENT environment variables. If I downgrade the plugin to version 1.10.5 then 100% of the jobs I kick off have the correct P4_CLIENT variables.
            Hide
            p4karl Karl Wirth added a comment -

            Hi James Bateman - Thanks for letting is know. I'll go back through the test harness to see if there's anything we have assumed that could be the difference.

            Show
            p4karl Karl Wirth added a comment - Hi James Bateman - Thanks for letting is know. I'll go back through the test harness to see if there's anything we have assumed that could be the difference.
            Hide
            p4karl Karl Wirth added a comment -

            Verified still sporadically broken in 1.10.7. P4_CLIENT is wrong. The full expansion for me is correct and matches the rest of the job:

            P4: saving built changes.
            Found last change 2240 on syncID jenkins-NODE_NAME-JENKINS-60213-ConcurrentVariables-EXECUTOR_NUMBER
            ... p4 login -s 
            ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
            ... p4 info 
            ... p4 info 
            ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
            ... p4 login -s 
            ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
            ... p4 info 
            ... p4 info 
            ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
            ... done
            
            [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins6561639748343680378.sh
            ==========================================
            WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
            P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-4
            EXECUTOR NUMBER:      3
            CLIENT_FROM_VARS:     jenkins-master-JENKINS-60213-ConcurrentVariables-3
            ==========================================
            Finished: SUCCESS
            

            Note for dev: In the above ${P4_CLIENT} is wrong but expansion is correct.

             

             

             

            Show
            p4karl Karl Wirth added a comment - Verified still sporadically broken in 1.10.7. P4_CLIENT is wrong. The full expansion for me is correct and matches the rest of the job: P4: saving built changes. Found last change 2240 on syncID jenkins-NODE_NAME-JENKINS-60213-ConcurrentVariables-EXECUTOR_NUMBER ... p4 login -s ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... p4 info ... p4 info ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... p4 login -s ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... p4 info ... p4 info ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... done [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins6561639748343680378.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-4 EXECUTOR NUMBER: 3 CLIENT_FROM_VARS: jenkins-master-JENKINS-60213-ConcurrentVariables-3 ========================================== Finished: SUCCESS Note for dev: In the above ${P4_CLIENT} is wrong but expansion is correct.      
            Hide
            msmeeth Matthew Smeeth added a comment - - edited

            Hi, I've written an automated test to reproduce this issue, Looking into it further, it looks to me like the P4_CLIENT variable is actually correct, and it's actually the EXECUTOR NUMBER that is wrong in your script.

             

            I can demonstrate this by putting an extra bit of debugging into the p4 plugin when running jobs. I have modified my build so that it outputs the executor number as the job is executing and this is what I get:
             

            Obtained concurrentBuildsTest/jenkinsfile from p4-brunoCred-//depot/... //jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}/...
            Running in Durability level: MAX_SURVIVABILITY
            [Pipeline] Start of Pipeline
            [Pipeline] node
            Running on Jenkins in /var/jenkins_home/workspace/concurrentBuildsTestProject
            [Pipeline] {
            [Pipeline] stage
            [Pipeline] { (testStage)
            [Pipeline] script
            [Pipeline] {
            [Pipeline] dir
            Running in /var/jenkins_home/workspace/concurrentBuildsTestProject/test
            [Pipeline] {
            [Pipeline] checkout
            Executor no at runtime:0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Extra debugging I've added to the p4 plugin
             ......
             ......
             ......
             + concurrentBuildsTest/script.sh
             ================================= 
             WORKSPACE: /var/jenkins_home/workspace/concurrentBuildsTestProject <<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT
             P4_CLIENT: jenkins-master-concurrentBuildsTestProject-0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT
             EXECUTOR NUMBER: 1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Wrong executor number
            =================================
            ERROR - Bad Workspace 
            ERROR - Bad P4 Client
            ......
            ......
            Finished: FAILURE
            

             

            I have run this several times and every time the job fails, the result is the same, the workspace and client are correct, and the executor number outputted by the script is incorrect.

            I'm theorising that this is because when builds are running concurrently, in between the sync and the running of the script, the executor number gets overridden by the other jobs, hence it ends up incorrect when the script runs. Therefore I don't think the script we're using is suitable for testing this issue.

             

            It's worth noting I am using the latest changes which contains https://github.com/jenkinsci/p4-plugin/pull/113. Which fixes an issue very similar to this, where the workspace was being set wrong. I believe this was why you were getting a different p4 client and workspace.

             

            James Bateman, when available please can you retest your original scenario using a build with https://github.com/jenkinsci/p4-plugin/pull/113 in. As I suspect this may fix your issue.

             

            Show
            msmeeth Matthew Smeeth added a comment - - edited Hi, I've written an automated test to reproduce this issue, Looking into it further, it looks to me like the P4_CLIENT variable is actually correct, and it's actually the EXECUTOR NUMBER that is wrong in your script.   I can demonstrate this by putting an extra bit of debugging into the p4 plugin when running jobs. I have modified my build so that it outputs the executor number as the job is executing and this is what I get:   Obtained concurrentBuildsTest/jenkinsfile from p4-brunoCred- //depot/... //jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}/... Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in / var /jenkins_home/workspace/concurrentBuildsTestProject [Pipeline] { [Pipeline] stage [Pipeline] { (testStage) [Pipeline] script [Pipeline] { [Pipeline] dir Running in / var /jenkins_home/workspace/concurrentBuildsTestProject/test [Pipeline] { [Pipeline] checkout Executor no at runtime:0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Extra debugging I've added to the p4 plugin ...... ...... ...... + concurrentBuildsTest/script.sh ================================= WORKSPACE: / var /jenkins_home/workspace/concurrentBuildsTestProject <<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT P4_CLIENT: jenkins-master-concurrentBuildsTestProject-0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT EXECUTOR NUMBER: 1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Wrong executor number ================================= ERROR - Bad Workspace ERROR - Bad P4 Client ...... ...... Finished: FAILURE   I have run this several times and every time the job fails, the result is the same, the workspace and client are correct, and the executor number outputted by the script is incorrect. I'm theorising that this is because when builds are running concurrently, in between the sync and the running of the script, the executor number gets overridden by the other jobs, hence it ends up incorrect when the script runs. Therefore I don't think the script we're using is suitable for testing this issue.   It's worth noting I am using the latest changes which contains  https://github.com/jenkinsci/p4-plugin/pull/113 . Which fixes an issue very similar to this, where the workspace was being set wrong. I believe this was why you were getting a different p4 client and workspace.   James Bateman , when available please can you retest your original scenario using a build with  https://github.com/jenkinsci/p4-plugin/pull/113  in. As I suspect this may fix your issue.  
            Hide
            msmeeth Matthew Smeeth added a comment -

            There seems to be underlying issues with the executor number in Jenkins core. 

            JENKINS-48882, JENKINS-24679, JENKINS-7357, JENKINS-4756.

            Our testing confirms that we calculate the correct executor number for the client, however the environment variable EXECUTOR_NUMBER set by Jenkins (hudson.model.Computer) seems always be incorrect with concurrent execution.

            Closing this issue as we have resolved the client name part of the issue. Please raise this against Jenkins core if you require the EXECUTOR_NUMBER to be fixed.

            Show
            msmeeth Matthew Smeeth added a comment - There seems to be underlying issues with the executor number in Jenkins core.  JENKINS-48882 , JENKINS-24679 , JENKINS-7357 , JENKINS-4756 . Our testing confirms that we calculate the correct executor number for the client, however the environment variable EXECUTOR_NUMBER set by Jenkins (hudson.model.Computer) seems always be incorrect with concurrent execution. Closing this issue as we have resolved the client name part of the issue. Please raise this against Jenkins core if you require the EXECUTOR_NUMBER to be fixed.
            Hide
            tomreed81 Tom Reed added a comment - - edited

            I have been battling a similar issue.

            My situation is that the P4_CLIENT I build uses the variables "jenkins-${JOB_NAME}-{NODE_NAME}"

            The NODE_NAME is wrong a few times a night with 6 parallel builds on 6 different nodes. About 40-050 builds are queued a night, and the wrong P4_CLIENT is causing the wrong P4_CHANGELIST to be reported.

            Show
            tomreed81 Tom Reed added a comment - - edited I have been battling a similar issue. My situation is that the P4_CLIENT I build uses the variables "jenkins-${JOB_NAME}-{NODE_NAME}" The NODE_NAME is wrong a few times a night with 6 parallel builds on 6 different nodes. About 40-050 builds are queued a night, and the wrong P4_CLIENT is causing the wrong P4_CHANGELIST to be reported.

              People

              Assignee:
              msmeeth Matthew Smeeth
              Reporter:
              jbateman James Bateman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: