[JENKINS-60213] P4_CLIENT environment variable non-reentrant

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: p4-plugin
Labels:
- P4_A
- P4_CLIENT
- P4_VERIFY

Similar Issues:
Powered by SuggestiMate

Show

Version 1.10.6 of the p4-plugin causes issues when multiple jobs are ran in parallel.

The result is that the p4 created environment variables of one job can corrupt those in another job that is running at the same time.

This bug wasn't in 1.10.5

Steps to reproduce issue:

Create a new Jenkins test job with the following settings
- General:
  - Execute concurrent builds if necessary
  - Use custom workspace
    - Directory
      /tmp/${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
- Source Code Management
  - Perforce Software:
    - Workspace behaviour: Template (view generated for each node)
      - Template workspace: Your test template
      - Workspace Name Format being
        
        jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
Add a build step

To see the issue:

Kick off two runs of the newly creates client

Result:

Pre 1.10.5 both would have worked and both will set the expected P4 environment variables.

1.10.6 gets confused. So in my case:

Job1 Environment variables

P4_CLIENT	_jenkins-master-ClientClashTest-0
WORKSPACE	/tmp/master-ClientClashTest-0

Job2 Environment variables (THIS IS THE ISSUE – Wrong P4_CLIENT value)

P4_CLIENT	_jenkins-master-ClientClashTest-0
WORKSPACE	/tmp/master-ClientClashTest-1

—

If I reinstall the 1.10.5 (or earlier) plugin then for job 2 I would see:

P4_CLIENT	_jenkins-master-ClientClashTest-1
WORKSPACE	/tmp/master-ClientClashTest-1

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Build History.png
9 kB
2019-11-20 11:05
JENKINS-60213__JobConfig.png
254 kB
2019-11-20 11:12
screenshot-1.png
103 kB
2019-11-20 11:07

is related to

JENKINS-48882 Build Executor number is not consistent with EXECUTOR_NUMBER

Open

James Bateman added a comment - 2019-11-25 18:12 - edited

Looks like your copy-and-paste went wrong, all the example in your last message are exactly the same (basically there is no way that each job can use the same bash script – i.e. /tmp/jenkins6285754148517933441.sh)

James Bateman added a comment - 2019-11-25 18:12 - edited Looks like your copy-and-paste went wrong, all the example in your last message are exactly the same (basically there is no way that each job can use the same bash script – i.e. /tmp/jenkins6285754148517933441.sh )

Karl Wirth added a comment - 2019-11-26 11:30

Hi jbateman - Urgh. Sorry. You are correct. My fault for manually editing the URL.

30:
[JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh
==========================================
WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables
P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-0
EXECUTOR NUMBER:     0
==========================================31:
[JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins1481999870837323074.sh
==========================================
WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-6
EXECUTOR NUMBER:     6
==========================================32:
[JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins3645232859628494475.sh
==========================================
WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3
P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-3
EXECUTOR NUMBER:     3
==========================================

Are you able to test the nightly at the following location to prove that it fixes the bug?

https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fp4-plugin/detail/PR-93/38/artifacts

Karl Wirth added a comment - 2019-11-26 11:30 Hi jbateman - Urgh. Sorry. You are correct. My fault for manually editing the URL. 30: [JENKINS-60213-ConcurrentVariables] $ /bin/bash /tmp/jenkins6285754148517933441.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-0 EXECUTOR NUMBER: 0 ==========================================31: [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins1481999870837323074.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-6 EXECUTOR NUMBER: 6 ==========================================32: [JENKINS-60213-ConcurrentVariables@3] $ /bin/bash /tmp/jenkins3645232859628494475.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@3 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-3 EXECUTOR NUMBER: 3 ========================================== Are you able to test the nightly at the following location to prove that it fixes the bug? https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fp4-plugin/detail/PR-93/38/artifacts

James Bateman added a comment - 2019-11-26 18:12

It is still broken

I have installed the groovy plugin and before my bash script ran this script

import jenkins.model.*
import hudson.PluginWrapper

println '-' * 80
println 'Jenkins Version = ' + Jenkins.instance.getVersion()

PluginWrapper p4 = Jenkins.instance.pluginManager.getPlugin('p4')
if(p4) {
    println p4.getDisplayName() + ':'
    println '    | Short name = ' + p4.getShortName()
    println '    | Version = ' + p4.getVersion()
    println '    | Plugin URL = ' + p4.getUrl()
    println '    | Is actived = ' + (p4.isActive() ? 'Yes' : 'No')
    println '    | Is enabled = ' + (p4.isEnabled() ? 'Yes' : 'No')
}

println '-' * 80

The output from this script confirms that I'm using your patched version of the p4 plugin, the output being:

--------------------------------------------------------------------------------
Jenkins Version = 2.190.2
P4 Plugin:
    | Short name = p4
    | Version = 1.10.7-SNAPSHOT (private-a1cfcc02-jenkins)
    | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
    | Is actived = Yes
    | Is enabled = Yes
--------------------------------------------------------------------------------

Kicking off two jobs one after another (i.e. clicking on 'Build Now' twice in short session), I get this output

1st job:

[master-ClientClashTest-0] $ /bin/bash /tmp/jenkins1556703573399779365.sh
=================================
WORKSPACE:       /tmp/master-ClientClashTest-0
P4_CLIENT:       jenkins-master-ClientClashTest-1
EXECUTOR NUMBER: 0
=================================

This job has been corrupted since the P4_CLIENT name should be:
- jenkins-master-ClientClashTest-0
and not
- jenkins-master-ClientClashTest-1

2nd job:

[master-ClientClashTest-1] $ /bin/bash /tmp/jenkins10551339048100737039.sh
=================================
WORKSPACE:       /tmp/master-ClientClashTest-1
P4_CLIENT:       jenkins-master-ClientClashTest-1
EXECUTOR NUMBER: 1
=================================

This job run is OK

PS. If you wanted the bash build step to fail in cases where the items do not match up correctly (which makes it easier to find problematic runs), then you could change the bash script in your test job to look like this:

#!/bin/bash

echo "================================="
echo "WORKSPACE:       ${WORKSPACE}"
echo "P4_CLIENT:       ${P4_CLIENT}"
echo "EXECUTOR NUMBER: ${EXECUTOR_NUMBER}"
echo "================================="

EXITCODE=0
if [ "${WORKSPACE##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad Workspace"; EXITCODE=1; fi
if [ "${P4_CLIENT##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad P4 Client"; EXITCODE=1; fi

exit $EXITCODE

Note: The above script assumes your 'Workspace Name Format' mapping looks like this:
- jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
and your 'custom workspace' / 'Directory name' looks something like this:
- /tmp/${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}

James Bateman added a comment - 2019-11-26 18:12 It is still broken I have installed the groovy plugin and before my bash script ran this script import jenkins.model.* import hudson.PluginWrapper println '-' * 80 println 'Jenkins Version = ' + Jenkins.instance.getVersion() PluginWrapper p4 = Jenkins.instance.pluginManager.getPlugin( 'p4' ) if (p4) { println p4.getDisplayName() + ':' println ' | Short name = ' + p4.getShortName() println ' | Version = ' + p4.getVersion() println ' | Plugin URL = ' + p4.getUrl() println ' | Is actived = ' + (p4.isActive() ? 'Yes' : 'No' ) println ' | Is enabled = ' + (p4.isEnabled() ? 'Yes' : 'No' ) } println '-' * 80 The output from this script confirms that I'm using your patched version of the p4 plugin, the output being: -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7-SNAPSHOT ( private -a1cfcc02-jenkins) | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- Kicking off two jobs one after another (i.e. clicking on 'Build Now' twice in short session), I get this output 1st job: [master-ClientClashTest-0] $ /bin/bash /tmp/jenkins1556703573399779365.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-0 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 0 ================================= This job has been corrupted since the P4_CLIENT name should be: jenkins-master-ClientClashTest- 0 and not jenkins-master-ClientClashTest- 1 2nd job: [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins10551339048100737039.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-1 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 1 ================================= This job run is OK PS. If you wanted the bash build step to fail in cases where the items do not match up correctly (which makes it easier to find problematic runs) , then you could change the bash script in your test job to look like this: #!/bin/bash echo "=================================" echo "WORKSPACE: ${WORKSPACE}" echo "P4_CLIENT: ${P4_CLIENT}" echo "EXECUTOR NUMBER: ${EXECUTOR_NUMBER}" echo "=================================" EXITCODE=0 if [ "${WORKSPACE##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad Workspace" ; EXITCODE=1; fi if [ "${P4_CLIENT##*-}" != "${EXECUTOR_NUMBER}" ]; then echo "*** ERROR - Bad P4 Client" ; EXITCODE=1; fi exit $EXITCODE Note: The above script assumes your ' Workspace Name Format ' mapping looks like this: jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER} and your ' custom workspace ' / ' Directory name' looks something like this: /tmp/${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}

Karl Wirth added a comment - 2019-11-29 15:52

Hi jbateman - I need to apologise. The nightly I gave you did not contain the fix. Are you able to test 1.10.7 that we shipped yesterday that does contain the correct bug fix?

Karl Wirth added a comment - 2019-11-29 15:52 Hi jbateman - I need to apologise. The nightly I gave you did not contain the fix. Are you able to test 1.10.7 that we shipped yesterday that does contain the correct bug fix?

James Bateman added a comment - 2019-12-02 12:36

Plugin 1.10.7 still has the same issues

Job #78: (BAD)

[EnvInject] - Injecting environment variables from a build step.
--------------------------------------------------------------------------------
Jenkins Version = 2.190.2
P4 Plugin:
    | Short name = p4
    | Version = 1.10.7
    | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
    | Is actived = Yes
    | Is enabled = Yes
--------------------------------------------------------------------------------
[master-ClientClashTest-1] $ /bin/bash /tmp/jenkins822701625673778739.sh
=================================
WORKSPACE:       /tmp/master-ClientClashTest-1
P4_CLIENT:       jenkins-master-ClientClashTest-0
EXECUTOR NUMBER: 1
=================================
*** ERROR - Bad P4 Client

Job #79: (BAD)

[EnvInject] - Injecting environment variables from a build step.
--------------------------------------------------------------------------------
Jenkins Version = 2.190.2
P4 Plugin:
    | Short name = p4
    | Version = 1.10.7
    | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
    | Is actived = Yes
    | Is enabled = Yes
--------------------------------------------------------------------------------
[master-ClientClashTest-0] $ /bin/bash /tmp/jenkins825491192926150409.sh
=================================
WORKSPACE:       /tmp/master-ClientClashTest-0
P4_CLIENT:       jenkins-master-ClientClashTest-1
EXECUTOR NUMBER: 0
=================================
*** ERROR - Bad P4 Client

Job #80: (GOOD)

[EnvInject] - Injecting environment variables from a build step.
--------------------------------------------------------------------------------
Jenkins Version = 2.190.2
P4 Plugin:
    | Short name = p4
    | Version = 1.10.7
    | Plugin URL = https://wiki.jenkins-ci.org/display/JENKINS/P4+Plugin
    | Is actived = Yes
    | Is enabled = Yes
--------------------------------------------------------------------------------
[master-ClientClashTest-1] $ /bin/bash /tmp/jenkins14602602394965857665.sh
=================================
WORKSPACE:       /tmp/master-ClientClashTest-1
P4_CLIENT:       jenkins-master-ClientClashTest-1
EXECUTOR NUMBER: 1
=================================
Finished: SUCCESS

Summary:

So 2 of 3 of the above jobs on a server running the latest plug (1.10.7) ended up with corrupt P4_CLIENT environment variables. If I downgrade the plugin to version 1.10.5 then 100% of the jobs I kick off have the correct P4_CLIENT variables.

James Bateman added a comment - 2019-12-02 12:36 Plugin 1.10.7 still has the same issues Job #78: (BAD) [EnvInject] - Injecting environment variables from a build step. -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7 | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins822701625673778739.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-1 P4_CLIENT: jenkins-master-ClientClashTest-0 EXECUTOR NUMBER: 1 ================================= *** ERROR - Bad P4 Client Job #79: (BAD) [EnvInject] - Injecting environment variables from a build step. -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7 | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- [master-ClientClashTest-0] $ /bin/bash /tmp/jenkins825491192926150409.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-0 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 0 ================================= *** ERROR - Bad P4 Client Job #80: (GOOD) [EnvInject] - Injecting environment variables from a build step. -------------------------------------------------------------------------------- Jenkins Version = 2.190.2 P4 Plugin: | Short name = p4 | Version = 1.10.7 | Plugin URL = https: //wiki.jenkins-ci.org/display/JENKINS/P4+Plugin | Is actived = Yes | Is enabled = Yes -------------------------------------------------------------------------------- [master-ClientClashTest-1] $ /bin/bash /tmp/jenkins14602602394965857665.sh ================================= WORKSPACE: /tmp/master-ClientClashTest-1 P4_CLIENT: jenkins-master-ClientClashTest-1 EXECUTOR NUMBER: 1 ================================= Finished: SUCCESS Summary: So 2 of 3 of the above jobs on a server running the latest plug (1.10.7) ended up with corrupt P4_CLIENT environment variables. If I downgrade the plugin to version 1.10.5 then 100% of the jobs I kick off have the correct P4_CLIENT variables.

Karl Wirth added a comment - 2019-12-02 15:31

Hi jbateman - Thanks for letting is know. I'll go back through the test harness to see if there's anything we have assumed that could be the difference.

Karl Wirth added a comment - 2019-12-02 15:31 Hi jbateman - Thanks for letting is know. I'll go back through the test harness to see if there's anything we have assumed that could be the difference.

Karl Wirth added a comment - 2019-12-03 17:11

Verified still sporadically broken in 1.10.7. P4_CLIENT is wrong. The full expansion for me is correct and matches the rest of the job:

P4: saving built changes.
Found last change 2240 on syncID jenkins-NODE_NAME-JENKINS-60213-ConcurrentVariables-EXECUTOR_NUMBER
... p4 login -s 
... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
... p4 info 
... p4 info 
... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
... p4 login -s 
... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
... p4 info 
... p4 info 
... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 
... done

[JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins6561639748343680378.sh
==========================================
WORKSPACE:            /var/lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2
P4_CLIENT:            jenkins-master-JENKINS-60213-ConcurrentVariables-4
EXECUTOR NUMBER:      3
CLIENT_FROM_VARS:     jenkins-master-JENKINS-60213-ConcurrentVariables-3
==========================================
Finished: SUCCESS

Note for dev: In the above ${P4_CLIENT} is wrong but expansion is correct.

Karl Wirth added a comment - 2019-12-03 17:11 Verified still sporadically broken in 1.10.7. P4_CLIENT is wrong. The full expansion for me is correct and matches the rest of the job: P4: saving built changes. Found last change 2240 on syncID jenkins-NODE_NAME-JENKINS-60213-ConcurrentVariables-EXECUTOR_NUMBER ... p4 login -s ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... p4 info ... p4 info ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... p4 login -s ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... p4 info ... p4 info ... p4 client -o jenkins-master-JENKINS-60213-ConcurrentVariables-3 ... done [JENKINS-60213-ConcurrentVariables@2] $ /bin/bash /tmp/jenkins6561639748343680378.sh ========================================== WORKSPACE: / var /lib/jenkins/workspace/JENKINS-60213-ConcurrentVariables@2 P4_CLIENT: jenkins-master-JENKINS-60213-ConcurrentVariables-4 EXECUTOR NUMBER: 3 CLIENT_FROM_VARS: jenkins-master-JENKINS-60213-ConcurrentVariables-3 ========================================== Finished: SUCCESS Note for dev: In the above ${P4_CLIENT} is wrong but expansion is correct.

Matthew Smeeth added a comment - 2019-12-24 15:17 - edited

Hi, I've written an automated test to reproduce this issue, Looking into it further, it looks to me like the P4_CLIENT variable is actually correct, and it's actually the EXECUTOR NUMBER that is wrong in your script.

I can demonstrate this by putting an extra bit of debugging into the p4 plugin when running jobs. I have modified my build so that it outputs the executor number as the job is executing and this is what I get:

Obtained concurrentBuildsTest/jenkinsfile from p4-brunoCred-//depot/... //jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}/...
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/concurrentBuildsTestProject
[Pipeline] {
[Pipeline] stage
[Pipeline] { (testStage)
[Pipeline] script
[Pipeline] {
[Pipeline] dir
Running in /var/jenkins_home/workspace/concurrentBuildsTestProject/test
[Pipeline] {
[Pipeline] checkout
Executor no at runtime:0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Extra debugging I've added to the p4 plugin
 ......
 ......
 ......
 + concurrentBuildsTest/script.sh
 ================================= 
 WORKSPACE: /var/jenkins_home/workspace/concurrentBuildsTestProject <<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT
 P4_CLIENT: jenkins-master-concurrentBuildsTestProject-0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT
 EXECUTOR NUMBER: 1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Wrong executor number
=================================
ERROR - Bad Workspace 
ERROR - Bad P4 Client
......
......
Finished: FAILURE

I have run this several times and every time the job fails, the result is the same, the workspace and client are correct, and the executor number outputted by the script is incorrect.

I'm theorising that this is because when builds are running concurrently, in between the sync and the running of the script, the executor number gets overridden by the other jobs, hence it ends up incorrect when the script runs. Therefore I don't think the script we're using is suitable for testing this issue.

It's worth noting I am using the latest changes which contains https://github.com/jenkinsci/p4-plugin/pull/113. Which fixes an issue very similar to this, where the workspace was being set wrong. I believe this was why you were getting a different p4 client and workspace.

jbateman, when available please can you retest your original scenario using a build with https://github.com/jenkinsci/p4-plugin/pull/113 in. As I suspect this may fix your issue.

Matthew Smeeth added a comment - 2019-12-24 15:17 - edited Hi, I've written an automated test to reproduce this issue, Looking into it further, it looks to me like the P4_CLIENT variable is actually correct, and it's actually the EXECUTOR NUMBER that is wrong in your script. I can demonstrate this by putting an extra bit of debugging into the p4 plugin when running jobs. I have modified my build so that it outputs the executor number as the job is executing and this is what I get: Obtained concurrentBuildsTest/jenkinsfile from p4-brunoCred- //depot/... //jenkins-${NODE_NAME}${JOB_NAME}${EXECUTOR_NUMBER}/... Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in / var /jenkins_home/workspace/concurrentBuildsTestProject [Pipeline] { [Pipeline] stage [Pipeline] { (testStage) [Pipeline] script [Pipeline] { [Pipeline] dir Running in / var /jenkins_home/workspace/concurrentBuildsTestProject/test [Pipeline] { [Pipeline] checkout Executor no at runtime:0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Extra debugging I've added to the p4 plugin ...... ...... ...... + concurrentBuildsTest/script.sh ================================= WORKSPACE: / var /jenkins_home/workspace/concurrentBuildsTestProject <<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT P4_CLIENT: jenkins-master-concurrentBuildsTestProject-0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< CORRECT EXECUTOR NUMBER: 1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Wrong executor number ================================= ERROR - Bad Workspace ERROR - Bad P4 Client ...... ...... Finished: FAILURE I have run this several times and every time the job fails, the result is the same, the workspace and client are correct, and the executor number outputted by the script is incorrect. I'm theorising that this is because when builds are running concurrently, in between the sync and the running of the script, the executor number gets overridden by the other jobs, hence it ends up incorrect when the script runs. Therefore I don't think the script we're using is suitable for testing this issue. It's worth noting I am using the latest changes which contains https://github.com/jenkinsci/p4-plugin/pull/113 . Which fixes an issue very similar to this, where the workspace was being set wrong. I believe this was why you were getting a different p4 client and workspace. jbateman , when available please can you retest your original scenario using a build with https://github.com/jenkinsci/p4-plugin/pull/113 in. As I suspect this may fix your issue.

Matthew Smeeth added a comment - 2020-03-11 11:02

There seems to be underlying issues with the executor number in Jenkins core.

JENKINS-48882, JENKINS-24679, ~~JENKINS-7357~~, ~~JENKINS-4756~~.

Our testing confirms that we calculate the correct executor number for the client, however the environment variable EXECUTOR_NUMBER set by Jenkins (hudson.model.Computer) seems always be incorrect with concurrent execution.

Closing this issue as we have resolved the client name part of the issue. Please raise this against Jenkins core if you require the EXECUTOR_NUMBER to be fixed.

Matthew Smeeth added a comment - 2020-03-11 11:02 There seems to be underlying issues with the executor number in Jenkins core. JENKINS-48882 , JENKINS-24679 , JENKINS-7357 , JENKINS-4756 . Our testing confirms that we calculate the correct executor number for the client, however the environment variable EXECUTOR_NUMBER set by Jenkins (hudson.model.Computer) seems always be incorrect with concurrent execution. Closing this issue as we have resolved the client name part of the issue. Please raise this against Jenkins core if you require the EXECUTOR_NUMBER to be fixed.

Tom Reed added a comment - 2020-03-13 18:10 - edited

I have been battling a similar issue.

My situation is that the P4_CLIENT I build uses the variables "jenkins-${JOB_NAME}-{NODE_NAME}"

The NODE_NAME is wrong a few times a night with 6 parallel builds on 6 different nodes. About 40-050 builds are queued a night, and the wrong P4_CLIENT is causing the wrong P4_CHANGELIST to be reported.

Tom Reed added a comment - 2020-03-13 18:10 - edited I have been battling a similar issue. My situation is that the P4_CLIENT I build uses the variables "jenkins-${JOB_NAME}-{NODE_NAME}" The NODE_NAME is wrong a few times a night with 6 parallel builds on 6 different nodes. About 40-050 builds are queued a night, and the wrong P4_CLIENT is causing the wrong P4_CHANGELIST to be reported.

Assignee:: Matthew Smeeth

Reporter:: James Bateman

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2019-11-19 16:14

Updated:: 2020-04-21 15:32

Resolved:: 2020-03-11 11:02

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: James Bateman added a comment - 2019-11-25 18:12, Edited by James Bateman - 2019-11-25 18:17

Expand comment: James Bateman added a comment - 2019-11-25 18:12, Edited by James Bateman - 2019-11-25 18:17

Collapse comment: Karl Wirth added a comment - 2019-11-26 11:30

Expand comment: Karl Wirth added a comment - 2019-11-26 11:30

Collapse comment: James Bateman added a comment - 2019-11-26 18:12

1st job:

2nd job:

Expand comment: James Bateman added a comment - 2019-11-26 18:12

Collapse comment: Karl Wirth added a comment - 2019-11-29 15:52

Expand comment: Karl Wirth added a comment - 2019-11-29 15:52

Collapse comment: James Bateman added a comment - 2019-12-02 12:36

Expand comment: James Bateman added a comment - 2019-12-02 12:36

Collapse comment: Karl Wirth added a comment - 2019-12-02 15:31

Expand comment: Karl Wirth added a comment - 2019-12-02 15:31

Collapse comment: Karl Wirth added a comment - 2019-12-03 17:11

Expand comment: Karl Wirth added a comment - 2019-12-03 17:11

Collapse comment: Matthew Smeeth added a comment - 2019-12-24 15:17, Edited by Matthew Smeeth - 2019-12-24 15:20

Expand comment: Matthew Smeeth added a comment - 2019-12-24 15:17, Edited by Matthew Smeeth - 2019-12-24 15:20

Collapse comment: Matthew Smeeth added a comment - 2020-03-11 11:02

Expand comment: Matthew Smeeth added a comment - 2020-03-11 11:02

Collapse comment: Tom Reed added a comment - 2020-03-13 18:10, Edited by Tom Reed - 2020-03-13 18:11

Expand comment: Tom Reed added a comment - 2020-03-13 18:10, Edited by Tom Reed - 2020-03-13 18:11

People

Dates