-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Master:
Jenkins ver. 2.32.2
Windows Server 2012 R2 64 bit
jre 1.8.0_121 64 bit
Running Jenkins directly installed as windows service
Installed Plugins
ace-editor 1.1 true false
ant 1.4 true false
antisamy-markup-formatter 1.5 true false
authentication-tokens 1.3 true false
bouncycastle-api 2.16.0 true false
branch-api 2.0.7 true false
build-blocker-plugin 1.7.3 true false
build-flow-plugin 0.20 true false
build-name-setter 1.6.5 true false
build-timeout 1.18 true false
build-user-vars-plugin 1.5 true false
buildgraph-view 1.5.1 true false
buildresult-trigger 0.17 true false
cloudbees-folder 5.18 true false
compact-columns 1.10 true false
conditional-buildstep 1.3.5 true false
config-file-provider 2.15.6 true false
credentials 2.1.13 true false
credentials-binding 1.10 true false
cvs 2.13 true false
display-url-api 1.1.1 true false
docker-commons 1.6 true false
docker-workflow 1.10 true false
durable-task 1.13 true false
dynamicparameter 0.2.0 true false
email-ext 2.57 true false
envinject 1.93.1 true false
extended-choice-parameter 0.76 true false
extensible-choice-parameter 1.3.4 true false
external-monitor-job 1.7 true false
git 3.1.0 true false
git-client 2.3.0 true false
git-server 1.7 true false
gradle 1.26 true false
groovy 1.30 true false
groovy-label-assignment 1.2.0 true false
groovy-postbuild 2.3.1 true false
handlebars 1.1.1 true false
http_request 1.8.13 true false
icon-shim 2.0.3 true false
javadoc 1.4 true false
jobConfigHistory 2.15 true false
jquery 1.11.2-0 true false
jquery-detached 1.2.1 true false
junit 1.20 true false
ldap 1.14 true false
leastload 1.0.3 true false
log-parser 2.0 true false
mailer 1.19 true false
mapdb-api 1.0.9.0 true false
mask-passwords 2.9 true false
matrix-auth 1.4 true false
matrix-project 1.8 true false
maven-plugin 2.15.1 true false
momentjs 1.1.1 true false
monitoring 1.63.0 true false
node-iterator-api 1.5.0 true false
nodelabelparameter 1.7.2 true false
pam-auth 1.3 true false
parameterized-trigger 2.33 true false
pipeline-build-step 2.4 true false
pipeline-graph-analysis 1.3 true false
pipeline-input-step 2.5 true false
pipeline-milestone-step 1.3 true false
pipeline-model-api 1.0.2 true false
pipeline-model-declarative-agent 1.0.2 true false
pipeline-model-definition 1.0.2 true false
pipeline-rest-api 2.5 true false
pipeline-stage-step 2.2 true false
pipeline-stage-tags-metadata 1.0.2 true false
pipeline-stage-view 2.5 true false
plain-credentials 1.4 true false
promoted-builds 2.28.1 true false
rebuild 1.25 true false
repository-connector 1.1.3 true false
resource-disposer 0.6 true false
role-strategy 2.3.2 true false
run-condition 1.0 true false
scm-api 2.0.8 true false
script-security 1.27 true false
scriptler 2.9 true false
simple-theme-plugin 0.3 true false
sitemonitor 0.5 true false
ssh-credentials 1.13 true false
ssh-slaves 1.13 true false
structs 1.6 true false
subversion 2.7.1 true false
testng-plugin 1.14 true false
token-macro 2.0 true false
translation 1.15 true false
uno-choice 1.5.2 true false
update-sites-manager 2.0.0 true false
validating-string-parameter 2.3 true false
view-job-filters 1.27 true false
vsphere-cloud 2.15 true false
windows-exe-runner 1.2 true false
windows-slaves 1.2 true false
workflow-aggregator 2.5 true false
workflow-api 2.11 true false
workflow-basic-steps 2.4 true false
workflow-cps 2.29 true false
workflow-cps-global-lib 2.7 true false
workflow-durable-task-step 2.9 true false
workflow-job 2.10 true false
workflow-multibranch 2.13 true false
workflow-scm-step 2.4 true false
workflow-step-api 2.9 true false
workflow-support 2.13 true false
ws-cleanup 0.32 true false
xunit 1.102 true false
Slave
vpshere slave
Windows XP 32 bit
Slave runs jenkins jar using jnlp launch
jdk1.7.0_40
Master: Jenkins ver. 2.32.2 Windows Server 2012 R2 64 bit jre 1.8.0_121 64 bit Running Jenkins directly installed as windows service Installed Plugins ace-editor 1.1 true false ant 1.4 true false antisamy-markup-formatter 1.5 true false authentication-tokens 1.3 true false bouncycastle-api 2.16.0 true false branch-api 2.0.7 true false build-blocker-plugin 1.7.3 true false build-flow-plugin 0.20 true false build-name-setter 1.6.5 true false build-timeout 1.18 true false build-user-vars-plugin 1.5 true false buildgraph-view 1.5.1 true false buildresult-trigger 0.17 true false cloudbees-folder 5.18 true false compact-columns 1.10 true false conditional-buildstep 1.3.5 true false config-file-provider 2.15.6 true false credentials 2.1.13 true false credentials-binding 1.10 true false cvs 2.13 true false display-url-api 1.1.1 true false docker-commons 1.6 true false docker-workflow 1.10 true false durable-task 1.13 true false dynamicparameter 0.2.0 true false email-ext 2.57 true false envinject 1.93.1 true false extended-choice-parameter 0.76 true false extensible-choice-parameter 1.3.4 true false external-monitor-job 1.7 true false git 3.1.0 true false git-client 2.3.0 true false git-server 1.7 true false gradle 1.26 true false groovy 1.30 true false groovy-label-assignment 1.2.0 true false groovy-postbuild 2.3.1 true false handlebars 1.1.1 true false http_request 1.8.13 true false icon-shim 2.0.3 true false javadoc 1.4 true false jobConfigHistory 2.15 true false jquery 1.11.2-0 true false jquery-detached 1.2.1 true false junit 1.20 true false ldap 1.14 true false leastload 1.0.3 true false log-parser 2.0 true false mailer 1.19 true false mapdb-api 1.0.9.0 true false mask-passwords 2.9 true false matrix-auth 1.4 true false matrix-project 1.8 true false maven-plugin 2.15.1 true false momentjs 1.1.1 true false monitoring 1.63.0 true false node-iterator-api 1.5.0 true false nodelabelparameter 1.7.2 true false pam-auth 1.3 true false parameterized-trigger 2.33 true false pipeline-build-step 2.4 true false pipeline-graph-analysis 1.3 true false pipeline-input-step 2.5 true false pipeline-milestone-step 1.3 true false pipeline-model-api 1.0.2 true false pipeline-model-declarative-agent 1.0.2 true false pipeline-model-definition 1.0.2 true false pipeline-rest-api 2.5 true false pipeline-stage-step 2.2 true false pipeline-stage-tags-metadata 1.0.2 true false pipeline-stage-view 2.5 true false plain-credentials 1.4 true false promoted-builds 2.28.1 true false rebuild 1.25 true false repository-connector 1.1.3 true false resource-disposer 0.6 true false role-strategy 2.3.2 true false run-condition 1.0 true false scm-api 2.0.8 true false script-security 1.27 true false scriptler 2.9 true false simple-theme-plugin 0.3 true false sitemonitor 0.5 true false ssh-credentials 1.13 true false ssh-slaves 1.13 true false structs 1.6 true false subversion 2.7.1 true false testng-plugin 1.14 true false token-macro 2.0 true false translation 1.15 true false uno-choice 1.5.2 true false update-sites-manager 2.0.0 true false validating-string-parameter 2.3 true false view-job-filters 1.27 true false vsphere-cloud 2.15 true false windows-exe-runner 1.2 true false windows-slaves 1.2 true false workflow-aggregator 2.5 true false workflow-api 2.11 true false workflow-basic-steps 2.4 true false workflow-cps 2.29 true false workflow-cps-global-lib 2.7 true false workflow-durable-task-step 2.9 true false workflow-job 2.10 true false workflow-multibranch 2.13 true false workflow-scm-step 2.4 true false workflow-step-api 2.9 true false workflow-support 2.13 true false ws-cleanup 0.32 true false xunit 1.102 true false Slave vpshere slave Windows XP 32 bit Slave runs jenkins jar using jnlp launch jdk1.7.0_40
Testing Jenkins upgrade from 2.7.2 to 2.32.2 with installed plugins upgraded as well.
We use a lot of flow jobs and post upgrade they don't work when combined with a label parameter with a value that isn't 'master' i.e. we can't get a flow job to run on a slave using the label parameter.
- As per screen shots, create a simple flow job with a single Label parameter
- Build the job with the label value of master and it works.
- Rebuild the job using a label for a connected slave that has free executors, the job permanently sits in the Build Queue - 'Waiting for next available executor'
- Alternatively, change the job configuration to no longer have a parameter but select 'Restrict where this project can be run' and set the slave name. Rebuild the job and same issue - sits in Build Queue.
- With the job sat in the Build Queue I can submit a separate 'free style project' also configured with a Label parameter for the same slave name and it completes ok.
During the upgrade steps neither if the two obvious plugins were upgraded:
- Node and Label parameter plugin still at 1.7.2
- Build Flow plugin still at 0.20
The Node and Label parameter had dependencies that were upgraded, but reverting both, and using the Jenkins auto restart option from the plugin page, does not solve the issue.
- parameterized-trigger Upgraded from 2.32 to 2.33
- token-macro Upgraded from 1.12.1 to 2.0
I'm about to restart the entire upgrade process again to try to narrow down which change causes this, but any advice/thoughts would be appreciated.
[JENKINS-42538] Build Flow jobs stuck waiting on next available executor when using label parameter to restrict job to vpshere slave
Elliott Jones
added a comment - Tried this with a clean install windows intall of 2.32.2 using just the Build Flow and Node and Label parameter plugins:
| ant | 1.4 | true | false |
| antisamy-markup-formatter | 1.5 | true | false |
| bouncycastle-api | 2.16.0 | true | false |
| build-flow-plugin | 0.20 | true | false |
| display-url-api | 1.1.1 | true | false |
| external-monitor-job | 1.7 | true | false |
| icon-shim | 2.0.3 | true | false |
| javadoc | 1.4 | true | false |
| jquery | 1.11.2-0 | true | false |
| junit | 1.20 | true | false |
| ldap | 1.14 | true | false |
| mailer | 1.19 | true | false |
| matrix-auth | 1.4 | true | false |
| matrix-project | 1.8 | true | false |
| nodelabelparameter | 1.7.2 | true | false |
| pam-auth | 1.3 | true | false |
| script-security | 1.27 | true | false |
| structs | 1.6 | true | false |
| token-macro | 2.0 | true | false |
| windows-slaves | 1.2 | true | false |
Issue cannot be reproduced, the flow job completes ok when using a slave label that isn't master. Key difference the slave used in this instance was a 'permanent agent' rather than a vsphere slave in envt where problem shown. Will try adding to the mix.
Elliott Jones
added a comment - Tried this with a clean install windows intall of 2.32.2 using just the Build Flow and Node and Label parameter plugins:
ant
1.4
true
false
antisamy-markup-formatter
1.5
true
false
bouncycastle-api
2.16.0
true
false
build-flow-plugin
0.20
true
false
display-url-api
1.1.1
true
false
external-monitor-job
1.7
true
false
icon-shim
2.0.3
true
false
javadoc
1.4
true
false
jquery
1.11.2-0
true
false
junit
1.20
true
false
ldap
1.14
true
false
mailer
1.19
true
false
matrix-auth
1.4
true
false
matrix-project
1.8
true
false
nodelabelparameter
1.7.2
true
false
pam-auth
1.3
true
false
script-security
1.27
true
false
structs
1.6
true
false
token-macro
2.0
true
false
windows-slaves
1.2
true
false
Issue cannot be reproduced, the flow job completes ok when using a slave label that isn't master. Key difference the slave used in this instance was a 'permanent agent' rather than a vsphere slave in envt where problem shown. Will try adding to the mix. That seems to be the key factor, during my upgrade I upgraded vSphere Plugin from 2.4 to 2.15. Downgrading back to 2.4 and the flow job works with the label parameter correctly. Updating ticket details appropriately.
Marking minor as workaround is simple for me, plugin downgrade.
Elliott Jones
added a comment - - edited That seems to be the key factor, during my upgrade I upgraded vSphere Plugin from 2.4 to 2.15. Downgrading back to 2.4 and the flow job works with the label parameter correctly. Updating ticket details appropriately.
Marking minor as workaround is simple for me, plugin downgrade. Elliott Jones
added a comment - I also had this issue https://issues.jenkins-ci.org/browse/JENKINS-41384 and thus had to apply jenkins.slaves.DefaultJnlpSlaveReceiver.disableStrictVerification=true
Elliott Jones
added a comment - I also had this issue https://issues.jenkins-ci.org/browse/JENKINS-41384 and thus had to apply jenkins.slaves.DefaultJnlpSlaveReceiver.disableStrictVerification=true 
pjdarton
added a comment - elliottjones Can you see if the issue is fixed in version 2.7 but broken in version 2.8?
FYI a change was made that disallowed "flyweight tasks" from running on vSphere nodes. This was done because vSphere nodes can be turned off, restarted etc when they've finished their main work and, if a "flyweight task" happens to be running on them when that happens, it'll break the "flyweight task" process.
I'm unfamiliar with flow jobs but I do wonder if these make use of "flyweight tasks" to run and that'd be why they're unable to run on vSphere nodes on newer versions of the plugin.
If that's the case, it'd be relatively straightforward to make the "don't allow flyweight tasks to run here" behavior a configurable tick-box option...
pjdarton
added a comment - elliottjones Can you see if the issue is fixed in version 2.7 but broken in version 2.8?
FYI a change was made that disallowed "flyweight tasks" from running on vSphere nodes. This was done because vSphere nodes can be turned off, restarted etc when they've finished their main work and, if a "flyweight task" happens to be running on them when that happens, it'll break the "flyweight task" process.
I'm unfamiliar with flow jobs but I do wonder if these make use of "flyweight tasks" to run and that'd be why they're unable to run on vSphere nodes on newer versions of the plugin.
If that's the case, it'd be relatively straightforward to make the "don't allow flyweight tasks to run here" behavior a configurable tick-box option... 
I thought it could be related to this issue https://issues.jenkins-ci.org/browse/JENKINS-34969 due to similar plugins but nothing on that ticket helps.