The pop-up says that is waiting for the next available executor, which is "master", which is the only one satisfying this label. It is idle, but Jenkins is still waiting.
The configuration worked fine before upgrading. Restarting Jenkins does not help.
I have the same issue. I'm on 2.222.1. It ONLY happened when I upgraded to 2.222.1 from 2.204.1
I have a label_expression on a matrix project set to "master_node".
When I run it, it says "(pending—Waiting for next available executor on ‘master’)"
Note that it says "master", not "master_node". I have no idea why it's looking for something called "master", not "master_node".
I'm attaching a gif of what I see.
Here are the logs that Daniel Beck requested too:
Alex Gray
added a comment - I have the same issue. I'm on 2.222.1. It ONLY happened when I upgraded to 2.222.1 from 2.204.1
I have a label_expression on a matrix project set to "master_node".
When I run it, it says "(pending—Waiting for next available executor on ‘master’)"
Note that it says "master", not "master_node". I have no idea why it's looking for something called "master", not "master_node".
I'm attaching a gif of what I see.
Here are the logs that Daniel Beck requested too:
Apr 08, 2020 12:16:14 AM FINEST hudson.model.Queue
Queue.Snapshot{waitingList=[];blockedProjects=[hudson.model.Queue$BlockedItem:hudson.model.FreeStyleProject@502099a9[util-slave-manager]:152871];buildables=[hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@598a8bc[util-check-for-container-spread]:152734, hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@4fe1c89a[util-generate-datadog-monitors]:152745];pendings=[]} → Queue.Snapshot{waitingList=[];blockedProjects=[hudson.model.Queue$BlockedItem:hudson.model.FreeStyleProject@502099a9[util-slave-manager]:152871];buildables=[hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@598a8bc[util-check-for-container-spread]:152734, hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@4fe1c89a[util-generate-datadog-monitors]:152745];pendings=[]}; leftItems={152845=hudson.model.Queue$LeftItem:hudson.matrix.MatrixProject@33385024[util-check-for-low-ips]:152845, 152866=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@2e88aabc[util-check-hung-ecs-tasks-prod-ca]:152866, 152865=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@1c17e4db[util-check-hung-ecs-tasks-prod]:152865, 152819=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@502099a9[util-slave-manager]:152819, 152868=hudson.model.Queue$LeftItem:ExecutorStepExecution.PlaceholderTask{runId=util-alert-jenkins-vs-jumpcloud-users#4201,label=,context=CpsStepContext[4:node]:Owner[util-alert-jenkins-vs-jumpcloud-users/4201:util-alert-jenkins-vs-jumpcloud-users #4201],cookie=f8461e2f-66d2-48fb-9acc-79bb1b98a0ad,auth=null}:152868, 152867=hudson.model.Queue$LeftItem:org.jenkinsci.plugins.workflow.job.WorkflowJob@13b79664[util-alert-jenkins-vs-jumpcloud-users]:152867, 152870=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@188db571[util-check-hung-ecs-tasks-stage]:152870, 152872=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@1ae51106[databases-without-termination-protection]:152872, 152869=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@60957b04[util-check-asg-thrashing]:152869}
If you configure your job, and select "Restrict where this project can be run" and enter "master_node" as the node label, you'll get "Waiting for next available executor on 'master'". NOT "master_node". It's like it's tripping up on the underscore. If I change that to "masternode", then it correctly says "Waiting for next available executor on 'masternodde'" (but it still doesn't work, even if I have a node with that label.)
Hope this bit of information helps!
Alex Gray
added a comment - One more piece of information:
If you configure your job, and select "Restrict where this project can be run" and enter "master_node" as the node label, you'll get "Waiting for next available executor on 'master'". NOT "master_node". It's like it's tripping up on the underscore. If I change that to "masternode", then it correctly says "Waiting for next available executor on 'masternodde'" (but it still doesn't work, even if I have a node with that label.)
Hope this bit of information helps!
Ran into the same problem. Everything is fine using version 2.204.6 but the problem shows up with version 2.205. So as previously suggested, https://github.com/jenkinsci/jenkins/pull/3983 seems the likely cause.
After some investigation it seems that the cause of the problem is the use of a label used to refer to the master node in jobs (for node restriction purposes). For instance, we have the "dispatch" label configured for the "master" node and use it in multi-configuration jobs which are the ones that get stuck.
If I modify a stuck job to use "master" instead of the "DISPATCH" label, the job gets triggered as before. Same thing if I just configure the job to no longer have node restrictions.
Alain Campeau
added a comment - Ran into the same problem. Everything is fine using version 2.204.6 but the problem shows up with version 2.205. So as previously suggested, https://github.com/jenkinsci/jenkins/pull/3983 seems the likely cause.
After some investigation it seems that the cause of the problem is the use of a label used to refer to the master node in jobs (for node restriction purposes). For instance, we have the "dispatch" label configured for the "master" node and use it in multi-configuration jobs which are the ones that get stuck.
If I modify a stuck job to use "master" instead of the "DISPATCH" label, the job gets triggered as before. Same thing if I just configure the job to no longer have node restrictions.
I tried to reproduce this issue, but failed to do so.
If any of you could try to figure out instructions how to reproduce this problem from scratch, please provide detailed and complete instructions how to do that.
Daniel Beck
added a comment - I tried to reproduce this issue, but failed to do so.
If any of you could try to figure out instructions how to reproduce this problem from scratch, please provide detailed and complete instructions how to do that.
I've managed to reproduce from scratch on a newly set up Jenkins server (Windows) configured with a single node/agent (Windows). This way I've stripped off a lot of things from our production setup as to eliminate as many possible causes for this.
Here are the minimum steps I needed to repro:
Install latest Jenkins LTS release (2.222.3) on a Windows machine with the default set of plugins
Configure the master node with a label such as DISPATCH
Add a new node and configure it as a new agent. Due to limited resources I configured the new node/agent to run on the same Windows machine. Configure it with its own label such as WINDOWS and make sure "Usage" configuration is set to "Only build jobs with label expressions matching the node"
Created a new Multi-configuration project job:
Configure this job's "Restrict where this project can run" setting so its "Label Expression" value is the one specified for master, so DISPATCH
Configure this job's "Configuration Matrix" by adding:
a "User-defined matrix" axis with a "Name" of "TARGET" and "Values" of "XboxOne PS4 Switch" (any strings to mimic building for various platforms)
a "Slaves" axis with a "Name" of "TARGET_POOL" and make sure to check the "WINDOWS" checkbox - Configure this job's "Build" section by adding a dummy "Execute Windows batch command" whose content is simply an "@echo Hello world!"
If I launch this job using Jenkins 2.222.3, 2.205 or anything in between, the job is stuck waiting for an executor on master when there are 2 available and the agent using the WINDOWS label is free with at least a single executor.
If I launch this job using Jenkins 2.204.6 or earlier, the job successfully launches and sequentially runs all three XboxOne, PS4 and Switch configurations on the sole agent using the WINDOWS label while the job itself "runs" on the master node (even though all it does is dispatch really).
On our production server we use the "Dynamic Axis" plugin to dynamically build an axis of all platforms to build and have multiple Windows, Linux and Mac build machines using an OS-specific labels. But for the sake of keeping these repro steps simple, I've dropped all but one OS and removed the "Dynamic Axis" plugin usage. It doesn't logically make much sense but shows the behavior difference starting with the Jenkins 2.205 release.
Alain Campeau
added a comment - I've managed to reproduce from scratch on a newly set up Jenkins server (Windows) configured with a single node/agent (Windows). This way I've stripped off a lot of things from our production setup as to eliminate as many possible causes for this.
Here are the minimum steps I needed to repro:
Install latest Jenkins LTS release (2.222.3) on a Windows machine with the default set of plugins
Configure the master node with a label such as DISPATCH
Add a new node and configure it as a new agent. Due to limited resources I configured the new node/agent to run on the same Windows machine. Configure it with its own label such as WINDOWS and make sure "Usage" configuration is set to "Only build jobs with label expressions matching the node"
Created a new Multi-configuration project job:
Configure this job's "Restrict where this project can run" setting so its "Label Expression" value is the one specified for master, so DISPATCH
Configure this job's "Configuration Matrix" by adding:
a "User-defined matrix" axis with a "Name" of "TARGET" and "Values" of "XboxOne PS4 Switch" (any strings to mimic building for various platforms)
a "Slaves" axis with a "Name" of "TARGET_POOL" and make sure to check the "WINDOWS" checkbox - Configure this job's "Build" section by adding a dummy "Execute Windows batch command" whose content is simply an "@echo Hello world!"
If I launch this job using Jenkins 2.222.3, 2.205 or anything in between, the job is stuck waiting for an executor on master when there are 2 available and the agent using the WINDOWS label is free with at least a single executor.
If I launch this job using Jenkins 2.204.6 or earlier, the job successfully launches and sequentially runs all three XboxOne, PS4 and Switch configurations on the sole agent using the WINDOWS label while the job itself "runs" on the master node (even though all it does is dispatch really).
On our production server we use the "Dynamic Axis" plugin to dynamically build an axis of all platforms to build and have multiple Windows, Linux and Mac build machines using an OS-specific labels. But for the sake of keeping these repro steps simple, I've dropped all but one OS and removed the "Dynamic Axis" plugin usage. It doesn't logically make much sense but shows the behavior difference starting with the Jenkins 2.205 release.
acampeau Thanks for these steps, I'll try to reproduce them when I have some time.
About
Configure this job's "Restrict where this project can run" setting so its "Label Expression" value is the one specified for master, so DISPATCH
What happens when you don't check that box, or specify "master" here? Would that be a viable workaround for this problem, and if not, why not?
Daniel Beck
added a comment - acampeau Thanks for these steps, I'll try to reproduce them when I have some time.
About
Configure this job's "Restrict where this project can run" setting so its "Label Expression" value is the one specified for master, so DISPATCH
What happens when you don't check that box, or specify "master" here? Would that be a viable workaround for this problem, and if not, why not?
I have pinned Jenkins to 2.204.6 on our sever to be able to use it, but it is increasingly complaining about unfixed vulnerabilities.
Peter Krefting
added a comment - Any news on this?
I have pinned Jenkins to 2.204.6 on our sever to be able to use it, but it is increasingly complaining about unfixed vulnerabilities.
The problem persists in the current version (2.289.3), when I try to start a tied job manually after upgrading to this version, it blocks in "Waiting for next available executor on 'master'" despite no other jobs being active across the system.
Peter Krefting
added a comment - The problem persists in the current version (2.289.3), when I try to start a tied job manually after upgrading to this version, it blocks in "Waiting for next available executor on 'master'" despite no other jobs being active across the system.
We tried to use the Job-Restriction plugin to allow only a certain job on master. But that also blocks the execution of other jobs even though they are actually not executed on the master but (always) on slaves.
For these "slave jobs" it says in the build queue: "(pending—Waiting for next available executor on ‘master’)".
So it looks to be caused by this issue.
Valentin Maechler
added a comment - What's the status of this issue?
We tried to use the Job-Restriction plugin to allow only a certain job on master. But that also blocks the execution of other jobs even though they are actually not executed on the master but (always) on slaves.
For these "slave jobs" it says in the build queue: "(pending—Waiting for next available executor on ‘master’)".
So it looks to be caused by this issue.
Unassigned
Peter Krefting
Votes:
3Vote for this issue
Watchers:
7Start watching this issue
Created:
Updated:
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1741401201318},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1741401201319},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1741401201342},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":true},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true,"lastViewed":1741401201350},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Bonfire Browser","id":"customfield_10229","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Operating System","id":"customfield_10231","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Screen Resolution","id":"customfield_10244","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire URL","id":"customfield_10237","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire User Agent","id":"customfield_10226","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire jQuery Version","id":"customfield_10252","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Business Value","id":"customfield_10333","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Development","id":"customfield_10720","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic Color","id":"customfield_10328","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10325","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10327","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10326","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10331","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Flagged","id":"customfield_10330","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"GitHub Users to Authorize as Committers","id":"customfield_10323","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Issue Tracker","id":"customfield_11320","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10228","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10230","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Document Mode","id":"customfield_10258","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10232","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10233","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10234","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10236","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10238","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10239","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10245","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10246","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10247","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10248","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10249","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10250","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10240","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10241","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10242","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10243","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10225","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10227","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10251","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10253","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10254","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10255","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10256","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10257","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Meeting minutes URL","id":"customfield_10020","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":false},{"name":"New Repository Name","id":"customfield_10321","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Original story points","id":"customfield_11423","key":"com.atlassian.jpo:jpo-custom-field-original-story-points","isShown":true},{"name":"Parent Link","id":"customfield_11420","key":"com.atlassian.jpo:jpo-custom-field-parent","isShown":false},{"name":"Plugin Description","id":"customfield_10322","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Raised During","id":"customfield_10220","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Raised During","id":"customfield_10221","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Rank","id":"customfield_10324","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Released As","id":"customfield_10620","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Repository URL","id":"customfield_10320","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true},{"name":"Similar Issues","id":"customfield_10520","key":"com.suggestimate:similar-issues-custom-field","isShown":true},{"name":"Sprint","id":"customfield_10329","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_10332","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Team","id":"customfield_11424","key":"com.atlassian.teams:rm-teams-custom-field-team","isShown":true},{"name":"Test Sessions","id":"customfield_10222","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10223","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10224","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Testing Status","id":"customfield_10259","key":"com.atlassian.bonfire.plugin:bonfire-testing-status-cft","isShown":false},{"name":"URL","id":"customfield_10000","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Target end","id":"customfield_11422","key":"com.atlassian.jpo:jpo-custom-field-baseline-end","isShown":true},{"name":"Target start","id":"customfield_11421","key":"com.atlassian.jpo:jpo-custom-field-baseline-start","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1741401201343},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\"\n id=\"searcher-type\"\n multiple=\"multiple\"\n name=\"type\"\n data-max-inline-results-displayed=\"100\"\n data-placeholder-text=\"Find Issue Types...\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14673&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/images/icons/issuetypes/epic.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10001\"\n title=\"Epic\"\n value=\"10001\">Epic</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14680&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_4\"\n title=\"Improvement\"\n value=\"4\">Improvement</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14681&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14670&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Patch\"\n value=\"5\">Patch</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14685&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10002\"\n title=\"Story\"\n value=\"10002\">Story</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14688&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":" \n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> <select class=\"js-project-checkboxmultiselect\"\n data-placeholder-text=\"Find Projects...\"\n id=\"searcher-pid\"\n multiple=\"multiple\"\n name=\"pid\">\n <optgroup label=\"Recent Projects\">\n </optgroup>\n <optgroup label=\"All Projects\" >\n \n <option data-icon=\"/secure/projectavatar?pid=10172&size=small\"\n title=\"Jenkins\"\n value=\"10172\">\n Jenkins (JENKINS)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10050&size=small\"\n title=\"test\"\n value=\"10050\">\n test (TEST)\n </option>\n </optgroup>\n </select>\n </div>\n \n\n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://issues.jenkins.io/secure/useravatar?size=xsmall&avatarId=10293\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> <select class=\"select js-default-checkboxmultiselectstatuslozenge\"\n data-placeholder-text=\"Find Statuses...\"\n id=\"searcher-status\"\n multiple=\"multiple\"\n name=\"status\"\n data-max-inline-results-displayed=\"100\"\n data-footer-text=\"-88 more options. Continue typing to refine further.\" data-status-lozenge=\"true\">\n <optgroup >\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/needinfo.png\" value=\"10001\" title=\"Untriaged\" data-simple-status=\"{"id":"10001","name":"Untriaged","description":"New issue sits in this state until a developer agrees that this is a security issue","iconUrl":"/images/icons/statuses/needinfo.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Untriaged</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"The issue is open and ready for the assignee to start work on it.","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/reopened.png\" value=\"4\" title=\"Reopened\" data-simple-status=\"{"id":"4","name":"Reopened","description":"This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.","iconUrl":"/images/icons/statuses/reopened.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Reopened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10005\" title=\"In Review\" data-simple-status=\"{"id":"10005","name":"In Review","description":"","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"10002\" title=\"Fix Prepared\" data-simple-status=\"{"id":"10002","name":"Fix Prepared","description":"A fix is implemented and is waiting for the next security release","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Fix Prepared</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10000\" title=\"Verified\" data-simple-status=\"{"id":"10000","name":"Verified","description":"Verified","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10203\" title=\"Fixed but Unreleased\" data-simple-status=\"{"id":"10203","name":"Fixed but Unreleased","description":"This change has been implemented and merged, but not yet released.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Fixed but Unreleased</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"5\" title=\"Resolved\" data-simple-status=\"{"id":"5","name":"Resolved","description":"A developer had implemented a fix and is waiting for a feedback from the reporter.","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Resolved</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10003\" title=\"To Do\" data-simple-status=\"{"id":"10003","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10004\" title=\"Done\" data-simple-status=\"{"id":"10004","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Done</option>\n </optgroup>\n</select>\n </div>\n \n","validSearcher":true,"isShown":true},"labels":{"name":"Label","viewHtml":" <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldlabels\">Label:</label><span id=\"fieldlabels\" class=\"fieldValue\">\n \n regression\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-labels\" >\n <label for=\"searcher-labels\">Labels</label> <select class=\"js-label-checkboxmultiselect\" multiple=\"multiple\" id=\"searcher-labels\" name=\"labels\" data-placeholder-text=\"Find Labels...\">\n <option value=\"regression\" title=\"regression\" selected=\"selected\">regression</option>\n </select>\n </div>\n \n","jql":"labels = regression","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Bug
Blocker
Alain Campeau
added a comment - 
Chris McAfee
added a comment - - edited We are seeing this problem on LTS 2.222.4
oleg_nenashev , can you take a look at this one? 
Peter Krefting
added a comment - 
Valentin Maechler
added a comment - 
I have the same issue. I'm on 2.222.1. It ONLY happened when I upgraded to 2.222.1 from 2.204.1
I have a label_expression on a matrix project set to "master_node".
When I run it, it says "(pending—Waiting for next available executor on ‘master’)"
Note that it says "master", not "master_node". I have no idea why it's looking for something called "master", not "master_node".
I'm attaching a gif of what I see.
Here are the logs that Daniel Beck requested too:
Apr 08, 2020 12:16:14 AM FINEST hudson.model.Queue Queue.Snapshot{waitingList=[];blockedProjects=[hudson.model.Queue$BlockedItem:hudson.model.FreeStyleProject@502099a9[util-slave-manager]:152871];buildables=[hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@598a8bc[util-check-for-container-spread]:152734, hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@4fe1c89a[util-generate-datadog-monitors]:152745];pendings=[]} → Queue.Snapshot{waitingList=[];blockedProjects=[hudson.model.Queue$BlockedItem:hudson.model.FreeStyleProject@502099a9[util-slave-manager]:152871];buildables=[hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@598a8bc[util-check-for-container-spread]:152734, hudson.model.Queue$BuildableItem:hudson.matrix.MatrixProject@4fe1c89a[util-generate-datadog-monitors]:152745];pendings=[]}; leftItems={152845=hudson.model.Queue$LeftItem:hudson.matrix.MatrixProject@33385024[util-check-for-low-ips]:152845, 152866=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@2e88aabc[util-check-hung-ecs-tasks-prod-ca]:152866, 152865=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@1c17e4db[util-check-hung-ecs-tasks-prod]:152865, 152819=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@502099a9[util-slave-manager]:152819, 152868=hudson.model.Queue$LeftItem:ExecutorStepExecution.PlaceholderTask{runId=util-alert-jenkins-vs-jumpcloud-users#4201,label=,context=CpsStepContext[4:node]:Owner[util-alert-jenkins-vs-jumpcloud-users/4201:util-alert-jenkins-vs-jumpcloud-users #4201],cookie=f8461e2f-66d2-48fb-9acc-79bb1b98a0ad,auth=null}:152868, 152867=hudson.model.Queue$LeftItem:org.jenkinsci.plugins.workflow.job.WorkflowJob@13b79664[util-alert-jenkins-vs-jumpcloud-users]:152867, 152870=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@188db571[util-check-hung-ecs-tasks-stage]:152870, 152872=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@1ae51106[databases-without-termination-protection]:152872, 152869=hudson.model.Queue$LeftItem:hudson.model.FreeStyleProject@60957b04[util-check-asg-thrashing]:152869}