Since the update to kubernetes plugin version 1.27.2 we've found that build tasks are canceled and terminated with "FAILURE" status when the namespace's ResourceQuota is temporarily exceeded.
Steps to reproduce:
Create a resource quota in namespace where the kubernetes plugin creates pods
Manually run a pod that uses up most or all of the quota, e.g. with
kubectl run quotauserpod --command --attach=false -i --image=alpine:latest --requests='cpu=2,memory=100Mi' --limits='cpu=2,memory=100Mi' -- sh -c 'cat'
Run a Jenkins job using the kubernetes plugin that has resource requests and limits that would fit into the quota if no other pods were running
Expected behaviour: Jenkins retries creating the pod until it succeeds
Actual behaviour: Jenkins cancels the job, sets its status to "FAILURE" and logs the following error message:
14:56:12 ERROR: Unable to create pod podnamespace/buildpodname.
14:56:12 Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/podnamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "buildpodname" is forbidden: exceeded quota: quota-s, requested: limits.cpu=2500m,requests.cpu=2, used: limits.cpu=2,requests.cpu=2, limited: limits.cpu=2500m,requests.cpu=2500m.
I have downgraded the kubernetes plugin to 1.26.4 and have been able to verify that the scheduling works as expected, i.e. jenkins retries pod creation until it succeeds.
This is a major issue for us since we run Jenkins as a service for our developers and can't control job scheduling to ensure that all parallel running jobs don't exceed the resource quota. Moreover, some job runtimes vary widely, which would make scheduling even more challenging.
I believe the new behaviour was introduced in commit 6e7c0c374ac53244aabc1a9008aed250c5937ac0 (plugin v1.27.1).
It seems that with this change all HTTP 4xx-errors errors lead to the behaviour described above, including when a namespace's resource quota has been exceeded.
dlandtwing, thank you for creating this issue. We have just arrived to the same conclusion - the behaviour of scheduling dynamic agents on k8s changed.
It affects us in the same way - we do not want to give unrestricted resources to jenkins namespace and have to impose quote, but due to the dynamic nature of tasks, sometimes the quota is exceeded and it is reasonable to expect to retry. Once the other jobs finish and destroy pods, new pods would be able to be scheduled.
Egor Ermakov
added a comment - dlandtwing , thank you for creating this issue. We have just arrived to the same conclusion - the behaviour of scheduling dynamic agents on k8s changed.
It affects us in the same way - we do not want to give unrestricted resources to jenkins namespace and have to impose quote, but due to the dynamic nature of tasks, sometimes the quota is exceeded and it is reasonable to expect to retry. Once the other jobs finish and destroy pods, new pods would be able to be scheduled.
We have the same use case as kenota. We are using a multi tenant Kubernetes and every team have their own quotas for scheduling builds. For the user experience is very bad to have the build failed without meaningful message.
The previous behavior was better, retrying until some resources are free (other builds to finish)
Thanks
Valentin Delaye
added a comment - Hi,
We have the same use case as kenota . We are using a multi tenant Kubernetes and every team have their own quotas for scheduling builds. For the user experience is very bad to have the build failed without meaningful message.
The previous behavior was better, retrying until some resources are free (other builds to finish)
Thanks
It just got merged recently. So expect the change should be in a near-future release of the plugin, or can install a copy now from the recent master build output.
Charles K
added a comment - - edited I submitted a PR which restores the behavior prior to v1.27.1 (retry rather than fail if quota exceeded / conflict): https://github.com/jenkinsci/kubernetes-plugin/pull/930
It just got merged recently. So expect the change should be in a near-future release of the plugin, or can install a copy now from the recent master build output.
Unassigned
Dominik Landtwing
Votes:
6Vote for this issue
Watchers:
10Start watching this issue
Created:
Updated:
Resolved:
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1740709992090},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1740709992090},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1740709992095},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":true},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true,"lastViewed":1740709992102},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Bonfire Browser","id":"customfield_10229","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Operating System","id":"customfield_10231","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Screen Resolution","id":"customfield_10244","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire URL","id":"customfield_10237","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire User Agent","id":"customfield_10226","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire jQuery Version","id":"customfield_10252","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Business Value","id":"customfield_10333","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Development","id":"customfield_10720","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic Color","id":"customfield_10328","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10325","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10327","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10326","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10331","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Flagged","id":"customfield_10330","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"GitHub Users to Authorize as Committers","id":"customfield_10323","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Issue Tracker","id":"customfield_11320","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10228","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10230","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Document Mode","id":"customfield_10258","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10232","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10233","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10234","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10236","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10238","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10239","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10245","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10246","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10247","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10248","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10249","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10250","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10240","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10241","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10242","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10243","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10225","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10227","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10251","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10253","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10254","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10255","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10256","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10257","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Meeting minutes URL","id":"customfield_10020","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":false},{"name":"New Repository Name","id":"customfield_10321","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Original story points","id":"customfield_11423","key":"com.atlassian.jpo:jpo-custom-field-original-story-points","isShown":true},{"name":"Parent Link","id":"customfield_11420","key":"com.atlassian.jpo:jpo-custom-field-parent","isShown":false},{"name":"Plugin Description","id":"customfield_10322","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Raised During","id":"customfield_10220","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Raised During","id":"customfield_10221","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Rank","id":"customfield_10324","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Released As","id":"customfield_10620","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Repository URL","id":"customfield_10320","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true},{"name":"Similar Issues","id":"customfield_10520","key":"com.suggestimate:similar-issues-custom-field","isShown":true},{"name":"Sprint","id":"customfield_10329","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_10332","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Team","id":"customfield_11424","key":"com.atlassian.teams:rm-teams-custom-field-team","isShown":true},{"name":"Test Sessions","id":"customfield_10222","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10223","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10224","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Testing Status","id":"customfield_10259","key":"com.atlassian.bonfire.plugin:bonfire-testing-status-cft","isShown":false},{"name":"URL","id":"customfield_10000","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Target end","id":"customfield_11422","key":"com.atlassian.jpo:jpo-custom-field-baseline-end","isShown":true},{"name":"Target start","id":"customfield_11421","key":"com.atlassian.jpo:jpo-custom-field-baseline-start","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1740709992097},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\"\n id=\"searcher-type\"\n multiple=\"multiple\"\n name=\"type\"\n data-max-inline-results-displayed=\"100\"\n data-placeholder-text=\"Find Issue Types...\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14673&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/images/icons/issuetypes/epic.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10001\"\n title=\"Epic\"\n value=\"10001\">Epic</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14680&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_4\"\n title=\"Improvement\"\n value=\"4\">Improvement</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14681&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14670&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Patch\"\n value=\"5\">Patch</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14685&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10002\"\n title=\"Story\"\n value=\"10002\">Story</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14688&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":" \n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> <select class=\"js-project-checkboxmultiselect\"\n data-placeholder-text=\"Find Projects...\"\n id=\"searcher-pid\"\n multiple=\"multiple\"\n name=\"pid\">\n <optgroup label=\"Recent Projects\">\n </optgroup>\n <optgroup label=\"All Projects\" >\n \n <option data-icon=\"/secure/projectavatar?pid=10172&size=small\"\n title=\"Jenkins\"\n value=\"10172\">\n Jenkins (JENKINS)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10050&size=small\"\n title=\"test\"\n value=\"10050\">\n test (TEST)\n </option>\n </optgroup>\n </select>\n </div>\n \n\n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://issues.jenkins.io/secure/useravatar?size=xsmall&avatarId=10293\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> <select class=\"select js-default-checkboxmultiselectstatuslozenge\"\n data-placeholder-text=\"Find Statuses...\"\n id=\"searcher-status\"\n multiple=\"multiple\"\n name=\"status\"\n data-max-inline-results-displayed=\"100\"\n data-footer-text=\"-88 more options. Continue typing to refine further.\" data-status-lozenge=\"true\">\n <optgroup >\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/needinfo.png\" value=\"10001\" title=\"Untriaged\" data-simple-status=\"{"id":"10001","name":"Untriaged","description":"New issue sits in this state until a developer agrees that this is a security issue","iconUrl":"/images/icons/statuses/needinfo.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Untriaged</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"The issue is open and ready for the assignee to start work on it.","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/reopened.png\" value=\"4\" title=\"Reopened\" data-simple-status=\"{"id":"4","name":"Reopened","description":"This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.","iconUrl":"/images/icons/statuses/reopened.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Reopened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10005\" title=\"In Review\" data-simple-status=\"{"id":"10005","name":"In Review","description":"","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"10002\" title=\"Fix Prepared\" data-simple-status=\"{"id":"10002","name":"Fix Prepared","description":"A fix is implemented and is waiting for the next security release","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Fix Prepared</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10000\" title=\"Verified\" data-simple-status=\"{"id":"10000","name":"Verified","description":"Verified","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10203\" title=\"Fixed but Unreleased\" data-simple-status=\"{"id":"10203","name":"Fixed but Unreleased","description":"This change has been implemented and merged, but not yet released.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Fixed but Unreleased</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"5\" title=\"Resolved\" data-simple-status=\"{"id":"5","name":"Resolved","description":"A developer had implemented a fix and is waiting for a feedback from the reporter.","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Resolved</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10003\" title=\"To Do\" data-simple-status=\"{"id":"10003","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10004\" title=\"Done\" data-simple-status=\"{"id":"10004","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Done</option>\n </optgroup>\n</select>\n </div>\n \n","validSearcher":true,"isShown":true},"labels":{"name":"Label","viewHtml":" <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldlabels\">Label:</label><span id=\"fieldlabels\" class=\"fieldValue\">\n \n quota\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-labels\" >\n <label for=\"searcher-labels\">Labels</label> <select class=\"js-label-checkboxmultiselect\" multiple=\"multiple\" id=\"searcher-labels\" name=\"labels\" data-placeholder-text=\"Find Labels...\">\n <option value=\"quota\" title=\"quota\" selected=\"selected\">quota</option>\n </select>\n </div>\n \n","jql":"labels = quota","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
dlandtwing, thank you for creating this issue. We have just arrived to the same conclusion - the behaviour of scheduling dynamic agents on k8s changed.
It affects us in the same way - we do not want to give unrestricted resources to jenkins namespace and have to impose quote, but due to the dynamic nature of tasks, sometimes the quota is exceeded and it is reasonable to expect to retry. Once the other jobs finish and destroy pods, new pods would be able to be scheduled.