* Jenkins 2.176.2
* Master(x1) and Build(x6) nodes are running Windows 10 64B Enterprise
* Java Version 8 Update 201 (build 1.8.0_201-b09) Oracle
* Jenkins is running on Master from command line call
* Jenkins is running on Slaves as a service
* Web browser: Chrome Version 76.0.3809.132 (Official Build) (64-bit)
* Jenkins 2.176.2
* Master(x1) and Build(x6) nodes are running Windows 10 64B Enterprise
* Java Version 8 Update 201 (build 1.8.0_201-b09) Oracle
* Jenkins is running on Master from command line call
* Jenkins is running on Slaves as a service
* Web browser: Chrome Version 76.0.3809.132 (Official Build) (64-bit)
Jenkins slave nodes hangs for up to 12+ minutes after build phase completes.
I have a master node and 7 nodes in the build pool. The master is configured to only run jobs labeled master. Builds are always only done from the build nodes.
There are no post-build steps configured.
Hang/delay is observed if the build step passes and/or fails.
Example Console Output follows
---------------------------------------------------------------------------
....
2019-09-16 16:51:45 ERROR: last command returned failure: 1
2019-09-16 16:51:45
2019-09-16 16:51:45 build.bat failed with error code '1'
2019-09-16 16:51:45
2019-09-16 16:51:45 Build step 'Execute Windows batch command' marked build as failure
2019-09-16 17:03:48 Finished: FAILURE
---------------------------------------------------------------------------
Observe that at 16:51:45 the Build step for Execute Windows batch command finishes. However the build step continues for another 12 minutes before finally completed with the FAILURE notification.
There are 6 nodes in the build pool being triggered for the builds
The issue doesn't always happen. It seems to depend upon what is being done in the job. For example, if the job is to update perforce there is no noticeable delay. If it's to do a simple compile using Visual Studio there doesn't seem to be a delay for that either. By simple compile I mean few objects that don't seem to trigger parallel compilation.
When it does happen it appears to be with bigger Visual Studio builds that have parallel compilation enabled. Moreover, I've noticed that there may be multiple MSBuild.exe processes still running even after Jenkins reports "build.bat existing with success"
For example, during my most recent reproduction, there were 5 MSBuild.exe processes lingering after Jenkins reported the script exited with success, but the build didn't return the final result until ~8m later.
The MSBuild.exe processes would slowly go away one by one.
Once all of the MSBuild.exe processes terminated, the Jenkins job reported it's final "Finished: SUCCESS" result.
John Rocha
added a comment - Update:
The issue doesn't always happen. It seems to depend upon what is being done in the job. For example, if the job is to update perforce there is no noticeable delay. If it's to do a simple compile using Visual Studio there doesn't seem to be a delay for that either. By simple compile I mean few objects that don't seem to trigger parallel compilation.
When it does happen it appears to be with bigger Visual Studio builds that have parallel compilation enabled. Moreover, I've noticed that there may be multiple MSBuild.exe processes still running even after Jenkins reports "build.bat existing with success"
For example, during my most recent reproduction, there were 5 MSBuild.exe processes lingering after Jenkins reported the script exited with success, but the build didn't return the final result until ~8m later.
The MSBuild.exe processes would slowly go away one by one.
Once all of the MSBuild.exe processes terminated, the Jenkins job reported it's final " Finished: SUCCESS " result.
If parallel compiles are enabled and used, the default is for the MSBuild.exe process to stay around so it can be re-used by future compiles. This seems to cause a problem with the remote jenkins build pools.
The MSBuild.exe reuse/linger functionality can be disabled by passing /nr:false for the build process.
When I added this flag it resolved my issue.
This problem doesn't happen if I am building without a build pool (i.e. one jenkins node that does it's own compiles). It only occurs when I go to a master/slave-build-pool scenario. Then it occurs when building on the slave nodes.
John Rocha
added a comment - Root cause - User calling error
I ran the script manually from the CLI and observed that the MSBuild.exe processes never went away. Ever.
I Googled for this and found this stackoverflow description/solution
If parallel compiles are enabled and used, the default is for the MSBuild.exe process to stay around so it can be re-used by future compiles. This seems to cause a problem with the remote jenkins build pools.
The MSBuild.exe reuse/linger functionality can be disabled by passing /nr:false for the build process.
When I added this flag it resolved my issue.
This problem doesn't happen if I am building without a build pool (i.e. one jenkins node that does it's own compiles). It only occurs when I go to a master/slave-build-pool scenario. Then it occurs when building on the slave nodes.
Unassigned
John Rocha
Votes:
0Vote for this issue
Watchers:
1Start watching this issue
Created:
Updated:
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1741241102732},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1741241102732},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1741241102770},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":true},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true,"lastViewed":1741241102778},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Bonfire Browser","id":"customfield_10229","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Operating System","id":"customfield_10231","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Screen Resolution","id":"customfield_10244","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire URL","id":"customfield_10237","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire User Agent","id":"customfield_10226","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire jQuery Version","id":"customfield_10252","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Business Value","id":"customfield_10333","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Development","id":"customfield_10720","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic Color","id":"customfield_10328","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10325","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10327","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10326","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10331","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Flagged","id":"customfield_10330","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"GitHub Users to Authorize as Committers","id":"customfield_10323","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Issue Tracker","id":"customfield_11320","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10228","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10230","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Document Mode","id":"customfield_10258","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10232","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10233","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10234","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10236","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10238","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10239","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10245","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10246","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10247","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10248","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10249","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10250","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10240","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10241","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10242","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10243","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10225","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10227","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10251","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10253","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10254","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10255","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10256","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10257","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Meeting minutes URL","id":"customfield_10020","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":false},{"name":"New Repository Name","id":"customfield_10321","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Original story points","id":"customfield_11423","key":"com.atlassian.jpo:jpo-custom-field-original-story-points","isShown":true},{"name":"Parent Link","id":"customfield_11420","key":"com.atlassian.jpo:jpo-custom-field-parent","isShown":false},{"name":"Plugin Description","id":"customfield_10322","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Raised During","id":"customfield_10220","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Raised During","id":"customfield_10221","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Rank","id":"customfield_10324","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Released As","id":"customfield_10620","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Repository URL","id":"customfield_10320","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true},{"name":"Similar Issues","id":"customfield_10520","key":"com.suggestimate:similar-issues-custom-field","isShown":true},{"name":"Sprint","id":"customfield_10329","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_10332","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Team","id":"customfield_11424","key":"com.atlassian.teams:rm-teams-custom-field-team","isShown":true},{"name":"Test Sessions","id":"customfield_10222","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10223","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10224","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Testing Status","id":"customfield_10259","key":"com.atlassian.bonfire.plugin:bonfire-testing-status-cft","isShown":false},{"name":"URL","id":"customfield_10000","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Target end","id":"customfield_11422","key":"com.atlassian.jpo:jpo-custom-field-baseline-end","isShown":true},{"name":"Target start","id":"customfield_11421","key":"com.atlassian.jpo:jpo-custom-field-baseline-start","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1741241102771},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\"\n id=\"searcher-type\"\n multiple=\"multiple\"\n name=\"type\"\n data-max-inline-results-displayed=\"100\"\n data-placeholder-text=\"Find Issue Types...\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14673&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/images/icons/issuetypes/epic.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10001\"\n title=\"Epic\"\n value=\"10001\">Epic</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14680&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_4\"\n title=\"Improvement\"\n value=\"4\">Improvement</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14681&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14670&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Patch\"\n value=\"5\">Patch</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14685&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10002\"\n title=\"Story\"\n value=\"10002\">Story</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14688&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":" \n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> <select class=\"js-project-checkboxmultiselect\"\n data-placeholder-text=\"Find Projects...\"\n id=\"searcher-pid\"\n multiple=\"multiple\"\n name=\"pid\">\n <optgroup label=\"Recent Projects\">\n </optgroup>\n <optgroup label=\"All Projects\" >\n \n <option data-icon=\"/secure/projectavatar?pid=10172&size=small\"\n title=\"Jenkins\"\n value=\"10172\">\n Jenkins (JENKINS)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10050&size=small\"\n title=\"test\"\n value=\"10050\">\n test (TEST)\n </option>\n </optgroup>\n </select>\n </div>\n \n\n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://issues.jenkins.io/secure/useravatar?size=xsmall&avatarId=10293\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> <select class=\"select js-default-checkboxmultiselectstatuslozenge\"\n data-placeholder-text=\"Find Statuses...\"\n id=\"searcher-status\"\n multiple=\"multiple\"\n name=\"status\"\n data-max-inline-results-displayed=\"100\"\n data-footer-text=\"-88 more options. Continue typing to refine further.\" data-status-lozenge=\"true\">\n <optgroup >\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/needinfo.png\" value=\"10001\" title=\"Untriaged\" data-simple-status=\"{"id":"10001","name":"Untriaged","description":"New issue sits in this state until a developer agrees that this is a security issue","iconUrl":"/images/icons/statuses/needinfo.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Untriaged</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"The issue is open and ready for the assignee to start work on it.","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/reopened.png\" value=\"4\" title=\"Reopened\" data-simple-status=\"{"id":"4","name":"Reopened","description":"This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.","iconUrl":"/images/icons/statuses/reopened.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Reopened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10005\" title=\"In Review\" data-simple-status=\"{"id":"10005","name":"In Review","description":"","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"10002\" title=\"Fix Prepared\" data-simple-status=\"{"id":"10002","name":"Fix Prepared","description":"A fix is implemented and is waiting for the next security release","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Fix Prepared</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10000\" title=\"Verified\" data-simple-status=\"{"id":"10000","name":"Verified","description":"Verified","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10203\" title=\"Fixed but Unreleased\" data-simple-status=\"{"id":"10203","name":"Fixed but Unreleased","description":"This change has been implemented and merged, but not yet released.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Fixed but Unreleased</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"5\" title=\"Resolved\" data-simple-status=\"{"id":"5","name":"Resolved","description":"A developer had implemented a fix and is waiting for a feedback from the reporter.","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Resolved</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10003\" title=\"To Do\" data-simple-status=\"{"id":"10003","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10004\" title=\"Done\" data-simple-status=\"{"id":"10004","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Done</option>\n </optgroup>\n</select>\n </div>\n \n","validSearcher":true,"isShown":true},"labels":{"name":"Label","viewHtml":" <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldlabels\">Label:</label><span id=\"fieldlabels\" class=\"fieldValue\">\n \n delay\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-labels\" >\n <label for=\"searcher-labels\">Labels</label> <select class=\"js-label-checkboxmultiselect\" multiple=\"multiple\" id=\"searcher-labels\" name=\"labels\" data-placeholder-text=\"Find Labels...\">\n <option value=\"delay\" title=\"delay\" selected=\"selected\">delay</option>\n </select>\n </div>\n \n","jql":"labels = delay","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Bug
Critical
John Rocha
added a comment - 
Update:
The issue doesn't always happen. It seems to depend upon what is being done in the job. For example, if the job is to update perforce there is no noticeable delay. If it's to do a simple compile using Visual Studio there doesn't seem to be a delay for that either. By simple compile I mean few objects that don't seem to trigger parallel compilation.
When it does happen it appears to be with bigger Visual Studio builds that have parallel compilation enabled. Moreover, I've noticed that there may be multiple MSBuild.exe processes still running even after Jenkins reports "build.bat existing with success"
For example, during my most recent reproduction, there were 5 MSBuild.exe processes lingering after Jenkins reported the script exited with success, but the build didn't return the final result until ~8m later.
The MSBuild.exe processes would slowly go away one by one.
Once all of the MSBuild.exe processes terminated, the Jenkins job reported it's final "Finished: SUCCESS" result.