-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Hudson ver. 1.393, matrix project, matrixtieparent plugin, post-build-batch-task plugin
I have a matrix project that completely builds on a slave node ("Build on multiple nodes", "Tie parent build to a node" vie matrixtieparent plugin). As a "Post-build Action", "Invoke batch tasks" is configured to run a batch task of the same project. Usually, this works fine and the batch task is execured after the matrix configurations are build. However, when the salve node becomes idle, the bach task is still in the build queue and stays there forever with "pending - Waiting for next available executor on MySlaveNode").
When a new job is added to the build queue, the pending job is finally executed.
This usually happen when may builds are in the queue. However, we also see it when hudson is idle and we start one project build manually.
- is related to
-
-
- Open
-
[JENKINS-7033] Job in build queue is not executed
Axel Heider
added a comment - Unfortunately, we have not found a way to repoduce this every time. However, there is a high change that this happen when I start a jon manually with master and slave idle.
Configuration:
Master: Windows, 4 executors
Slave: Linux, 2 executors, launched via SSH, "use exclusive for tied jobs, keep slave online as much as possible.
Axel Heider
added a comment - Unfortunately, we have not found a way to repoduce this every time. However, there is a high change that this happen when I start a jon manually with master and slave idle.
Configuration:
Master: Windows, 4 executors
Slave: Linux, 2 executors, launched via SSH, "use exclusive for tied jobs, keep slave online as much as possible. Axel Heider
added a comment - I wonder that I'm the only one seeing this happen - and it appears do be reproducible every time by now.
Axel Heider
added a comment - I wonder that I'm the only one seeing this happen - and it appears do be reproducible every time by now. Now I have found a way to reproduce this always:
- Configuratuon: Windows Master, Linux Salve
- Matrix Job triggered by SVN commit runs fully on Linux Slave (also the parent vie matrix-tie-parent plugin)
- Job has one Batch task configured
- this batch Task is triggered as post build action (via "Invoke batch tasks")
When I start the build job manually, all matrix builds run and the batch task is put in the queue. But then the batch task does not run, it stays in thew queue. And it stays in the queue until another job arrives in the queue.
So, is it a problem with the post-build-Invoke-batch-task-plugin and the way it adds a job? It appears the problem always occurs for job on slaves, it occurs rarely for jobs on the master node.
Axel Heider
added a comment - - edited Now I have found a way to reproduce this always:
Configuratuon: Windows Master, Linux Salve
Matrix Job triggered by SVN commit runs fully on Linux Slave (also the parent vie matrix-tie-parent plugin)
Job has one Batch task configured
this batch Task is triggered as post build action (via "Invoke batch tasks")
When I start the build job manually, all matrix builds run and the batch task is put in the queue. But then the batch task does not run, it stays in thew queue. And it stays in the queue until another job arrives in the queue.
So, is it a problem with the post-build-Invoke-batch-task-plugin and the way it adds a job? It appears the problem always occurs for job on slaves, it occurs rarely for jobs on the master node.
Axel Heider
added a comment - Problem seen with Hudson 1.393 and 1.394 in Windows Master today. A matrix project building on the windows mater is configured to run 3 batch jobs after building. All three job remain in the build queue without getting executed.
Work around is now that an empty dummy project is build every 10 minutes so the build queue gets flushed.
Axel Heider
added a comment - Problem seen with Hudson 1.393 and 1.394 in Windows Master today. A matrix project building on the windows mater is configured to run 3 batch jobs after building. All three job remain in the build queue without getting executed.
Work around is now that an empty dummy project is build every 10 minutes so the build queue gets flushed. Axel Heider
added a comment - Problem still exists in 1.396 Axel Heider
added a comment - Todays I've seen another strange issue, however I canno reproduce it. I've set up a Dummy job that is triggered by many matrix jobs when they are done. This is supposed to add something to the build queue, so the neglected batch tasks from the matrix jobs waiting there finally get triggered. Now the Dummy job is also waiting in the build queue with the timeout/clock symbol. All executors are available but nothing happens. If I start the Dummy job manually, nothing happens, too. I have to add a new job to the queue to get it finally flushed to the executors.
Please have a review on the build queue handling.
Axel Heider
added a comment - Todays I've seen another strange issue, however I canno reproduce it. I've set up a Dummy job that is triggered by many matrix jobs when they are done. This is supposed to add something to the build queue, so the neglected batch tasks from the matrix jobs waiting there finally get triggered. Now the Dummy job is also waiting in the build queue with the timeout/clock symbol. All executors are available but nothing happens. If I start the Dummy job manually, nothing happens, too. I have to add a new job to the queue to get it finally flushed to the executors.
Please have a review on the build queue handling. Axel Heider
added a comment - Have not seen this for a long time now, assume it was some internal race condition that has ben resolved.
Axel Heider
added a comment - Have not seen this for a long time now, assume it was some internal race condition that has ben resolved. I do have the problem on Jenkins 1.518 running on Windows and one specific job started to behave like this.
SCM pollling (git) finds changes and adds the job to the queue. The queue is configured to wait for 300 seconds before building and the polling is configured at every 2 minutes.
The job ends-up always in the queue, never building but if I manually try to build it it will work.
C:\JenkinsWar>java -DJENKINS_HOME=C:\Jenkins -jar jenkins.war Running from: C:\JenkinsWar\jenkins.war webroot: System.getProperty("JENKINS_HOME") Jun 20, 2013 11:20:44 AM winstone.Logger logInternal INFO: Beginning extraction from war file Jenkins home directory: C:\Jenkins found at: System.getProperty("JENKINS_HOME") Jun 20, 2013 11:20:53 AM winstone.Logger logInternal INFO: HTTP Listener started: port=8080 Jun 20, 2013 11:20:53 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled Jun 20, 2013 11:20:55 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization Jun 20, 2013 11:20:56 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin translation.jpi is disabled Jun 20, 2013 11:20:56 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins Jun 20, 2013 11:20:56 AM hudson.plugins.greenballs.PluginImpl start INFO: Green Balls! Jun 20, 2013 11:20:56 AM jenkins.InitReactorRunner$1 onAttained INFO: Prepared all plugins Jun 20, 2013 11:21:01 AM jenkins.InitReactorRunner$1 onAttained INFO: Started all plugins Jun 20, 2013 11:21:15 AM jenkins.InitReactorRunner$1 onAttained INFO: Augmented all extensions Jun 20, 2013 11:21:17 AM jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs Jun 20, 2013 11:21:19 AM org.jenkinsci.main.modules.sshd.SSHD start INFO: Started SSHD at port 49165 Jun 20, 2013 11:21:19 AM jenkins.InitReactorRunner$1 onAttained INFO: Completed initialization Jun 20, 2013 11:21:19 AM hudson.TcpSlaveAgentListener <init> INFO: JNLP slave agent listener started on TCP port 49166 Jun 20, 2013 11:21:19 AM hudson.WebAppMain$2 run INFO: Jenkins is fully up and running Jun 20, 2013 11:21:53 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Triggering #58 Jun 20, 2013 11:23:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:25:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:27:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:29:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:31:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:33:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:35:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue
Can you find/post steps to reproduce the issue? How are the slave nodes configured?