-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Jenkins 1.580.1 LTS and Gearman plugin 0.1.1
Jenkins 1.625.3 LTS and Gearman plugin 1.3.3 with https://review.openstack.org/#/c/252768/
We have a setup with one Jenkins master and Zuul triggers the job through the Jenkins Gearman plugin.
Sometimes no new jobs will be scheduled even though all slots are available.
A workaround for the slaves is to disconnect/connect the slave and then new job would be scheduled again.
For the master the only way to get jobs to be scheduled again is to restart the Jenkins service.
When this happens on one node jobs would still be scheduled on other nodes.
Attaching server thread log for gearman threads when no jobs are currently running and jobs are scheduled in the queue.
Also attaching a trunkated jenkins.log by (grep -C 2 10.33.14.26_manager)
Let me know if you need more logs or other info, I would be happy to help
- is related to
-
-
- Resolved
-
[JENKINS-25867] Gearman won't schedule new jobs even though there are slots available on master
| Assignee | New: Khai Do [ zaro ] |
Khai Do
added a comment - It would help if you can test without zuul to verify that this is a gearman-plugin issue. I would try 2 things:
1. Make sure the jenkins job is set to 'Execute concurrent builds if necessary', more info here: https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin?focusedCommentId=74875603#comment-74875603
2. Instead of zuul, use the simple gearman client to schedule the jobs, https://github.com/zaro0508/gearman-plugin-client
Khai Do
added a comment - It would help if you can test without zuul to verify that this is a gearman-plugin issue. I would try 2 things:
1. Make sure the jenkins job is set to 'Execute concurrent builds if necessary', more info here: https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin?focusedCommentId=74875603#comment-74875603
2. Instead of zuul, use the simple gearman client to schedule the jobs, https://github.com/zaro0508/gearman-plugin-client 1. We already have concurrent builds enabled on all jobs.
2. We have the same issue when scheduling through the simple gearman client.
The job will start when assigning it to another label than master.
The job will not start when assigning it to master label, the label has 6 free executors.
The command I executed was: python gear_client.py -s localhost -p 4730 –
function build:~Update_Scripts --wait
Thanks for the link to the simple gearman client, it was very neat to use to try things out.
Please let me know if I can help in any other way.
Khai Do
added a comment - lets work on this in https://storyboard.openstack.org/#!/story/2000030 Khai Do
added a comment - Sorry to switch again but storyboard isn't really working for us. Lets switch back to using jenkins issue tracker.
Khai Do
added a comment - Sorry to switch again but storyboard isn't really working for us. Lets switch back to using jenkins issue tracker. Khai Do
added a comment - I've noticed that there is a bug when using the "OFFLINE_NODE_WHEN_COMPLETE=true" parameter when there are multiple executors. When used with multiple executors the node gets offline and the jobs get unregistered but it never re-registers therefore no new job requests get executed. Looks like you have setup your jenkins with multiple executors so was wondering if you are using this parameter?
Khai Do
added a comment - I've noticed that there is a bug when using the "OFFLINE_NODE_WHEN_COMPLETE=true" parameter when there are multiple executors. When used with multiple executors the node gets offline and the jobs get unregistered but it never re-registers therefore no new job requests get executed. Looks like you have setup your jenkins with multiple executors so was wondering if you are using this parameter? Khai Do
made changes -
| Link |
New:
This issue is related to |
Placed a bounty of 200$ for this issue on freedomsponsors.
https://freedomsponsors.org/issue/595/gearman-wont-schedule-new-jobs-even-though-there-are-slots-available-on-master?alert=SPONSOR#