-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
mesos 0.17 jenkins 2.138.4 centos 7
There is a critical problem in mesos plugins 0.17. When a lot of job building in jenkins, jenkins master don't allocate mesos nodes and Mesos master produce a lot of outstanding offers.There is a critical problem in mesos plugins 0.17. When a lot of job building in jenkins, jenkins master don't allocate mesos nodes and Mesos master produce a lot of outstanding offers.We execute java command jstack -l 58(which is PID of jenkins master), as follow:
"Thread-28255" #93222 prio=3 os_prio=0 tid=0x00007f89c0975432 nid=0xe7 runnable (0x00007f89c09432)
java.lang.Thread.state: RUNNABLE
at java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.trySplit(LinkedBlockingQueue.java 890)
at java.util.stream.AbstractTask.compute(AbstractTask.java:297)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
......
Lockeked owable synchronizers:
-<0x000000006089c0975> (a java.util.concurrent.locks.ReentranLock$NonfairSync)
-<0x000000006089c0968> (a java.util.concurrent.locks.ReentranLock$NonfairSync) "mesos-offer-processor-6berferd-32re-45fd-12erdf6578uy-0026" #103 prio=5 os_prio=0 tid=0x00007f89c0973451 nid=0xf7 waiting on condition[0x00007f89c0971000] java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000006089c0975> (a java.util.concurrent.locks.ReentranLock$NonfairSync)
......
Thread mesos-offer-processor is always at WAITING status because Thread-28255 is own the ReetranLock.There, jenkins doesn't allocate the mesos nodes.The below code which in mesos plugin produce the Thread-28255.
public void offerRescinded(SchedulerDriver driver, OfferId offerId)
{ LOGGER.info("Rescinded offer"+offerId); offerQueue.remove(offerId); }We use the solution which is remove the code "offerQueue.remove(offerId);". However, I think there should be better solution for question?