Critical The mesos-plugin has a slave retention strategy that only allows a slave to be used once. Once that slave has been used for a build it is taken offline. Ever since upgrading to Jenkins 2.103, this error started appearing in the logs and the slaves are not being taken offline.
=============== Full stack trace:
Jan 23, 2018 4:52:33 PM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent MesosCloud failed to launch
java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave
at hudson.XmlFile.write(XmlFile.java:201)
at jenkins.model.Nodes.persistNode(Nodes.java:160)
at jenkins.model.Nodes.addNode(Nodes.java:142)
at jenkins.model.Jenkins.addNode(Jenkins.java:2053)
at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241)
at hudson.model.Queue._withLock(Queue.java:1370)
at hudson.model.Queue.withLock(Queue.java:1247)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:207)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
at hudson.XmlFile.write(XmlFile.java:194)
... 18 more
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.mesos.MesosRetentionStrategy#checkLock for class org.jenkinsci.plugins.mesos.MesosRetentionStrategy
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
... 31 more
Caused by: java.lang.UnsupportedOperationException: Refusing to marshal java.util.concurrent.locks.ReentrantLock for security reasons; see [https://jenkins.io/redirect/class-filter/]
at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:530)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
... 40 more
- relates to
-
JENKINS-50303 Mesos Offer Selection Attributes Mesos Plugin JEP-200
-
- Resolved
-
- links to
[JENKINS-49117] Jenkins 2.103 breaks mesos slave retention strategies
Issue is here: https://github.com/jenkinsci/mesos-plugin/blob/1f0a7e24b866030a9f5f5f0149a62df10786215e/src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java#L41
Surprisingly, PCT didn't catch it. PCT run is here: https://github.com/jenkinsci/mesos-plugin/pull/309
I have created https://github.com/jenkinsci/mesos-plugin/pull/310 with a hotfix. chr0n1x would you be able to verify the patch? Previously Mesos plugin passed PCT during JEP-200 testing, so the current test suite is not good enough
oleg_nenashev sorry, I had to revert the instance. It's not prod but we just got it up and other teammates are using it so I can't really reconstruct this specific env/configuration.
Code changed in jenkins
User: Oleg Nenashev
Path:
src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java
http://jenkins-ci.org/commit/mesos-plugin/bb2b25558fdc7a0d57104d48bd7cd0e9016499d5
Log:
JENKINS-49117 - Stop serializing ReentrantLock to the disk in MesosRetentionStrategy
Code changed in jenkins
User: Vincent Latombe
Path:
Jenkinsfile
pom.xml
src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java
http://jenkins-ci.org/commit/mesos-plugin/0e5df152e462979c46a609ed1ea8e9b5c80a9ece
Log:
Merge pull request #310 from oleg-nenashev/JENKINS-49117-jep-200
JENKINS-49117 - Stop serializing ReentrantLocks to the disk
Compare: https://github.com/jenkinsci/mesos-plugin/compare/3e7002c19e71...0e5df152e462
I'd guess it is a bad idea to serialize Lock objects to the disk: java.util.concurrent.locks.ReentrantLock