[JENKINS-49117] Jenkins 2.103 breaks mesos slave retention strategies

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: core
Labels:
- JEP-200
Environment:
Jenkins 2.103
Mesos-Plugin 0.15.0
SSH Slaves 1.25

Similar Issues:
Powered by SuggestiMate

Show

The mesos-plugin has a slave retention strategy that only allows a slave to be used once. Once that slave has been used for a build it is taken offline. Ever since upgrading to Jenkins 2.103, this error started appearing in the logs and the slaves are not being taken offline.

=============== Full stack trace:

Jan 23, 2018 4:52:33 PM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent MesosCloud failed to launch
java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave
	at hudson.XmlFile.write(XmlFile.java:201)
	at jenkins.model.Nodes.persistNode(Nodes.java:160)
	at jenkins.model.Nodes.addNode(Nodes.java:142)
	at jenkins.model.Jenkins.addNode(Jenkins.java:2053)
	at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241)
	at hudson.model.Queue._withLock(Queue.java:1370)
	at hudson.model.Queue.withLock(Queue.java:1247)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:207)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:194)
        ... 18 more
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.mesos.MesosRetentionStrategy#checkLock for class org.jenkinsci.plugins.mesos.MesosRetentionStrategy
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        ... 31 more
Caused by: java.lang.UnsupportedOperationException: Refusing to marshal java.util.concurrent.locks.ReentrantLock for security reasons; see [https://jenkins.io/redirect/class-filter/]
	at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:530)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
        ... 40 more

relates to

JENKINS-50303 Mesos Offer Selection Attributes Mesos Plugin JEP-200

Resolved

links to

https://github.com/jenkinsci/mesos-plugin/pull/310

Kevin R. created issue - 2018-01-23 17:37

Kevin R. made changes - 2018-01-23 17:52

Description

Original: Full stack trace:

{{Jan 23, 2018 4:52:33 PM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent MesosCloud failed to launch java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave at hudson.XmlFile.write(XmlFile.java:201) at jenkins.model.Nodes.persistNode(Nodes.java:160) at jenkins.model.Nodes.addNode(Nodes.java:142) at jenkins.model.Jenkins.addNode(Jenkins.java:2053) at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241) at hudson.model.Queue._withLock(Queue.java:1370) at hudson.model.Queue.withLock(Queue.java:1247) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:207) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:194) ... 18 more Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.mesos.MesosRetentionStrategy#checkLock for class org.jenkinsci.plugins.mesos.MesosRetentionStrategy at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) ... 31 more Caused by: java.lang.UnsupportedOperationException: Refusing to marshal java.util.concurrent.locks.ReentrantLock for security reasons; see https://jenkins.io/redirect/class-filter/ at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:530) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) ... 40 more}}

New: The mesos-plugin has a slave retention strategy that only allows a slave to be used once. Once that slave has been used for a build it is taken offline. Ever since upgrading to Jenkins 2.103, this error started appearing in the logs and the slaves are not being taken offline.

=============== Full stack trace:

{{Jan 23, 2018 4:52:33 PM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent MesosCloud failed to launch java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave at hudson.XmlFile.write(XmlFile.java:201) at jenkins.model.Nodes.persistNode(Nodes.java:160) at jenkins.model.Nodes.addNode(Nodes.java:142) at jenkins.model.Jenkins.addNode(Jenkins.java:2053) at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241) at hudson.model.Queue._withLock(Queue.java:1370) at hudson.model.Queue.withLock(Queue.java:1247) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:207) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:194) ... 18 more Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.mesos.MesosRetentionStrategy#checkLock for class org.jenkinsci.plugins.mesos.MesosRetentionStrategy at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) ... 31 more Caused by: java.lang.UnsupportedOperationException: Refusing to marshal java.util.concurrent.locks.ReentrantLock for security reasons; see [https://jenkins.io/redirect/class-filter/] at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:530) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) ... 40 more}}

Kevin R. made changes - 2018-01-23 17:52

Environment

Original: Jenkins 2.103
Mesos-Plugin 0.15.0

New: Jenkins 2.103
Mesos-Plugin 0.15.0
SSH Slaves 1.25

Oleg Nenashev made changes - 2018-01-23 22:45

Labels

New: JEP-200

Oleg Nenashev added a comment - 2018-01-23 22:59

I'd guess it is a bad idea to serialize Lock objects to the disk: java.util.concurrent.locks.ReentrantLock

Oleg Nenashev added a comment - 2018-01-23 22:59 I'd guess it is a bad idea to serialize Lock objects to the disk: java.util.concurrent.locks.ReentrantLock

Oleg Nenashev added a comment - 2018-01-23 23:04

Issue is here: https://github.com/jenkinsci/mesos-plugin/blob/1f0a7e24b866030a9f5f5f0149a62df10786215e/src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java#L41
Surprisingly, PCT didn't catch it. PCT run is here: https://github.com/jenkinsci/mesos-plugin/pull/309

Oleg Nenashev added a comment - 2018-01-23 23:04 Issue is here: https://github.com/jenkinsci/mesos-plugin/blob/1f0a7e24b866030a9f5f5f0149a62df10786215e/src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java#L41 Surprisingly, PCT didn't catch it. PCT run is here: https://github.com/jenkinsci/mesos-plugin/pull/309

Oleg Nenashev made changes - 2018-01-24 08:26

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Oleg Nenashev made changes - 2018-01-24 08:26

Assignee

New: Oleg Nenashev [ oleg_nenashev ]

Oleg Nenashev made changes - 2018-01-24 08:26

Status

Original: In Progress [ 3 ]

New: In Review [ 10005 ]

Oleg Nenashev added a comment - 2018-01-24 08:27

I have created https://github.com/jenkinsci/mesos-plugin/pull/310 with a hotfix. chr0n1x would you be able to verify the patch? Previously Mesos plugin passed PCT during JEP-200 testing, so the current test suite is not good enough

Oleg Nenashev added a comment - 2018-01-24 08:27 I have created https://github.com/jenkinsci/mesos-plugin/pull/310 with a hotfix. chr0n1x would you be able to verify the patch? Previously Mesos plugin passed PCT during JEP-200 testing, so the current test suite is not good enough

Oleg Nenashev made changes - 2018-01-24 08:27

Remote Link

New: This issue links to "https://github.com/jenkinsci/mesos-plugin/pull/310 (Web Link)" [ 19932 ]

Assignee:: Oleg Nenashev

Reporter:: Kevin R.

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2018-01-23 17:37

Updated:: 2018-03-21 13:35

Resolved:: 2018-01-29 10:09

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Oleg Nenashev added a comment - 2018-01-23 22:59

Expand comment: Oleg Nenashev added a comment - 2018-01-23 22:59

Collapse comment: Oleg Nenashev added a comment - 2018-01-23 23:04

Expand comment: Oleg Nenashev added a comment - 2018-01-23 23:04

Collapse comment: Oleg Nenashev added a comment - 2018-01-24 08:27

Expand comment: Oleg Nenashev added a comment - 2018-01-24 08:27

People

Dates