Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49117

Jenkins 2.103 breaks mesos slave retention strategies

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 2.103
      Mesos-Plugin 0.15.0
      SSH Slaves 1.25
    • Similar Issues:

      Description

      The mesos-plugin has a slave retention strategy that only allows a slave to be used once. Once that slave has been used for a build it is taken offline. Ever since upgrading to Jenkins 2.103, this error started appearing in the logs and the slaves are not being taken offline.

      =============== Full stack trace:

      Jan 23, 2018 4:52:33 PM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent MesosCloud failed to launch
      java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave
      	at hudson.XmlFile.write(XmlFile.java:201)
      	at jenkins.model.Nodes.persistNode(Nodes.java:160)
      	at jenkins.model.Nodes.addNode(Nodes.java:142)
      	at jenkins.model.Jenkins.addNode(Jenkins.java:2053)
      	at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241)
      	at hudson.model.Queue._withLock(Queue.java:1370)
      	at hudson.model.Queue.withLock(Queue.java:1247)
      	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:207)
      	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
      	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Slave#retentionStrategy for class org.jenkinsci.plugins.mesos.MesosSlave
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      	at hudson.XmlFile.write(XmlFile.java:194)
              ... 18 more
      Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.mesos.MesosRetentionStrategy#checkLock for class org.jenkinsci.plugins.mesos.MesosRetentionStrategy
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
      	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
      	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
      	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
      	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              ... 31 more
      Caused by: java.lang.UnsupportedOperationException: Refusing to marshal java.util.concurrent.locks.ReentrantLock for security reasons; see [https://jenkins.io/redirect/class-filter/]
      	at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:530)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
      	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              ... 40 more
      

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I'd guess it is a bad idea to serialize Lock objects to the disk: java.util.concurrent.locks.ReentrantLock

            Show
            oleg_nenashev Oleg Nenashev added a comment - I'd guess it is a bad idea to serialize Lock objects to the disk: java.util.concurrent.locks.ReentrantLock
            Show
            oleg_nenashev Oleg Nenashev added a comment - Issue is here: https://github.com/jenkinsci/mesos-plugin/blob/1f0a7e24b866030a9f5f5f0149a62df10786215e/src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java#L41 Surprisingly, PCT didn't catch it. PCT run is here: https://github.com/jenkinsci/mesos-plugin/pull/309
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I have created https://github.com/jenkinsci/mesos-plugin/pull/310 with a hotfix. Kevin R. would you be able to verify the patch? Previously Mesos plugin passed PCT during JEP-200 testing, so the current test suite is not good enough

            Show
            oleg_nenashev Oleg Nenashev added a comment - I have created https://github.com/jenkinsci/mesos-plugin/pull/310 with a hotfix. Kevin R. would you be able to verify the patch? Previously Mesos plugin passed PCT during JEP-200 testing, so the current test suite is not good enough
            Hide
            chr0n1x Kevin R. added a comment -

            Oleg Nenashev sorry, I had to revert the instance. It's not prod but we just got it up and other teammates are using it so I can't really reconstruct this specific env/configuration.

            Show
            chr0n1x Kevin R. added a comment - Oleg Nenashev sorry, I had to revert the instance. It's not prod but we just got it up and other teammates are using it so I can't really reconstruct this specific env/configuration.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java
            http://jenkins-ci.org/commit/mesos-plugin/bb2b25558fdc7a0d57104d48bd7cd0e9016499d5
            Log:
            JENKINS-49117 - Stop serializing ReentrantLock to the disk in MesosRetentionStrategy

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java http://jenkins-ci.org/commit/mesos-plugin/bb2b25558fdc7a0d57104d48bd7cd0e9016499d5 Log: JENKINS-49117 - Stop serializing ReentrantLock to the disk in MesosRetentionStrategy
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Vincent Latombe
            Path:
            Jenkinsfile
            pom.xml
            src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java
            http://jenkins-ci.org/commit/mesos-plugin/0e5df152e462979c46a609ed1ea8e9b5c80a9ece
            Log:
            Merge pull request #310 from oleg-nenashev/JENKINS-49117-jep-200

            JENKINS-49117 - Stop serializing ReentrantLocks to the disk

            Compare: https://github.com/jenkinsci/mesos-plugin/compare/3e7002c19e71...0e5df152e462

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: Jenkinsfile pom.xml src/main/java/org/jenkinsci/plugins/mesos/MesosRetentionStrategy.java http://jenkins-ci.org/commit/mesos-plugin/0e5df152e462979c46a609ed1ea8e9b5c80a9ece Log: Merge pull request #310 from oleg-nenashev/ JENKINS-49117 -jep-200 JENKINS-49117 - Stop serializing ReentrantLocks to the disk Compare: https://github.com/jenkinsci/mesos-plugin/compare/3e7002c19e71...0e5df152e462
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            It has been fixed in 0.15.1

            Show
            oleg_nenashev Oleg Nenashev added a comment - It has been fixed in 0.15.1
            Hide
            chr0n1x Kevin R. added a comment -

            Oleg Nenashev good to know, thanks!

            Show
            chr0n1x Kevin R. added a comment - Oleg Nenashev good to know, thanks!

              People

              Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              chr0n1x Kevin R.
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: