Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71539

Running multiple "pipelineJob" item updates one by one occasionally throws java.util.ConcurrentModificationException

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • job-dsl-plugin
    • None

      Hi Jenkins Team,

      We have an intermittent problem with periodical job item updates when this update is being triggered.

      Update logic is straightforward:
      1. A list of automatically managed jobs is composed, they are nested inside folders but overall the structure is rather simple, projects are not interlinked within themselves. The list is basically a list of branches of selected remote repositories so their only difference is their names and SCM address.
      2. For every item in the list a basic job DSL call is invoked thus updating target item to make all jobs up to date.

      Full update process takes about a hour to complete and no parallel updates are made – every item is updated once and then the it goes to the next in order.

      We first encountered this problem about half a year ago and this is not happening every launch of update, sometimes this is not reproducible at all (so no errors are shown in build log) leading to successful build.

      Least known version of Jenkins core application to have this issue is 2.355 and it is still reproducible on recent versions 2.409 and 2.411 (at the moment of creation of this issue).

      All job DSL definitions are mostly the same, an example follows:

      pipelineJob('TEST-PROJECT/test-subproject/release_branch') {
  properties {
    triggers{
      pollSCM {scmpoll_spec("*/2 * * * *")}
    }
  }
  authorization {
    permissions('CommonUsers', [ 'hudson.model.Item.Build' ])
  }
  definition {
    cps {
      sandbox(true)
      script( """// Basic build pipeline
                 |@Library(value='libraryName', changelog=false)_
                 |runBuild()
    }
  }
}

      No extra actions performed (direct XML manipulations, etc), only basic job settings and definition are refreshed but this script text.

      When processing DSL script text with jobDsl(scriptText: text) following Exception occurs thus effectively failing the build:

      java.util.ConcurrentModificationException
        at java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
        at java.base/java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:741)
        at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
Caused: java.lang.RuntimeException: Failed to serialize javaposse.jobdsl.plugin.actions.GeneratedObjectsRunAction#modifiedObjects for class javaposse.jobdsl.plugin.actions.GeneratedJobsBuildAction
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
        at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 9ae3d5e8-2b20-4ec3-b42f-c06611c8cbd0
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1303)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1292)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:1265)
        at hudson.util.XStream2.toXMLUTF8(XStream2.java:386)
        at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1252)
        at hudson.BulkChange.commit(BulkChange.java:98)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1599)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:509)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
        at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

      Additional observations:
      1. The same issue has been reported for Job DSL plugin version 1.81 and presumably got fixed by 1.81.1 but now it happens again with 1.83 and 1.84 – their changelog provides no evidence that some functionality has been broken.
      2. Problem will not show itself every time this update runs, sometimes is takes a week or even two to reproduce (5 to 10 runs or even more builds).
      3. No specific item is affected by this Exception, it can be thrown for any item in the list – be it 100th or 2000th item in list. Removing an item does not alter this behavior.
      4. No user interaction occurs during update process – this has been proven by temporary locking all user from access to particular Jenkins instance.
      5. Sometimes the problem is not reproducible by hand, sometimes it takes about 20 manual runs to happen.
      6. Upgrading or downgrading to recent versions of Jenkins or Job DSL plugins did not alter this behavior.
      7. Less intensive job updates (like partial list update or precise item update) almost never triggers this error, it mostly shows up when more than 5000 items are updated one by one (never in parallel).

      Notes on similar issues:
      1. JENKINS-69064 looks like depicting the same issue but it shows "Fixed but unreleased" status.
      2. JENKINS-48974 looks similar but our templates never use Grab.
      3. JENKINS-69064 looks fixed but shows "Fixed but unreleased" as well.

      The most problematic part in all of this: failing update job build causes executor thread deadlock where this update runs, only full instance restart drops this build from executor process.

      Killing the build through UI or by Admin Console commands never helps: it just stucks on executor in "zombie" state. Build queue shows it as in "in progress" but the build itself is "finished with FAILURE" if you try to check its status via Groovy model or by viewing build page.

      One current workaround is to increase executor count and then occasionally reboot Jenkins from the full off state.

      Any help with the issue is greatly appreciated because it is really annoying to manage every time it happens. Let me know if I can provide any more details on the matter or how this can be investigated next time it happens.

      Thanks in advance for any hints and help!

            jamietanna Jamie Tanna
            pawel_labut Pawel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: