Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52249

Parallel build led to AIOOBE in Plot.savePlotData

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • plot-plugin
    • None
    • The latest and greatest:
      * Jenkins (core) 2.129
      * Plot Plugin 2.1.0

      2 parallel builds (scripted pipeline) executed the "same" plot step at very, very much the same time (as far as I can tell it was seemingly definitely at least the same second, but of course it must have been really the very, very same time) leading to one of the two builds failing with:

      java.lang.ArrayIndexOutOfBoundsException: 2
          at hudson.plugins.plot.Plot.savePlotData(Plot.java:967)
          at hudson.plugins.plot.Plot.addBuild(Plot.java:717)
          at hudson.plugins.plot.PlotBuilder.perform(PlotBuilder.java:223)
          at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
          at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
          at hudson.security.ACL.impersonate(ACL.java:290)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      
      • ... whereas the other one was successful
      • Presumably the plot CSV file in the pipeline job folder was corrupted by that, because the next build (not in parallel anymore) failed with the same symptom.
      • But must have kind of fixed the corrupt plot CSV file in the pipeline job folder. But in a rather forceful way...
      • Because subsequent builds (also not in parallel anymore) then succeeded.

      Sad side effects:

      • Please note:
        • there are in fact 4 consecutive plot step calls
        • seemingly the 3rd one collided
      • The "interesting"/sad result is:
        • in the plot CSV files of the plot step calls #1 and #2 "only" the values/entries/lines of the successful build are missing
        • in the plot CSV file of the colliding plot step call #3 there is a huge gap of values/entries/lines: in fact the bulk of the history is lost and the remaining file has less than 10% of the size it had before
        • in the plot CSV file of the plot step call #4 (which was skipped due to the failures) the values/entries/lines of the two failing builds are missing; which makes of course sense

          [JENKINS-52249] Parallel build led to AIOOBE in Plot.savePlotData

          (IMHO not related to JENKINS-6913 or JENKINS-43708; but maybe similar to old Jenkins core bugs JENKINS-22516 and JENKINS-25930?)

          jglick Can standard JVM synchronization mechanism (e.g. synchronized keyword for synchronized methods) be used in (a) Jenkins plugin code (that is called via pipelines that are executed via groovy CPS whatever magic) and (b) shared Jenkins pipeline library code (that again is called via pipelines that are executed via groovy CPS whatever magic)? Or is something like https://gist.github.com/pditommaso/4616192e50cd333487211d1e37e81d58 necessary? Thanks for any hints!

          Reinhold Füreder added a comment - (IMHO not related to JENKINS-6913 or JENKINS-43708 ; but maybe similar to old Jenkins core bugs JENKINS-22516 and JENKINS-25930 ?) jglick Can standard JVM synchronization mechanism (e.g. synchronized keyword for synchronized methods) be used in (a) Jenkins plugin code (that is called via pipelines that are executed via groovy CPS whatever magic) and (b) shared Jenkins pipeline library code (that again is called via pipelines that are executed via groovy CPS whatever magic)? Or is something like https://gist.github.com/pditommaso/4616192e50cd333487211d1e37e81d58 necessary? Thanks for any hints!

          Jesse Glick added a comment -

          reinholdfuereder (a) yes, sure, which is why this is a bug in the plot plugin; (b) no, since this is a virtual machine using green threads, but there is a lock step.

          Jesse Glick added a comment - reinholdfuereder (a) yes, sure, which is why this is a bug in the plot plugin; (b) no, since this is a virtual machine using green threads, but there is a lock step.

          jglick Sorry for this potentially irritating or annoying follow-up question (that is only partially related with this issue):

          Let's claim the synchronization is in both cases – that is for (a) Jenkins plugin code providing a pipeline step and (b) shared Jenkins pipeline library code – needed for protecting a shared file in the pipeline job folder ("/var/lib/jenkins/jobs/...") that is read and written by each pipeline execution (= pipeline build); and this shared file access is being done in (a) and (b) respectively.

          My naive expectation would have been that when calling the pipeline step – that internally reads and writes a shared file in the pipeline from (a) from either a Jenkinsfile or from (b) – then any standard JVM synchronization mechanism does not help either, because it is also called in context of the "virtual machine using green threads" (as you wrote)?

          I try to depict this:

          • "Jenkinsfile" -> custom step from custom shared pipeline library -> "plot" step from "Plot" plugin -> synchronized method in "Plot" plugin
          • "Jenkinsfile" -> custom step from custom shared pipeline library -> synchronized method in custom shared pipeline library

          I see one (most of the times) disadvantage of using the lock step for such a let's call it rather low-level synchronization: the resource will not just be kind of living forever, but especially bloating up the "Lockable Resources" view (and the "Lockable Resources" section in "Configure System"), see JENKINS-38906; but maybe I am just stubborn and such a distinction in low-level vs. high-level synchronization is unfair?

          Reinhold Füreder added a comment - jglick Sorry for this potentially irritating or annoying follow-up question (that is only partially related with this issue): Let's claim the synchronization is in both cases – that is for (a) Jenkins plugin code providing a pipeline step and (b) shared Jenkins pipeline library code – needed for protecting a shared file in the pipeline job folder ("/var/lib/jenkins/jobs/...") that is read and written by each pipeline execution (= pipeline build); and this shared file access is being done in (a) and (b) respectively. My naive expectation would have been that when calling the pipeline step – that internally reads and writes a shared file in the pipeline from (a) from either a Jenkinsfile or from (b) – then any standard JVM synchronization mechanism does not help either, because it is also called in context of the "virtual machine using green threads" (as you wrote)? I try to depict this: "Jenkinsfile" -> custom step from custom shared pipeline library -> "plot" step from "Plot" plugin -> synchronized method in "Plot" plugin "Jenkinsfile" -> custom step from custom shared pipeline library -> synchronized method in custom shared pipeline library I see one (most of the times) disadvantage of using the lock step for such a let's call it rather low-level synchronization: the resource will not just be kind of living forever, but especially bloating up the "Lockable Resources" view (and the "Lockable Resources" section in "Configure System"), see JENKINS-38906 ; but maybe I am just stubborn and such a distinction in low-level vs. high-level synchronization is unfair?

          Since it happened again (the 2nd time overall and as expected rather seldom: the 2nd occurrence >6 months after the 1st one; but again with loss of historical plot data) I really implemented the suggested workaround and changed my shared Jenkins pipeline library code to use a lock step as a so-called critical section.

          Reinhold Füreder added a comment - Since it happened again (the 2nd time overall and as expected rather seldom: the 2nd occurrence >6 months after the 1st one; but again with loss of historical plot data) I really implemented the suggested workaround and changed my shared Jenkins pipeline library code to use a lock step as a so-called critical section.

            vgaidarji Veaceslav Gaidarji
            reinholdfuereder Reinhold Füreder
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: