Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47224

Pipeline builds failing/hanging under load: RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript ___cps___6009

XMLWordPrintable

      When we have many pipeline build running simultaneously, we see them starting to hang, never completing or failing (screenshot attached)

      In the logs it shows that it failed with the following error:

      org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript __cps__6009

      We're not actually calling this method anywhere, so I think it's happening within jenkins workflow framework itself:

      Here's a bigger chunk of the logs of one of the failed builds:

      Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7"
      Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T12:39:06-07:00)
      Maven home: /opt/apache-maven-3.5.0
      Java version: 1.8.0_144, vendor: Oracle Corporation
      Java home: /usr/java/jdk1.8.0_144/jre
      Default locale: en_US, platform encoding: ANSI_X3.4-1968
      OS name: "linux", version: "2.6.32-642.3.1.el6.x86_64", arch: "amd64", family: "unix"
      [jenkins-maven-event-spy] INFO generate /u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7/maven-spy-20170929-114954-962.log ...
      [INFO] Scanning for projects...
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/maven-metadata.xml
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-resources-plugin/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/maven-metadata.xml
      (812 B at 1.8 kB/s)
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-install-plugin/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/maven-metadata.xml
      (623 B at 9.2 kB/s)
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/codehaus/mojo/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
      (13 kB at 133 kB/s)
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
      (20 kB at 122 kB/s)
      [INFO] Downloaded:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-metadata.xml
      (567 B at 1.5 kB/s)
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/codehaus/mojo/versions-maven-plugin/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/versions-maven-plugin/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/versions-maven-plugin/maven-metadata.xml
      (742 B at 11 kB/s)
      [INFO]
      [INFO] ------------------------------------------------------------------------
      [INFO] Building Stores WCS Experience API 1.0-SNAPSHOT
      [INFO] ------------------------------------------------------------------------
      [INFO]
      [INFO]

            • maven-clean-plugin:2.5:clean (default-clean) @ stores-wcs-exp-api ---*
              [INFO]
              [INFO]
            • versions-maven-plugin:2.5:lock-snapshots (default-cli) @ stores-wcs-exp-api ---*
              [INFO] ------------------------------------------------------------------------
              [INFO] BUILD SUCCESS
              [INFO] ------------------------------------------------------------------------
              [INFO] Total time: 3.179 s
              [INFO] Finished at: 2017-09-29T11:49:58-07:00
              [INFO] Final Memory: 20M/304M
              [INFO] ------------------------------------------------------------------------
              [jenkins-maven-event-spy] INFO generated /u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7/maven-spy-20170929-114954-962.log
              [Pipeline] }
              [withMaven] Archive artifact pom.xml under com/petco/api/stores-wcs-exp-api/1.0-SNAPSHOT/stores-wcs-exp-api-1.0-SNAPSHOT.pom
              [withMaven] Scan Tasks for Maven artifact MavenArtifact{com.petco.api:stores-wcs-exp-api::1.0-SNAPSHOT} in source directory src/main/java
              [TASKS] Scanning folder '/u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3' for files matching the pattern 'src/main/java/*/' - excludes: null
              [TASKS] Found 2 files to scan for tasks
              [TASKS] Found 0 open tasks.
              [Pipeline] End of Pipeline
              [BFA] Scanning build for known causes...
              [BFA] No failure causes found
              [BFA] Done. 0s
              org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript __cps__6009
              at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectStaticMethod(StaticWhitelist.java:189)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onStaticCall(SandboxInterceptor.java:150)
              at org.kohsuke.groovy.sandbox.impl.Checker$2.call(Checker.java:179)
              at org.kohsuke.groovy.sandbox.impl.Checker.checkedStaticCall(Checker.java:183)
              at org.kohsuke.groovy.sandbox.impl.Checker$checkedStaticCall.callStatic(Unknown Source)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
              at WorkflowScript.<clinit>(WorkflowScript)
              Caused: java.lang.ExceptionInInitializerError
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
              at java.lang.reflect.Constructor.newInstance(Unknown Source)
              at java.lang.Class.newInstance(Unknown Source)
              at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:434)
              at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:613)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:588)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:612)
              at hudson.model.RunMap.retrieve(RunMap.java:225)
              at hudson.model.RunMap.retrieve(RunMap.java:57)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:345)
              at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:362)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.getPreviousBuild(WorkflowRun.java:226)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.getPreviousBuild(WorkflowRun.java:132)
              at hudson.plugins.analysis.core.BuildHistory.getAction(BuildHistory.java:137)
              at hudson.plugins.analysis.core.BuildHistory.getReferenceAction(BuildHistory.java:123)
              at hudson.plugins.analysis.core.BuildHistory.getReferenceAnnotations(BuildHistory.java:106)
              at hudson.plugins.analysis.core.BuildResult.initialize(BuildResult.java:278)
              at hudson.plugins.analysis.core.BuildResult.<init>(BuildResult.java:222)
              at hudson.plugins.tasks.TasksResult.<init>(TasksResult.java:92)
              at hudson.plugins.tasks.TasksResult.<init>(TasksResult.java:59)
              at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:187)
              at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:68)
              at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:295)
              at org.jenkinsci.plugins.pipeline.maven.reporters.TasksScannerReporter.process(TasksScannerReporter.java:100)
              at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:126)
              at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$Callback.finished(WithMavenStepExecution.java:770)
              at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
              at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
              at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:165)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
              at java.util.concurrent.FutureTask.run(Unknown Source)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
              at java.util.concurrent.FutureTask.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)
              Finished: FAILURE

        1. Screen Shot 2017-09-29 at 13.01.55.png
          Screen Shot 2017-09-29 at 13.01.55.png
          16 kB
        2. Screen Shot 2017-11-15 at 13.55.11.png
          Screen Shot 2017-11-15 at 13.55.11.png
          47 kB
        3. stacktrace.txt
          13 kB
        4. stacktrace.txt
          12 kB

          [JENKINS-47224] Pipeline builds failing/hanging under load: RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript ___cps___6009

          Andrew Bayer added a comment -

          What versions of workflow-cps and script-security plugins do you have installed?

          Andrew Bayer added a comment - What versions of workflow-cps and script-security plugins do you have installed?

          Andrew Bayer added a comment -

          ping - still looking for those versions.

          Andrew Bayer added a comment - ping - still looking for those versions.

          Phil Grayson added a comment -

          Hi, we see this sometimes too.

          script-security v1.34

          workflow-cps v2.41

          Phil Grayson added a comment - Hi, we see this sometimes too. script-security v1.34 workflow-cps v2.41

          Phil Grayson added a comment - - edited

          I see the script approval requests every so often but have never seen the error trigger until now.

          I ran job that had not changed and got this error. Re-running the job afterward worked successfully. Therefore it appears to be a transient^ ^error. In my situation the Jenkins server had been restarted within 5 minutes of running the job but I don't have any evidence to suggest it was related

          stacktrace.txt stacktrace.txt

          (sorry for two .txt file uploads. One contains the full job output).

          Phil Grayson added a comment - - edited I see the script approval requests every so often but have never seen the error trigger until now. I ran job that had not changed and got this error. Re-running the job afterward worked successfully. Therefore it appears to be a transient^ ^error. In my situation the Jenkins server had been restarted within 5 minutes of running the job but I don't have any evidence to suggest it was related stacktrace.txt   stacktrace.txt (sorry for two .txt file uploads. One contains the full job output).

          Sam Van Oort added a comment -

          abayer Please could you take a look? I swear we fixed something like this in a recent update but nothing is jumping out at me.

          Sam Van Oort added a comment - abayer Please could you take a look? I swear we fixed something like this in a recent update but nothing is jumping out at me.

          Andrew Bayer added a comment - - edited

          svanoort - yeah, this is related to the reparsing-on-load stuff, i.e. JENKINS-39489. The RejectedAccessException here is technically correct, but it never should have gotten to that point in the first place. Fixing JENKINS-39489 will also fix this, since it's only possible when we're reparsing scripts for a completed build in a context other than actually executing a Pipeline.

          Andrew Bayer added a comment - - edited svanoort - yeah, this is related to the reparsing-on-load stuff, i.e. JENKINS-39489 . The RejectedAccessException here is technically correct, but it never should have gotten to that point in the first place. Fixing JENKINS-39489 will also fix this, since it's only possible when we're reparsing scripts for a completed build in a context other than actually executing a Pipeline.

          Sam Van Oort added a comment -

          abayer Maybe I'm missing a nuance here, but why are we reparsing the script? Or did you mean to link another JIRA? JENKINS-39489 is mostly concerned with moving where the scripts are persisted to somewhere other than the build itself to reduce the disk hit of loading the build.

          Sam Van Oort added a comment - abayer Maybe I'm missing a nuance here, but why are we reparsing the script? Or did you mean to link another JIRA? JENKINS-39489 is mostly concerned with moving where the scripts are persisted to somewhere other than the build itself to reduce the disk hit of loading the build.

          Andrew Bayer added a comment -

          Ah, right, sorry, JENKINS-45585 - so looks like you already fixed it. =)

          Andrew Bayer added a comment - Ah, right, sorry, JENKINS-45585 - so looks like you already fixed it. =)

          Sam Van Oort added a comment -

          Ah yeah, the lazy-load of the execution DOES mean we don't have a risk of reparsing the Groovy, excellent. ymenager you should have your fix available in workflow-job 2.18 (you'll want to pick up the latest workflow-cps though due to other fixes to related issues though).

          Sam Van Oort added a comment - Ah yeah, the lazy-load of the execution DOES mean we don't have a risk of reparsing the Groovy, excellent. ymenager you should have your fix available in workflow-job 2.18 (you'll want to pick up the latest workflow-cps though due to other fixes to related issues though).

          Sam Van Oort added a comment -

          Not a true duplicate of JENKINS-45585, but the fix to that issue resolves this one.

          Sam Van Oort added a comment - Not a true duplicate of JENKINS-45585 , but the fix to that issue resolves this one.

            Unassigned Unassigned
            ymenager ymenager
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: