Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38576

Long build parameters result in shell scripts crashing

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • durable-task-plugin
    • None
    • Jenkins v2.23, pipeline 2.4, Debian Linux 64 bit (wheezy)

      I set up a test pipeline project, which has two string parameters, diff1 and diff2. The pipeline groovy text is as follows

      stage('Initiating from Master') {
          node('master') {
              writeFile file:"encoded_patch1.txt", text:diff1
              archive "encoded_patch1.txt" // So we can check the whole string was received
              writeFile file:"encoded_patch2.txt", text:diff2
              archive "encoded_patch2.txt" // So we can check the whole string was received
              sh "echo hello"
          }
      }
      

      I then created a python script to call this

      from __future__ import print_function
      import requests
      import platform, subprocess
      import json
      import pprint
      import os
      import sys
      import re
      
      diff1_length = int(sys.argv[1])
      diff2_length = int(sys.argv[2])
      diff1_text = "w" + ("x" * diff1_length) + "y"
      diff2_text = "a" + ("b" * diff2_length) + "c"
      diff1_param = ("{\"name\": \"diff1\", \"value\": \"%s\"}") % diff1_text
      diff2_param = ("{\"name\": \"diff2\", \"value\": \"%s\"}") % diff2_text
      
      form_data = {
          "json" : ("{\"parameter\": [%s, %s]}") % (diff1_param, diff2_param),
      }
      
      jenkins_url = "http://my.ser.ver.url:8080"
      job_url = jenkins_url +"/job/TestParameterSizeBug/build"
      user = "Me"
      password = "Password"
      
      build_request_result = requests.post(job_url, data=form_data, auth=(user, password))
      print ("Build Reguest returned %s\n" % build_request_result)
      

      Now when I run the python script, all is well so long as both parameters are kept below 13064. But if either of them is 13064 or more, then I get the following error

      java.io.IOException: Cannot run program "nohup" (in directory "/var/lib/jenkins/workspace/TestParameterSizeBug"): error=7, Argument list too long
      	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
      	at hudson.Proc$LocalProc.<init>(Proc.java:243)
      	at hudson.Proc$LocalProc.<init>(Proc.java:212)
      	at hudson.Launcher$LocalLauncher.launch(Launcher.java:815)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:381)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:147)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:158)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:109)
      	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
      	at WorkflowScript.run(WorkflowScript:7)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:48)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor441.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: error=7, Argument list too long
      	at java.lang.UNIXProcess.forkAndExec(Native Method)
      	at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
      	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
      	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
      	at hudson.Proc$LocalProc.<init>(Proc.java:243)
      	at hudson.Proc$LocalProc.<init>(Proc.java:212)
      	at hudson.Launcher$LocalLauncher.launch(Launcher.java:815)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:381)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:147)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:158)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:109)
      	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
      	... 29 more
      Finished: FAILURE
      

      Note that both strings appear to be received correctly (that's why I made the first and last characters different, so I could verify that)

      Note also that the result is the same if I comment out the writeFile and archive lines.

          [JENKINS-38576] Long build parameters result in shell scripts crashing

          Jesse Glick added a comment -

          Possibly a Java or Linux limitation, TBD. Environment variables are not being passed in argument lists; perhaps the error code 7 refers more generally to other things that are too long.

          Workaround should simply be to wrap in

          withEnv(['diff1=', 'diff2=']) {…}
          

          The proper solution is of course to use file parameters, once implemented.

          Jesse Glick added a comment - Possibly a Java or Linux limitation, TBD. Environment variables are not being passed in argument lists; perhaps the error code 7 refers more generally to other things that are too long. Workaround should simply be to wrap in withEnv([ 'diff1=' , 'diff2=' ]) {…} The proper solution is of course to use file parameters, once implemented.

          Jon Hodgson added a comment -

          The workaround seems successful, thank you. Tested with an 8 megabyte parameter and no crashing.

          As for the "proper solution" well I commented on the issue of the lack of file parameters months ago when I started working on this, and you have no idea how much time and pain have gone into getting the string parameter workaround to function properly, so I would love to.

          But until they are implemented it's a bit like telling me the proper filling for my pillow is unicorn hair

          Jon Hodgson added a comment - The workaround seems successful, thank you. Tested with an 8 megabyte parameter and no crashing. As for the "proper solution" well I commented on the issue of the lack of file parameters months ago when I started working on this, and you have no idea how much time and pain have gone into getting the string parameter workaround to function properly, so I would love to. But until they are implemented it's a bit like telling me the proper filling for my pillow is unicorn hair

          Jon Hodgson added a comment -

          I seem to have encountered another pronlem.

          The winEnv workaround is working fine around my shell scripts calls, but it doesn't seem to help around a checkout call

              withEnv(['rep_diff=']) {
                  checkout([$class: 'MercurialSCM', clean:true, credentialsId: credentials, revision: revision, source: url, subdir: destination])
              }
          

          gives

          [Source] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" showconfig paths.default
          ERROR: Failed to determine whether workspace can be reused
          java.io.IOException: Cannot run program "hg" (in directory "/var/lib/jenkins/workspace/WPF-TryBuild/impOSCar/Source"): error=7, Argument list too long
          	at java.lang.ProcessBuilder.
          

          Jon Hodgson added a comment - I seem to have encountered another pronlem. The winEnv workaround is working fine around my shell scripts calls, but it doesn't seem to help around a checkout call withEnv([ 'rep_diff=' ]) { checkout([$class: 'MercurialSCM' , clean: true , credentialsId: credentials, revision: revision, source: url, subdir: destination]) } gives [Source] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" showconfig paths.default ERROR: Failed to determine whether workspace can be reused java.io.IOException: Cannot run program "hg" (in directory "/var/lib/jenkins/workspace/WPF-TryBuild/impOSCar/Source"): error=7, Argument list too long at java.lang.ProcessBuilder.

          Jesse Glick added a comment -

          Not sure offhand, that could be a bug in the mercurial plugin.

          Jesse Glick added a comment - Not sure offhand, that could be a bug in the mercurial plugin.

          Ben Hines added a comment -

          We encountered this as well. I bet Jenkins could detect parameters over a certain length and warn about this, at least, which would avoid the pain involved in tracking down the cryptic error message. (took us a while to figure out this was the problem

          Ben Hines added a comment - We encountered this as well. I bet Jenkins could detect parameters over a certain length and warn about this, at least, which would avoid the pain involved in tracking down the cryptic error message. (took us a while to figure out this was the problem

          Jaya Mantri added a comment - - edited

          jglick and jonhodgson I am also getting similar issue with the pipeline. I tried to implement the fix suggested but it didn't work out for me. I use the same code as jonhodgson wrapping into withEnv clause. It gives following error to me. Will really appreciate a quick response  on it (Please see ***Refer 1)

          -------------------------------------

          ***Refer 1

          withEnv(['diff1=', 'diff2=']) {
          stage('Initiating from Master') {
          node()

          { writeFile file:"encoded_patch1.txt", text:diff1 archive "encoded_patch1.txt" // So we can check the whole string was received writeFile file:"encoded_patch2.txt", text:diff2 archive "encoded_patch2.txt" // So we can check the whole string was received sh "echo hello" }

          } }

           
          [Pipeline] End of Pipelinegroovy.lang.MissingPropertyException: No such property: jobParams for class: groovy.lang.Binding
          at groovy.lang.Binding.getVariable(Binding.java:63)
          at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:264)
          at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288)
          at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:292)
          at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
          at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
          at WorkflowScript.run(WorkflowScript:1)
          at __cps.transform__(Native Method)

          ---------------------------------------------------

          Jaya Mantri added a comment - - edited jglick and jonhodgson I am also getting similar issue with the pipeline. I tried to implement the fix suggested but it didn't work out for me. I use the same code as jonhodgson wrapping into withEnv clause. It gives following error to me. Will really appreciate a quick response  on it (Please see ***Refer 1) ------------------------------------- ***Refer 1 withEnv( ['diff1=', 'diff2='] ) { stage('Initiating from Master') { node() { writeFile file:"encoded_patch1.txt", text:diff1 archive "encoded_patch1.txt" // So we can check the whole string was received writeFile file:"encoded_patch2.txt", text:diff2 archive "encoded_patch2.txt" // So we can check the whole string was received sh "echo hello" } } }   [Pipeline] End of Pipelinegroovy.lang.MissingPropertyException: No such property: jobParams for class: groovy.lang.Binding at groovy.lang.Binding.getVariable(Binding.java:63) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:264) at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288) at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:292) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29) at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20) at WorkflowScript.run(WorkflowScript:1) at __ cps.transform __(Native Method) ---------------------------------------------------

          Ben Hines added a comment -

          We ended up using the 'file parameter' workaround. However, due to a separate jenkins issue (file parameters don't work with pipeline currently), we are using this OTHER workaround to support that

           

          https://issues.jenkins-ci.org/browse/JENKINS-27413

          https://bitbucket.org/janvrany/jenkins-27413-workaround-library

           

           

          Ben Hines added a comment - We ended up using the 'file parameter' workaround. However, due to a separate jenkins issue (file parameters don't work with pipeline currently), we are using this OTHER workaround to support that   https://issues.jenkins-ci.org/browse/JENKINS-27413 https://bitbucket.org/janvrany/jenkins-27413-workaround-library    

          Jaya Mantri added a comment -

          Thanks benh57. But have you tried the approach mentioned by jglick jonhodgson. It didn't worked out for me, so just want to know what am I doing wrong here

           

          Jaya Mantri added a comment - Thanks benh57 . But have you tried the approach mentioned by jglick jonhodgson . It didn't worked out for me, so just want to know what am I doing wrong here  

          Ben Hines added a comment -

          Yes, the withEnv() wrapper works for fixing 'sh', but it does not work for scm steps such as git.  You need to override the specific environment variable(s) which are too long.

          Ben Hines added a comment - Yes, the withEnv() wrapper works for fixing 'sh', but it does not work for scm steps such as git.  You need to override the specific environment variable(s) which are too long.

            Unassigned Unassigned
            jonhodgson Jon Hodgson
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: