Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48271

Strings with exclamation mark passed to sh step get single quoted

    XMLWordPrintable

Details

    Description

      Using this as a minimal example:

       

      script {
          def testString1 = "echo a="
          def testString2 = "!123"
          def testString3 = "b="
          def testString4 = "2937"
      
          print "testString1+2 = "+testString1+testString2
          print "testString3+4 = "+testString3+testString4
          print "total: "+testString1+testString2+testString3+testString4
      
          sh "echo ${testString1}${testString2}"
          sh "eval ${testString1}${testString2}"
          
          sh "echo ${testString1}${testString2} ${testString3}${testString4}"
          sh "eval ${testString1}${testString2} ${testString3}${testString4}"
      }
      

       

      The result while running the pipeline is:

       

      [Pipeline] script
      [Pipeline] {
      [Pipeline] echo
      testString1+2 = echo a=!123
      [Pipeline] echo
      testString3+4 = b=2937
      [Pipeline] echo
      total: echo a=!123b=2937
      [Pipeline] sh
      [pipeName] Running shell script
      + echo echo 'a=!123'
      echo a=!123
      [Pipeline] sh
      [pipeName] Running shell script
      + eval echo 'a=!123'
      ++ echo 'a=!123'
      a=!123
      [Pipeline] sh
      [pipeName] Running shell script
      + echo echo 'a=!123' b=2937
      echo a=!123 b=2937
      [Pipeline] sh
      [pipeName] Running shell script
      + eval echo 'a=!123' b=2937
      ++ echo 'a=!123' b=2937
      a=!123 b=2937
      [Pipeline] }
      [Pipeline] // script
      

       

      Notice that last echo treats strings differently depending on whether or not the exclamation mark is contained within a contiguous string.

      This creates issues when, for example, creating maven options dynamically at runtime. Trying to generate 

      -Dopt1=test -Dopt2=test!1

       leads to a string

      -Dopt1=test '-Dopt2=test!1'

      which then doesn't work.

      This treatment of strings lacking consistency, and despite all my efforts to try to trim the string (or create the string within a shell script, etc), ultimately this issue is a bottleneck.

      Let me know if I am missing something, I will happily provide more information if needed.

       

       

       

      Attachments

        Activity

          Your example has single quotes only in the lines that start with a plus sign. Those look like trace output from bash -x:

          $ bash -x -c "echo abc d!e"
          + echo abc 'd!e'
          abc d!e
          

          Indeed, durable-task-plugin adds the -x option: https://github.com/jenkinsci/durable-task-plugin/blob/durable-task-1.17/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L119

          I'm pretty sure the single quotes exist only in the trace output and are never seen by maven or other commands that the shell scripts run.

          kon Kalle Niemitalo added a comment - Your example has single quotes only in the lines that start with a plus sign. Those look like trace output from bash -x: $ bash -x -c "echo abc d!e" + echo abc 'd!e' abc d!e Indeed, durable-task-plugin adds the -x option: https://github.com/jenkinsci/durable-task-plugin/blob/durable-task-1.17/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L119 I'm pretty sure the single quotes exist only in the trace output and are never seen by maven or other commands that the shell scripts run.

          Unfortunately the single quotes DO get carried over to commands such as maven, and commands like this when built as a string and eval'd then fail:

          def testString1 = "mvn clean test "
          def testString2 = "-Denv=beta"
          def testString3 = "-Dpw=Password1!"
          def testString4 = "-Dpw2=test2"
          
          sh "echo ${testString1} ${testString2} ${testString3} ${testString4}"
          sh "eval ${testString1} ${testString2} ${testString3} ${testString4}"
          
          leads to:
          mvn clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2

          The eco suppresses single quotes, but when trying to create such a string (or when using the withMaven() plugin, either way) we end up with that issue still.

          cmehdy Mehdy Chaillou added a comment - Unfortunately the single quotes DO get carried over to commands such as maven, and commands like this when built as a string and eval'd then fail: def testString1 = "mvn clean test " def testString2 = "-Denv=beta" def testString3 = "-Dpw=Password1!" def testString4 = "-Dpw2=test2" sh "echo ${testString1} ${testString2} ${testString3} ${testString4}" sh "eval ${testString1} ${testString2} ${testString3} ${testString4}" leads to: mvn clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 The eco suppresses single quotes, but when trying to create such a string (or when using the withMaven() plugin, either way) we end up with that issue still.
          abayer Andrew Bayer added a comment -

          Very interesting. Need to figure out where that quoting is happening. I have verified that this doesn't happen in Scripted Pipeline, only in Declarative, so that helps narrow it down. =)

          abayer Andrew Bayer added a comment - Very interesting. Need to figure out where that quoting is happening. I have verified that this doesn't happen in Scripted Pipeline, only in Declarative, so that helps narrow it down. =)
          abayer Andrew Bayer added a comment -

          Ah, no, I'm wrong, it's reproducible in Scripted with this:

          node {
            def testString1 = "non_existing_thingie clean test "
            def testString2 = "-Denv=beta"
            def testString3 = "-Dpw=Password1!"
            def testString4 = "-Dpw2=test2"
            assert testString3 == "-Dpw=Password1" + '!'
            def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}"
            echo "fts: ${fullTestString}"
            sh "echo ${fullTestString}"
            sh "eval ${fullTestString}"
          }
          

          fts is fine, but as soon as it gets passed to sh, the single quotes show up.

          abayer Andrew Bayer added a comment - Ah, no, I'm wrong, it's reproducible in Scripted with this: node { def testString1 = "non_existing_thingie clean test " def testString2 = "-Denv=beta" def testString3 = "-Dpw=Password1!" def testString4 = "-Dpw2=test2" assert testString3 == "-Dpw=Password1" + '!' def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}" echo "fts: ${fullTestString}" sh "echo ${fullTestString}" sh "eval ${fullTestString}" } fts is fine, but as soon as it gets passed to sh , the single quotes show up.

          abayer, did your tests emit single quotes in the trace output only, or somewhere else as well?

          kon Kalle Niemitalo added a comment - abayer , did your tests emit single quotes in the trace output only, or somewhere else as well?
          abayer Andrew Bayer added a comment -

          Both, looks like -

          [jenkins-48271] Running shell script
          + echo non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + eval non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          ++ non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          
          abayer Andrew Bayer added a comment - Both, looks like - [jenkins-48271] Running shell script + echo non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + eval non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 ++ non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2

          That last line starting with two plus signs is also trace output from the shell. It has two plus signs because it describes a nested evaluation caused by the eval keyword.

          kon Kalle Niemitalo added a comment - That last line starting with two plus signs is also trace output from the shell. It has two plus signs because it describes a nested evaluation caused by the eval keyword.
          abayer Andrew Bayer added a comment -

          And

          def testString5 = "non_existing_thingie -Dpw=Password1!"
          sh "eval ${testString5}"
          

          resulted in

          + eval non_existing_thingie '-Dpw=Password1!'
          ++ non_existing_thingie '-Dpw=Password1!'
          
          abayer Andrew Bayer added a comment - And def testString5 = "non_existing_thingie -Dpw=Password1!" sh "eval ${testString5}" resulted in + eval non_existing_thingie '-Dpw=Password1!' ++ non_existing_thingie '-Dpw=Password1!'
          abayer Andrew Bayer added a comment -

          Ok, I could be wrong:
          This:

          node {
              writeFile(file:'non_existing_thingie',text:'''
          #!/bin/bash 
          echo $@
          ''')
              sh "chmod +x non_existing_thingie"
              def testString1 = "./non_existing_thingie clean test "
              def testString2 = "-Denv=beta"
              def testString3 = "-Dpw=Password1!"
              def testString4 = "-Dpw2=test2"
              assert testString3 == "-Dpw=Password1" + '!'
              def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}"
              echo "fts: ${fullTestString}"
              sh "echo ${fullTestString}"
              sh "eval ${fullTestString}"
              def testString5 = "./non_existing_thingie -Dpw=Password1!"
              sh "eval ${testString5}"
          }
          

          resulted in

          [Pipeline] node
          Running on centos in /home/jenkins/workspace/bug-reproduction/jenkins-48271
          [Pipeline] {
          [Pipeline] writeFile
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + chmod +x non_existing_thingie
          [Pipeline] echo
          fts: ./non_existing_thingie clean test  -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + echo ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + eval ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          ++ ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
          clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
          [Pipeline] sh
          [jenkins-48271] Running shell script
          + eval ./non_existing_thingie '-Dpw=Password1!'
          ++ ./non_existing_thingie '-Dpw=Password1!'
          -Dpw=Password1!
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          
          abayer Andrew Bayer added a comment - Ok, I could be wrong: This: node { writeFile(file: 'non_existing_thingie' ,text:''' #!/bin/bash echo $@ ''') sh "chmod +x non_existing_thingie" def testString1 = "./non_existing_thingie clean test " def testString2 = "-Denv=beta" def testString3 = "-Dpw=Password1!" def testString4 = "-Dpw2=test2" assert testString3 == "-Dpw=Password1" + '!' def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}" echo "fts: ${fullTestString}" sh "echo ${fullTestString}" sh "eval ${fullTestString}" def testString5 = "./non_existing_thingie -Dpw=Password1!" sh "eval ${testString5}" } resulted in [Pipeline] node Running on centos in /home/jenkins/workspace/bug-reproduction/jenkins-48271 [Pipeline] { [Pipeline] writeFile [Pipeline] sh [jenkins-48271] Running shell script + chmod +x non_existing_thingie [Pipeline] echo fts: ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + echo ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + eval ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 ++ ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + eval ./non_existing_thingie '-Dpw=Password1!' ++ ./non_existing_thingie '-Dpw=Password1!' -Dpw=Password1! [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS

          I don't see why one would use "eval" here in the first place, since ${testString1}${testString2} is expanding Groovy variables rather than shell variables.

          cmehdy, what is the error you get from maven? Does it fail slowly enough that you have time to run "ps" during the build and check the command-line arguments that way?

          kon Kalle Niemitalo added a comment - I don't see why one would use "eval" here in the first place, since ${testString1}${testString2} is expanding Groovy variables rather than shell variables. cmehdy , what is the error you get from maven? Does it fail slowly enough that you have time to run "ps" during the build and check the command-line arguments that way?
          cmehdy Mehdy Chaillou added a comment - - edited

          I'm creating runtime options on the fly in a few different places, and put it all together at the shell step (with or without the withMaven() plugin).

          With mvnTest being a groovy function (mvnTest.groovy in /vars with a call() defined, to return a string like:

          -Dpassword1=test -Dpassword2=test!2

          The following steps:  

          script {
            mvnCommand = mvnTest(suiteFile)
          }
          
          withMaven(maven: "M3", //){
           options: [
           junitPublisher(disabled:true),
           artifactsPublisher(disabled: true),
           findbugsPublisher(disabled: true),
           openTasksPublisher(disabled: true)
           ]) {
           sh "mvn ${mvnCommand}"
          }

          end up calling maven this way:

          + mvn clean test -Dpasssword1=test '-Dpassword2=test!2' -Dotherstuff=blah
           [INFO] Scanning for projects...

          Something which doesn't show, were I to just try to 'print mvnCommand', or to sh "echo ${mvnCommand}".

           

          Interestingly, and following your suggestion, doing a `ps` while the step is being executed shows that the parameters are passed without single or double quotes.

          I've had to remove some sensitive data so I hope you'll forgive me if it's not the most readable, but you get the point.

           

           

          cmehdy Mehdy Chaillou added a comment - - edited I'm creating runtime options on the fly in a few different places, and put it all together at the shell step (with or without the withMaven() plugin). With mvnTest being a groovy function (mvnTest.groovy in /vars with a call() defined, to return a string like: -Dpassword1=test -Dpassword2=test!2 The following steps:   script {   mvnCommand = mvnTest(suiteFile) } withMaven(maven: "M3" , //){ options: [ junitPublisher(disabled: true ), artifactsPublisher(disabled: true ), findbugsPublisher(disabled: true ), openTasksPublisher(disabled: true ) ]) { sh "mvn ${mvnCommand}" } end up calling maven this way: + mvn clean test -Dpasssword1=test '-Dpassword2=test!2' -Dotherstuff=blah [INFO] Scanning for projects... Something which doesn't show, were I to just try to 'print mvnCommand', or to sh "echo ${mvnCommand}".   Interestingly, and following your suggestion, doing a `ps` while the step is being executed shows that the parameters are passed without single or double quotes. I've had to remove some sensitive data so I hope you'll forgive me if it's not the most readable, but you get the point.    
          abayer Andrew Bayer added a comment -

          So it sounds like this isn't actually an issue, if the Maven process got the argument correctly. I'm going to close this as not a defect.

          abayer Andrew Bayer added a comment - So it sounds like this isn't actually an issue, if the Maven process got the argument correctly. I'm going to close this as not a defect.

          People

            Unassigned Unassigned
            cmehdy Mehdy Chaillou
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: