Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Not A Defect
-
MacOS 10.13.1
Jenkins 2.92
JDK 1.8 Oracle
Description
Using this as a minimal example:
script {
def testString1 = "echo a="
def testString2 = "!123"
def testString3 = "b="
def testString4 = "2937"
print "testString1+2 = "+testString1+testString2
print "testString3+4 = "+testString3+testString4
print "total: "+testString1+testString2+testString3+testString4
sh "echo ${testString1}${testString2}"
sh "eval ${testString1}${testString2}"
sh "echo ${testString1}${testString2} ${testString3}${testString4}"
sh "eval ${testString1}${testString2} ${testString3}${testString4}"
}
The result while running the pipeline is:
[Pipeline] script
[Pipeline] {
[Pipeline] echo
testString1+2 = echo a=!123
[Pipeline] echo
testString3+4 = b=2937
[Pipeline] echo
total: echo a=!123b=2937
[Pipeline] sh
[pipeName] Running shell script
+ echo echo 'a=!123'
echo a=!123
[Pipeline] sh
[pipeName] Running shell script
+ eval echo 'a=!123'
++ echo 'a=!123'
a=!123
[Pipeline] sh
[pipeName] Running shell script
+ echo echo 'a=!123' b=2937
echo a=!123 b=2937
[Pipeline] sh
[pipeName] Running shell script
+ eval echo 'a=!123' b=2937
++ echo 'a=!123' b=2937
a=!123 b=2937
[Pipeline] }
[Pipeline] // script
Notice that last echo treats strings differently depending on whether or not the exclamation mark is contained within a contiguous string.
This creates issues when, for example, creating maven options dynamically at runtime. Trying to generate
-Dopt1=test -Dopt2=test!1
leads to a string
-Dopt1=test '-Dopt2=test!1'
which then doesn't work.
This treatment of strings lacking consistency, and despite all my efforts to try to trim the string (or create the string within a shell script, etc), ultimately this issue is a bottleneck.
Let me know if I am missing something, I will happily provide more information if needed.
Attachments
Activity
Mehdy Chaillou
created issue -
| Field | Original Value | New Value |
|---|---|---|
| Component/s | pipeline-model-definition-plugin [ 21706 ] | |
| Component/s | pipeline [ 21692 ] |
| Assignee | Andrew Bayer [ abayer ] |
Mehdy Chaillou
added a comment - Unfortunately the single quotes DO get carried over to commands such as maven, and commands like this when built as a string and eval'd then fail:
def testString1 = "mvn clean test "
def testString2 = "-Denv=beta"
def testString3 = "-Dpw=Password1!"
def testString4 = "-Dpw2=test2"
sh "echo ${testString1} ${testString2} ${testString3} ${testString4}"
sh "eval ${testString1} ${testString2} ${testString3} ${testString4}"
leads to:
mvn clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
The eco suppresses single quotes, but when trying to create such a string (or when using the withMaven() plugin, either way) we end up with that issue still.
Mehdy Chaillou
added a comment - Unfortunately the single quotes DO get carried over to commands such as maven, and commands like this when built as a string and eval'd then fail:
def testString1 = "mvn clean test "
def testString2 = "-Denv=beta"
def testString3 = "-Dpw=Password1!"
def testString4 = "-Dpw2=test2"
sh "echo ${testString1} ${testString2} ${testString3} ${testString4}"
sh "eval ${testString1} ${testString2} ${testString3} ${testString4}"
leads to:
mvn clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
The eco suppresses single quotes, but when trying to create such a string (or when using the withMaven() plugin, either way) we end up with that issue still. Very interesting. Need to figure out where that quoting is happening. I have verified that this doesn't happen in Scripted Pipeline, only in Declarative, so that helps narrow it down. =)
| Component/s | workflow-durable-task-step-plugin [ 21715 ] | |
| Component/s | pipeline-model-definition-plugin [ 21706 ] |
| Assignee | Andrew Bayer [ abayer ] |
Ah, no, I'm wrong, it's reproducible in Scripted with this:
node {
def testString1 = "non_existing_thingie clean test "
def testString2 = "-Denv=beta"
def testString3 = "-Dpw=Password1!"
def testString4 = "-Dpw2=test2"
assert testString3 == "-Dpw=Password1" + '!'
def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}"
echo "fts: ${fullTestString}"
sh "echo ${fullTestString}"
sh "eval ${fullTestString}"
}
fts is fine, but as soon as it gets passed to sh, the single quotes show up.
| Summary | Declarative Pipelines: Strings with exclamation mark get single quoted | Strings with exclamation mark passed to sh step get single quoted |
Kalle Niemitalo
added a comment - abayer, did your tests emit single quotes in the trace output only, or somewhere else as well?
Kalle Niemitalo
added a comment - abayer , did your tests emit single quotes in the trace output only, or somewhere else as well? Both, looks like -
[jenkins-48271] Running shell script + echo non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2 [Pipeline] sh [jenkins-48271] Running shell script + eval non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2 ++ non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
Kalle Niemitalo
added a comment - That last line starting with two plus signs is also trace output from the shell. It has two plus signs because it describes a nested evaluation caused by the eval keyword.
Kalle Niemitalo
added a comment - That last line starting with two plus signs is also trace output from the shell. It has two plus signs because it describes a nested evaluation caused by the eval keyword. And
def testString5 = "non_existing_thingie -Dpw=Password1!" sh "eval ${testString5}"
resulted in
+ eval non_existing_thingie '-Dpw=Password1!' ++ non_existing_thingie '-Dpw=Password1!'
Ok, I could be wrong:
This:
node {
writeFile(file:'non_existing_thingie',text:'''
#!/bin/bash
echo $@
''')
sh "chmod +x non_existing_thingie"
def testString1 = "./non_existing_thingie clean test "
def testString2 = "-Denv=beta"
def testString3 = "-Dpw=Password1!"
def testString4 = "-Dpw2=test2"
assert testString3 == "-Dpw=Password1" + '!'
def fullTestString = "${testString1} ${testString2} ${testString3} ${testString4}"
echo "fts: ${fullTestString}"
sh "echo ${fullTestString}"
sh "eval ${fullTestString}"
def testString5 = "./non_existing_thingie -Dpw=Password1!"
sh "eval ${testString5}"
}
resulted in
[Pipeline] node
Running on centos in /home/jenkins/workspace/bug-reproduction/jenkins-48271
[Pipeline] {
[Pipeline] writeFile
[Pipeline] sh
[jenkins-48271] Running shell script
+ chmod +x non_existing_thingie
[Pipeline] echo
fts: ./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
[Pipeline] sh
[jenkins-48271] Running shell script
+ echo ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
./non_existing_thingie clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
[Pipeline] sh
[jenkins-48271] Running shell script
+ eval ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
++ ./non_existing_thingie clean test -Denv=beta '-Dpw=Password1!' -Dpw2=test2
clean test -Denv=beta -Dpw=Password1! -Dpw2=test2
[Pipeline] sh
[jenkins-48271] Running shell script
+ eval ./non_existing_thingie '-Dpw=Password1!'
++ ./non_existing_thingie '-Dpw=Password1!'
-Dpw=Password1!
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Kalle Niemitalo
added a comment - I don't see why one would use "eval" here in the first place, since ${testString1}${testString2} is expanding Groovy variables rather than shell variables.
cmehdy, what is the error you get from maven? Does it fail slowly enough that you have time to run "ps" during the build and check the command-line arguments that way?
Kalle Niemitalo
added a comment - I don't see why one would use "eval" here in the first place, since ${testString1}${testString2} is expanding Groovy variables rather than shell variables.
cmehdy , what is the error you get from maven? Does it fail slowly enough that you have time to run "ps" during the build and check the command-line arguments that way? Mehdy Chaillou
made changes -
| Attachment | image-2017-11-29-16-40-26-819.png [ 40536 ] |
I'm creating runtime options on the fly in a few different places, and put it all together at the shell step (with or without the withMaven() plugin).
With mvnTest being a groovy function (mvnTest.groovy in /vars with a call() defined, to return a string like:
-Dpassword1=test -Dpassword2=test!2
The following steps:
script {
mvnCommand = mvnTest(suiteFile)
}
withMaven(maven: "M3", //){
options: [
junitPublisher(disabled:true),
artifactsPublisher(disabled: true),
findbugsPublisher(disabled: true),
openTasksPublisher(disabled: true)
]) {
sh "mvn ${mvnCommand}"
}
end up calling maven this way:
+ mvn clean test -Dpasssword1=test '-Dpassword2=test!2' -Dotherstuff=blah [INFO] Scanning for projects...
Something which doesn't show, were I to just try to 'print mvnCommand', or to sh "echo ${mvnCommand}".
Interestingly, and following your suggestion, doing a `ps` while the step is being executed shows that the parameters are passed without single or double quotes.
I've had to remove some sensitive data so I hope you'll forgive me if it's not the most readable, but you get the point.
Mehdy Chaillou
added a comment - - edited I'm creating runtime options on the fly in a few different places, and put it all together at the shell step (with or without the withMaven() plugin).
With mvnTest being a groovy function (mvnTest.groovy in /vars with a call() defined, to return a string like:
-Dpassword1=test -Dpassword2=test!2
The following steps:
script {
mvnCommand = mvnTest(suiteFile)
}
withMaven(maven: "M3" , //){
options: [
junitPublisher(disabled: true ),
artifactsPublisher(disabled: true ),
findbugsPublisher(disabled: true ),
openTasksPublisher(disabled: true )
]) {
sh "mvn ${mvnCommand}"
}
end up calling maven this way:
+ mvn clean test -Dpasssword1=test '-Dpassword2=test!2' -Dotherstuff=blah
[INFO] Scanning for projects...
Something which doesn't show, were I to just try to 'print mvnCommand', or to sh "echo ${mvnCommand}".
Interestingly, and following your suggestion, doing a `ps` while the step is being executed shows that the parameters are passed without single or double quotes.
I've had to remove some sensitive data so I hope you'll forgive me if it's not the most readable, but you get the point.
So it sounds like this isn't actually an issue, if the Maven process got the argument correctly. I'm going to close this as not a defect.
| Resolution | Not A Defect [ 7 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
Your example has single quotes only in the lines that start with a plus sign. Those look like trace output from bash -x:
Indeed, durable-task-plugin adds the -x option: https://github.com/jenkinsci/durable-task-plugin/blob/durable-task-1.17/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L119
I'm pretty sure the single quotes exist only in the trace output and are never seen by maven or other commands that the shell scripts run.