Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64751

Environment variables with a trailing double quotes are not escaped correctly on Windows

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • docker-workflow-plugin
    • None
    • Jenkins Version: 2.274
      Docker Pipeline Plugin: 1.25
      Linux master
      Windows agent
    • 1.28-r2

      Seeing an no output hang at the first step run by a docker agent when there is an environment variable with a value that ends with a double quote.

      Like when the github PR being built has a double quote at the end of it's title.

      https://github.com/jenkinsci/docker-workflow-plugin/pull/220 did fix the issue for double quotes in other positions.

       

      In my case, where only github PR titles were an issue, the following is a workaround.

      env.CHANGE_TITLE = env.CHANGE_TITLE.replaceAll('"','\'')

      That prevented the hang that had occurred when the PR title (env.CHANGE_ENV) was

      Checking if "quotes" are correctly ("escaped") "even with trailing ones"

      Note the hang did not occur when PR title (env.CHANGE_ENV) was:

      Checking if "quotes" are correctly ("escaped")

       

          [JENKINS-64751] Environment variables with a trailing double quotes are not escaped correctly on Windows

          Paul "TBBle" Hampson added a comment -

          Possibly the same issue as JENKINS-65933

          Paul "TBBle" Hampson added a comment - Possibly the same issue as JENKINS-65933

          Paul "TBBle" Hampson added a comment - - edited

          We have been hitting this for ages on our CI cluster, and were mystified because a rebuild would often (but not always) fix the issue. It turned out that the problematic env-vars were all coming from the GitLab Branch Source plugin, e.g. the git commit message, and a rebuild locally did not have them and so did not reproduce the issue, but a rebuild triggered from GitLab's UI did send the env-vars again and reproduce.

          For future reference, I put together a Jenkinsfile to demonstrate/test the issue in a couple of variants:

          // Tag is 1809 for ltsc2019, ltsc2022 for ltsc2022
// See https://hub.docker.com/_/microsoft-windows-nanoserver for other options
String baseImage = "mcr.microsoft.com/windows/nanoserver:1809"

// Make this a node name or label for Windows nodes with Docker installed.
String nodeSelector = "docker-windows-lite"

// Outputs TESTENV both inside and outside a container
Closure runTest = {
    node(nodeSelector) {
        stage('test') {
            echo "Running on ${env.NODE_NAME}"
            bat 'echo TEST: [%TESTENV%]'
            docker.image(baseImage).inside() {
                timeout(time: 10, unit: 'SECONDS') {
                    bat 'echo PASS: [%TESTENV%]'
                }
            }
        }
    }
}

parallel emptyEnv: {
    withEnv(['TESTENV=']) {
        runTest()
    }
}, envVar: {
    withEnv(['TESTENV=SIMPLE_VALUE']) {
        runTest()
    }
}, envVarWithSpaces: {
    withEnv(['TESTENV=VALUE WITH SPACES']) {
        runTest()
    }
}, envVarWithQuoteAtEnd: {
    withEnv(['TESTENV=VALUE_WITH_QUOTE_AT_THE_END"']) {
        runTest()
    }
}, envVarWithQuoteInTheMiddle: {
    withEnv(['TESTENV=VALUE_WITH_QUOTE"IN_THE_MIDDLE']) {
        runTest()
    }
}, envVarWithSpacesAndQuoteAtEnd: {
    withEnv(['TESTENV=VALUE WITH SPACES AND QUOTE AT THE END"']) {
        runTest()
    }
}, envVarWithSpacesAndQuoteInTheMiddle: {
    withEnv(['TESTENV=VALUE WITH QUOTE"IN THE MIDDLE']) {
        runTest()
    }
}, multilineEnvVar: {
    withEnv(['''\
TESTENV=SIMPLE_VALUE
A_SECOND_LINE''']) {
        runTest()
    }
}, multilineEnvVarWithSpaces: {
    withEnv(['''\
TESTENV=VALUE WITH SPACES
A SECOND LINE''']) {
        runTest()
    }
}, multilineEnvVarWithQuoteAtEnd: {
    withEnv(['''\
TESTENV=VALUE_WITH_QUOTE
AT_END_OF_SECOND_LINE"''']) {
        runTest()
    }
}, multilineEnvVarWithQuoteInTheMiddle: {
    withEnv(['''\
TESTENV=VALUE_WITH_QUOTE
IN_THE_MIDDLE"OF_SECOND_LINE''']) {
        runTest()
    }
}, multilineEnvVarWithSpacesAndQuoteAtEnd: {
    withEnv(['''\
TESTENV=VALUE WITH SPACES AND QUOTE
AT END OF SECOND LINE"''']) {
        runTest()
    }
}, multilineEnvVarWithSpacesAndQuoteInTheMiddle: {
    withEnv(['''\
TESTENV=VALUE WITH SPACES AND QUOTE
IN THE MIDDLE"OF SECOND LINE''']) {
        runTest()
    }
}, failFast: false

          In our environment (Docker Pipeline Plugin 1.26, Jenkins 2.303.1), the following stages show the failure:

          • envVarWithQuoteAtEnd
          • envVarWithQuoteInTheMiddle
          • envVarWithSpacesAndQuoteInTheMiddle
          • multilineEnvVarWithQuoteAtEnd
          • multilineEnvVarWithQuoteInTheMiddle
          • multilineEnvVarWithSpacesAndQuoteInTheMiddle

          I did check elsewhere that the two examples given by allenbenz gave the same results as originally described, so I suspect that balanced quotes in the middle don't introduce the issue, as my tests here with a single double-quote in the middle all fail. I observe that multiline doesn't actually seem to be relevant; I noticed that at least for the GitLab Branch Source plugin, the commit message env-var always includes the trailing newline so it's always multiline, and it works if there's no quotes there.

          It is interesting that spaces inline makes the single-quote-at-the-end pass. The quote definitely comes through to the batch file in the container so I'm not sure why that is, or why it's different to allenbenz's tests. Again, it might be related to having unbalanced quotes changing the result.

          Note that PR 220 actually fixed this problem when starting the container (adding WindowsUtil.quoteArgument here), the remaining problem we're seeing is when running commands inside the container, i.e. here.

          I suspect the fix is as simple as changing prefix.add(e) to prefix.add(WindowsUtil.quoteArgument(e)) and similar for prefix.addAll(envReduced), except this is not Windows-specific code so it'd have to be conditional on super.IsUnix().

          Paul "TBBle" Hampson added a comment - - edited We have been hitting this for ages on our CI cluster, and were mystified because a rebuild would often (but not always) fix the issue. It turned out that the problematic env-vars were all coming from the GitLab Branch Source plugin, e.g. the git commit message, and a rebuild locally did not have them and so did not reproduce the issue, but a rebuild triggered from GitLab's UI did send the env-vars again and reproduce. For future reference, I put together a Jenkinsfile to demonstrate/test the issue in a couple of variants: // Tag is 1809 for ltsc2019, ltsc2022 for ltsc2022 // See https://hub.docker.com/_/microsoft-windows-nanoserver for other options String baseImage = "mcr.microsoft.com/windows/nanoserver:1809" // Make this a node name or label for Windows nodes with Docker installed. String nodeSelector = "docker-windows-lite" // Outputs TESTENV both inside and outside a container Closure runTest = { node(nodeSelector) { stage( 'test' ) { echo "Running on ${env.NODE_NAME}" bat 'echo TEST: [%TESTENV%]' docker.image(baseImage).inside() { timeout(time: 10, unit: 'SECONDS' ) { bat 'echo PASS: [%TESTENV%]' } } } } } parallel emptyEnv: { withEnv([ 'TESTENV=' ]) { runTest() } }, envVar: { withEnv([ 'TESTENV=SIMPLE_VALUE' ]) { runTest() } }, envVarWithSpaces: { withEnv([ 'TESTENV=VALUE WITH SPACES' ]) { runTest() } }, envVarWithQuoteAtEnd: { withEnv([ 'TESTENV=VALUE_WITH_QUOTE_AT_THE_END"' ]) { runTest() } }, envVarWithQuoteInTheMiddle: { withEnv([ 'TESTENV=VALUE_WITH_QUOTE"IN_THE_MIDDLE' ]) { runTest() } }, envVarWithSpacesAndQuoteAtEnd: { withEnv([ 'TESTENV=VALUE WITH SPACES AND QUOTE AT THE END"' ]) { runTest() } }, envVarWithSpacesAndQuoteInTheMiddle: { withEnv([ 'TESTENV=VALUE WITH QUOTE"IN THE MIDDLE' ]) { runTest() } }, multilineEnvVar: { withEnv(['''\ TESTENV=SIMPLE_VALUE A_SECOND_LINE''']) { runTest() } }, multilineEnvVarWithSpaces: { withEnv(['''\ TESTENV=VALUE WITH SPACES A SECOND LINE''']) { runTest() } }, multilineEnvVarWithQuoteAtEnd: { withEnv(['''\ TESTENV=VALUE_WITH_QUOTE AT_END_OF_SECOND_LINE"''']) { runTest() } }, multilineEnvVarWithQuoteInTheMiddle: { withEnv(['''\ TESTENV=VALUE_WITH_QUOTE IN_THE_MIDDLE"OF_SECOND_LINE''']) { runTest() } }, multilineEnvVarWithSpacesAndQuoteAtEnd: { withEnv(['''\ TESTENV=VALUE WITH SPACES AND QUOTE AT END OF SECOND LINE"''']) { runTest() } }, multilineEnvVarWithSpacesAndQuoteInTheMiddle: { withEnv(['''\ TESTENV=VALUE WITH SPACES AND QUOTE IN THE MIDDLE"OF SECOND LINE''']) { runTest() } }, failFast: false In our environment (Docker Pipeline Plugin 1.26, Jenkins 2.303.1), the following stages show the failure: envVarWithQuoteAtEnd envVarWithQuoteInTheMiddle envVarWithSpacesAndQuoteInTheMiddle multilineEnvVarWithQuoteAtEnd multilineEnvVarWithQuoteInTheMiddle multilineEnvVarWithSpacesAndQuoteInTheMiddle I did check elsewhere that the two examples given by allenbenz gave the same results as originally described, so I suspect that balanced quotes in the middle don't introduce the issue, as my tests here with a single double-quote in the middle all fail. I observe that multiline doesn't actually seem to be relevant; I noticed that at least for the GitLab Branch Source plugin, the commit message env-var always includes the trailing newline so it's always multiline, and it works if there's no quotes there. It is interesting that spaces inline makes the single-quote-at-the-end pass. The quote definitely comes through to the batch file in the container so I'm not sure why that is, or why it's different to allenbenz 's tests. Again, it might be related to having unbalanced quotes changing the result. Note that PR 220 actually fixed this problem when starting the container (adding WindowsUtil.quoteArgument here ), the remaining problem we're seeing is when running commands inside the container, i.e. here . I suspect the fix is as simple as changing prefix.add(e) to prefix.add(WindowsUtil.quoteArgument(e)) and similar for prefix.addAll(envReduced) , except this is not Windows-specific code so it'd have to be conditional on super.IsUnix() .

          Paul "TBBle" Hampson added a comment - - edited

          Separately concerning for this issue, is that docker event shows that in the failure cases, no exec_create event occurs, suggesting that the CLI or dockerd rejected the call, but somehow Jenkins doesn't see a failure there. I'm not sure if docker is not returning an error code, or if somehow Jenkins is missing it, but I can see error logs from the attempt to call docker exec ps (which fails on Windows and probably should also be addressed), so failures from inside the container come through okay.

          If this was working correctly, then this bug would be a "fails" rather than "hangs" problem.

          Paul "TBBle" Hampson added a comment - - edited Separately concerning for this issue, is that docker event shows that in the failure cases, no exec_create event occurs, suggesting that the CLI or dockerd rejected the call, but somehow Jenkins doesn't see a failure there. I'm not sure if docker is not returning an error code, or if somehow Jenkins is missing it, but I can see error logs from the attempt to call docker exec ps (which fails on Windows and probably should also be addressed), so failures from inside the container come through okay. If this was working correctly, then this bug would be a "fails" rather than "hangs" problem.

          Paul "TBBle" Hampson added a comment -

          I've drafted the fix described above as PR 245, but have not yet tested it.

          Paul "TBBle" Hampson added a comment - I've drafted the fix described above as PR 245 , but have not yet tested it.

          Basil Crow added a comment -

          Fixed in jenkinsci/docker-workflow-plugin#245. Released in 1.28.

          Basil Crow added a comment - Fixed in jenkinsci/docker-workflow-plugin#245 . Released in 1.28 .

            p_hampson Paul "TBBle" Hampson
            allenbenz Allen Benz
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: