Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67573

Powershell Pipeline Step hangs job if bogus command is run

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • durable-task-plugin
    • None
    • Jenkins 2.289.3, durable-task-pipeline 1.39
      also Jenkins 2.319.1, durable-task-pipeline 493.v195aefbb0ff2

      This is related to multiple issues that are resolved as fix, but there are still ongoing issues:

      Running a powershell step where the command is not found OR where the command does NOT generate any stdout when returnStdout: true will hang the job

      Use this for the pipeline to duplicate issue:

      pipeline {
         agent {
           label "john-windows"
         }
         stages {
           stage('gobbledygook - will pass') {
             steps{
               // powershell script: 'gobbledygook'
               powershell script: 'date'
             }
           }
           stage('gobbledygook - will fail') {
             steps{
               powershell returnStdout: true, script: 'gobbledygook'
             }
           }
         }
       }

       

      The jobs will hang and will need to be aborted.  Here is he console output:

       

       14:05:18 [Pipeline] Start of Pipeline
       14:05:19 [Pipeline] node
       14:05:34 Still waiting to schedule task
       14:05:34 ‘john-windows’ is offline
       14:05:51 Running on john-windows in c:\jenkins\workspace\test-powershell-stdout
       14:05:51 [Pipeline] {
       14:05:51 [Pipeline] stage
       14:05:51 [Pipeline] { (gobbledygook - will pass)
       14:05:51 [Pipeline] powershell
       14:05:57 
       14:05:57 Wednesday, January 12, 2022 2:05:57 PM
       14:05:58 [Pipeline] }
       14:05:58 [Pipeline] // stage
       14:05:58 [Pipeline] stage
       14:05:58 [Pipeline] { (gobbledygook - will fail)
       14:05:58 [Pipeline] powershell
       14:06:03 Exception calling "WriteAllLines" with "2" argument(s): "Value cannot be null.
       14:06:03 Parameter name: contents"
       14:06:03 At line:1 char:456
       14:06:03 + ... utput.txt"; [IO.File]::WriteAllLines("c:\jenkins\workspace\test-power ...
       14:06:03 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       14:06:03 + CategoryInfo : NotSpecified: ( [], MethodInvocationException
       14:06:03 + FullyQualifiedErrorId : ArgumentNullException
       14:06:03
       14:06:03 gobbledygook : The term 'gobbledygook' is not recognized as the name of a cmdlet, function, script file, or operable
       14:06:03 program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
       14:06:03 At C:\jenkins\workspace\test-powershell-stdout@tmp\durable-58ebe341\powershellScript.ps1:1 char:1
       14:06:03 + gobbledygook
       14:06:03 + ~~~~~~~~~~~~
       14:06:03 + CategoryInfo : ObjectNotFound: (gobbledygook:String) [], ParentContainsErrorRecordException
       14:06:03 + FullyQualifiedErrorId : CommandNotFoundException
       14:06:03
       14:06:03 14:06:03.275691 common.go:40: exit status 1
       14:06:04 Cannot contact john-windows: java.io.FileNotFoundException: c:\jenkins\workspace\test-powershell-stdout@tmp\durable-58ebe341\output.txt (The system cannot find the file specified)
       14:07:32 Aborted by Lengeling John M
       14:07:32 Sending interrupt signal to process
       14:07:47 Click here to forcibly terminate running steps
       14:07:52 After 20s process did not stop
       14:07:52 [Pipeline] }
       14:07:52 [Pipeline] // stage
       14:07:52 [Pipeline] }
       14:07:52 [Pipeline] // node
       14:07:52 [Pipeline] End of Pipeline
       14:07:52 Finished: ABORTED

       

      This issue appears related to other tickets but are marked as resolved and fixed in older versions of durable-task-plugin.

       

      https://issues.jenkins.io/browse/JENKINS-59529

      https://issues.jenkins.io/browse/JENKINS-46508

          [JENKINS-67573] Powershell Pipeline Step hangs job if bogus command is run

          Mark Waite added a comment - - edited

          I can't duplicate the problem as described. Steps I took:

          1. Create a new Pipeline job that uses the Pipeline script
            pipeline {
               agent {
                 label "windows"
               }
               stages {
                 stage('Pass') {
                   steps{
                     powershell script: 'date'
                   }
                 }
                 stage('Fail') {
                   steps{
                     powershell returnStdout: true, script: 'gobbledygook'
                   }
                 }
               }
             }
            
          2. Save the Pipeline job
          3. Run the Pipeline job, confirm that it terminates as expected

          Plugin versions that I was using are listed in my plugins.txt file.

          Does the issue require that I must checkout the Pipeline from SCM?

          Is there something else that makes the problem visible to you and not to me?

          Are all your plugins currently up to date?

          Are you running a different Windows version on the agent than the Windows 10 that I'm running on my agents?

          Are you running a different Powershell version?

          Mark Waite added a comment - - edited I can't duplicate the problem as described. Steps I took: Create a new Pipeline job that uses the Pipeline script pipeline { agent { label "windows" } stages { stage( 'Pass' ) { steps{ powershell script: 'date' } } stage( 'Fail' ) { steps{ powershell returnStdout: true , script: 'gobbledygook' } } } } Save the Pipeline job Run the Pipeline job, confirm that it terminates as expected Plugin versions that I was using are listed in my plugins.txt file . Does the issue require that I must checkout the Pipeline from SCM? Is there something else that makes the problem visible to you and not to me? Are all your plugins currently up to date? Are you running a different Windows version on the agent than the Windows 10 that I'm running on my agents? Are you running a different Powershell version?

          John Lengeling added a comment - - edited

          I took the exact script you tried except changed the label value.  The job hangs.

          I am not using scm to checkout the pipeline.  The pipeline is entered directly in the job.

          I am using latest LTS 2.319.2 and latest plugins.   Durable task plugin version: 493.v195aefbb0ff2 (which I think is the plugin for the powershell() step)
          I see this issue also on 2.319.1, 2.289.3 with earlier plugin versions.  Like: Durable task 1.39 

          The Windows agent is Windows 10:

              c:\jenkins>ver
              Microsoft Windows [Version 10.0.19042.1348]
          

          Powershell version:

          PS C:\jenkins> $psversiontable
          Name Value
          ---- -----
          PSVersion 5.1.19041.1320
          PSEdition Desktop
          PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
          BuildVersion 10.0.19041.1320
          CLRVersion 4.0.30319.42000
          WSManStackVersion 3.0
          PSRemotingProtocolVersion 2.3
          SerializationVersion 1.1.0.1

          The last message of the output in the console before clicking abort is "Cannot contact john-windows: java.io.FileNotFoundException: c:\jenkins\workspace\test-powershell-stdout@tmp\durable-58ebe341\output.txt (The system cannot find the file specified)"  The contents of the folder mentioned is:

          c:\jenkins\workspace\test-powershell-stdout@tmp\durable-3d4099da>dir
           Volume in drive C has no label.
           Volume Serial Number is 9E3C-A775
          Directory of c:\jenkins\workspace\test-powershell-stdout@tmp\durable-3d4099da
          01/13/2022 11:31 AM <DIR> .
          01/13/2022 11:31 AM <DIR> ..
          01/13/2022 11:31 AM 997 jenkins-log.txt
          01/13/2022 11:31 AM 1 jenkins-result.txt
          01/13/2022 11:31 AM 15 powershellScript.ps1
          01/13/2022 11:31 AM 0 temporaryOutput.txt
           4 File(s) 1,013 bytes
           2 Dir(s) 76,536,225,792 bytes free
          

          Would there be some system logging to enable to help?

          John Lengeling added a comment - - edited I took the exact script you tried except changed the label value.  The job hangs. I am not using scm to checkout the pipeline.  The pipeline is entered directly in the job. I am using latest LTS 2.319.2 and latest plugins.   Durable task plugin version: 493.v195aefbb0ff2 (which I think is the plugin for the powershell() step) I see this issue also on 2.319.1, 2.289.3 with earlier plugin versions.  Like: Durable task 1.39  The Windows agent is Windows 10:     c:\jenkins>ver     Microsoft Windows [Version 10.0.19042.1348] Powershell version: PS C:\jenkins> $psversiontable Name Value ---- ----- PSVersion 5.1.19041.1320 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.19041.1320 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1 The last message of the output in the console before clicking abort is " Cannot contact john-windows: java.io.FileNotFoundException: c:\jenkins\workspace\test-powershell-stdout@tmp\durable-58ebe341\output.txt (The system cannot find the file specified) "  The contents of the folder mentioned is: c:\jenkins\workspace\test-powershell-stdout@tmp\durable-3d4099da>dir Volume in drive C has no label. Volume Serial Number is 9E3C-A775 Directory of c:\jenkins\workspace\test-powershell-stdout@tmp\durable-3d4099da 01/13/2022 11:31 AM <DIR> . 01/13/2022 11:31 AM <DIR> .. 01/13/2022 11:31 AM 997 jenkins-log.txt 01/13/2022 11:31 AM 1 jenkins-result.txt 01/13/2022 11:31 AM 15 powershellScript.ps1 01/13/2022 11:31 AM 0 temporaryOutput.txt 4 File(s) 1,013 bytes 2 Dir(s) 76,536,225,792 bytes free Would there be some system logging to enable to help?

          Mark Waite added a comment -

          With the job definition:

          pipeline {
             agent {
               label "windows"
             }
             stages {
               stage('Passes') {
                 steps{
                     bat 'ver'
                     powershell script: 'date'
                     powershell script: '$psversiontable'
                 }
               }
               stage('Hangs') {
                 steps{
                   powershell returnStdout: true, script: 'gobbledygook'
                 }
               }
             }
           }
          

          I see my Windows version is slightly newer than yours and my Powershell version is the same version as yours. Neither of those seem likely to cause the different behavior you see compared to the behavior that I see.

          Microsoft Windows [Version 10.0.19044.1466]
          
          Name                           Value                                                                                   
          ----                           -----                                                                                   
          PSVersion                      5.1.19041.1320                                                                          
          PSEdition                      Desktop                                                                                 
          PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                 
          BuildVersion                   10.0.19041.1320                                                                         
          CLRVersion                     4.0.30319.42000                                                                         
          WSManStackVersion              3.0                                                                                     
          PSRemotingProtocolVersion      2.3                                                                                     
          SerializationVersion           1.1.0.1  
          

          I am running my controller on a Linux machine. Where is your controller running?

          the message "Cannot contact ..." may indicate that the network connection between the controller and the agent is having issues. Do you see the same problem if you use a different Windows agent?

          Mark Waite added a comment - With the job definition: pipeline { agent { label "windows" } stages { stage( 'Passes' ) { steps{ bat 'ver' powershell script: 'date' powershell script: '$psversiontable' } } stage( 'Hangs' ) { steps{ powershell returnStdout: true , script: 'gobbledygook' } } } } I see my Windows version is slightly newer than yours and my Powershell version is the same version as yours. Neither of those seem likely to cause the different behavior you see compared to the behavior that I see. Microsoft Windows [Version 10.0.19044.1466] Name Value ---- ----- PSVersion 5.1.19041.1320 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.19041.1320 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1 I am running my controller on a Linux machine. Where is your controller running? the message "Cannot contact ..." may indicate that the network connection between the controller and the agent is having issues. Do you see the same problem if you use a different Windows agent?

            Unassigned Unassigned
            johnlengeling John Lengeling
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: