Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68211

Parallel stages fails with no exception reported in Build logs/System logs

XMLWordPrintable

      I have a jenkins job running on a Openshift pod.

      It has 2 stages running in parallel.

      1. Running few stages in sequence : mostly mvn commands
      1. Running individual stage : helm commands

      My pipeline syntax looks something like :

      parallel{ 
  stage('Parallel-Process-1'){ 
    stages{ 
      stage('Build Stage'){ 
        //mvn commands
      } 
      stage('Upload docker image'){ 
        //mvn command 
      }
    }
  } 
  stage('Parallel-Process-2'){ 
     //helm commands 
  } 
}

      With this, Pipeline is failing in one of the child stages of Parallel-Process-1

      In the logs, there is no exception. mvn process logs are disconnected and post-action starts.

      Jenkins somehow(incorrectly) seems to get a signal that the build stage has completed and moves to the post activity.

       

      [DEBUG] Writing tracking file /home/jenkins/.m2/repository/org/apache/maven/doxia/doxia-core/1.8/_remote.repositories 
|DEBUG| Single MS CI | Build and UT | Publishing Junit test results Recording test results No test report files were found. Configuration error?

      Findings :

      1. mvn process did not get terminated (I could see the complete success logs in the pods workspace)

      Question : Why Jenkins is not able to monitor the nested parallel process ?

      Tools : Jenkins-2.235.2 Declarative Pipeline plugin-1.5.1 Job type : Multibranch (Workflow)

          [JENKINS-68211] Parallel stages fails with no exception reported in Build logs/System logs

          Suraj Waragade created issue -
          Suraj Waragade made changes -
          Description Original: I have a jenkins job running on a Openshift pod.

          It has 2 stages running in parallel.
           # Build Stages : mvn command

           # Deploy Stage : helm commands

          My pipeline syntax looks something like :
          {code:java}
          // Declarative

          parallel{

            stage('Group Stage Execution'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }

            stage('Deploy Stage'){
              //helm commands }
            }

          }
          {code}
          With this, mvn command in `Build Stage` is killed by some process.

          There is no exception in the build logs/Jenkins logs

          -------------------

          Then I made a slight change in the pipeline syntax :
          {code:java}
          parallel{

            stage('Group Stage Execution'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }

            stage('Deploy Stage Execution'){
              stages{
                stage('Deploy Stage'){
                //helm commands
                }
              }
            }

          }
          {code}
          I just wrapped `Deploy Stage` as the other stages.

          With this, the frequency of failure has dropped quite a bit.

           

          Question 1 : Is the process really getting killed ? Who is killing the process and why ?

          Question 2 : How does second syntax seems to tackle the issue when it is doing the exact same thing ?

                      		New: I have a jenkins job running on a Openshift pod.

          It has 2 stages running in parallel.
           # Build Stages : mvn command

           # Deploy Stage : helm commands

          My pipeline syntax looks something like :
          {code:java}
          // Declarative

          parallel{

            stage('Group Stage Execution'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }

            stage('Deploy Stage'){
              //helm commands }
            }

          }
          {code}
          With this, mvn command in `Build Stage` is killed by some process. (happens 8/10 times, runs fine otherwise)

          There is no exception in the build logs/Jenkins logs

          -------------------

          Then I made a slight change in the pipeline syntax :
          {code:java}
          parallel{

            stage('Group Stage Execution'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }

            stage('Deploy Stage Execution'){
              stages{
                stage('Deploy Stage'){
                //helm commands
                }
              }
            }

          }
          {code}
          I just wrapped `Deploy Stage` as the other stages.

          With this, the frequency of failure has dropped quite a bit. ( 2/10 failures )

           

          Question 1 : Is the process really getting killed ? Who is killing the process and why ?

          Question 2 : How does second syntax seems to tackle the issue when it is doing the exact same thing ?

           
          Suraj Waragade made changes -
          Description Original: I have a jenkins job running on a Openshift pod.

          It has 2 stages running in parallel.
           # Build Stages : mvn command

           # Deploy Stage : helm commands

          My pipeline syntax looks something like :
          {code:java}
          // Declarative

          parallel{

            stage('Group Stage Execution'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }

            stage('Deploy Stage'){
              //helm commands }
            }

          }
          {code}
          With this, mvn command in `Build Stage` is killed by some process. (happens 8/10 times, runs fine otherwise)

          There is no exception in the build logs/Jenkins logs

          -------------------

          Then I made a slight change in the pipeline syntax :
          {code:java}
          parallel{

            stage('Group Stage Execution'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }

            stage('Deploy Stage Execution'){
              stages{
                stage('Deploy Stage'){
                //helm commands
                }
              }
            }

          }
          {code}
          I just wrapped `Deploy Stage` as the other stages.

          With this, the frequency of failure has dropped quite a bit. ( 2/10 failures )

           

          Question 1 : Is the process really getting killed ? Who is killing the process and why ?

          Question 2 : How does second syntax seems to tackle the issue when it is doing the exact same thing ?

                      		New: I have a jenkins job running on a Openshift pod.

          It has 2 stages running in parallel.
           # Running few stages in sequence : mostly mvn commands

           # Running individual stage : helm commands

          My pipeline syntax looks something like :

           

          {{}}
          {code:java}
          parallel{
            stage('Parallel-Process-1'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }
            stage('Parallel-Process-2'){
               //helm commands
            }
          }{code}
          {{}}

          With this, Pipeline is failing in one of the child stages of {{Parallel-Process-1}}

          In the logs, there is no exception. mvn process logs are disconnected and post-action starts.

          Jenkins somehow(incorrectly) seems to get a signal that the build stage has completed and moves to the post activity.

           

          {{}}
          {code:java}
          [DEBUG] Writing tracking file /home/jenkins/.m2/repository/org/apache/maven/doxia/doxia-core/1.8/_remote.repositories
          |DEBUG| Single MS CI | Build and UT | Publishing Junit test results Recording test results No test report files were found. Configuration error?{code}
          {{}}

          Findings :
           # mvn process did not get terminated (I could see the complete success logs in the pods workspace)

          Question : Why Jenkins is not able to monitor the nested parallel process ?

          Tools : Jenkins-2.235.2 Declarative Pipeline plugin-1.5.1 Job type : Multibranch (Workflow)
          Suraj Waragade made changes -
          Description Original: I have a jenkins job running on a Openshift pod.

          It has 2 stages running in parallel.
           # Running few stages in sequence : mostly mvn commands

           # Running individual stage : helm commands

          My pipeline syntax looks something like :

           

          {{}}
          {code:java}
          parallel{
            stage('Parallel-Process-1'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }
            stage('Parallel-Process-2'){
               //helm commands
            }
          }{code}
          {{}}

          With this, Pipeline is failing in one of the child stages of {{Parallel-Process-1}}

          In the logs, there is no exception. mvn process logs are disconnected and post-action starts.

          Jenkins somehow(incorrectly) seems to get a signal that the build stage has completed and moves to the post activity.

           

          {{}}
          {code:java}
          [DEBUG] Writing tracking file /home/jenkins/.m2/repository/org/apache/maven/doxia/doxia-core/1.8/_remote.repositories
          |DEBUG| Single MS CI | Build and UT | Publishing Junit test results Recording test results No test report files were found. Configuration error?{code}
          {{}}

          Findings :
           # mvn process did not get terminated (I could see the complete success logs in the pods workspace)

          Question : Why Jenkins is not able to monitor the nested parallel process ?

          Tools : Jenkins-2.235.2 Declarative Pipeline plugin-1.5.1 Job type : Multibranch (Workflow)           		New: I have a jenkins job running on a Openshift pod.

          It has 2 stages running in parallel.
           # Running few stages in sequence : mostly mvn commands

           # Running individual stage : helm commands

          My pipeline syntax looks something like :
          {code:java}
          parallel{
            stage('Parallel-Process-1'){
              stages{
                stage('Build Stage'){
                  //mvn commands
                }
                stage('Upload docker image'){
                  //mvn command
                }
              }
            }
            stage('Parallel-Process-2'){
               //helm commands
            }
          }{code}
          With this, Pipeline is failing in one of the child stages of {{Parallel-Process-1}}

          In the logs, there is no exception. mvn process logs are disconnected and post-action starts.

          Jenkins somehow(incorrectly) seems to get a signal that the build stage has completed and moves to the post activity.

           
          {code:java}
          [DEBUG] Writing tracking file /home/jenkins/.m2/repository/org/apache/maven/doxia/doxia-core/1.8/_remote.repositories
          |DEBUG| Single MS CI | Build and UT | Publishing Junit test results Recording test results No test report files were found. Configuration error?{code}
          Findings :
           # mvn process did not get terminated (I could see the complete success logs in the pods workspace)

          Question : Why Jenkins is not able to monitor the nested parallel process ?

          Tools : Jenkins-2.235.2 Declarative Pipeline plugin-1.5.1 Job type : Multibranch (Workflow)

          Mark Waite added a comment -

          Closing as "Cannot reproduce"

          Mark Waite added a comment - Closing as "Cannot reproduce"
          Mark Waite made changes -
          Resolution New: Cannot Reproduce [ 5 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

            Unassigned Unassigned
            swaragade Suraj Waragade
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: