Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21145

Several concurrent builds waits for each other

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi,

      We have an upstream project(with configuration matrix) which trigers downstream one. Downstream project has "Throttle Concurrent Builds" option enabled with "Throttle this project alone", "Maximum Total Concurrent Builds=2", "Maximum Concurrent Builds Per Node=2"

      Both upstream and downstream projects have "xUnit" related steps in "Post-build Actions"

      Issue there is when we have triggered 10 concurrent builds(#100 - #109) of the downstream project and the third one (#102) requires more time to finish - all further builds (#103 - #109) will be locked until this third build finish.

      Other example, when two builds there were executing simultaneously, one of them failed and took long time to finish.
      It looks like successful one was waiting for unsuccessful one.

      Just compare end of logs for these pairs of tests.
      Pay attention on time stamps: gap between "Stopping recording" and "Sending e-mails":

      jenkins/view/QA/job/_SIP/job/pjsip_c/2900/console
      jenkins/view/QA/job/_SIP/job/pjsip_c/2901/console

      11:07:30 [xUnit] [INFO] - Starting to record.
      11:07:30 [xUnit] [INFO] - Processing JUnit
      11:07:30 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
      11:07:35 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      11:07:35 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      11:07:35 [xUnit] [INFO] - Setting the build status to FAILURE
      11:07:35 [xUnit] [INFO] - Stopping recording.
      11:07:35 Build step 'Publish xUnit test result report' changed build result to FAILURE
      11:07:35 Sending e-mails to:
      11:08:05 Notifying upstream projects of job completion
      11:08:05 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=390
      11:08:05 Finished: FAILURE

      VS

      10:56:40 [xUnit] [INFO] - Setting the build status to SUCCESS
      10:56:40 [xUnit] [INFO] - Stopping recording.
      11:07:34 Sending e-mails to:
      11:08:04 Notifying upstream projects of job completion
      11:08:04 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=380
      11:08:04 Finished: SUCCESS

      And:
      jenkins/view/QA/job/_SIP/job/pjsip_c/2904/console
      jenkins/view/QA/job/_SIP/job/pjsip_c/2905/console

      10:34:52 [xUnit] [INFO] - Starting to record.
      10:34:52 [xUnit] [INFO] - Processing JUnit
      10:34:53 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
      10:34:57 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      10:34:57 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      10:34:57 [xUnit] [INFO] - Setting the build status to FAILURE
      10:34:57 [xUnit] [INFO] - Stopping recording.
      10:34:57 Build step 'Publish xUnit test result report' changed build result to FAILURE
      10:34:57 Sending e-mails to:
      10:35:36 Notifying upstream projects of job completion
      10:35:36 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=390
      10:35:36 Finished: FAILURE

      VS

      10:21:28 [xUnit] [INFO] - Setting the build status to SUCCESS
      10:21:28 [xUnit] [INFO] - Stopping recording.
      10:34:57 Sending e-mails to:
      10:35:31 Notifying upstream projects of job completion
      10:35:31 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=371
      10:35:31 Finished: SUCCESS

      Could you please advise how we can workaround this issue. Any help is appreciated.

        Attachments

          Issue Links

            Activity

            vbaskoff Vitalii Baskov created issue -
            vbaskoff Vitalii Baskov made changes -
            Field Original Value New Value
            Environment Gentoo Jenkins 1.532.1
            vbaskoff Vitalii Baskov made changes -
            Description Hi,

            We have an upstream project(with configuration matrix) which tigers downstream one. Downstream project has "Throttle Concurrent Builds" tick enabled with "Throttle this project alone", "Maximum Total Concurrent Builds=2", "Maximum Concurrent Builds Per Nod=2"

            Both upstream and downstream projects have "xUnit" related steps in "Post-build Actions"

            Issue there is when we have triggered 10 concurrent builds(#100 - #109) of the downstream project and the third one (#102) requires more time to finish - all further builds (#103 - #109) will be locked until this third build finish.

            Other example, when two builds there were executing simultaneously, one of them failed and took long time to finish.
            It looks like successful one was waiting for unsuccessful one.

            Just compare end of logs for these pairs of tests:
            https://jenkins/view/QA/job/_SIP/job/pjsip_c/2900/console
            https://jenkins/view/QA/job/_SIP/job/pjsip_c/2901/console


            11:07:30 [xUnit] [INFO] - Starting to record.
            11:07:30 [xUnit] [INFO] - Processing JUnit
            11:07:30 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
            11:07:35 [xUnit] [INFO] - Check 'Failed Tests' threshold.
            11:07:35 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
            11:07:35 [xUnit] [INFO] - Setting the build status to FAILURE
            11:07:35 [xUnit] [INFO] - Stopping recording.
            11:07:35 Build step 'Publish xUnit test result report' changed build result to FAILURE
            11:07:35 Sending e-mails to:
            11:08:05 Notifying upstream projects of job completion
            11:08:05 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=390
            11:08:05 Finished: FAILURE

            VS

            10:56:40 [xUnit] [INFO] - Setting the build status to SUCCESS
            10:56:40 [xUnit] [INFO] - Stopping recording.
            11:07:34 Sending e-mails to:
            11:08:04 Notifying upstream projects of job completion
            11:08:04 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=380
            11:08:04 Finished: SUCCESS


            And:
            https://jenkins/view/QA/job/_SIP/job/pjsip_c/2904/console
            https://jenkins/view/QA/job/_SIP/job/pjsip_c/2905/console

            10:34:52 [xUnit] [INFO] - Starting to record.
            10:34:52 [xUnit] [INFO] - Processing JUnit
            10:34:53 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
            10:34:57 [xUnit] [INFO] - Check 'Failed Tests' threshold.
            10:34:57 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
            10:34:57 [xUnit] [INFO] - Setting the build status to FAILURE
            10:34:57 [xUnit] [INFO] - Stopping recording.
            10:34:57 Build step 'Publish xUnit test result report' changed build result to FAILURE
            10:34:57 Sending e-mails to:
            10:35:36 Notifying upstream projects of job completion
            10:35:36 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=390
            10:35:36 Finished: FAILURE

            VS

            10:21:28 [xUnit] [INFO] - Setting the build status to SUCCESS
            10:21:28 [xUnit] [INFO] - Stopping recording.
            10:34:57 Sending e-mails to:
            10:35:31 Notifying upstream projects of job completion
            10:35:31 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=371
            10:35:31 Finished: SUCCESS


            Could you prompt us how to overcome this issue? Any help is appreciated.
            Hi,

            We have an upstream project(with configuration matrix) which trigers downstream one. Downstream project has "Throttle Concurrent Builds" option enabled with "Throttle this project alone", "Maximum Total Concurrent Builds=2", "Maximum Concurrent Builds Per Node=2"

            Both upstream and downstream projects have "xUnit" related steps in "Post-build Actions"

            Issue there is when we have triggered 10 concurrent builds(#100 - #109) of the downstream project and the third one (#102) requires more time to finish - all further builds (#103 - #109) will be locked until this third build finish.

            Other example, when two builds there were executing simultaneously, one of them failed and took long time to finish.
            It looks like successful one was waiting for unsuccessful one.

            Just compare end of logs for these pairs of tests.
            Pay attention on time stamps: gap between "Stopping recording" and "Sending e-mails":

            jenkins/view/QA/job/_SIP/job/pjsip_c/2900/console
            jenkins/view/QA/job/_SIP/job/pjsip_c/2901/console


            11:07:30 [xUnit] [INFO] - Starting to record.
            11:07:30 [xUnit] [INFO] - Processing JUnit
            11:07:30 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
            11:07:35 [xUnit] [INFO] - Check 'Failed Tests' threshold.
            11:07:35 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
            11:07:35 [xUnit] [INFO] - Setting the build status to FAILURE
            11:07:35 [xUnit] [INFO] - Stopping recording.
            11:07:35 Build step 'Publish xUnit test result report' changed build result to FAILURE
            11:07:35 Sending e-mails to:
            11:08:05 Notifying upstream projects of job completion
            11:08:05 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=390
            11:08:05 Finished: FAILURE

            VS

            10:56:40 [xUnit] [INFO] - Setting the build status to SUCCESS
            10:56:40 [xUnit] [INFO] - Stopping recording.
            11:07:34 Sending e-mails to:
            11:08:04 Notifying upstream projects of job completion
            11:08:04 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=380
            11:08:04 Finished: SUCCESS


            And:
            jenkins/view/QA/job/_SIP/job/pjsip_c/2904/console
            jenkins/view/QA/job/_SIP/job/pjsip_c/2905/console

            10:34:52 [xUnit] [INFO] - Starting to record.
            10:34:52 [xUnit] [INFO] - Processing JUnit
            10:34:53 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
            10:34:57 [xUnit] [INFO] - Check 'Failed Tests' threshold.
            10:34:57 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
            10:34:57 [xUnit] [INFO] - Setting the build status to FAILURE
            10:34:57 [xUnit] [INFO] - Stopping recording.
            10:34:57 Build step 'Publish xUnit test result report' changed build result to FAILURE
            10:34:57 Sending e-mails to:
            10:35:36 Notifying upstream projects of job completion
            10:35:36 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=390
            10:35:36 Finished: FAILURE

            VS

            10:21:28 [xUnit] [INFO] - Setting the build status to SUCCESS
            10:21:28 [xUnit] [INFO] - Stopping recording.
            10:34:57 Sending e-mails to:
            10:35:31 Notifying upstream projects of job completion
            10:35:31 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=371
            10:35:31 Finished: SUCCESS


            Could you please advise how we can workaround this issue. Any help is appreciated.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Post-Build actions (publishers) define the locking behavior in concurrent builds on their own. I've re-assigned the issue to the xunit plugin

            Show
            oleg_nenashev Oleg Nenashev added a comment - Post-Build actions (publishers) define the locking behavior in concurrent builds on their own. I've re-assigned the issue to the xunit plugin
            oleg_nenashev Oleg Nenashev made changes -
            Component/s xunit [ 15636 ]
            Component/s core [ 15593 ]
            Assignee Gregory Boissinot [ gbois ]
            Hide
            danielbeck Daniel Beck added a comment -

            This looks more like email-ext before 2.37 when it synchronized on BuildStepMonitor to send accurate subjects ('Still Unstable', 'Fixed', etc.). (JENKINS-16376)

            What version of email-ext plugin do you use?

            Show
            danielbeck Daniel Beck added a comment - This looks more like email-ext before 2.37 when it synchronized on BuildStepMonitor to send accurate subjects ('Still Unstable', 'Fixed', etc.). ( JENKINS-16376 ) What version of email-ext plugin do you use?
            Hide
            danielbeck Daniel Beck added a comment -

            Given that this was filed before Email-ext 2.37 with JENKINS-16376 fix was out, I'm resolving this one optimistically

            Show
            danielbeck Daniel Beck added a comment - Given that this was filed before Email-ext 2.37 with JENKINS-16376 fix was out, I'm resolving this one optimistically
            danielbeck Daniel Beck made changes -
            Assignee Gregory Boissinot [ gbois ]
            Resolution Cannot Reproduce [ 5 ]
            Status Open [ 1 ] Resolved [ 5 ]
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-16376 [ JENKINS-16376 ]
            danielbeck Daniel Beck made changes -
            Component/s email-ext [ 15538 ]
            Component/s xunit [ 15636 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 153007 ] JNJira + In-Review [ 194430 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              vbaskoff Vitalii Baskov
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: