Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21145

Several concurrent builds waits for each other

XMLWordPrintable

      Hi,

      We have an upstream project(with configuration matrix) which trigers downstream one. Downstream project has "Throttle Concurrent Builds" option enabled with "Throttle this project alone", "Maximum Total Concurrent Builds=2", "Maximum Concurrent Builds Per Node=2"

      Both upstream and downstream projects have "xUnit" related steps in "Post-build Actions"

      Issue there is when we have triggered 10 concurrent builds(#100 - #109) of the downstream project and the third one (#102) requires more time to finish - all further builds (#103 - #109) will be locked until this third build finish.

      Other example, when two builds there were executing simultaneously, one of them failed and took long time to finish.
      It looks like successful one was waiting for unsuccessful one.

      Just compare end of logs for these pairs of tests.
      Pay attention on time stamps: gap between "Stopping recording" and "Sending e-mails":

      jenkins/view/QA/job/_SIP/job/pjsip_c/2900/console
      jenkins/view/QA/job/_SIP/job/pjsip_c/2901/console

      11:07:30 [xUnit] [INFO] - Starting to record.
      11:07:30 [xUnit] [INFO] - Processing JUnit
      11:07:30 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
      11:07:35 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      11:07:35 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      11:07:35 [xUnit] [INFO] - Setting the build status to FAILURE
      11:07:35 [xUnit] [INFO] - Stopping recording.
      11:07:35 Build step 'Publish xUnit test result report' changed build result to FAILURE
      11:07:35 Sending e-mails to:
      11:08:05 Notifying upstream projects of job completion
      11:08:05 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=390
      11:08:05 Finished: FAILURE

      VS

      10:56:40 [xUnit] [INFO] - Setting the build status to SUCCESS
      10:56:40 [xUnit] [INFO] - Stopping recording.
      11:07:34 Sending e-mails to:
      11:08:04 Notifying upstream projects of job completion
      11:08:04 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=380
      11:08:04 Finished: SUCCESS

      And:
      jenkins/view/QA/job/_SIP/job/pjsip_c/2904/console
      jenkins/view/QA/job/_SIP/job/pjsip_c/2905/console

      10:34:52 [xUnit] [INFO] - Starting to record.
      10:34:52 [xUnit] [INFO] - Processing JUnit
      10:34:53 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
      10:34:57 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      10:34:57 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      10:34:57 [xUnit] [INFO] - Setting the build status to FAILURE
      10:34:57 [xUnit] [INFO] - Stopping recording.
      10:34:57 Build step 'Publish xUnit test result report' changed build result to FAILURE
      10:34:57 Sending e-mails to:
      10:35:36 Notifying upstream projects of job completion
      10:35:36 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=390
      10:35:36 Finished: FAILURE

      VS

      10:21:28 [xUnit] [INFO] - Setting the build status to SUCCESS
      10:21:28 [xUnit] [INFO] - Stopping recording.
      10:34:57 Sending e-mails to:
      10:35:31 Notifying upstream projects of job completion
      10:35:31 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=371
      10:35:31 Finished: SUCCESS

      Could you please advise how we can workaround this issue. Any help is appreciated.

            Unassigned Unassigned
            vbaskoff Vitalii Baskov
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: