-
Bug
-
Resolution: Fixed
-
Critical
-
This issue is platform independent and happens for any kind of job which sends out final status emails.
When you have a job configured to send out status emails for a job (like success, fail, still failing) the jobs will block the node for up to 15 minutes or even more, only to send out the email. Given that some of our jobs only run 2min this is an absolutely large overhead we have to get rid of soon. It's mainly blocking our internal QA team from signing off of new releases.
Here an example of the last lines of a job which has debug mode enabled for email-ext:
00:09:31.082 Archiving artifacts
00:09:31.093 Recording test results
00:09:31.218 Checking for post-build
00:09:31.218 Performing post-build step
00:09:31.219 Checking if email needs to be generated
00:09:31.219 Email was triggered for: Success
00:09:31.219 Sending email for trigger: Success
00:09:31.219 NOT overriding default server settings, using Mailer to create session
00:24:25.439 messageContentType = text/plain; charset=UTF-8
00:24:25.446 Adding recipients from recipient list
00:31:41.324 Successfully created MimeMessage
00:31:41.324 Sending email to: mozmill-ci@mozilla.org
00:31:41.523 Finished: SUCCESS
As you can see the job is running 9:31 minutes. Then when trying to add the content (i hope that's right here) it takes about 15 minutes, and again 6 more minutes to create the MimeMessage.
During all that time Jenkins has a dramatically high cpu load and it consuming nearly all CPU power with 99% load. As more jobs are running concurrently as worse the situation is becoming.
Right now we are using version 2.32, and we haven't upgraded to the latest yet given that we don't see features or fixes included we would benefit.
Detailed information for our current problems can be found in our own issue tracker: https://github.com/mozilla/mozmill-ci/issues/301
We would appreciate a quick fix if possible. I would be around if you need more information. You can also reach me via IRC. My nickname is whimboo.
Thanks
[JENKINS-20078] email-ext plugin takes a huge amount of time to send out final status emails (30min)
Henrik Skupin
created issue -
Henrik Skupin
made changes -
| Summary | Original: email-ext plugin takes a huge amount of time to send out final status emails (15min) | New: email-ext plugin takes a huge amount of time to send out final status emails (30min) |
Henrik Skupin
added a comment - Sure, so as recipient we have a static entry:
https://github.com/mozilla/mozmill-ci/blob/master/config/production/jenkins.patch#L1300
As content we have this:
https://github.com/mozilla/mozmill-ci/blob/master/jenkins-master/jobs/mozilla-aurora_update/config.xml#L197
As it has been shown about 2 hours ago, a restart of Jenkins temporarily made the problem go away:
02:06:08.483 Checking for post-build
02:06:08.484 Performing post-build step
02:06:08.484 Checking if email needs to be generated
02:06:08.484 Email was triggered for: Success
02:06:08.484 Sending email for trigger: Success
02:06:08.485 NOT overriding default server settings, using Mailer to create session
02:06:25.838 messageContentType = text/plain; charset=UTF-8
02:06:25.841 Adding recipients from recipient list
02:06:34.349 Successfully created MimeMessage
02:06:34.350 Sending email to: mozmill-ci@mozilla.org
02:06:34.571 Finished: SUCCESS
It's not the first time we have seen this problem. Sadly I haven't reported it earlier.
Regarding version 2.35.1, was there any change since 2.32 I'm not aware of, and which could have changed that?
Henrik Skupin
added a comment - Sure, so as recipient we have a static entry:
https://github.com/mozilla/mozmill-ci/blob/master/config/production/jenkins.patch#L1300
As content we have this:
https://github.com/mozilla/mozmill-ci/blob/master/jenkins-master/jobs/mozilla-aurora_update/config.xml#L197
As it has been shown about 2 hours ago, a restart of Jenkins temporarily made the problem go away:
02:06:08.483 Checking for post-build
02:06:08.484 Performing post-build step
02:06:08.484 Checking if email needs to be generated
02:06:08.484 Email was triggered for: Success
02:06:08.484 Sending email for trigger: Success
02:06:08.485 NOT overriding default server settings, using Mailer to create session
02:06:25.838 messageContentType = text/plain; charset=UTF-8
02:06:25.841 Adding recipients from recipient list
02:06:34.349 Successfully created MimeMessage
02:06:34.350 Sending email to: mozmill-ci@mozilla.org
02:06:34.571 Finished: SUCCESS
It's not the first time we have seen this problem. Sadly I haven't reported it earlier.
Regarding version 2.35.1, was there any change since 2.32 I'm not aware of, and which could have changed that? Henrik Skupin
added a comment - Just to add, the times in my last comment is not the duration but the real time. So it took about 26s, which probably also not that fast.
Henrik Skupin
added a comment - Just to add, the times in my last comment is not the duration but the real time. So it took about 26s, which probably also not that fast. Henrik Skupin
added a comment - Sure. It's here:
https://github.com/mozilla/mozmill-ci/blob/master/jenkins-master/config.xml
For the various instances (staging, production) we simply patch this global config with the changes in the jenkins.patch file, which I have referenced above.
Henrik Skupin
added a comment - Sure. It's here:
https://github.com/mozilla/mozmill-ci/blob/master/jenkins-master/config.xml
For the various instances (staging, production) we simply patch this global config with the changes in the jenkins.patch file, which I have referenced above. Also, can you increase the level for logging of Jenkins stuff to FINE and see if there is anything in there regarding email stuff.
It doesn't look like that includes anything related to email-ext, usually there is another file for the global config of each plugin. hudson.plugins.emailext.SOMETHING.xml in the same directory as the global config.
Henrik Skupin
added a comment - Yes it is 'hudson.plugins.emailext.ExtendedEmailPublisher.xml' but we do not include it in our repository because it will be auto-generated when Jenkins is getting started.
I will check how to set Jenkins logging into FINE mode and we might hopefully see some results soon. Thanks for letting me know that option.
Henrik Skupin
added a comment - Yes it is 'hudson.plugins.emailext.ExtendedEmailPublisher.xml' but we do not include it in our repository because it will be auto-generated when Jenkins is getting started.
I will check how to set Jenkins logging into FINE mode and we might hopefully see some results soon. Thanks for letting me know that option.
How is your recipient list generated? Is it a static list, or do you use tokens? Same for the content of the email, please add those items to the JENKINS ticket at a minimum. Also, 2.35.1 is the latest version of the plugin.