When analysing the stack traces, we see a lock between the email-ext token resolution and the BFA:

"Executor #4 for be-bld-jen-02 : executing xxx #936" prio=10 tid=0x00007fc45edfb000 nid=0x2040 waiting for monitor entry [0x00007fc439f05000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jenkinsci.plugins.tokenmacro.DataBoundTokenMacro.evaluate(DataBoundTokenMacro.java:167)
	- waiting to lock <0x00000006c58d5888> (a com.sonyericsson.jenkins.plugins.bfa.tokens.Token)
	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expand(TokenMacro.java:190)
	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:242)
	at hudson.plugins.emailext.plugins.ContentBuilder.transformText(ContentBuilder.java:63)
	at hudson.plugins.emailext.ExtendedEmailPublisher.getContent(ExtendedEmailPublisher.java:698)

Upon BFA token resolution, the code asks the build to be scanned again, but this checks only if it the BFA action is there. It seems that there is a kind of a conflict here.

We use email-ext 2.36 and token-macro 1.11.

When we remove the BFA token from the mail generator, it's OK.

Since the org.jenkinsci.plugins.tokenmacro.DataBoundTokenMacro#evaluate(hudson.model.AbstractBuild<?,?>, hudson.model.TaskListener, java.lang.String, java.util.Map<java.lang.String,java.lang.String>, com.google.common.collect.ListMultimap<java.lang.String,java.lang.String>) method is synchronised (since version 1.11), we have indeed a bottleneck when several builds need to resolve the token.

It seems to be necessary because this class has some instance variables and cannot be shared between different threads.

So in the end, this seems more a problem with the token-macro plugin than with the BFA or the email-ext plugin. I'll go and see if there is already an issue about this.

Moving this issue to the token-macro-plugin component.

I've created the https://github.com/jenkinsci/token-macro-plugin/pull/21 pull request in an attempt to solve this bottleneck issue.

We have also seen some performance issues in the BFA, but only on our less powerful machines, that
have other performance issues as well, so we haven't had the need to look further into the issues.
As a Jenkins instance gets bigger, with more jobs, more rules and longer logs to analyze, this could
get out of hand, we just haven't seen it in our installation yet, but it seems you have.

Good find on the token-macro plugin, if the pull request can get merged, it seems like a cheap solution.

Code changed in jenkins
User: Damien Coraboeuf
Path:
src/main/java/org/jenkinsci/plugins/tokenmacro/DataBoundTokenMacro.java
http://jenkins-ci.org/commit/token-macro-plugin/f9054f3f3becc5744e0874a31634282285829e54
Log:
JENKINS-32331 Removed bottleneck synchronisation

Code changed in jenkins
User: Damien Coraboeuf
Path:
src/main/java/org/jenkinsci/plugins/tokenmacro/DataBoundTokenMacro.java
http://jenkins-ci.org/commit/token-macro-plugin/803399668417c58a14052b74fae9f21cb4f03601
Log:
JENKINS-32331 Using a concurrent map for the setters

Code changed in jenkins
User: Alex Earl
Path:
src/main/java/org/jenkinsci/plugins/tokenmacro/DataBoundTokenMacro.java
http://jenkins-ci.org/commit/token-macro-plugin/68e4779779c09e7a12ba64c7c184cdc38166db7a
Log:
Merge pull request #21 from dcoraboeuf/jenkins-32331-bottleneck

JENKINS-32331 Removed bottleneck synchronisation

Compare: https://github.com/jenkinsci/token-macro-plugin/compare/c28636e1aa69...68e4779779c0

Will be fixed in 2.0

damien_coraboeuf, t_westling, can you check my PR: https://github.com/jenkinsci/build-failure-analyzer-plugin/pull/57 ? Or at least join to discussion