We have reduced the number of concurrent scans per build from 3 to 1. It seems better now, but we're monitoring the situation. Especially, it could now cause the analysis to be longer than before.

It would be nice to have a BFA dashboard showing the number of concurrent scans currently running. I'll try to work on this.

When analysing the stack traces, we see a lock between the email-ext token resolution and the BFA:

"Executor #4 for be-bld-jen-02 : executing xxx #936" prio=10 tid=0x00007fc45edfb000 nid=0x2040 waiting for monitor entry [0x00007fc439f05000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jenkinsci.plugins.tokenmacro.DataBoundTokenMacro.evaluate(DataBoundTokenMacro.java:167)
	- waiting to lock <0x00000006c58d5888> (a com.sonyericsson.jenkins.plugins.bfa.tokens.Token)
	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expand(TokenMacro.java:190)
	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:242)
	at hudson.plugins.emailext.plugins.ContentBuilder.transformText(ContentBuilder.java:63)
	at hudson.plugins.emailext.ExtendedEmailPublisher.getContent(ExtendedEmailPublisher.java:698)

Upon BFA token resolution, the code asks the build to be scanned again, but this checks only if it the BFA action is there. It seems that there is a kind of a conflict here.

We use email-ext 2.36 and token-macro 1.11.

When we remove the BFA token from the mail generator, it's OK.

Since the org.jenkinsci.plugins.tokenmacro.DataBoundTokenMacro#evaluate(hudson.model.AbstractBuild<?,?>, hudson.model.TaskListener, java.lang.String, java.util.Map<java.lang.String,java.lang.String>, com.google.common.collect.ListMultimap<java.lang.String,java.lang.String>) method is synchronised (since version 1.11), we have indeed a bottleneck when several builds need to resolve the token.

It seems to be necessary because this class has some instance variables and cannot be shared between different threads.

So in the end, this seems more a problem with the token-macro plugin than with the BFA or the email-ext plugin. I'll go and see if there is already an issue about this.

Moving this issue to the token-macro-plugin component.

I've created the https://github.com/jenkinsci/token-macro-plugin/pull/21 pull request in an attempt to solve this bottleneck issue.

We have also seen some performance issues in the BFA, but only on our less powerful machines, that
have other performance issues as well, so we haven't had the need to look further into the issues.
As a Jenkins instance gets bigger, with more jobs, more rules and longer logs to analyze, this could
get out of hand, we just haven't seen it in our installation yet, but it seems you have.

Good find on the token-macro plugin, if the pull request can get merged, it seems like a cheap solution.