vkuklinatjoom, thanks for reporting this, and thanks for the recreate steps. I'll take a shot at this recreate in the morning. If this is the bug I'm hoping it is, this has been showing itself for a long time, and has been super annoying to try and recreate.

I'm going to see if this will show itself on versions of GHBS older than 2.5.0 as well (see JENKINS-43194). I'm pretty sure we've been chasing this since well before that release.

vkuklinatjoom - I want to make sure I'm doing this right. Here's what I've done.

1. Created a multibranch project on a GitHub repo. It includes a basic text file so I can easily create merge conflicts.

2. Create a new branch based off master, make changes to the text file, and push those changes to a new branch change1 on GitHub of the same name.

3. Scan the repository from Jenkins. The branch change1 appears on Jenkins.

4. Do a local checkout of master, intentionally change the text file in such a way that when I try to create a pull request from change1 into master, the PR will not be mergeable. Push the change to master.

5. Switch back to github.com and create the PR from change1 to master. As expected, it shows as not being mergeable.

6. Scan the repository from Jenkinis. The change1 branch disappears from the Branches tab, which is what I would expect, given I'm using the default Discover Branches behavior of "Exclude branches that are also filed as PRs."

7. Copy the github cache folder to someplace where I can mess around with it. I've found the correct file, knowing I'd created PR number 737 to my test repo:

$ grep -R -i '737.*merge' *.0 
54d724cbddb40d04caa4c58007667fba.0:https://api.github.com/repos/kshultzCB/creates-and-deletes/contents/?ref=refs%2Fpull%2F737%2Fmerge

Based upon that, the filenames I'm after should be 54d724cbddb40d04caa4c58007667fba.0 and 54d724cbddb40d04caa4c58007667fba.1. When I zcat 54d724cbddb40d04caa4c58007667fba.1, what I get back is json. I've formatted it so it's pretty, but this is what's inside:

{
    "message": "No commit found for the ref refs/pull/737/merge",
    "documentation_url": "https://developer.github.com/v3/repos/contents/"
}

What I'm expecting is that I'll fix the PR so it can be merged, rescan the repository from Jenkins, and hit our "Jenkinsfile not found" problem.

Have I got this right so far?

Pretty much that's it (the contents of the GHCache files are exactly the same that we have encountered) but there are several things and steps you need to check or follow:

1. You should have github cache enabled with its size set to some value that the entry will not have been purged for a while (like an hour to make the pain of the developers really an issue)
2. Step 8 should be - fix the conflict and push the new head into the source branch, so that the PR is mergeable now, but it will not get discovered as it is marked as the one that does have a Jenkinsfile/contents due to false-negative from the GHCache. 

In some (or may be all) scenarios the "JenkinsFile not found" is not shown after purported step #8 as the PR branch is considered as "Dead", therefore not buildable. We can also see the logs like 'Ressurrect dead branch' or smth like this which points somewhere here: 
https://github.com/jenkinsci/branch-api-plugin/blob/847348f6c741f9ac350c138d6267e41a081b89e4/src/main/java/jenkins/branch/MultiBranchProject.java

Hence, I would suggest a step #9:

Trigger the 'redeliver hook payload' button in github (you can do it with github webhooks page and resend the hook json once again) and see that nothing happens as if the (Multi)Branch (supposedly) plugin just skips the branch as not buildable/Dead.

Fantastic. Thanks so much! You've done some really excellent detective work on this.

No real fear of the cache getting wiped out from under me, I'm just running the plugin locally via mvnDebug hpi:run. I'm using manually invoked repository scans rather than webhooks, but your comment leads me to believe that having webhooks available would be useful. I'll set that up next.

Might be able to write a unit test that simulates this too. I'll look at that after I see this happen

Another question, if you don't mind. You wrote:

In some (or may be all) scenarios the "JenkinsFile not found" is not shown after purported step #8 as the PR branch is considered as "Dead", therefore not buildable. We can also see the logs like 'Ressurrect dead branch' or smth like this which points somewhere here:
https://github.com/jenkinsci/branch-api-plugin/blob/847348f6c741f9ac350c138d6267e41a081b89e4/src/main/java/jenkins/branch/MultiBranchProject.java

Sounds like you do not always see the "Jenkinsfile not found" message in your scan logs. Is that correct? If yes, that tracks with what we've been seeing before. The "Jenkinsfile not found" error has been very elusive and difficult to recreate.

You are vey welcome, sir.

Wrt the question I think (still not 100% sure) that the "Jenkinsfile not found" is shown the first time it misses the contents of the merge head. All the consequent attempts to redeliver the payload or rescan the branches end with the branch just considered "dead"/not buildable.