Yes, "random" is a poor choice of word, "based on current build executor" would be better wording. The idea that drives the display number to be based on build executor number is that display names generated based on build executors would bring greater stability to the build than display numbers that are purely random.
If you can't use the "Let Xvfb choose display name" option, I suggest that you reserve a port range that and use that as "Xvfb display name offset".
Randomized display names would require some kind of collision detection which would bring (IMHO) unneeded complexity to the Xvfb Jenkins wrapper.

I have a uniquely awkward set of conditions. I have boards with 2 ssh daemons, on different ports, connected as single-executor nodes in Jenkins. It's done this way because the second instance is a slightly different CPU architecture/ABI, and I can't have that distinction in Jenkins between executors on the same node. The two environments are totally separate... Except for sharing port space, so if the bare system uses Xvfb on port 6001, running Xvfb on 6001 in the chroot will fail (and fail the build)

Do you have different jobs for each of your daemons? If so use different offsets for each.

I could find a way of adding slave into the mix, such that display number is based on slave, executor number and offset, would that help?

Same jobs on both

Code changed in jenkins
User: Zoran Regvart
Path:
src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java
src/main/resources/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper/help-displayName.html
http://jenkins-ci.org/commit/xvfb-plugin/0aa3519c0b6c8dd21289a8df11cf8ae4db18801d
Log:
JENKINS-26848 Does not actually randomize DISPLAY

Change the wording in the help to better reflect the actual display name
generation

Code changed in jenkins
User: Zoran Regvart
Path:
src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java
src/main/resources/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper/config.jelly
src/main/resources/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper/help-parallelBuild.html
http://jenkins-ci.org/commit/xvfb-plugin/259e422e60a33fa7b232b00bf8aeede354544ece
Log:
JENKINS-26848 Does not actually randomize DISPLAY

Implement new feature for parallel builds, display names take
100*computer index offset if used.

I've released version 1.0.16-beta1 that you can try from the experimental update center (read about it at http://jenkins-ci.org/content/experimental-plugins-update-center).

A new feature was added that helps with your situation called parallel builds, under Advanced settings of the job configuration you need to set I’m running this job in parallel on same node. With this set, an offset tied to the node the executor is running on is added into the calculation of the display name.

In that case display name is 100 * node + executor + offset. For instance display name for a job running on slave 3 under executor 4 with offset of 20 will be 324.

I would much appreciate if you could test the beta release and confirm if it works for you so that I can release the next version of the plugin.

Amazing. I'll try that next time I'm near a computer - I'd rather not break a production system from a smartphone

Doesn't seem to be in http://ftp-chi.osuosl.org/pub/jenkins/updates/experimental/update-center.json or http://updates.jenkins-ci.org/download/plugins/xvfb/ - do you have a direct link to the .hpi I can use?