I'm seeing a similar problem for matrix jobs with build timeout set. I've seen one case where a matrix job with a 2x2 configuration times out, but the configurations appear to keep going and cannot be stopped. At the bottom of the console output for each configuration, I see the "Build timed out (after 120 minutes). Marking the build as aborted." entries in the logs, but the spinner at the bottom is still moving and each of the configuration jobs appear to keep trying to execute. I tried to stop each configuration build the normal way (clicking the red X), which didn't work after multiple attempts, so I looked at the threads via the JavaMelody Monitoring plugin and all 4 of them are in RUNNABLE state. Trying to kill each thread does not work either, so I'm stuck here with the only choice of restarting Jenkins.
Nor does the build timeout seem to apply if the build fails to even start, e.g. if the target node is offline.