Certain of our jobs will intermittently hang forever. When this happens they cannot be canceled via the Jenkins webapp. Clicking on the red [x] has no effect and Jenkins must be restarted in order for the offending jobs to be cleared from the queue.
Interestingly, once Jenkins restarts these jobs show as up as having completed successfully. So the jobs are finishing but Jenkins isn't "noticing". The "cancel" functionality is probably failing because the job has, in fact, already completed and there's nothing to cancel.
The affected jobs use a combination of bash and ruby scripts that are executed from within the "Execute shell" section in the job configuration.
I will update this ticket with a thread dump the next time I notice a hang happening.