LastFailureColumn, shown by default in AllView, is calling Job.getLastFailedBuild which then keeps on loading builds until it finds a failed build, with no cutoff. The fix for that would be to either impose a maximum number of builds to search back from (10? 20?); or do the search asynchronously after the page loads, which might still put some load on the server but would at least not stop you from browsing.
The same would apply to the last success column if all your recent builds failed, etc.