Yeah I think Vivek Pandey is going to need to look at this as it seems, to me, like the pagination needs to change because at the moment it is expecting the full resultset to be created by the function bound to the endpoint (e.g. AbstractPipelineImpl.getActivities()) and then the filtering (start, limit etc) happens further up the call stack in Stapler (I think). The fact that AbstractPipelineImpl.getActivities() needs to read all runs on a Job in order for the pagination to work back in stapler is where this is choking imo i.e. you want the first 30 runs but are reading all 5000 runs into a collection and only then the filtering happens. Filtering earlier in the process would work better I think.
getBuilds(hudson.model.Fingerprint.RangeSet) would not really help here as far as I can see.
If changing the pagination is a big problem then one possible quick solution here might be to not use the Stapler pagination and, instead, to do the pagination "manually' for getting activity (where there can be large resultsets).