Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31083

Performance issue when querying changelog information

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • p4-plugin
    • Jenkins ver. 1.625.1
      P4 plugin 1.3.2
      OS: RHEL 6.4
      Java: 1.8.0_60

      We have recently switched from the perforce-plugin to p4-plugin but are seeing a performance issue on our perforce server. The p4-plugin seems to be querying more changes from the perforce server than necessary.
      For example I see it doing:

      p4 changes -m100 //myclientname/...@0,2126849

      which returns 100 changelists and p4-plugin then proceeds to do "p4 change -o <change> , p4 user -o <user> and p4 describe -s <change> for every one of those 100 changes.

      But the previous build in jenkins for that job was changelist 2125434 so if p4-plugin had done the following query:

      p4 changes -m100 //myclientname/...@2125434,2126849

      it would have only returned one change (as many changes were outside the view of this client workspace) and then the load on the main perforce server would be much reduced.

      Why isn't p4-plugin using the changelist number of the previous build to reduce the amount of queries?

          [JENKINS-31083] Performance issue when querying changelog information

          Paul Allen added a comment -

          Originally I did exactly that (used the previous build to get the changelist number), however that lead to numerous issues (if the previous build didn't occur was aborted etc...)

          The plugin uses 'p4 cstat' to calculate the starting change. If your use the 'Auto clean and sync' option then the range for 'p4 changes' will be smaller. For the initial 'first' build or if you use 'Forced clean and sync' option, the range will start from 0. I had hoped that limiting it to 100 changes would minimise the impact, perhaps your change lists are large and it's the describe that is expensive.

          I agree that the whole change-list reporting could be more optimal, if you have any ideas or want to help contribute let me know.

          Paul Allen added a comment - Originally I did exactly that (used the previous build to get the changelist number), however that lead to numerous issues (if the previous build didn't occur was aborted etc...) The plugin uses 'p4 cstat' to calculate the starting change. If your use the 'Auto clean and sync' option then the range for 'p4 changes' will be smaller. For the initial 'first' build or if you use 'Forced clean and sync' option, the range will start from 0. I had hoped that limiting it to 100 changes would minimise the impact, perhaps your change lists are large and it's the describe that is expensive. I agree that the whole change-list reporting could be more optimal, if you have any ideas or want to help contribute let me know.

          Perhaps p4-plugin should only use this behavior if the previous build changelist number is not available from the jenkins previous build data.
          Using 'p4 cstat' could be problematic for us as we are sharing workspaces between multiple jobs in order to save disk space which would probably mean that "p4 cstat" would not be accurate for that job. We are using the "one workspace per executor" model similar to that described in the thread https://swarm.workshop.perforce.com/changes/15665#comments
          It would be nice to have some native support for this in p4-plugin.

          I am happy to contribute to try to solve this as at the moment Jenkins is putting so much load on our perforce replica that it is becoming unusable for the normal users.

          Andrew Barnish added a comment - Perhaps p4-plugin should only use this behavior if the previous build changelist number is not available from the jenkins previous build data. Using 'p4 cstat' could be problematic for us as we are sharing workspaces between multiple jobs in order to save disk space which would probably mean that "p4 cstat" would not be accurate for that job. We are using the "one workspace per executor" model similar to that described in the thread https://swarm.workshop.perforce.com/changes/15665#comments It would be nice to have some native support for this in p4-plugin. I am happy to contribute to try to solve this as at the moment Jenkins is putting so much load on our perforce replica that it is becoming unusable for the normal users.

          Jason Davis added a comment -

          We've noticed the same behavior with performance when clicking on the changes link. We thought it might be because it appears the changes per build aren't currently reflecting what's actually in the build (creating a different ticket for that). We have decent VM with a replica close our our build systems, so it's unclear, other than the explanation here, why there would be a delay when using the various build or project "changes" link.

          Jason Davis added a comment - We've noticed the same behavior with performance when clicking on the changes link. We thought it might be because it appears the changes per build aren't currently reflecting what's actually in the build (creating a different ticket for that). We have decent VM with a replica close our our build systems, so it's unclear, other than the explanation here, why there would be a delay when using the various build or project "changes" link.

          Not seeing this issue anymore so closing it.

          Andrew Barnish added a comment - Not seeing this issue anymore so closing it.

            p4paul Paul Allen
            barnish Andrew Barnish
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: