Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11241

AccuRev: reduce load and increase speed for SCM polling

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • accurev-plugin
    • None
    • Jenkins v1.432, AccuRev Plugin v0.6.18

      This feature should be usable per job setting so that each job/project can either use the old style or this style because not every project likes to have the extra required stream.

      – -- – -- –

      Here is a suggestion for a more effective AccuRev polling to decide whether a CI build should be started.

      1) One time only: Create a sub-stream of Project named Project_Last_CI and set time-basis to "now"
      accurev mkstream -s Project_Last_CI -b Project -t now

      2) On each polling cycle, check to see if there are any changes in Project (promotes into Project, its parent, its grandparent, any xlinks, etc.)
      accurev diff -a -i -v Project -V Project_Last_CI

      3) If the accurev diff command produces any output, it is time to kick off a CI build
      a) Update Last_CI stream time-basis
      accurev chstream -s Project_Last_CI -t now
      b) Perform regular hist information being done today to capture transactions
      c) workspace update
      d) build

      Because the diff command normally takes just 4-5 seconds, even for a large stream, this will create a significant load reduction. It should actually enable the project to run the polling more frequently while simultaneously creating less load on the AccuRev server.

          [JENKINS-11241] AccuRev: reduce load and increase speed for SCM polling

          robsimon created issue -
          robsimon made changes -
          Description Original: This should feature should be usable per job setting so that each job/project can either use the old style or this style because not every project likes to have the extra stream.

          -- -- -- -- --

          Here is a suggestion for a more effective AccuRev polling to decide whether a CI build should be started.

          1) One time only: Create a sub-stream of Project named Project_Last_CI and set time-basis to "now"
          accurev mkstream -s Project_Last_CI -b Project -t now

          2) On each polling cycle, check to see if there are any changes in Project (promotes into Project, its parent, its grandparent, any xlinks, etc.)
          accurev diff -a -i -v Project -V Project_Last_CI

          3) If the accurev diff command produces any output, it is time to kick off a CI build
          a) Update Last_CI stream time-basis
          accurev chstream -s Project_Last_CI -t now
          b) Perform regular hist information being done today to capture transactions
          c) workspace update
          d) build

          Because the diff command normally takes just 4-5 seconds, even for a large stream, this will create a significant load reduction. It should actually enable the project to run the polling more frequently while simultaneously creating less load on the AccuRev server.
          New: This feature should be usable per job setting so that each job/project can either use the old style or this style because not every project likes to have the extra required stream.

          -- -- -- -- --

          Here is a suggestion for a more effective AccuRev polling to decide whether a CI build should be started.

          1) One time only: Create a sub-stream of Project named Project_Last_CI and set time-basis to "now"
          accurev mkstream -s Project_Last_CI -b Project -t now

          2) On each polling cycle, check to see if there are any changes in Project (promotes into Project, its parent, its grandparent, any xlinks, etc.)
          accurev diff -a -i -v Project -V Project_Last_CI

          3) If the accurev diff command produces any output, it is time to kick off a CI build
          a) Update Last_CI stream time-basis
          accurev chstream -s Project_Last_CI -t now
          b) Perform regular hist information being done today to capture transactions
          c) workspace update
          d) build

          Because the diff command normally takes just 4-5 seconds, even for a large stream, this will create a significant load reduction. It should actually enable the project to run the polling more frequently while simultaneously creating less load on the AccuRev server.
          Joshua Sherwood made changes -
          Assignee Original: Scott Tatum [ statler ] New: Joshua Sherwood [ jsherwood_accurev ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 141460 ] New: JNJira + In-Review [ 175486 ]

          Arno Moonen added a comment -

          Once you know a transaction ID, you can probably quickly check for changes using the accurev update command.
          This does not provide you with a means to get all the individual transactions, but it does allow you to check for changes with a single command (and if changes are found, the current mechanism can still be used to list the actual changes).

          The command would like this:

          accurev update -s "Some Stream Name" -t 219585-now -fx
          

          ...in which Some Stream Name is ofcourse the name of the stream and 219585 is the ID of the last transaction that was built.

          In case of no changes it will return an empty acResponse element:

          <?xml version="1.0" encoding="utf-8"?>
          <acResponse
              command="update"
              TaskId="13595"/>
          

          Otherwise it will list the elements that have newer versions available, something like:

          <?xml version="1.0" encoding="utf-8"?>
          <acResponse
              command="update"
              TaskId="13749">
            <progress
                phase="Preparing for update"
                increment="100"/>
            <progress
                phase="Calculating changes"
                increment="100"
                number="1"/>
            <progress
                phase="results"
                number="1"/>
            <message>Would replace content of &quot;some/source/file.c&quot; .
          </message>
            <Change
                What="version"
                step="1">
              <Stream1
                  Name="some\source\file.c"
                  eid="676"
                  Version="320/4"
                  NamedVersion="workspace_arno/4"
                  IsDir="no"
                  elemType="text"/>
              <Stream2
                  Name="some\source\file.c"
                  eid="676"
                  Version="320/1"
                  NamedVersion="workspace_arno/1"
                  VirtualVersion="320/1"
                  IsDir="no"
                  elemType="text"/>
            </Change>
            <element
                location="ssome\source\file.c"/>
          </acResponse>
          

          This should also work for workspaces, regardless of the fact if they have been checked out or not (can be done from any machine).

          Arno Moonen added a comment - Once you know a transaction ID, you can probably quickly check for changes using the accurev update command. This does not provide you with a means to get all the individual transactions, but it does allow you to check for changes with a single command (and if changes are found, the current mechanism can still be used to list the actual changes). The command would like this: accurev update -s "Some Stream Name" -t 219585-now -fx ...in which Some Stream Name is ofcourse the name of the stream and 219585 is the ID of the last transaction that was built. In case of no changes it will return an empty acResponse element: <?xml version= "1.0" encoding= "utf-8" ?> <acResponse command= "update" TaskId= "13595" /> Otherwise it will list the elements that have newer versions available, something like: <?xml version= "1.0" encoding= "utf-8" ?> <acResponse command= "update" TaskId= "13749" > <progress phase= "Preparing for update" increment= "100" /> <progress phase= "Calculating changes" increment= "100" number= "1" /> <progress phase= "results" number= "1" /> <message>Would replace content of &quot;some/source/file.c&quot; . </message> <Change What= "version" step= "1" > <Stream1 Name= "some\source\file.c" eid= "676" Version= "320/4" NamedVersion= "workspace_arno/4" IsDir= "no" elemType= "text" /> <Stream2 Name= "some\source\file.c" eid= "676" Version= "320/1" NamedVersion= "workspace_arno/1" VirtualVersion= "320/1" IsDir= "no" elemType= "text" /> </Change> <element location= "ssome\source\file.c" /> </acResponse> This should also work for workspaces, regardless of the fact if they have been checked out or not (can be done from any machine).

          Arno Moonen added a comment -

          PS: This would also get rid of the "false" triggers mentioned in JENKINS-2637, because update does take these rules into account.

          Arno Moonen added a comment - PS: This would also get rid of the "false" triggers mentioned in JENKINS-2637 , because update does take these rules into account.

          Arno Moonen added a comment -

          After the update command, the hist command can be used with the -l option to get only the changes of the changed files mentioned in the update.

          I'll try to add this "optimized" approach as an option to the plug-in, as soon as I find some time to do so.

          Arno Moonen added a comment - After the update command, the hist command can be used with the -l option to get only the changes of the changed files mentioned in the update. I'll try to add this "optimized" approach as an option to the plug-in, as soon as I find some time to do so.
          Joseph Petersen (old) made changes -
          Assignee Original: Joshua Sherwood [ jsherwood_accurev ] New: Joseph Petersen (old) [ casz ]

          Joseph Petersen (old) added a comment - - edited

          arnom Thanks for poking me, to take a look.

          I few word of caution. I would like to avoid the option hell we currently are in.
          Already being going through my head how to better optimise polling to avoid the 4 and soon to be 6 kinds of transactions with hist -k (kind).

          I would like to just pull all transaction with transaction id to another transaction id.
          So to get the current depot transaction id to avoid the issue with multiple "now" calls that changes in time

          So never build before:
          Just trigger the build or do not depending on commen practice in other SCM pluigins

          For polling in pseudo code:

          latestId = accurev hist -p depot -t now
          previousId = build.previousBuild.getTransactionId()
          accurev update -s stream -t previousId-latestId
          "if any changes"
          accurev hist -s stream -t previousId-latestId
          

          after that
          filter all the transaction based on plugin rules cause they are common to build/changelog behaviour

          Then filter based on rules setup in job. Add extension behaviour like in git-plugin
          Should we ignore changes from user
          Should we ignore changes from this path
          etc.

          Then
          Generate changelog from all the transactions that was not filtered and build!

          Joseph Petersen (old) added a comment - - edited arnom Thanks for poking me, to take a look. I few word of caution. I would like to avoid the option hell we currently are in. Already being going through my head how to better optimise polling to avoid the 4 and soon to be 6 kinds of transactions with hist -k (kind). I would like to just pull all transaction with transaction id to another transaction id. So to get the current depot transaction id to avoid the issue with multiple "now" calls that changes in time So never build before: Just trigger the build or do not depending on commen practice in other SCM pluigins For polling in pseudo code: latestId = accurev hist -p depot -t now previousId = build.previousBuild.getTransactionId() accurev update -s stream -t previousId-latestId " if any changes" accurev hist -s stream -t previousId-latestId after that filter all the transaction based on plugin rules cause they are common to build/changelog behaviour Then filter based on rules setup in job. Add extension behaviour like in git-plugin Should we ignore changes from user Should we ignore changes from this path etc. Then Generate changelog from all the transactions that was not filtered and build!

          Arno Moonen added a comment -

          In your final call, will hist give you both upstream and downstream changes?
          In general it seems to be a good approach.

          So far I also do not see any trouble when expanding on this detection algorithm, so it can work with workspaces as well (assuming you would use accurev pop to get the sources in the end, instead of a workspace or reference stream).
          The cool thing being that, if we can check individual workspaces, we can kind of copy the options that exists for automatically creating a pipeline for a Git branch, and do this for a workspace.
          That way, if I keep my changes, Jenkins CI can verify my changes before I promote them, without me having to configure Jenkins CI each time I reparent my workspace.
          This is definitely something that sparked some interest at the client I'm currently working for.

          Arno Moonen added a comment - In your final call, will hist give you both upstream and downstream changes? In general it seems to be a good approach. So far I also do not see any trouble when expanding on this detection algorithm, so it can work with workspaces as well (assuming you would use accurev pop to get the sources in the end, instead of a workspace or reference stream). The cool thing being that, if we can check individual workspaces, we can kind of copy the options that exists for automatically creating a pipeline for a Git branch, and do this for a workspace. That way, if I keep my changes, Jenkins CI can verify my changes before I promote them, without me having to configure Jenkins CI each time I reparent my workspace. This is definitely something that sparked some interest at the client I'm currently working for.

            Unassigned Unassigned
            robsimon robsimon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: