Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11241

AccuRev: reduce load and increase speed for SCM polling

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • accurev-plugin
    • None
    • Jenkins v1.432, AccuRev Plugin v0.6.18

      This feature should be usable per job setting so that each job/project can either use the old style or this style because not every project likes to have the extra required stream.

      – -- – -- –

      Here is a suggestion for a more effective AccuRev polling to decide whether a CI build should be started.

      1) One time only: Create a sub-stream of Project named Project_Last_CI and set time-basis to "now"
      accurev mkstream -s Project_Last_CI -b Project -t now

      2) On each polling cycle, check to see if there are any changes in Project (promotes into Project, its parent, its grandparent, any xlinks, etc.)
      accurev diff -a -i -v Project -V Project_Last_CI

      3) If the accurev diff command produces any output, it is time to kick off a CI build
      a) Update Last_CI stream time-basis
      accurev chstream -s Project_Last_CI -t now
      b) Perform regular hist information being done today to capture transactions
      c) workspace update
      d) build

      Because the diff command normally takes just 4-5 seconds, even for a large stream, this will create a significant load reduction. It should actually enable the project to run the polling more frequently while simultaneously creating less load on the AccuRev server.

          [JENKINS-11241] AccuRev: reduce load and increase speed for SCM polling

          Arno Moonen added a comment -

          Once you know a transaction ID, you can probably quickly check for changes using the accurev update command.
          This does not provide you with a means to get all the individual transactions, but it does allow you to check for changes with a single command (and if changes are found, the current mechanism can still be used to list the actual changes).

          The command would like this:

          accurev update -s "Some Stream Name" -t 219585-now -fx
          

          ...in which Some Stream Name is ofcourse the name of the stream and 219585 is the ID of the last transaction that was built.

          In case of no changes it will return an empty acResponse element:

          <?xml version="1.0" encoding="utf-8"?>
          <acResponse
              command="update"
              TaskId="13595"/>
          

          Otherwise it will list the elements that have newer versions available, something like:

          <?xml version="1.0" encoding="utf-8"?>
          <acResponse
              command="update"
              TaskId="13749">
            <progress
                phase="Preparing for update"
                increment="100"/>
            <progress
                phase="Calculating changes"
                increment="100"
                number="1"/>
            <progress
                phase="results"
                number="1"/>
            <message>Would replace content of &quot;some/source/file.c&quot; .
          </message>
            <Change
                What="version"
                step="1">
              <Stream1
                  Name="some\source\file.c"
                  eid="676"
                  Version="320/4"
                  NamedVersion="workspace_arno/4"
                  IsDir="no"
                  elemType="text"/>
              <Stream2
                  Name="some\source\file.c"
                  eid="676"
                  Version="320/1"
                  NamedVersion="workspace_arno/1"
                  VirtualVersion="320/1"
                  IsDir="no"
                  elemType="text"/>
            </Change>
            <element
                location="ssome\source\file.c"/>
          </acResponse>
          

          This should also work for workspaces, regardless of the fact if they have been checked out or not (can be done from any machine).

          Arno Moonen added a comment - Once you know a transaction ID, you can probably quickly check for changes using the accurev update command. This does not provide you with a means to get all the individual transactions, but it does allow you to check for changes with a single command (and if changes are found, the current mechanism can still be used to list the actual changes). The command would like this: accurev update -s "Some Stream Name" -t 219585-now -fx ...in which Some Stream Name is ofcourse the name of the stream and 219585 is the ID of the last transaction that was built. In case of no changes it will return an empty acResponse element: <?xml version= "1.0" encoding= "utf-8" ?> <acResponse command= "update" TaskId= "13595" /> Otherwise it will list the elements that have newer versions available, something like: <?xml version= "1.0" encoding= "utf-8" ?> <acResponse command= "update" TaskId= "13749" > <progress phase= "Preparing for update" increment= "100" /> <progress phase= "Calculating changes" increment= "100" number= "1" /> <progress phase= "results" number= "1" /> <message>Would replace content of &quot;some/source/file.c&quot; . </message> <Change What= "version" step= "1" > <Stream1 Name= "some\source\file.c" eid= "676" Version= "320/4" NamedVersion= "workspace_arno/4" IsDir= "no" elemType= "text" /> <Stream2 Name= "some\source\file.c" eid= "676" Version= "320/1" NamedVersion= "workspace_arno/1" VirtualVersion= "320/1" IsDir= "no" elemType= "text" /> </Change> <element location= "ssome\source\file.c" /> </acResponse> This should also work for workspaces, regardless of the fact if they have been checked out or not (can be done from any machine).

          Arno Moonen added a comment -

          PS: This would also get rid of the "false" triggers mentioned in JENKINS-2637, because update does take these rules into account.

          Arno Moonen added a comment - PS: This would also get rid of the "false" triggers mentioned in JENKINS-2637 , because update does take these rules into account.

          Arno Moonen added a comment -

          After the update command, the hist command can be used with the -l option to get only the changes of the changed files mentioned in the update.

          I'll try to add this "optimized" approach as an option to the plug-in, as soon as I find some time to do so.

          Arno Moonen added a comment - After the update command, the hist command can be used with the -l option to get only the changes of the changed files mentioned in the update. I'll try to add this "optimized" approach as an option to the plug-in, as soon as I find some time to do so.

          Joseph Petersen (old) added a comment - - edited

          arnom Thanks for poking me, to take a look.

          I few word of caution. I would like to avoid the option hell we currently are in.
          Already being going through my head how to better optimise polling to avoid the 4 and soon to be 6 kinds of transactions with hist -k (kind).

          I would like to just pull all transaction with transaction id to another transaction id.
          So to get the current depot transaction id to avoid the issue with multiple "now" calls that changes in time

          So never build before:
          Just trigger the build or do not depending on commen practice in other SCM pluigins

          For polling in pseudo code:

          latestId = accurev hist -p depot -t now
          previousId = build.previousBuild.getTransactionId()
          accurev update -s stream -t previousId-latestId
          "if any changes"
          accurev hist -s stream -t previousId-latestId
          

          after that
          filter all the transaction based on plugin rules cause they are common to build/changelog behaviour

          Then filter based on rules setup in job. Add extension behaviour like in git-plugin
          Should we ignore changes from user
          Should we ignore changes from this path
          etc.

          Then
          Generate changelog from all the transactions that was not filtered and build!

          Joseph Petersen (old) added a comment - - edited arnom Thanks for poking me, to take a look. I few word of caution. I would like to avoid the option hell we currently are in. Already being going through my head how to better optimise polling to avoid the 4 and soon to be 6 kinds of transactions with hist -k (kind). I would like to just pull all transaction with transaction id to another transaction id. So to get the current depot transaction id to avoid the issue with multiple "now" calls that changes in time So never build before: Just trigger the build or do not depending on commen practice in other SCM pluigins For polling in pseudo code: latestId = accurev hist -p depot -t now previousId = build.previousBuild.getTransactionId() accurev update -s stream -t previousId-latestId " if any changes" accurev hist -s stream -t previousId-latestId after that filter all the transaction based on plugin rules cause they are common to build/changelog behaviour Then filter based on rules setup in job. Add extension behaviour like in git-plugin Should we ignore changes from user Should we ignore changes from this path etc. Then Generate changelog from all the transactions that was not filtered and build!

          Arno Moonen added a comment -

          In your final call, will hist give you both upstream and downstream changes?
          In general it seems to be a good approach.

          So far I also do not see any trouble when expanding on this detection algorithm, so it can work with workspaces as well (assuming you would use accurev pop to get the sources in the end, instead of a workspace or reference stream).
          The cool thing being that, if we can check individual workspaces, we can kind of copy the options that exists for automatically creating a pipeline for a Git branch, and do this for a workspace.
          That way, if I keep my changes, Jenkins CI can verify my changes before I promote them, without me having to configure Jenkins CI each time I reparent my workspace.
          This is definitely something that sparked some interest at the client I'm currently working for.

          Arno Moonen added a comment - In your final call, will hist give you both upstream and downstream changes? In general it seems to be a good approach. So far I also do not see any trouble when expanding on this detection algorithm, so it can work with workspaces as well (assuming you would use accurev pop to get the sources in the end, instead of a workspace or reference stream). The cool thing being that, if we can check individual workspaces, we can kind of copy the options that exists for automatically creating a pipeline for a Git branch, and do this for a workspace. That way, if I keep my changes, Jenkins CI can verify my changes before I promote them, without me having to configure Jenkins CI each time I reparent my workspace. This is definitely something that sparked some interest at the client I'm currently working for.

          Joseph Petersen (old) added a comment - - edited

          The hist call would still have to travel upstream unless ignoreParentStream is true. Unless accurev would supply a better hist command.
          Nothing is stopping you from populating the contents from a workspace stream. You just have to type the workspace name instead of the stream name.

          What your talking about is multibranch pipeline
          Which is also definitely on my radar to support.
          Question is if I wanted to support workspaces
          or
          use Gated streams (which I am more inclined on) since organization wise you could have several workspaces under a staging stream produced by gated stream.
          or perhaps support both.

          Joseph Petersen (old) added a comment - - edited The hist call would still have to travel upstream unless ignoreParentStream is true. Unless accurev would supply a better hist command. Nothing is stopping you from populating the contents from a workspace stream. You just have to type the workspace name instead of the stream name. What your talking about is multibranch pipeline Which is also definitely on my radar to support. Question is if I wanted to support workspaces or use Gated streams (which I am more inclined on) since organization wise you could have several workspaces under a staging stream produced by gated stream. or perhaps support both.

          Arno Moonen added a comment -

          I would support both. People probably want to test their changes as early on as possible.
          It's been a while since I've worked on Jenkins plug-ins, but if I can help in anyway, please let me know.
          The organization I'm currently working for could really benefit from the multibranch feature.
          (We want to apply a branching scheme similar to Git Flow, where we have a branch per issue we are working on, so it is clear what is changed for that issue, even when we have merged in changes from other issues.)

          Arno Moonen added a comment - I would support both. People probably want to test their changes as early on as possible. It's been a while since I've worked on Jenkins plug-ins, but if I can help in anyway, please let me know. The organization I'm currently working for could really benefit from the multibranch feature. (We want to apply a branching scheme similar to Git Flow, where we have a branch per issue we are working on, so it is clear what is changed for that issue, even when we have merged in changes from other issues.)

          I had a good talk with my colleague today.
          The way multibranch works is by scanning the repo (in git term) for any jenkinsfile across branches and then for github you have the whole organization to scan each repo for jenkinsfile.

          And I totally agree, I think for multibranch you would still supply depot and stream, though if stream is left empty it will scan the whole depot.

          So scanning whole depot for jenkinsfile will create a job for each "stream, staging, gated, workspace" streams with a jenkinsfile. Perhaps nesting some of the streams into folders but I will look into that possibility.

          if you give the multibranch a stream name it will treat it as the top level stream but scan downwards from there. Cause in our case we have one very large depot with lots and lots of streams!

          Joseph Petersen (old) added a comment - I had a good talk with my colleague today. The way multibranch works is by scanning the repo (in git term) for any jenkinsfile across branches and then for github you have the whole organization to scan each repo for jenkinsfile. And I totally agree, I think for multibranch you would still supply depot and stream, though if stream is left empty it will scan the whole depot. So scanning whole depot for jenkinsfile will create a job for each "stream, staging, gated, workspace" streams with a jenkinsfile. Perhaps nesting some of the streams into folders but I will look into that possibility. if you give the multibranch a stream name it will treat it as the top level stream but scan downwards from there. Cause in our case we have one very large depot with lots and lots of streams!

          Arno Moonen added a comment -

          What we currently have looks a bit like this:

          • Depot
            • Libraries
              • LIB-A
              • LIB-B
            • Firmware
              • PRODUCT-A
                • PRODUCT-A-WIP
                  • PRODUCT-A workspace for Jenkins
                  • PRODUCT-A workspace user 1
                  • PRODUCT-A workspace user 2
              • PRODUCT-B
            • PC Utilities
              • UTIL-A
              • UTIL-B

          Libraries are generally pieces of firmware used by multiple products, via include / exclude rules.
          I would like to get rid of the workspace for Jenkins CI, because this only adds additional work when adding a new product to the CI environment.
          It would also be nice if the workspaces get automatically checked, so I can work on a specific issue/feature in there (or I might add an additional stream for that, on the same level as the workspaces currently are).

          Using Jenkinsfile would be nice, but if I remember correctly this file still needs to be in the root of the repository, which would not be practical for us.
          For example, for PRODUCT-A, we have a directory product\PRODUCT-A in which the makefile and other project specific files are located.
          To me this would be the ideal location for a Jenkinsfile, but the open source version of Jenkins does not allow me to specify a search location (again, if I remember correctly).

          Arno Moonen added a comment - What we currently have looks a bit like this: Depot Libraries LIB-A LIB-B Firmware PRODUCT-A PRODUCT-A-WIP PRODUCT-A workspace for Jenkins PRODUCT-A workspace user 1 PRODUCT-A workspace user 2 PRODUCT-B PC Utilities UTIL-A UTIL-B Libraries are generally pieces of firmware used by multiple products, via include / exclude rules. I would like to get rid of the workspace for Jenkins CI, because this only adds additional work when adding a new product to the CI environment. It would also be nice if the workspaces get automatically checked, so I can work on a specific issue/feature in there (or I might add an additional stream for that, on the same level as the workspaces currently are). Using Jenkinsfile would be nice, but if I remember correctly this file still needs to be in the root of the repository, which would not be practical for us. For example, for PRODUCT-A, we have a directory product\PRODUCT-A in which the makefile and other project specific files are located. To me this would be the ideal location for a Jenkinsfile, but the open source version of Jenkins does not allow me to specify a search location (again, if I remember correctly).

          Arno Moonen added a comment - - edited

          Small update: I no longer work at the company that could benefit from the multibranch support, but I still think they would be really enthusiastic about it.
          Also, I noticed since somewhere late 2017 it is now possible to enter a different path for the Jenkinsfile.

          @casz: Have you ever looked into this?

          Arno Moonen added a comment - - edited Small update: I no longer work at the company that could benefit from the multibranch support, but I still think they would be really enthusiastic about it. Also, I noticed since somewhere late 2017 it is now possible to enter a different path for the Jenkinsfile. @ casz : Have you ever looked into this?

            Unassigned Unassigned
            robsimon robsimon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: