• Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • accurev-plugin
    • None

      I'm finding that the "accurev show streams" command takes forever to run, and this causes the AccuRev GUI program's "View Streams" panel to take forever to load. I've also found that I have an awful lot of snapshots in my stream, most of which were created by the accurev plugin.

      It would be nice if these didn't appear, and the best way of making them not appear would be to do an "accurev remove stream XXX" on the snapshot that was created.
      (whether a stream is "removed" or not doesn't have any functional effect on it)

          [JENKINS-13483] Hide snapshots created by accurev plugin

          pjdarton added a comment -

          Would need to add yet another boolean tick-box option in the configuration to enable/disable this feature. This would only be shown if the snapshot feature was enabled.

          If stream removal was enabled, we'd need to do the command immediately after we'd created the snapshot, before we do the populate.

          pjdarton added a comment - Would need to add yet another boolean tick-box option in the configuration to enable/disable this feature. This would only be shown if the snapshot feature was enabled. If stream removal was enabled, we'd need to do the command immediately after we'd created the snapshot, before we do the populate.

          Jack Flynn added a comment -

          Hey there, I'm from accurev & was looking to get involved with some of these issues. I talked to Scott in the past and gave him some ideas which should take this plugin to the next level. I'm just looking at some low hanging fruit right now and this one got my attention. I still need to go through the source code but can you describe why a full "ac show streams" is being invoked in the first place? I'm pretty confident that there can be some new way to achieve the end results here. This and those hist calls are killers. I'll jump on those issues soon too but one at a time. I just need to see if anybody out there is listening and your name is all over the place. would love to get things rolling asap. Looking forward.

          --Jack

          Jack Flynn added a comment - Hey there, I'm from accurev & was looking to get involved with some of these issues. I talked to Scott in the past and gave him some ideas which should take this plugin to the next level. I'm just looking at some low hanging fruit right now and this one got my attention. I still need to go through the source code but can you describe why a full "ac show streams" is being invoked in the first place? I'm pretty confident that there can be some new way to achieve the end results here. This and those hist calls are killers. I'll jump on those issues soon too but one at a time. I just need to see if anybody out there is listening and your name is all over the place. would love to get things rolling asap. Looking forward. --Jack

          pjdarton added a comment -

          Hi Jack,

          Nice to see some attention from accurev themselves. Yes, my name's all over many of the recent changes as I did a number of bugfixes/improvements to make it "work for my needs" (and hopefully others' needs too). I spent quite a while with my colleague Martin McAuley and accurev support working out a way of making the build process reliable (answer: avoid doing an accurev login unless logged out) and suitable for use with our internal build labelling/packaging infrastructure, as well as working around a memory leak issue (I'm not sure that one is entirely fixed, but it's no longer causing problems).

          When you use the Accurev GUI to show the streams within a depot, unless you've hidden all the snapshots, the GUI takes /ages/ to display anything, and just as long to refresh after you've done anything with it - it makes the GUI fairly unusable as it's doing (and re-parsing) a "show streams" commands each time.
          I think that it also impacts on server performance and the time it takes to do various other commands - certainly, we found that general performance improved after I "remove stream"ed about 65000 snapshots in the depot we're working in (accurev's philosophy of "snapshots are lightweight, you can create as many as you like", which is used to negate any "why do I need to create a snapshot to do this" issues, is clearly not entirely true!).

          However, this would certainly qualify as low-hanging fruit - I reckon it'd only take an hour or two, once you've got the general modify/build/test cycle sorted out that is (which could take a little longer).
          I would also strongly recommend sorting out JENKINS-10937 as well on that basis - it's a definite bug that I introduced due to a lack of testing (I don't use workspaces on my Jenkins setup as I use many slaves, and I've also found that "accurev update" is not 100% reliable, hence I just brute-force a populate instead) and it should be easy to fix.

          As for "why are we doing a show streams" command in the first place: It's because of accurev's stream hierarchy/inheritance model - Jenkins needs to know if, were it to populate a new working area, it would get a different set of files to the last time it did so. As a promote to a parent stream can (and often will) affect child streams, Jenkins needs to also check the history of the parent stream. It was the performance problems with running these commands that lead to the "ignore parent changes" option being implemented, as that significantly reduces the amount of work done too.
          I guess the short answer to "why" is "because the accurev CLI didn't provide a good way of doing this", and hence we're stuck with it (as not everyone runs the latest version, especially once one has been bitten by major blocking issues on latest versions before).

          Looking forward, I'd like to see the plugin moving to the "new" Jenkins VCS-plugin API (it's no longer new, it's just newer than the API we're using), and making more use of accurev transaction numbers (if accurev had always supported a "populate this directory X from that stream Y up to transaction number Z" operation, much of this messing around with snapshots would not have been necessary, but the accurev server side hasn't always supported that and hence the plugin can't use it without introducing compatibility issues) in order to sort out builds missing triggering changes from history reports etc (basing things on time is unreliable and inaccurate, especially on MS Windows where keeping clocks accurate within 5 seconds is a challenge).

          Regards,

          Peter

          pjdarton added a comment - Hi Jack, Nice to see some attention from accurev themselves. Yes, my name's all over many of the recent changes as I did a number of bugfixes/improvements to make it "work for my needs" (and hopefully others' needs too). I spent quite a while with my colleague Martin McAuley and accurev support working out a way of making the build process reliable (answer: avoid doing an accurev login unless logged out) and suitable for use with our internal build labelling/packaging infrastructure, as well as working around a memory leak issue (I'm not sure that one is entirely fixed, but it's no longer causing problems). When you use the Accurev GUI to show the streams within a depot, unless you've hidden all the snapshots, the GUI takes /ages/ to display anything, and just as long to refresh after you've done anything with it - it makes the GUI fairly unusable as it's doing (and re-parsing) a "show streams" commands each time. I think that it also impacts on server performance and the time it takes to do various other commands - certainly, we found that general performance improved after I "remove stream"ed about 65000 snapshots in the depot we're working in (accurev's philosophy of "snapshots are lightweight, you can create as many as you like", which is used to negate any "why do I need to create a snapshot to do this" issues, is clearly not entirely true!). However, this would certainly qualify as low-hanging fruit - I reckon it'd only take an hour or two, once you've got the general modify/build/test cycle sorted out that is (which could take a little longer). I would also strongly recommend sorting out JENKINS-10937 as well on that basis - it's a definite bug that I introduced due to a lack of testing (I don't use workspaces on my Jenkins setup as I use many slaves, and I've also found that "accurev update" is not 100% reliable, hence I just brute-force a populate instead) and it should be easy to fix. As for "why are we doing a show streams" command in the first place: It's because of accurev's stream hierarchy/inheritance model - Jenkins needs to know if, were it to populate a new working area, it would get a different set of files to the last time it did so. As a promote to a parent stream can (and often will) affect child streams, Jenkins needs to also check the history of the parent stream. It was the performance problems with running these commands that lead to the "ignore parent changes" option being implemented, as that significantly reduces the amount of work done too. I guess the short answer to "why" is "because the accurev CLI didn't provide a good way of doing this", and hence we're stuck with it (as not everyone runs the latest version, especially once one has been bitten by major blocking issues on latest versions before). Looking forward, I'd like to see the plugin moving to the "new" Jenkins VCS-plugin API (it's no longer new, it's just newer than the API we're using), and making more use of accurev transaction numbers (if accurev had always supported a "populate this directory X from that stream Y up to transaction number Z" operation, much of this messing around with snapshots would not have been necessary, but the accurev server side hasn't always supported that and hence the plugin can't use it without introducing compatibility issues) in order to sort out builds missing triggering changes from history reports etc (basing things on time is unreliable and inaccurate, especially on MS Windows where keeping clocks accurate within 5 seconds is a challenge). Regards, Peter

          Jack Flynn added a comment -

          Ha, I hear you Peter. We are getting so many customer's that are using the plugin that we decided to put some more time investigating & helping out. You are right, a looping, show streams will take away resources from your accurev server & performance will suffer - especially when dealing with that many streams/snapshots.

          One thing that has helped our customers steer clear of a full show streams is the new -r & -R flags. -r will give all of the parent streams of a workspace (or stream too). I'm not sure of the accurev version when that little nugget was introduced. It may be the 5.x series. There has been so much work put into update & populate that I'm pretty confident that a clean "refresh" of a workspace (update & pop) can go away and just utilize update.

          I know you are concerned about compatibility issues but I think the new features/enhancements might be enough to draw a line in the sand & say if you upgrade accurev, you can get a new, optimized plugin. I know that there is a lot of old code "history" within that plugin and I would love to move past that - especially under the Jenkins fork.

          I'm not a big java head but I can easily write code in perl if you want to try to interpret into java. Let's keep the ball rolling and see what we can come up with. Very happy to help.

          --Jack

          Jack Flynn added a comment - Ha, I hear you Peter. We are getting so many customer's that are using the plugin that we decided to put some more time investigating & helping out. You are right, a looping, show streams will take away resources from your accurev server & performance will suffer - especially when dealing with that many streams/snapshots. One thing that has helped our customers steer clear of a full show streams is the new -r & -R flags. -r will give all of the parent streams of a workspace (or stream too). I'm not sure of the accurev version when that little nugget was introduced. It may be the 5.x series. There has been so much work put into update & populate that I'm pretty confident that a clean "refresh" of a workspace (update & pop) can go away and just utilize update. I know you are concerned about compatibility issues but I think the new features/enhancements might be enough to draw a line in the sand & say if you upgrade accurev, you can get a new, optimized plugin. I know that there is a lot of old code "history" within that plugin and I would love to move past that - especially under the Jenkins fork. I'm not a big java head but I can easily write code in perl if you want to try to interpret into java. Let's keep the ball rolling and see what we can come up with. Very happy to help. --Jack

            Unassigned Unassigned
            pjdarton pjdarton
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: