Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8224

snapshot view workspace filling up with .keep folders

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • clearcase-plugin
    • None
    • Hudson 1.387
      ClearCase Plugin 1.3.2
      Master: Linux Centos 5.3
      Slave: Linux Centos 4.7

      We are running a ClearCase build jobs on a Centos 4.7 Slave. We are currently using Snapshot views and for some reason which we cannot figure out sometimes when a build is run it creates/renames a previous view to a .keep folder. This is really troublesome as one of our target build projects is almost 40GB and we run out of space on the slave.

      Example of one view in the workspace:
      drwxr-xr-x 3 builder ccusers 4096 Nov 15 15:08 builder_RavenPlat1.3.0.0_Platform.keep.1
      drwxr-xr-x 3 builder ccusers 4096 Nov 25 23:41 builder_RavenPlat1.3.0.0_Platform.keep.2
      drwxr-xr-x 3 builder ccusers 4096 Nov 28 23:41 builder_RavenPlat1.3.0.0_Platform.keep.3
      drwxr-xr-x 3 builder ccusers 4096 Nov 30 23:42 builder_RavenPlat1.3.0.0_Platform

      In the above list, these are not the only builds, so its not every time that a build creates a keep folder. For example there were two builds done between the 28th and the 30th.

      So far we have been manually removing these keep folders but it seems to be non deterministic when the view is copied to a keep folder.

          [JENKINS-8224] snapshot view workspace filling up with .keep folders

          peterkline added a comment -

          An additional datapoint, we were in testing and we created a project and then immediately cloned it in hudson so that we have two projects building the same code on the same Clearcase stream. On one project it had created several .keep folders while the other one never did. These projects in Hudson are 100% identical except one is called test1 and the other test2 (view names also follow the same convention).

          peterkline added a comment - An additional datapoint, we were in testing and we created a project and then immediately cloned it in hudson so that we have two projects building the same code on the same Clearcase stream. On one project it had created several .keep folders while the other one never did. These projects in Hudson are 100% identical except one is called test1 and the other test2 (view names also follow the same convention).

          Vincent Latombe added a comment - - edited

          If you have .keep folders it means that for some reason the plugin detected that the view metadata were not valid. It can happen if you have connectivity problems with the Clearcase server.

          The view folder can be renamed in 2 cases :

          • the view tag associated with the existing view doesn't match the view tag that is defined in the job config
          • the view tag doesn't exist on clearcase server

          Vincent Latombe added a comment - - edited If you have .keep folders it means that for some reason the plugin detected that the view metadata were not valid. It can happen if you have connectivity problems with the Clearcase server. The view folder can be renamed in 2 cases : the view tag associated with the existing view doesn't match the view tag that is defined in the job config the view tag doesn't exist on clearcase server

          peterkline added a comment -

          Is there way to instead of creating a .keep folder to just trash the existing workspace folder on the slave? As mentioned I don't have the storage space for the plugin to determine it needs to rename the folder and stack up another 40GB on the disk.

          peterkline added a comment - Is there way to instead of creating a .keep folder to just trash the existing workspace folder on the slave? As mentioned I don't have the storage space for the plugin to determine it needs to rename the folder and stack up another 40GB on the disk.

          peterkline added a comment -

          Vincent,

          If you feel that this is not related I will open a new ticket.
          After reviewing the logs I can see where the build triggered and removed and then created a new view. It went about removing the old snapshot view but I guess it didn't remove the snapshot view storage. This is troubling because the source for the project didn't even change as this is building on an non-integration stream and we haven't rebased it since we created it (we are just testing build loads on the slave system). I assume this remove/create is what caused the .keep folder which means the rmview isn't working appropriately? Just realized that our ClearCase plugin is 1.3.3 and not 1.3.2.

          18:07:57 [RavenPlatMain_TEST3] $ cleartool desc -fmt %[found_bls]Xp\n stream:builder_RavenPlatMain_TEST3@/vobs/tlv_pvob
          18:07:57 baseline:RavenPlatMain_TEST_12_01_2010@/vobs/tlv_pvob
          18:07:57 [RavenPlatMain_TEST3] $ cleartool desc -fmt %[component]Xp\n baseline:RavenPlatMain_TEST_12_01_2010@/vobs/tlv_pvob
          18:07:57 component:tlv_comp@/vobs/tlv_pvob
          18:07:57 [RavenPlatMain_TEST3] $ cleartool lsview builder_RavenPlatMain_TEST3
          18:07:57 builder_RavenPlatMain_TEST3 /net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws
          18:07:57 [builder_RavenPlatMain_TEST3] $ cleartool lsview -cview -s
          18:07:57 builder_RavenPlatMain_TEST3
          18:07:57 [RavenPlatMain_TEST3] $ cleartool lsview builder_RavenPlatMain_TEST3
          18:07:57 builder_RavenPlatMain_TEST3 /net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws
          18:07:57 [builder_RavenPlatMain_TEST3] $ cleartool lsview -cview -s
          18:07:57 builder_RavenPlatMain_TEST3
          18:07:57 [RavenPlatMain_TEST3] $ cleartool rmview -force -tag builder_RavenPlatMain_TEST3
          18:07:57 Removing references from VOB "/vobs/tlv_pvob" ...
          18:07:57 Removed references to view "/net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws" from VOB "/vobs/tlv_pvob".
          18:07:59 [RavenPlatMain_TEST3] $ cleartool mkview -snapshot -stream builder_RavenPlatMain_TEST3@/vobs/tlv_pvob -tag builder_RavenPlatMain_TEST3 builder_RavenPlatMain_TEST3
          18:08:35 Selected Server Storage Location "raven-dev8_views".
          18:08:35 Created view.
          18:08:35 Host-local path: wsb-raven-dev8:/ccstore/builder/builder_RavenPlatMain_TEST3.vws
          18:08:35 Global path: /net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws
          18:08:35 It has the following rights:
          18:08:35 User : builder : rwx
          18:08:35 Group: ccusers : r-x
          18:08:35 Other: : r-x
          18:08:35 Created snapshot view directory "builder_RavenPlatMain_TEST3".
          18:08:36 Attached view to stream "builder_RavenPlatMain_TEST3".

          peterkline added a comment - Vincent, If you feel that this is not related I will open a new ticket. After reviewing the logs I can see where the build triggered and removed and then created a new view. It went about removing the old snapshot view but I guess it didn't remove the snapshot view storage. This is troubling because the source for the project didn't even change as this is building on an non-integration stream and we haven't rebased it since we created it (we are just testing build loads on the slave system). I assume this remove/create is what caused the .keep folder which means the rmview isn't working appropriately? Just realized that our ClearCase plugin is 1.3.3 and not 1.3.2. 18:07:57 [RavenPlatMain_TEST3] $ cleartool desc -fmt % [found_bls] Xp\n stream:builder_RavenPlatMain_TEST3@/vobs/tlv_pvob 18:07:57 baseline:RavenPlatMain_TEST_12_01_2010@/vobs/tlv_pvob 18:07:57 [RavenPlatMain_TEST3] $ cleartool desc -fmt % [component] Xp\n baseline:RavenPlatMain_TEST_12_01_2010@/vobs/tlv_pvob 18:07:57 component:tlv_comp@/vobs/tlv_pvob 18:07:57 [RavenPlatMain_TEST3] $ cleartool lsview builder_RavenPlatMain_TEST3 18:07:57 builder_RavenPlatMain_TEST3 /net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws 18:07:57 [builder_RavenPlatMain_TEST3] $ cleartool lsview -cview -s 18:07:57 builder_RavenPlatMain_TEST3 18:07:57 [RavenPlatMain_TEST3] $ cleartool lsview builder_RavenPlatMain_TEST3 18:07:57 builder_RavenPlatMain_TEST3 /net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws 18:07:57 [builder_RavenPlatMain_TEST3] $ cleartool lsview -cview -s 18:07:57 builder_RavenPlatMain_TEST3 18:07:57 [RavenPlatMain_TEST3] $ cleartool rmview -force -tag builder_RavenPlatMain_TEST3 18:07:57 Removing references from VOB "/vobs/tlv_pvob" ... 18:07:57 Removed references to view "/net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws" from VOB "/vobs/tlv_pvob". 18:07:59 [RavenPlatMain_TEST3] $ cleartool mkview -snapshot -stream builder_RavenPlatMain_TEST3@/vobs/tlv_pvob -tag builder_RavenPlatMain_TEST3 builder_RavenPlatMain_TEST3 18:08:35 Selected Server Storage Location "raven-dev8_views". 18:08:35 Created view. 18:08:35 Host-local path: wsb-raven-dev8:/ccstore/builder/builder_RavenPlatMain_TEST3.vws 18:08:35 Global path: /net/wsb-raven-dev8/ccstore/builder/builder_RavenPlatMain_TEST3.vws 18:08:35 It has the following rights: 18:08:35 User : builder : rwx 18:08:35 Group: ccusers : r-x 18:08:35 Other: : r-x 18:08:35 Created snapshot view directory "builder_RavenPlatMain_TEST3". 18:08:36 Attached view to stream "builder_RavenPlatMain_TEST3".

          Vincent Latombe added a comment - - edited

          It is probably related. Checking the logs against the implementation, it seems that according to the plugin, at the time of checkout :

          • the view tag exists
          • the view path doesn't exist <-- This shouldn't happen unless the view path has been deleted/renamed
          • so it deletes the view tag metadata and creates a new view.

          Here is the code snippet :
          FilePath filePath = new FilePath(workspace, viewPath);
          boolean viewPathExists = filePath.exists();

          where workspace is the job workspace and viewPath is the relative path in the workspace (in your case, builder_RavenPlatMain_TEST3)

          Basically this calls File#exists() on the slave with the given path. Maybe on Centos 4.7 there is something specific that makes this method return an incorrect value?

          Vincent Latombe added a comment - - edited It is probably related. Checking the logs against the implementation, it seems that according to the plugin, at the time of checkout : the view tag exists the view path doesn't exist <-- This shouldn't happen unless the view path has been deleted/renamed so it deletes the view tag metadata and creates a new view. Here is the code snippet : FilePath filePath = new FilePath(workspace, viewPath); boolean viewPathExists = filePath.exists(); where workspace is the job workspace and viewPath is the relative path in the workspace (in your case, builder_RavenPlatMain_TEST3) Basically this calls File#exists() on the slave with the given path. Maybe on Centos 4.7 there is something specific that makes this method return an incorrect value?

          peterkline added a comment -

          Is there any way to get a stripped down executable we can run X amount of times to see if we can get it the OS to report that the directory doesn't exist?

          peterkline added a comment - Is there any way to get a stripped down executable we can run X amount of times to see if we can get it the OS to report that the directory doesn't exist?

          VladDRussian added a comment -

          We've ran into the issue as well with 'Remove clearcase view on rename' true, 'dynamic view' false, 'remove when renamed' true.
          Win2k3 & Win2k8 systems with Hud/Jenk 1.382 / 1.413, CC plugin 1.3.1 - 1.3.6, jdk1.6.0_17 (master and slave boot).

          Would it be simpler to default a flag true for 'If issue rename directory *keep' Or 'if metadata issue delete and recreate view'. We are using it on INT streams and don't want any artifacts leftover (we don't use update).
          Thanks

          VladDRussian added a comment - We've ran into the issue as well with 'Remove clearcase view on rename' true, 'dynamic view' false, 'remove when renamed' true. Win2k3 & Win2k8 systems with Hud/Jenk 1.382 / 1.413, CC plugin 1.3.1 - 1.3.6, jdk1.6.0_17 (master and slave boot). Would it be simpler to default a flag true for 'If issue rename directory *keep' Or 'if metadata issue delete and recreate view'. We are using it on INT streams and don't want any artifacts leftover (we don't use update). Thanks

            Unassigned Unassigned
            peterkline peterkline
            Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated: