Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-2812

Clean Artifactory caches

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Caches of remote repositories take up ~80% of the storage required for our repositories.

      We should clean up these caches, ideally in a way that causes minimal disruption to the Jenkins project.

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment -

            +1 I think this is a good process. I like the safety provided by removing only artifacts that exist upstream and have matching checksums.

            Could you also report those cases where an artifact exists upstream with a non-matching checksum? That seems like a possible cause for future evaluation.

            Show
            markewaite Mark Waite added a comment - +1 I think this is a good process. I like the safety provided by removing only artifacts that exist upstream and have matching checksums. Could you also report those cases where an artifact exists upstream with a non-matching checksum? That seems like a possible cause for future evaluation.
            Hide
            danielbeck Daniel Beck added a comment -

            Could you also report those cases where an artifact exists upstream with a non-matching checksum? That seems like a possible cause for future evaluation.

            Sure, but note we're not processing all artifacts in the cache, that would be a separate project.

            FWIW I've now evaluated all files 1.4MB or larger in repo1-cache, which is ~120k files out of 5.7M, or about 2% of files, which make up 1.1 of 1.5 TB of storage used (~75%). So this looks like focusing on large files might be a viable strategy.

            I've asked Baruch as well whether this plan looks sane just in case Artifactory would be weird about this.

            Show
            danielbeck Daniel Beck added a comment - Could you also report those cases where an artifact exists upstream with a non-matching checksum? That seems like a possible cause for future evaluation. Sure, but note we're not processing all artifacts in the cache, that would be a separate project. FWIW I've now evaluated all files 1.4MB or larger in repo1-cache, which is ~120k files out of 5.7M, or about 2% of files, which make up 1.1 of 1.5 TB of storage used (~75%). So this looks like focusing on large files might be a viable strategy. I've asked Baruch as well whether this plan looks sane just in case Artifactory would be weird about this.
            Hide
            danielbeck Daniel Beck added a comment -

            I've asked Baruch as well whether this plan looks sane just in case Artifactory would be weird about this.

            Got confirmation that this plan looks sane so will proceed with it.

            Show
            danielbeck Daniel Beck added a comment - I've asked Baruch as well whether this plan looks sane just in case Artifactory would be weird about this. Got confirmation that this plan looks sane so will proceed with it.
            Hide
            danielbeck Daniel Beck added a comment -

            Cleaned up 1.2 TB across the three largest caches, limited to files 10+ MB in size.

            Show
            danielbeck Daniel Beck added a comment - Cleaned up 1.2 TB across the three largest caches, limited to files 10+ MB in size.
            Hide
            danielbeck Daniel Beck added a comment -

            Storage has caught up, and confirms we went from 3.56 to 2.42 TB storage with the above.

            Considering this resolved for now unless I hear back from JFrog otherwise.

            Show
            danielbeck Daniel Beck added a comment - Storage has caught up, and confirms we went from 3.56 to 2.42 TB storage with the above. Considering this resolved for now unless I hear back from JFrog otherwise.

              People

              Assignee:
              danielbeck Daniel Beck
              Reporter:
              danielbeck Daniel Beck
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: