Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5597

symlinks in archive trees lead to double archiving

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Centos 5.4
    • Similar Issues:
    • Released As:
      Jenkins 2.230

      Description

      If the tree you are archiving contains an internal symlink, the target files will be archived twice. This can lead to a very large increase in the size of the archived data and consequently, the time it takes to archive it.

      Example:

      /archive-root
      /big-directory
      /symlink -> big-directory

      Then every file in big directory will be archived twice.

      A fix would be for Hudson to detect internal symlinks and copy them rather than dereference them.

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The change was released in Jenkins 2.230. Thanks to Daniel Beck Wadeck Follonier Jeff Thompson for reviews!

            Show
            oleg_nenashev Oleg Nenashev added a comment - The change was released in Jenkins 2.230. Thanks to Daniel Beck Wadeck Follonier Jeff Thompson for reviews!
            Hide
            abhsha Abhishek Sharma added a comment -

            Hi,

            Can someone please merge the changes to mains. We're very badly hurt by this issue, and currently building Jenkins manually for this change.

            There is a pending merge request  https://github.com/jenkinsci/jenkins/pull/3947 that makes follow symlinks configurable.

             

            Thanks,

            Abhishek

            Show
            abhsha Abhishek Sharma added a comment - Hi, Can someone please merge the changes to mains. We're very badly hurt by this issue, and currently building Jenkins manually for this change. There is a pending merge request   https://github.com/jenkinsci/jenkins/pull/3947  that makes follow symlinks configurable.   Thanks, Abhishek
            Hide
            aakashsd Aakash Sudhanwa added a comment -

            Will this address a similar issue with stash/unstash as well ?

            We are using git to checkout sources - which have symbolic links. Then when the sources are stashed and unstashed the symbolic links are lost. They appear as separate directories. This is leading up to a series of issues including bloating up of the sanbox size.

            Show
            aakashsd Aakash Sudhanwa added a comment - Will this address a similar issue with stash/unstash as well ? We are using git to checkout sources - which have symbolic links. Then when the sources are stashed and unstashed the symbolic links are lost. They appear as separate directories. This is leading up to a series of issues including bloating up of the sanbox size.
            Hide
            mwinter69 Markus Winter added a comment -

            opened a pull request https://github.com/jenkinsci/jenkins/pull/3947 that make follow symlinks configurable

            Show
            mwinter69 Markus Winter added a comment - opened a pull request https://github.com/jenkinsci/jenkins/pull/3947 that make follow symlinks configurable
            Hide
            mwinter69 Markus Winter added a comment -

            Ran into an issue where a build made out of 1000 directory entries over 11 million for the ant DirectoryScanner because of symlinks to directories that again contains symlinks in a subfolder despite having an exclude pattern on the problematic folders.

            archive pattern: gen/**/*log

            exclude pattern: gen/out/modules/*/

            The symlinks were all below gen/out/modules but DirectoryScanner still tried to read everything in before applying the exclude.

            Agent process was started with -Xmx8g and ran oom.

             

            Show
            mwinter69 Markus Winter added a comment - Ran into an issue where a build made out of 1000 directory entries over 11 million for the ant DirectoryScanner because of symlinks to directories that again contains symlinks in a subfolder despite having an exclude pattern on the problematic folders. archive pattern: gen/**/*log exclude pattern: gen/out/modules/* / The symlinks were all below gen/out/modules but DirectoryScanner still tried to read everything in before applying the exclude. Agent process was started with -Xmx8g and ran oom.  

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              pgweiss pgweiss
              Votes:
              23 Vote for this issue
              Watchers:
              29 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: