• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • disk-usage-plugin
    • None

      normal du doesn't so disk usage shouldnt either.

      you risk circularities and jumping outside of jobs workspace if you do any follows

          [JENKINS-13247] disk usage should not follow symlinks

          Ben Golding added a comment -

          I see this issue also on Windows slaves.

          Example:
          Jenkins reports workspace 260GB ( http://<host>:8080/plugin/disk-usage/ )
          Windows properties on the workspace folder reports 49GB
          Cygwin 'du -hs' reports 49GB

          Ben Golding added a comment - I see this issue also on Windows slaves. Example: Jenkins reports workspace 260GB ( http://<host>:8080/plugin/disk-usage/ ) Windows properties on the workspace folder reports 49GB Cygwin 'du -hs' reports 49GB

          evernat added a comment - - edited

          You are certainly right.
          And the method hudson.Util.isSymLink(File) seems perfect to use on every file going through the disk-usage plugin.

          But, what would be the performance on a very large number of files?
          When using Java 6 vs Java 7 ?
          And when using Windows vs Linux?
          I do not know the answers to these questions, but you can try with a fork of the disk-usage plugin.

          evernat added a comment - - edited You are certainly right. And the method hudson.Util.isSymLink(File) seems perfect to use on every file going through the disk-usage plugin. But, what would be the performance on a very large number of files? When using Java 6 vs Java 7 ? And when using Windows vs Linux? I do not know the answers to these questions, but you can try with a fork of the disk-usage plugin.

          Ben Golding added a comment -

          I think performance is important, but less important than correctness.

          Also isn't the disk usage calculation anyway running as a background thread?

          You may also consider that following a symlink to a directory potentially causes more files/dirs to be examined, so in those cases it will probably be faster overall, due to fewer total number of files/dirs.

          Ben Golding added a comment - I think performance is important, but less important than correctness. Also isn't the disk usage calculation anyway running as a background thread? You may also consider that following a symlink to a directory potentially causes more files/dirs to be examined, so in those cases it will probably be faster overall, due to fewer total number of files/dirs.

          evernat added a comment -

          > performance is important, but less important than correctness
          Yes, of course correctness is important, but less important than the plugin causing unavailability of Jenkins for some of the people, including you.

          I do not say that this should not be changed; I say that the change should certainly be tested with a large number of files, in several environments, before being pushed.

          evernat added a comment - > performance is important, but less important than correctness Yes, of course correctness is important, but less important than the plugin causing unavailability of Jenkins for some of the people, including you. I do not say that this should not be changed; I say that the change should certainly be tested with a large number of files, in several environments, before being pushed.

          centic added a comment -

          What about only checking for symlinks for directories?

          centic added a comment - What about only checking for symlinks for directories?

          Ben Golding added a comment -

          My suggestion is to make this part of the next plugin release, but controlled by a global option (and keep the default behaviour unchanged - follow all symlinks).

          The global option could also control different behaviour for file/directory symlink, as 'centic' suggested.

          I would be happy to help with testing.

          Ben Golding added a comment - My suggestion is to make this part of the next plugin release, but controlled by a global option (and keep the default behaviour unchanged - follow all symlinks). The global option could also control different behaviour for file/directory symlink, as 'centic' suggested. I would be happy to help with testing.

          When I wrote this ticket, the original environment was linux clearcase dynamic views.

          For those that haven't had suffered with clearcase in a unix environment, it
          creates a virtual file system /vobs (vobs/subdirs based on how you divide your typically monolithic repository)
          creates a sorta workspace /view/workspace_name
          inside the sorta workspace has /view/workspace_name/vobs/whatever
          also in the sorta workspace is /view/workspace_name/links_to_all_root_filesystem_elements

          so, if you follow smylinks you have infinite recursion.

          Greg Moncreaff added a comment - When I wrote this ticket, the original environment was linux clearcase dynamic views. For those that haven't had suffered with clearcase in a unix environment, it creates a virtual file system /vobs (vobs/subdirs based on how you divide your typically monolithic repository) creates a sorta workspace /view/workspace_name inside the sorta workspace has /view/workspace_name/vobs/whatever also in the sorta workspace is /view/workspace_name/links_to_all_root_filesystem_elements so, if you follow smylinks you have infinite recursion.

          evernat added a comment -

          @Ben Golding
          I am not against a configuration option.
          Of course, it seems better if it "just works" without configuration (in all environments), but otherwise configuration is supposed to be better than a "bug".

          Most important is that you may need to submit a github pull request with working code in order to make it accepted by the plugin maintainer.

          evernat added a comment - @Ben Golding I am not against a configuration option. Of course, it seems better if it "just works" without configuration (in all environments), but otherwise configuration is supposed to be better than a "bug". Most important is that you may need to submit a github pull request with working code in order to make it accepted by the plugin maintainer.

          Why we should calculate symlinks at all?
          Adding hudson.Util.isSymLink(File) wouldn't cause performance degradation because of page objects cashes in kernel.

          Kanstantsin Shautsou added a comment - Why we should calculate symlinks at all? Adding hudson.Util.isSymLink(File) wouldn't cause performance degradation because of page objects cashes in kernel.

          Code changed in jenkins
          User: Kanstantsin Shautsou
          Path:
          src/main/java/hudson/plugins/disk_usage/DiskUsageThread.java
          http://jenkins-ci.org/commit/disk-usage-plugin/361984ea3af33b8cbbb6910e2e8ad8e8bac431a3
          Log:
          JENKINS-13247 don't calculate symlinks

          Tested with one file and symlink to it in workspace.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kanstantsin Shautsou Path: src/main/java/hudson/plugins/disk_usage/DiskUsageThread.java http://jenkins-ci.org/commit/disk-usage-plugin/361984ea3af33b8cbbb6910e2e8ad8e8bac431a3 Log: JENKINS-13247 don't calculate symlinks Tested with one file and symlink to it in workspace.

          Code changed in jenkins
          User: Vojtěch Juránek
          Path:
          .gitignore
          src/main/java/hudson/plugins/disk_usage/DiskUsageThread.java
          http://jenkins-ci.org/commit/disk-usage-plugin/e1334059b37d17c0885fb940ce20fc41a1515163
          Log:
          Merge pull request #12 from KostyaSha/master

          JENKINS-13247 don't calculate symlink file size

          Compare: https://github.com/jenkinsci/disk-usage-plugin/compare/301ef38d342e...e1334059b37d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vojtěch Juránek Path: .gitignore src/main/java/hudson/plugins/disk_usage/DiskUsageThread.java http://jenkins-ci.org/commit/disk-usage-plugin/e1334059b37d17c0885fb940ce20fc41a1515163 Log: Merge pull request #12 from KostyaSha/master JENKINS-13247 don't calculate symlink file size Compare: https://github.com/jenkinsci/disk-usage-plugin/compare/301ef38d342e...e1334059b37d

          vjuranek added a comment -

          Hi,
          I merged PR #12. Before I release it, what the concerns about performance actually are about? On JDK7, there's native support for checking if the file is symlink, on older JDKs rest of the hudson.Util.isSymlink() will be used, but it doesn't seem me that is would cause any significant issues. If I miss something, please let me know and I'll make skipping symlinks optional.
          Thanks

          vjuranek added a comment - Hi, I merged PR #12. Before I release it, what the concerns about performance actually are about? On JDK7, there's native support for checking if the file is symlink, on older JDKs rest of the hudson.Util.isSymlink() will be used, but it doesn't seem me that is would cause any significant issues. If I miss something, please let me know and I'll make skipping symlinks optional. Thanks

          evernat added a comment -

          Hi vjuranek,
          Concerns about performance were the difference between before and after using isSymLink when there are a large number of files, for Windows vs Linux and for JDK6 vs JDK7 (and on Windows on JDK6, isSymLink uses jna).

          I do not say that there is a problem, I have suggested to test in order to know.
          Anyway, you know much more about this than me, so if you think that this should not be a problem, then it's ok and fixed now.
          Thanks to care.

          evernat added a comment - Hi vjuranek, Concerns about performance were the difference between before and after using isSymLink when there are a large number of files, for Windows vs Linux and for JDK6 vs JDK7 (and on Windows on JDK6, isSymLink uses jna). I do not say that there is a problem, I have suggested to test in order to know. Anyway, you know much more about this than me, so if you think that this should not be a problem, then it's ok and fixed now. Thanks to care.

          vjuranek added a comment -

          Actually I don't know much about windows and moreover haven't any win machine with large FS where I could test it. So if there is any volunteer how is willing to build the plugin from from trunk and test it, that would be great.

          vjuranek added a comment - Actually I don't know much about windows and moreover haven't any win machine with large FS where I could test it. So if there is any volunteer how is willing to build the plugin from from trunk and test it, that would be great.

          Everybody who wants performance i can advice update to java7.

          Kanstantsin Shautsou added a comment - Everybody who wants performance i can advice update to java7.

          Oleg Nenashev added a comment -

          Seems that fix resolves JENKINS-15974 as well. When are you going to kick-off new release?

          Oleg Nenashev added a comment - Seems that fix resolves JENKINS-15974 as well. When are you going to kick-off new release?

          centic added a comment -

          seems to be fixed and released now => resolving issue

          centic added a comment - seems to be fixed and released now => resolving issue

            vjuranek vjuranek
            moncreaff Greg Moncreaff
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: