Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30148

Allocate shorter workspace if it will be too long for reasonable use inside build

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None

      When using rich matrix axes or deep folders hierarchy, substantial part of maximal path name (255 on windows for instance) can be consumed by the path to workspace not leaving enough room for the build itself.

      Introduce threshold (global or per job) to define expected path length left for build. if the default allocation algorithm creates WS path so long that there will be not enough path length left for the build, hash should be used instead.

      Therefore, if the threshold will be configured to 1024 Jenkins will use hash on Windows all the time and on linux only if the workspace path exceeds 3072 characters (assuming 4096 is the maximal path allowed on linux).

          [JENKINS-30148] Allocate shorter workspace if it will be too long for reasonable use inside build

          Oliver Gondža created issue -
          Oliver Gondža made changes -
          Description Original: When using rich matrix axes or deep folders hierarchy, substantial part of maximal path name (255 on windows for instance) can be consumed with the path to workspace not leaving enough for the build itself.

          Introduce threshold (global or per job) to define expected path length left for build. if the default allocation algorithm creates path so long that there will be not enough path length left for the build, hash should be used instead.

          Therefore, if the threshold will be configured to 1024 Jenkins will use hash on Windows all the time and on linux only if the workspace path exceeds 3072 characters (assuming 4096 is the maximal path allowed on linux).
          New: When using rich matrix axes or deep folders hierarchy, substantial part of maximal path name (255 on windows for instance) can be consumed by the path to workspace not leaving enough room for the build itself.

          Introduce threshold (global or per job) to define expected path length left for build. if the default allocation algorithm creates WS path so long that there will be not enough path length left for the build, hash should be used instead.

          Therefore, if the threshold will be configured to 1024 Jenkins will use hash on Windows all the time and on linux only if the workspace path exceeds 3072 characters (assuming 4096 is the maximal path allowed on linux).
          Kanstantsin Shautsou made changes -
          Link New: This issue is related to JENKINS-25330 [ JENKINS-25330 ]
          Oliver Gondža made changes -
          Link New: This issue is related to JENKINS-25783 [ JENKINS-25783 ]

          Here is a plugin to address the folders part that adapts to maximal path length automatically: https://wiki.jenkins-ci.org/display/JENKINS/Short+Workspace+Path+Plugin

          Oliver Gondža added a comment - Here is a plugin to address the folders part that adapts to maximal path length automatically: https://wiki.jenkins-ci.org/display/JENKINS/Short+Workspace+Path+Plugin
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 165235 ] New: JNJira + In-Review [ 181886 ]

          Sorin Sbarnea added a comment -

          Did anyone managed to solve the workspace length issue for good? This new plugins doesn't work on master, making it almost useless because it would sort the problem only for same jobs.

          The length of the workspace is a serious issue that is causing serious issue with builds.

          I am aware of two permanent sources of failure caused by this:

          • shebang line limit breaking tools in python virtualenvl
          • ssh control path reaching its socket filename limits. control_path folder is supposed to be configured unique for each job in order to avoid occasional ssh connection hijacking. If you use the standard relative to home directory you will get hickjacking. 

          Both of these bugs are practically impossible to fix because they are kernel limits set at compile time and nobody wants to compile their own kernel.

          The only way to avoid these (an others) is to assure that Jenkins is minimizing the depth of the workspace directory. Other CI systems are smart and they use checksum folder names for avoiding this issue. 

          I know that it is "nice" to have named folders as it makes debugging easy. We don't really have to drop the current folders, instead we can create workspace in a short folder and transform the current folder in a symlink to the short one. This would keep compatibility with other tools that may make assumptions and also deal with that problem once for good.

          Sorin Sbarnea added a comment - Did anyone managed to solve the workspace length issue for good? This new plugins doesn't work on master, making it almost useless because it would sort the problem only for same jobs. The length of the workspace is a serious issue that is causing serious issue with builds. I am aware of two permanent sources of failure caused by this: shebang line limit breaking tools in python virtualenvl ssh control path reaching its socket filename limits. control_path folder is supposed to be configured unique for each job in order to avoid occasional ssh connection hijacking. If you use the standard relative to home directory you will get hickjacking.  Both of these bugs are practically impossible to fix because they are kernel limits set at compile time and nobody wants to compile their own kernel. The only way to avoid these (an others) is to assure that Jenkins is minimizing the depth of the workspace directory. Other CI systems are smart and they use checksum folder names for avoiding this issue.  I know that it is "nice" to have named folders as it makes debugging easy. We don't really have to drop the current folders, instead we can create workspace in a short folder and transform the current folder in a symlink to the short one. This would keep compatibility with other tools that may make assumptions and also deal with that problem once for good.
          Sorin Sbarnea made changes -
          Priority Original: Minor [ 4 ] New: Major [ 3 ]

          ssbarnea, note that there are several reasons why the path length can bloat and you have not mentioned what is yours. While the reason why the plugin does not work on master is purely technical, it is generally discouraged to use master for building.

          Oliver Gondža added a comment - ssbarnea , note that there are several reasons why the path length can bloat and you have not mentioned what is yours. While the reason why the plugin does not work on master is purely technical, it is generally discouraged to use master for building.

          Jesse Glick added a comment -

          Solved for branch projects in JENKINS-34564 because the problem was acute for these (and also these often had job names including dangerous characters). I proposed a more general solution to how Jenkins manages workspaces, which would deal with not just length but special characters and cleanup.

          Jesse Glick added a comment - Solved for branch projects in  JENKINS-34564 because the problem was acute for these (and also these often had job names including dangerous characters). I proposed a more general solution to how Jenkins manages workspaces, which would deal with not just length but special characters and cleanup.
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-2111 [ JENKINS-2111 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-34564 [ JENKINS-34564 ]

            Unassigned Unassigned
            olivergondza Oliver Gondža
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: