Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25783

very long build directory paths are created by Matrix project, and difficult directcory names

    • Icon: Improvement Improvement
    • Resolution: Duplicate
    • Icon: Minor Minor
    • matrix-project-plugin
    • Jenkins 1.576, Matrix Plugin: 1.4, windows, Unix.

      Hi.

      An issue related to 6048, but not just about special characters, but too long path names.

      The Matrix Project plugin is picking the multi-configuration directory names, based on both the configuration axis names and their values, and this leads to very long sub-directory paths, not to mention names with inconvenient characters in them, as my options include hyphens as part of their values. (Both Windows and Unix allow hyphens in their directory names, but then to traverse these directories interactively is inconvenient as it requires escaping.)

      Example:
      I have three axis, one for the slave name, (whose parameter name defaults to "label"), a configuration build option for either 32 bit or 64 bit, and a debug or release build type option.

      label
      slaves names: company_server_1, company_server_2
      bit_option:
      values" --enable32, --enable64
      type:
      values: debug, release

      This leads to directory names such as:

      .../label/company_server_1/bit_option/--enable32/type/release
      .../label/company_server_2/bit_option/--enable64/type/debug

      In other words, 2 sub-directories as added for each axis consisting of a parameter name, and one of its possible values. Having 3 axis means 6 sub-directories are presently used.

      In some cases, when added to the root of the file system used for the slave connection, I can end up a base build path of 100 characters, leading me to build problems when the build themselves contains other very long path names.

      Trying to cd into a directory starting with --enable doesn't work, these have to be quoted.

      I can do various things to shorten the axis parameter names, but not too much about the values, nor the root of the file system for the slave connection.

      For example, I can make the label just "l", and the "bits_options" just "b", and the type option just t, but cannot change the options values themselves. This still leads to 6 sub-directories as follows:

      .../l/company_server_1/b/--enable32/t/release
      .../l/company_server_2/b/--enable64/t/debug

      Existing suggestion not workable:
      This is a enhancement that is not needed for all projects. Invoking the Jenkins at startup with the MatrixConfiguration.useShortWorkspaceName as described in the 6048 issue would mean all projects get shortened hashname directories, which is not a desirable fix. Other users do not want or need this. The enhancement needs to be able to be applied on a per-project basis.

      Suggestions:
      1) It is NOT necessary to include the parameter name as part of the directory path, as only the parameter values need to be used. In my example earlier, this means we do not need to include the "label" (or "l"), or "bit_option" (or 'b"), or "type" (or "t") in the directory path. Only the parameter values (and not also their names) are required. This would shorten the above example by 22 characters as coded originally, and a a further 6 characters (and 3 sub-directories) less after doing everything I can presently due to shorten the path. Only the following 3 sub-directory paths would be needed, and not 6 as are presently used.
      .../company_server_1/--enable32/release
      .../company_server_2/--enable64/debug

      So,the first suggestion is to either drop using the parameter name as part of the directory entirely, or at least, have a tick box which defaults to "on" for compatibility with existing behaviour, but which could be turned off.

      2) the second suggestion is to allow the specifying of a directory name for each value. For example, in my situation, I would like to map the following values to shorter directory names:

      parameter : company_server_1 ---> s1
      parameter : company_server_2 ---> s2
      parameter : debug ---> d
      parameter : release ---> r
      parameter : --enable32 ---> 32
      parameter : --enable64 ---> 64

      if I could specify the directory name to be used with each value, then with the above changes included, my 3 axis of parameters can be specified as 3 very short directory names, without special characters, as in:
      .../s1/32/r
      .../s2/64/d

      which is considerably shorter then then the present method using 6 sub-directories which makes me skimp on parameter names.

      .../label/company_server_1/bit_option/--enable32/type/release
      .../label/company_server_2/bit_option/--enable64/type/debug

          [JENKINS-25783] very long build directory paths are created by Matrix project, and difficult directcory names

          Daniel Beck added a comment - - edited

          (note to self: read first, then comment )

          Daniel Beck added a comment - - edited (note to self: read first, then comment )

          Jordan Spiker added a comment -

          Label expression axis can also create bad paths. If I have "label1&&label2" expression, the path then include /job/label1&&label2/...

          This makes for some nasty workarounds in shell scripts.

          Jordan Spiker added a comment - Label expression axis can also create bad paths. If I have "label1&&label2" expression, the path then include /job/label1&&label2/... This makes for some nasty workarounds in shell scripts.

          Daniel Beck added a comment -

          spikerjenk2: Wouldn't it suffice to just consistently double-quote references to the folder? What specifically breaks?

          Daniel Beck added a comment - spikerjenk2 : Wouldn't it suffice to just consistently double-quote references to the folder? What specifically breaks?

          Pascal Bach added a comment -

          You are right if the path was quoted correctly everywhere there would be no issue. However if you build a whole linux distribution with hundreds of third party packages it is very likely that some components get it wrong. To fix this is a huge effort and not possible to fix in the short term.
          So I also would appreciate an option that would allow to get more "standard" paths.

          Pascal Bach added a comment - You are right if the path was quoted correctly everywhere there would be no issue. However if you build a whole linux distribution with hundreds of third party packages it is very likely that some components get it wrong. To fix this is a huge effort and not possible to fix in the short term. So I also would appreciate an option that would allow to get more "standard" paths.

          Jordan Spiker added a comment - - edited

          @Daniel @Pascal

          The worst offender is python virtualenv. Wrapper scripts insert the shebang of the interpreter, which is in the workspace. So "myProg.py" would have

          #!/path/to/my/job/label1&&label2/venv/bin/python
          ...
          

          I unfortunately have to wrap everything in a shell script to re-write the shebang.

          I'd also like to add (most) linux kernels limit the shebang line to 127 characters, which is easily outdone with big matrix jobs.

          Jordan Spiker added a comment - - edited @Daniel @Pascal The worst offender is python virtualenv. Wrapper scripts insert the shebang of the interpreter, which is in the workspace. So "myProg.py" would have #!/path/to/my/job/label1&&label2/venv/bin/python ... I unfortunately have to wrap everything in a shell script to re-write the shebang. I'd also like to add (most) linux kernels limit the shebang line to 127 characters, which is easily outdone with big matrix jobs.

          Daniel Beck added a comment - - edited

          FWIW there's a global (Jenkins-wide) option to use short sub-workspace paths for matrix projects: hudson.matrix.MatrixConfiguration.useShortWorkspaceName

          More information on:
          https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

          Daniel Beck added a comment - - edited FWIW there's a global (Jenkins-wide) option to use short sub-workspace paths for matrix projects: hudson.matrix.MatrixConfiguration.useShortWorkspaceName More information on: https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

          Craig Ringer added a comment - - edited

          Daniel, very useful hint. Thankyou. The applicability of that property sure isn't limited to Cygwin.

          Some Windows kernels have bugs relating to deep paths (well short of the official path limit). I've had to abbreviate all my labels to get things to work, and even then it's troublesome.

          On *nix, label expressions containing shell metacharacters are "fun". Yes, everything should be carefully quoted. It's a pain when you're working with things like third party shell script snippets in configure scripts though.

          Some form of mapping would be very helpful. Or at least exposing this as an advanced matrix build option instead of a system property.

          Craig Ringer added a comment - - edited Daniel, very useful hint. Thankyou. The applicability of that property sure isn't limited to Cygwin. Some Windows kernels have bugs relating to deep paths (well short of the official path limit). I've had to abbreviate all my labels to get things to work, and even then it's troublesome. On *nix, label expressions containing shell metacharacters are "fun". Yes, everything should be carefully quoted. It's a pain when you're working with things like third party shell script snippets in configure scripts though. Some form of mapping would be very helpful. Or at least exposing this as an advanced matrix build option instead of a system property.

          Jesse Glick added a comment -

          Better solved more generically, as in JENKINS-30148.

          Jesse Glick added a comment - Better solved more generically, as in  JENKINS-30148 .

            Unassigned Unassigned
            rickybobpat Rick Patterson
            Votes:
            5 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: