Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25783

very long build directory paths are created by Matrix project, and difficult directcory names

    XMLWordPrintable

Details

    Description

      Hi.

      An issue related to 6048, but not just about special characters, but too long path names.

      The Matrix Project plugin is picking the multi-configuration directory names, based on both the configuration axis names and their values, and this leads to very long sub-directory paths, not to mention names with inconvenient characters in them, as my options include hyphens as part of their values. (Both Windows and Unix allow hyphens in their directory names, but then to traverse these directories interactively is inconvenient as it requires escaping.)

      Example:
      I have three axis, one for the slave name, (whose parameter name defaults to "label"), a configuration build option for either 32 bit or 64 bit, and a debug or release build type option.

      label
      slaves names: company_server_1, company_server_2
      bit_option:
      values" --enable32, --enable64
      type:
      values: debug, release

      This leads to directory names such as:

      .../label/company_server_1/bit_option/--enable32/type/release
      .../label/company_server_2/bit_option/--enable64/type/debug

      In other words, 2 sub-directories as added for each axis consisting of a parameter name, and one of its possible values. Having 3 axis means 6 sub-directories are presently used.

      In some cases, when added to the root of the file system used for the slave connection, I can end up a base build path of 100 characters, leading me to build problems when the build themselves contains other very long path names.

      Trying to cd into a directory starting with --enable doesn't work, these have to be quoted.

      I can do various things to shorten the axis parameter names, but not too much about the values, nor the root of the file system for the slave connection.

      For example, I can make the label just "l", and the "bits_options" just "b", and the type option just t, but cannot change the options values themselves. This still leads to 6 sub-directories as follows:

      .../l/company_server_1/b/--enable32/t/release
      .../l/company_server_2/b/--enable64/t/debug

      Existing suggestion not workable:
      This is a enhancement that is not needed for all projects. Invoking the Jenkins at startup with the MatrixConfiguration.useShortWorkspaceName as described in the 6048 issue would mean all projects get shortened hashname directories, which is not a desirable fix. Other users do not want or need this. The enhancement needs to be able to be applied on a per-project basis.

      Suggestions:
      1) It is NOT necessary to include the parameter name as part of the directory path, as only the parameter values need to be used. In my example earlier, this means we do not need to include the "label" (or "l"), or "bit_option" (or 'b"), or "type" (or "t") in the directory path. Only the parameter values (and not also their names) are required. This would shorten the above example by 22 characters as coded originally, and a a further 6 characters (and 3 sub-directories) less after doing everything I can presently due to shorten the path. Only the following 3 sub-directory paths would be needed, and not 6 as are presently used.
      .../company_server_1/--enable32/release
      .../company_server_2/--enable64/debug

      So,the first suggestion is to either drop using the parameter name as part of the directory entirely, or at least, have a tick box which defaults to "on" for compatibility with existing behaviour, but which could be turned off.

      2) the second suggestion is to allow the specifying of a directory name for each value. For example, in my situation, I would like to map the following values to shorter directory names:

      parameter : company_server_1 ---> s1
      parameter : company_server_2 ---> s2
      parameter : debug ---> d
      parameter : release ---> r
      parameter : --enable32 ---> 32
      parameter : --enable64 ---> 64

      if I could specify the directory name to be used with each value, then with the above changes included, my 3 axis of parameters can be specified as 3 very short directory names, without special characters, as in:
      .../s1/32/r
      .../s2/64/d

      which is considerably shorter then then the present method using 6 sub-directories which makes me skimp on parameter names.

      .../label/company_server_1/bit_option/--enable32/type/release
      .../label/company_server_2/bit_option/--enable64/type/debug

      Attachments

        Issue Links

          Activity

            rickybobpat Rick Patterson created issue -
            danielbeck Daniel Beck added a comment - - edited

            (note to self: read first, then comment )

            danielbeck Daniel Beck added a comment - - edited (note to self: read first, then comment )
            spikerjenk2 Jordan Spiker added a comment -

            Label expression axis can also create bad paths. If I have "label1&&label2" expression, the path then include /job/label1&&label2/...

            This makes for some nasty workarounds in shell scripts.

            spikerjenk2 Jordan Spiker added a comment - Label expression axis can also create bad paths. If I have "label1&&label2" expression, the path then include /job/label1&&label2/... This makes for some nasty workarounds in shell scripts.
            danielbeck Daniel Beck added a comment -

            spikerjenk2: Wouldn't it suffice to just consistently double-quote references to the folder? What specifically breaks?

            danielbeck Daniel Beck added a comment - spikerjenk2 : Wouldn't it suffice to just consistently double-quote references to the folder? What specifically breaks?
            bachp Pascal Bach added a comment -

            You are right if the path was quoted correctly everywhere there would be no issue. However if you build a whole linux distribution with hundreds of third party packages it is very likely that some components get it wrong. To fix this is a huge effort and not possible to fix in the short term.
            So I also would appreciate an option that would allow to get more "standard" paths.

            bachp Pascal Bach added a comment - You are right if the path was quoted correctly everywhere there would be no issue. However if you build a whole linux distribution with hundreds of third party packages it is very likely that some components get it wrong. To fix this is a huge effort and not possible to fix in the short term. So I also would appreciate an option that would allow to get more "standard" paths.
            spikerjenk2 Jordan Spiker added a comment - - edited

            @Daniel @Pascal

            The worst offender is python virtualenv. Wrapper scripts insert the shebang of the interpreter, which is in the workspace. So "myProg.py" would have

            #!/path/to/my/job/label1&&label2/venv/bin/python
            ...
            

            I unfortunately have to wrap everything in a shell script to re-write the shebang.

            I'd also like to add (most) linux kernels limit the shebang line to 127 characters, which is easily outdone with big matrix jobs.

            spikerjenk2 Jordan Spiker added a comment - - edited @Daniel @Pascal The worst offender is python virtualenv. Wrapper scripts insert the shebang of the interpreter, which is in the workspace. So "myProg.py" would have #!/path/to/my/job/label1&&label2/venv/bin/python ... I unfortunately have to wrap everything in a shell script to re-write the shebang. I'd also like to add (most) linux kernels limit the shebang line to 127 characters, which is easily outdone with big matrix jobs.
            danielbeck Daniel Beck added a comment - - edited

            FWIW there's a global (Jenkins-wide) option to use short sub-workspace paths for matrix projects: hudson.matrix.MatrixConfiguration.useShortWorkspaceName

            More information on:
            https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

            danielbeck Daniel Beck added a comment - - edited FWIW there's a global (Jenkins-wide) option to use short sub-workspace paths for matrix projects: hudson.matrix.MatrixConfiguration.useShortWorkspaceName More information on: https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties
            ringerc Craig Ringer added a comment - - edited

            Daniel, very useful hint. Thankyou. The applicability of that property sure isn't limited to Cygwin.

            Some Windows kernels have bugs relating to deep paths (well short of the official path limit). I've had to abbreviate all my labels to get things to work, and even then it's troublesome.

            On *nix, label expressions containing shell metacharacters are "fun". Yes, everything should be carefully quoted. It's a pain when you're working with things like third party shell script snippets in configure scripts though.

            Some form of mapping would be very helpful. Or at least exposing this as an advanced matrix build option instead of a system property.

            ringerc Craig Ringer added a comment - - edited Daniel, very useful hint. Thankyou. The applicability of that property sure isn't limited to Cygwin. Some Windows kernels have bugs relating to deep paths (well short of the official path limit). I've had to abbreviate all my labels to get things to work, and even then it's troublesome. On *nix, label expressions containing shell metacharacters are "fun". Yes, everything should be carefully quoted. It's a pain when you're working with things like third party shell script snippets in configure scripts though. Some form of mapping would be very helpful. Or at least exposing this as an advanced matrix build option instead of a system property.
            olivergondza Oliver Gondža made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-30148 [ JENKINS-30148 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 159762 ] JNJira + In-Review [ 180117 ]
            jglick Jesse Glick added a comment -

            Better solved more generically, as in JENKINS-30148.

            jglick Jesse Glick added a comment - Better solved more generically, as in  JENKINS-30148 .
            jglick Jesse Glick made changes -
            Assignee Kohsuke Kawaguchi [ kohsuke ]
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]

            People

              Unassigned Unassigned
              rickybobpat Rick Patterson
              Votes:
              5 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: