Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46054

Unable to pull submodules that contain ".url" in the url

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      git-client-plugin fetches the list of submodules using:

      git config -f .gitmodules --get-regexp "^submodule\\.(.*)\\.url"

      Which returns entries in the form:

      submodule.<name>.url <value>

      It then uses 

      "Pattern.compile("^submodule\\.(.*)\\.url", Pattern.MULTILINE)"

      to extract the module name (CliGitAPIImpl.java#L1062).

      I have a thirdparty submodule called "urlesc" that lives in a repo that looks something like:

      submodule.src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty.urlesc.git

      If the value contains ".url" the regex will match too much of the line.

      It thinks the name is "src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty" and so tries to run:

      git config --get "submodule.src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty.url"

      which is incorrect.

      The regex should be something like 

      "^submodule\\.(.*)\\.url "

      (note the space after url) or 

      "^submodule\\.([^ ]*)\\.url"

      .

        Attachments

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-48818 Git plugin no longer updates submodules with spaces in the name

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-48997 The new submodule pattern matching breaks submodules with spaces in name

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

            Activity

            Ascending order - Click to sort in descending order
            gfranklin Greg Franklin created issue -
            gfranklin Greg Franklin made changes -
            Field Original Value New Value
            Description git-client-plugin fetches the list of submodules using:

            git config -f .gitmodules --get-regexp "^submodule\\.(.*)\\.url"

            Which returns entries in the form:

            submodule.<name>.url <value>

            It then uses "Pattern.compile("^submodule\\.(.*)\\.url", Pattern.MULTILINE)" to extract the module name (CliGitAPIImpl.java#L1062).

            I have a thirdparty submodule called "urlesc" that lives in a repo that looks something like:

            submodule.src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty.urlesc.git

            If the value contains ".url" the regex will match too much of the line.

            It thinks the name is "src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty" and so tries to run:

            git config --get "submodule.src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty.url"

            which is incorrect.

            The regex should be something like "^submodule\\.(.*)\\.url " (note the space after url) or "^submodule\\.([^ ]*)\\.url".             		git-client-plugin fetches the list of submodules using:

            {noformat}
            git config -f .gitmodules --get-regexp "^submodule\\.(.*)\\.url"
            {noformat}


            Which returns entries in the form:

            {noformat}
            submodule.<name>.url <value>
            {noformat}

            It then uses
            {noformat}
            "Pattern.compile("^submodule\\.(.*)\\.url", Pattern.MULTILINE)"
            {noformat}
             to extract the module name (CliGitAPIImpl.java#L1062).

            I have a thirdparty submodule called "urlesc" that lives in a repo that looks something like:

            submodule.src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty.urlesc.git

            If the value contains ".url" the regex will match too much of the line.

            It thinks the name is "src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty" and so tries to run:


            {noformat}
            git config --get "submodule.src/git.repo.local/urlesc.url git.repo.local:/gitroot/thirdparty.url"
            {noformat}


            which is incorrect.

            The regex should be something like
            {noformat}
            "^submodule\\.(.*)\\.url "
            {noformat}
             (note the space after url) or
            {noformat}
            "^submodule\\.([^ ]*)\\.url"
            {noformat}
            .
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            Hide
            Permalink
            markewaite Mark Waite added a comment - - edited

            Proposed pull request build will generate artifacts that you can use for testing.

            Please review the tests and the proposed new pattern.

            The problem in my first attempt was the trailing space on the pattern. The pattern is used in two places, one to parse a string, and the other as a regexp argument to git config. When used with git config, it is searching to match a key, and the key will never include a trailing space. As far as I can tell with the new regexp, the trailing space is not required.

            Show
            markewaite Mark Waite added a comment - - edited Proposed pull request build will generate artifacts that you can use for testing. Please review the tests and the proposed new pattern . The problem in my first attempt was the trailing space on the pattern. The pattern is used in two places, one to parse a string, and the other as a regexp argument to git config . When used with git config , it is searching to match a key, and the key will never include a trailing space. As far as I can tell with the new regexp, the trailing space is not required.
            markewaite Mark Waite made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java
            http://jenkins-ci.org/commit/git-client-plugin/725a9ac09ac9c7c5970f1bd3cf6c77503017e35f
            Log:
            Add JENKINS-46054 failing tests

            If the repository URL for a submodule includes ".url" in the
            URL, then the submodule query code in the command line git
            implementation gathers the wrong name as the name of the
            submodule.

            These tests check specific conditions with the regular
            expression string used to parse the "git config"
            output.

            They intentionally fail with the current regular expression.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java http://jenkins-ci.org/commit/git-client-plugin/725a9ac09ac9c7c5970f1bd3cf6c77503017e35f Log: Add JENKINS-46054 failing tests If the repository URL for a submodule includes ".url" in the URL, then the submodule query code in the command line git implementation gathers the wrong name as the name of the submodule. These tests check specific conditions with the regular expression string used to parse the "git config" output. They intentionally fail with the current regular expression.
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            http://jenkins-ci.org/commit/git-client-plugin/81392eda3c65e023d82ca69221070c080128cd26
            Log:
            Add JENKINS-46054 integration test

            Prior to the fix, this test fails on command line git but passes
            on JGit.

            After the fix, the test passes on both implementations.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/81392eda3c65e023d82ca69221070c080128cd26 Log: Add JENKINS-46054 integration test Prior to the fix, this test fails on command line git but passes on JGit. After the fix, the test passes on both implementations.
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            http://jenkins-ci.org/commit/git-client-plugin/52f681e6153eb088495edd497a82d6dc919dcae3
            Log:
            [Fixed JENKINS-46054] submodule repo URL with '.url' substring failed

            Modify the submodule config parsing regular expression to correctly
            extract the submodule name from the config output.

            Splits cli submodule URL regexp use into two cases.

            git config --get-regex applies the regex to match keys, and returns all
            matches (including substring matches).

            Thus, a config call:

            git config -f .gitmodules --get-regexp "^submodule\.([^ ]+)\.url"

            will report two lines of output if the submodule URL includes ".url":

            submodule.modules/JENKINS-46504.url.path modules/JENKINS-46504.url
            submodule.modules/JENKINS-46504.url.url https://github.com/MarkEWaite/JENKINS-46054.url

            The code originally used the same pattern for get-regexp and for output parsing.
            By using the same pattern in both places, it incorrectly took the first line
            of output as the URL of a submodule (when it is instead the path of a submodule).

            Fixes tests added in previous commits.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/52f681e6153eb088495edd497a82d6dc919dcae3 Log: [Fixed JENKINS-46054] submodule repo URL with '.url' substring failed Modify the submodule config parsing regular expression to correctly extract the submodule name from the config output. Splits cli submodule URL regexp use into two cases. git config --get-regex applies the regex to match keys, and returns all matches (including substring matches). Thus, a config call: git config -f .gitmodules --get-regexp "^submodule\.( [^ ] +)\.url" will report two lines of output if the submodule URL includes ".url": submodule.modules/ JENKINS-46504 .url.path modules/ JENKINS-46504 .url submodule.modules/ JENKINS-46504 .url.url https://github.com/MarkEWaite/JENKINS-46054.url The code originally used the same pattern for get-regexp and for output parsing. By using the same pattern in both places, it incorrectly took the first line of output as the URL of a submodule (when it is instead the path of a submodule). Fixes tests added in previous commits.
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java
            http://jenkins-ci.org/commit/git-client-plugin/b8bad8ab164e146e24eb38bc3a0715374eb25587
            Log:
            Merge pull request #284 from MarkEWaite/fix-submodule-url-regex

            Fix submodule url regex JENKINS-46054

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/c2b1ff48f3a3...b8bad8ab164e

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java http://jenkins-ci.org/commit/git-client-plugin/b8bad8ab164e146e24eb38bc3a0715374eb25587 Log: Merge pull request #284 from MarkEWaite/fix-submodule-url-regex Fix submodule url regex JENKINS-46054 Compare: https://github.com/jenkinsci/git-client-plugin/compare/c2b1ff48f3a3...b8bad8ab164e
            Hide
            Permalink
            markewaite Mark Waite added a comment - - edited

            Fixed in git client plugin 2.7.0

            Show
            markewaite Mark Waite added a comment - - edited Fixed in git client plugin 2.7.0
            markewaite Mark Waite made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            markewaite Mark Waite made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            Hide
            Permalink
            markewaite Mark Waite added a comment -

            See pull request 296 for changes to fix a regression reported as JENKINS-48818.

            Show
            markewaite Mark Waite added a comment - See pull request 296 for changes to fix a regression reported as JENKINS-48818 .
            markewaite Mark Waite made changes -
            Link This issue is related to JENKINS-48818 [ JENKINS-48818 ]
            markewaite Mark Waite made changes -
            Link This issue is related to JENKINS-48997 [ JENKINS-48997 ]
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java
            http://jenkins-ci.org/commit/git-client-plugin/69297552b3c36e084bea336448c6ad82763b303a
            Log:
            Fix both JENKINS-48818 and JENKINS-46054

            Add JENKINS-48818 test - space character in submodule remote name

            Submodule remote name which contains a space character was supported by
            git client plugin versions prior to 2.7.0. The bug fix for JENKINS-46054
            (support submodule update when the remote name or the remote URL end with
            '.url') broke support for submodule remote names which contain a space.

            This includes a test to show that a remote name containing a space
            character is not correctly handled by the 2.7.0 change.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java http://jenkins-ci.org/commit/git-client-plugin/69297552b3c36e084bea336448c6ad82763b303a Log: Fix both JENKINS-48818 and JENKINS-46054 Add JENKINS-48818 test - space character in submodule remote name Submodule remote name which contains a space character was supported by git client plugin versions prior to 2.7.0. The bug fix for JENKINS-46054 (support submodule update when the remote name or the remote URL end with '.url') broke support for submodule remote names which contain a space. This includes a test to show that a remote name containing a space character is not correctly handled by the 2.7.0 change.
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            http://jenkins-ci.org/commit/git-client-plugin/88fa4fe6599aa71da5e1c5a7b6cb0c96160a88f5
            Log:
            Don't ignore the GitClientTest integration test of JENKINS-46054

            The integration test is an end to end verification of the bug,
            while the unit tests check more details of the specific parsing
            problem that was the root of the bug.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/88fa4fe6599aa71da5e1c5a7b6cb0c96160a88f5 Log: Don't ignore the GitClientTest integration test of JENKINS-46054 The integration test is an end to end verification of the bug, while the unit tests check more details of the specific parsing problem that was the root of the bug.
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java
            http://jenkins-ci.org/commit/git-client-plugin/2dcbb2d62a980468d339e52f4f8dd6653473d5cd
            Log:
            Merge pull request #296 from MarkEWaite/explore-JENKINS-46054

            Fix JENKINS-48818 and keep JENKINS-46054 fixed

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/21914aaf97f0...2dcbb2d62a98

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/SubmodulePatternStringTest.java http://jenkins-ci.org/commit/git-client-plugin/2dcbb2d62a980468d339e52f4f8dd6653473d5cd Log: Merge pull request #296 from MarkEWaite/explore- JENKINS-46054 Fix JENKINS-48818 and keep JENKINS-46054 fixed Compare: https://github.com/jenkinsci/git-client-plugin/compare/21914aaf97f0...2dcbb2d62a98

              People

              Assignee:
              markewaite Mark Waite
              Reporter:
              gfranklin Greg Franklin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: