Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-864

cvs log fails with a lot of parameters on Unix

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: cvs-plugin
    • Labels:
      None
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      We have been getting a lot of cvs log failures recently on
      deadlock.netbeans.org, I think caused by there being many updated files in one
      build.

      1. Hudson does not report any actual error output from CVS, just that it "failed
      with exit code 255". (Ant's ChangeLogTask seems to capture all output even if
      the command failed, which is perhaps a bug in Ant.) This is not very helpful,
      but I suspect that it is a max command length limitation.

      2. Currently Hudson limits the max number of arguments on Windows only,
      otherwise defaulting to logging the whole working dir (which for a big project
      is unacceptably slow, but at least it works). For Unix, there is no limit.
      Probably there should be some (I hope rather higher) limit.

      3. Another option is to check if the log task fails when using an explicit list
      of arguments, and if so, automatically retry by logging the whole working dir.
      Then we could discard the hardcoded max arg limit.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Seems to work so long as CVS arguments are separated from actual filenames.

            Checking in AbstractCvsTask.java;
            /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/AbstractCvsTask.java,v
            <-- AbstractCvsTask.java
            new revision: 1.4; previous revision: 1.3
            done
            Checking in cvslib/RedirectingStreamHandler.java;
            /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/RedirectingStreamHandler.java,v
            <-- RedirectingStreamHandler.java
            new revision: 1.3; previous revision: 1.2
            done
            Checking in cvslib/ChangeLogTask.java;
            /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/ChangeLogTask.java,v
            <-- ChangeLogTask.java
            new revision: 1.13; previous revision: 1.12
            done

            Show
            jglick Jesse Glick added a comment - Seems to work so long as CVS arguments are separated from actual filenames. Checking in AbstractCvsTask.java; /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/AbstractCvsTask.java,v <-- AbstractCvsTask.java new revision: 1.4; previous revision: 1.3 done Checking in cvslib/RedirectingStreamHandler.java; /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/RedirectingStreamHandler.java,v <-- RedirectingStreamHandler.java new revision: 1.3; previous revision: 1.2 done Checking in cvslib/ChangeLogTask.java; /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/ChangeLogTask.java,v <-- ChangeLogTask.java new revision: 1.13; previous revision: 1.12 done
            Hide
            jglick Jesse Glick added a comment -

            Well actually it is possible to affect how CVS is run (though messy).
            Unfortunately when I tried it, it does not work correctly. For reasons I still
            do not understand, xargs cvs fails on a large enough example:

            $ /usr/bin/xargs cvs < /tmp/input > /tmp/output
            Unknown command: `some-file'

            CVS commands are:
            add Add a new file/directory to the repository

            where some-file was one of the arguments in the input, #2838, about 250Kb in. It
            seems that xargs is actually splitting up the command into chunks.

            Show
            jglick Jesse Glick added a comment - Well actually it is possible to affect how CVS is run (though messy). Unfortunately when I tried it, it does not work correctly. For reasons I still do not understand, xargs cvs fails on a large enough example: $ /usr/bin/xargs cvs < /tmp/input > /tmp/output Unknown command: `some-file' CVS commands are: add Add a new file/directory to the repository where some-file was one of the arguments in the input, #2838, about 250Kb in. It seems that xargs is actually splitting up the command into chunks.
            Hide
            jglick Jesse Glick added a comment -

            Created an attachment (id=96)
            Patch to use xargs

            Show
            jglick Jesse Glick added a comment - Created an attachment (id=96) Patch to use xargs
            Hide
            jglick Jesse Glick added a comment -

            Unfortunately Hudson does not control how CVS is actually invoked, so I guess I
            cannot use the xargs workaround.

            Show
            jglick Jesse Glick added a comment - Unfortunately Hudson does not control how CVS is actually invoked, so I guess I cannot use the xargs workaround.
            Hide
            jglick Jesse Glick added a comment -

            Browsing Solaris 9 man pages, it seems that xargs there (1) will truncate
            arguments to fit some max length (which seems to defeat the purpose but what do
            I know), (2) has no -0 option. (Mac OS X says -0 is accepted and POSIX does not
            specify -0.) It seems the lack of -0 can be worked around by separating params
            by newlines and preceding every char in every param with \ which I can confirm
            does work on Linux.

            Show
            jglick Jesse Glick added a comment - Browsing Solaris 9 man pages, it seems that xargs there (1) will truncate arguments to fit some max length (which seems to defeat the purpose but what do I know), (2) has no -0 option. (Mac OS X says -0 is accepted and POSIX does not specify -0.) It seems the lack of -0 can be worked around by separating params by newlines and preceding every char in every param with \ which I can confirm does work on Linux.
            Hide
            jglick Jesse Glick added a comment -

            In particular, it seems that on Linux the command must fit into a 128K buffer
            (at least on a machine with 4K pages??). The environment and fixed part of argv
            seem to consume a few K, so basically the command will fail if the total number
            of logged file paths (counting one nullsep char each) exceeds about 130000. (On
            my machine, 129790 seems to be the max.)

            xargs works fine. Perhaps I can patch CVSSCM to limit length on Windows as now
            (I assume Win has no workaround), else use /usr/bin/xargs if it exists, else
            hope for the best.

            Show
            jglick Jesse Glick added a comment - In particular, it seems that on Linux the command must fit into a 128K buffer (at least on a machine with 4K pages??). The environment and fixed part of argv seem to consume a few K, so basically the command will fail if the total number of logged file paths (counting one nullsep char each) exceeds about 130000. (On my machine, 129790 seems to be the max.) xargs works fine. Perhaps I can patch CVSSCM to limit length on Windows as now (I assume Win has no workaround), else use /usr/bin/xargs if it exists, else hope for the best.
            Hide
            jglick Jesse Glick added a comment -

            With approach #3, there are these possibilities:

            a. CVS command succeeds, fine.

            b. CVS command fails due to max arg length exceeded, quickly retried, succeeds
            more slowly.

            c. Fails due to max arg length, quickly retried, fails for some other reason,
            log says that other reason.

            d. Fails for some unrelated reason, retried w/o params, succeeds fortuitously, fine.

            e. Fails for some unrelated reason, retried w/o params, fails again (probably
            for the same reason such as connectivity problem).

            Case (e) is the only real drawback - if the failure is slow to occur, then the
            build will take a little longer to fail because CVS will be run twice. I think
            this is an acceptable trade-off given that the lack of case (b) is at least in
            my case causing builds to fail gratuitously every day and alarming and
            interrupting developers.

            Sample output:

            http://deadlock.netbeans.org/hudson/job/javadoc-nbms/1045/consoleText

            Cf. issue #389. Apparently 600 files is OK on Unix, but 2900 is not.

            Show
            jglick Jesse Glick added a comment - With approach #3, there are these possibilities: a. CVS command succeeds, fine. b. CVS command fails due to max arg length exceeded, quickly retried, succeeds more slowly. c. Fails due to max arg length, quickly retried, fails for some other reason, log says that other reason. d. Fails for some unrelated reason, retried w/o params, succeeds fortuitously, fine. e. Fails for some unrelated reason, retried w/o params, fails again (probably for the same reason such as connectivity problem). Case (e) is the only real drawback - if the failure is slow to occur, then the build will take a little longer to fail because CVS will be run twice. I think this is an acceptable trade-off given that the lack of case (b) is at least in my case causing builds to fail gratuitously every day and alarming and interrupting developers. Sample output: http://deadlock.netbeans.org/hudson/job/javadoc-nbms/1045/consoleText Cf. issue #389. Apparently 600 files is OK on Unix, but 2900 is not.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            Doing an automatic retry is attractive but that would require us to know what
            error to look for — I don't think we want to retry for every error.

            Any pointer to the failing build and output from cvs?

            Show
            kohsuke Kohsuke Kawaguchi added a comment - Doing an automatic retry is attractive but that would require us to know what error to look for — I don't think we want to retry for every error. Any pointer to the failing build and output from cvs?

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jglick Jesse Glick
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: