Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-864

cvs log fails with a lot of parameters on Unix

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • cvs-plugin
    • None
    • Platform: All, OS: All

      We have been getting a lot of cvs log failures recently on
      deadlock.netbeans.org, I think caused by there being many updated files in one
      build.

      1. Hudson does not report any actual error output from CVS, just that it "failed
      with exit code 255". (Ant's ChangeLogTask seems to capture all output even if
      the command failed, which is perhaps a bug in Ant.) This is not very helpful,
      but I suspect that it is a max command length limitation.

      2. Currently Hudson limits the max number of arguments on Windows only,
      otherwise defaulting to logging the whole working dir (which for a big project
      is unacceptably slow, but at least it works). For Unix, there is no limit.
      Probably there should be some (I hope rather higher) limit.

      3. Another option is to check if the log task fails when using an explicit list
      of arguments, and if so, automatically retry by logging the whole working dir.
      Then we could discard the hardcoded max arg limit.

          [JENKINS-864] cvs log fails with a lot of parameters on Unix

          Doing an automatic retry is attractive but that would require us to know what
          error to look for — I don't think we want to retry for every error.

          Any pointer to the failing build and output from cvs?

          Kohsuke Kawaguchi added a comment - Doing an automatic retry is attractive but that would require us to know what error to look for — I don't think we want to retry for every error. Any pointer to the failing build and output from cvs?

          Jesse Glick added a comment -

          With approach #3, there are these possibilities:

          a. CVS command succeeds, fine.

          b. CVS command fails due to max arg length exceeded, quickly retried, succeeds
          more slowly.

          c. Fails due to max arg length, quickly retried, fails for some other reason,
          log says that other reason.

          d. Fails for some unrelated reason, retried w/o params, succeeds fortuitously, fine.

          e. Fails for some unrelated reason, retried w/o params, fails again (probably
          for the same reason such as connectivity problem).

          Case (e) is the only real drawback - if the failure is slow to occur, then the
          build will take a little longer to fail because CVS will be run twice. I think
          this is an acceptable trade-off given that the lack of case (b) is at least in
          my case causing builds to fail gratuitously every day and alarming and
          interrupting developers.

          Sample output:

          http://deadlock.netbeans.org/hudson/job/javadoc-nbms/1045/consoleText

          Cf. issue #389. Apparently 600 files is OK on Unix, but 2900 is not.

          Jesse Glick added a comment - With approach #3, there are these possibilities: a. CVS command succeeds, fine. b. CVS command fails due to max arg length exceeded, quickly retried, succeeds more slowly. c. Fails due to max arg length, quickly retried, fails for some other reason, log says that other reason. d. Fails for some unrelated reason, retried w/o params, succeeds fortuitously, fine. e. Fails for some unrelated reason, retried w/o params, fails again (probably for the same reason such as connectivity problem). Case (e) is the only real drawback - if the failure is slow to occur, then the build will take a little longer to fail because CVS will be run twice. I think this is an acceptable trade-off given that the lack of case (b) is at least in my case causing builds to fail gratuitously every day and alarming and interrupting developers. Sample output: http://deadlock.netbeans.org/hudson/job/javadoc-nbms/1045/consoleText Cf. issue #389. Apparently 600 files is OK on Unix, but 2900 is not.

          Jesse Glick added a comment -

          In particular, it seems that on Linux the command must fit into a 128K buffer
          (at least on a machine with 4K pages??). The environment and fixed part of argv
          seem to consume a few K, so basically the command will fail if the total number
          of logged file paths (counting one nullsep char each) exceeds about 130000. (On
          my machine, 129790 seems to be the max.)

          xargs works fine. Perhaps I can patch CVSSCM to limit length on Windows as now
          (I assume Win has no workaround), else use /usr/bin/xargs if it exists, else
          hope for the best.

          Jesse Glick added a comment - In particular, it seems that on Linux the command must fit into a 128K buffer (at least on a machine with 4K pages??). The environment and fixed part of argv seem to consume a few K, so basically the command will fail if the total number of logged file paths (counting one nullsep char each) exceeds about 130000. (On my machine, 129790 seems to be the max.) xargs works fine. Perhaps I can patch CVSSCM to limit length on Windows as now (I assume Win has no workaround), else use /usr/bin/xargs if it exists, else hope for the best.

          Jesse Glick added a comment -

          Browsing Solaris 9 man pages, it seems that xargs there (1) will truncate
          arguments to fit some max length (which seems to defeat the purpose but what do
          I know), (2) has no -0 option. (Mac OS X says -0 is accepted and POSIX does not
          specify -0.) It seems the lack of -0 can be worked around by separating params
          by newlines and preceding every char in every param with \ which I can confirm
          does work on Linux.

          Jesse Glick added a comment - Browsing Solaris 9 man pages, it seems that xargs there (1) will truncate arguments to fit some max length (which seems to defeat the purpose but what do I know), (2) has no -0 option. (Mac OS X says -0 is accepted and POSIX does not specify -0.) It seems the lack of -0 can be worked around by separating params by newlines and preceding every char in every param with \ which I can confirm does work on Linux.

          Jesse Glick added a comment -

          Unfortunately Hudson does not control how CVS is actually invoked, so I guess I
          cannot use the xargs workaround.

          Jesse Glick added a comment - Unfortunately Hudson does not control how CVS is actually invoked, so I guess I cannot use the xargs workaround.

          Jesse Glick added a comment -

          Created an attachment (id=96)
          Patch to use xargs

          Jesse Glick added a comment - Created an attachment (id=96) Patch to use xargs

          Jesse Glick added a comment -

          Well actually it is possible to affect how CVS is run (though messy).
          Unfortunately when I tried it, it does not work correctly. For reasons I still
          do not understand, xargs cvs fails on a large enough example:

          $ /usr/bin/xargs cvs < /tmp/input > /tmp/output
          Unknown command: `some-file'

          CVS commands are:
          add Add a new file/directory to the repository

          where some-file was one of the arguments in the input, #2838, about 250Kb in. It
          seems that xargs is actually splitting up the command into chunks.

          Jesse Glick added a comment - Well actually it is possible to affect how CVS is run (though messy). Unfortunately when I tried it, it does not work correctly. For reasons I still do not understand, xargs cvs fails on a large enough example: $ /usr/bin/xargs cvs < /tmp/input > /tmp/output Unknown command: `some-file' CVS commands are: add Add a new file/directory to the repository where some-file was one of the arguments in the input, #2838, about 250Kb in. It seems that xargs is actually splitting up the command into chunks.

          Jesse Glick added a comment -

          Seems to work so long as CVS arguments are separated from actual filenames.

          Checking in AbstractCvsTask.java;
          /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/AbstractCvsTask.java,v
          <-- AbstractCvsTask.java
          new revision: 1.4; previous revision: 1.3
          done
          Checking in cvslib/RedirectingStreamHandler.java;
          /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/RedirectingStreamHandler.java,v
          <-- RedirectingStreamHandler.java
          new revision: 1.3; previous revision: 1.2
          done
          Checking in cvslib/ChangeLogTask.java;
          /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/ChangeLogTask.java,v
          <-- ChangeLogTask.java
          new revision: 1.13; previous revision: 1.12
          done

          Jesse Glick added a comment - Seems to work so long as CVS arguments are separated from actual filenames. Checking in AbstractCvsTask.java; /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/AbstractCvsTask.java,v <-- AbstractCvsTask.java new revision: 1.4; previous revision: 1.3 done Checking in cvslib/RedirectingStreamHandler.java; /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/RedirectingStreamHandler.java,v <-- RedirectingStreamHandler.java new revision: 1.3; previous revision: 1.2 done Checking in cvslib/ChangeLogTask.java; /shared/data/ccvs/repository/hudson/hudson/main/core/src/main/java/hudson/org/apache/tools/ant/taskdefs/cvslib/ChangeLogTask.java,v <-- ChangeLogTask.java new revision: 1.13; previous revision: 1.12 done

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: