Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2635

Performance issue

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Fixed
    • warnings-plugin
    • None
    • Platform: PC, OS: Linux

    Description

      Some parsers are really slow for my system (4mb log file size). In total the
      warnings plugin needs 1:45h to process the log with all parsers. Since I have
      six different builds this will need more than 10h just for parsing the logs ...
      Some tests with the unit tests showed that the MsBuild, gcc and erlang parsers
      are really slow. Is there a way of to make the parsers used for a build
      configurable or to speed up these parsers (e.g. the gnat parser and others need
      less than a second to parse the log).
      If the desired way is to speed up the parsers I can do some testing with
      "better" regular expressions ...

      Attachments

        Activity

          Code changed in hudson
          User: : drulli
          Path:
          trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/ErlcParser.java
          trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/GccParser.java
          trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/GnatParser.java
          trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/MsBuildParser.java
          http://fisheye4.cenqua.com/changelog/hudson/?cs=13370
          Log:
          [FIXED JENKINS-2635] Speedup for several slow regexp parsers.

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in hudson User: : drulli Path: trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/ErlcParser.java trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/GccParser.java trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/GnatParser.java trunk/hudson/plugins/warnings/src/main/java/hudson/plugins/warnings/parser/MsBuildParser.java http://fisheye4.cenqua.com/changelog/hudson/?cs=13370 Log: [FIXED JENKINS-2635] Speedup for several slow regexp parsers.
          drulli Ulli Hafner added a comment -

          Thanks for adding the patches. I think in the long term I need to provide a
          parser selection. (See issue
          https://hudson.dev.java.net/issues/show_bug.cgi?id=2462).

          drulli Ulli Hafner added a comment - Thanks for adding the patches. I think in the long term I need to provide a parser selection. (See issue https://hudson.dev.java.net/issues/show_bug.cgi?id=2462 ).
          bergerbd bergerbd added a comment -

          The patch do not break any test case and speeds up the parsing (let's say 10s
          total time for parsing my log).
          The GnatParser is a little bit more severe now. (Less false positives in the
          feature since it checks for the file endings (ads/adb))

          Hope the patch helps.

          bergerbd bergerbd added a comment - The patch do not break any test case and speeds up the parsing (let's say 10s total time for parsing my log). The GnatParser is a little bit more severe now. (Less false positives in the feature since it checks for the file endings (ads/adb)) Hope the patch helps.
          bergerbd bergerbd added a comment -

          Created an attachment (id=444)
          Activated multiline for erlang, gcc and msbuild

          bergerbd bergerbd added a comment - Created an attachment (id=444) Activated multiline for erlang, gcc and msbuild
          drulli Ulli Hafner added a comment -

          I'll look at all these regexps if it is possible to switch to the mutli-line
          mode as suggested in your patch. I haven't yet tried to test the parsers with a
          large file...

          If that does not help, I'll add an option to enable/configure the parsers in the
          config screen.

          drulli Ulli Hafner added a comment - I'll look at all these regexps if it is possible to switch to the mutli-line mode as suggested in your patch. I haven't yet tried to test the parsers with a large file... If that does not help, I'll add an option to enable/configure the parsers in the config screen.
          bergerbd bergerbd added a comment -

          Created an attachment (id=443)
          Sample patch for GCC parser

          bergerbd bergerbd added a comment - Created an attachment (id=443) Sample patch for GCC parser
          bergerbd bergerbd added a comment -

          I have made some additional tests with the GCC parser. It seems as if the single
          line mode is the reason for the performance issue. It would be much faster if it
          would be possible to specify the ^ and $ for the line based matcher (which is
          not possible due to the special meaning of these symbols in the line mode). The
          only possible solution is to switch to the multiline mode and specify the whole
          line with ^ and $.

          (timed on a Intel Core Duo @ 1.2GHz each core)
          original gcc-parser:
          original testcase -> 0.004 sec
          my log -> killed after 60min

          patched gcc-parser:
          original test -> 0.006 sec
          my log -> 1.24 sec

          bergerbd bergerbd added a comment - I have made some additional tests with the GCC parser. It seems as if the single line mode is the reason for the performance issue. It would be much faster if it would be possible to specify the ^ and $ for the line based matcher (which is not possible due to the special meaning of these symbols in the line mode). The only possible solution is to switch to the multiline mode and specify the whole line with ^ and $. (timed on a Intel Core Duo @ 1.2GHz each core) original gcc-parser: original testcase -> 0.004 sec my log -> killed after 60min patched gcc-parser: original test -> 0.006 sec my log -> 1.24 sec

          People

            drulli Ulli Hafner
            bergerbd bergerbd
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: