Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8915

huge performance drop in post build actions

    XMLWordPrintable

Details

    Description

      we have jenkins v 1.398 with analysis core 1.17, findbugs4.21, checkstyle3.13, pmd3.13, tasks4.14 and warnings3.14.
      lately we are noticing huge performance problems on our Jenkins instance after the build steps are finished, when jenkins is publishing the results. This becomes very noticeable on the projects having a lot of warnings (i.e. thousnads or tens of thousands) but mostly when two or more projects are building.
      I.e. i have a project that when built alone takes 2 min to execute all build steps and then one minute to publish all the results - this is acceptable and works fine IF there is no other projects building. The project is quite big and has a lot warnings (about 10000)
      But if I'm running two project like that together the build time expands to 6-10 min, but post build actions are taking up to 20 min or more. Some of the builds there were usually taking 8 min total could get stuck for 1hr 30min!

      We have jenkins running on a virtual host having 4 cores and 8gb RAM - it has 3 executors and no slaves.

      When I'm looking at the logs i see that our build system is finishing in acceptable times - ant reports build finished then it gets stuck on pretty much eery step publiching checktyle, pmd, warnings, then javadoc.

      At some point the log files gets huge in just few days (1.6Gb) over the weekend.

      Anybody experienced this?

      Attachments

        Issue Links

          Activity

            eguess74 eguess74 created issue -
            eguess74 eguess74 added a comment -

            In addition to that at the moments of slownesses i see that the log is full of the following error messages about severe error of RecordReaper javax.jmdns.impl.tasks.RecorReaper run

            Error while reaping records
            java.lang.IllegalArgumentException: The instance name component of fully qualified service cannot be empty
            at javax.jmdns.impl.ServiceInfoImpl.CheckQualifiednameMap
            at javax.jmdns.impl.ServiceInfoImpl.<init>
            at javax.jmdns.impl.DNSRecord$Pointer.getServiceInfo
            at javax.jmdns.impl.DNSRecord.getServiceInfo

            any ideas?

            Thanks!

            eguess74 eguess74 added a comment - In addition to that at the moments of slownesses i see that the log is full of the following error messages about severe error of RecordReaper javax.jmdns.impl.tasks.RecorReaper run Error while reaping records java.lang.IllegalArgumentException: The instance name component of fully qualified service cannot be empty at javax.jmdns.impl.ServiceInfoImpl.CheckQualifiednameMap at javax.jmdns.impl.ServiceInfoImpl.<init> at javax.jmdns.impl.DNSRecord$Pointer.getServiceInfo at javax.jmdns.impl.DNSRecord.getServiceInfo any ideas? Thanks!
            eguess74 eguess74 added a comment -

            found this issue JENKINS-8647 that seems to be resolved in 1.399
            Will see if the fix will improve performance.

            eguess74 eguess74 added a comment - found this issue JENKINS-8647 that seems to be resolved in 1.399 Will see if the fix will improve performance.
            markewaite Mark Waite added a comment -

            I would not expect JENKINS-8647 to resolve this issue. You might refer to http://jenkins.361315.n4.nabble.com/Has-warnings-plugin-become-CPU-intensive-td963935.html for more information. There have been other hints as well that the warnings processing may be slow when there are many warnings.

            markewaite Mark Waite added a comment - I would not expect JENKINS-8647 to resolve this issue. You might refer to http://jenkins.361315.n4.nabble.com/Has-warnings-plugin-become-CPU-intensive-td963935.html for more information. There have been other hints as well that the warnings processing may be slow when there are many warnings.
            markewaite Mark Waite added a comment - - edited

            JENKINS-5185 is a resolved bug which discusses warning parse time as well.

            markewaite Mark Waite added a comment - - edited JENKINS-5185 is a resolved bug which discusses warning parse time as well.
            drulli Ulli Hafner added a comment -

            Ok. Please let me know if the problem still exists with 1.399.

            drulli Ulli Hafner added a comment - Ok. Please let me know if the problem still exists with 1.399.
            eguess74 eguess74 added a comment - - edited

            well, it is probably not a good test but i have deployed the 1.399 and also removed warnings plugin from the server.
            The project that was at times taking 40 min to complete now took 4min 50 sec
            Another project that at some point took 1hr 30 min to complete now takes 9 min
            I'm taking it...

            OTOH
            as warnings plugin is scanning for many different types of warnings shouldn't this list be configurable on per server basis?? I.e. having about 450 jobs where 99,99% are java ant builds i would like to be able to switch off all unnecessary scanning targets once and for all... Makes sense?
            I.e. general configuration should provide a possibility to select the subset of warnings your interested in for the whole server and then you can filter them out even more (from the given subset) for each particular job if needed?

            eguess74 eguess74 added a comment - - edited well, it is probably not a good test but i have deployed the 1.399 and also removed warnings plugin from the server. The project that was at times taking 40 min to complete now took 4min 50 sec Another project that at some point took 1hr 30 min to complete now takes 9 min I'm taking it... OTOH as warnings plugin is scanning for many different types of warnings shouldn't this list be configurable on per server basis?? I.e. having about 450 jobs where 99,99% are java ant builds i would like to be able to switch off all unnecessary scanning targets once and for all... Makes sense? I.e. general configuration should provide a possibility to select the subset of warnings your interested in for the whole server and then you can filter them out even more (from the given subset) for each particular job if needed?
            drulli Ulli Hafner added a comment - - edited

            well, it is probably not a good test but i have deployed the 1.399 and also removed warnings plugin from the server.
            The project that was at times taking 40 min to complete now took 4min 50 sec
            Another project that at some point took 1hr 30 min to complete now takes 9 min
            I'm taking it...

            Did you try with warnings plug-in and 1.399?

            as warnings plugin is scanning for many different types of warnings shouldn't this list be configurable on per server basis?? I.e. having about 450 jobs where 99,99% are java ant builds i would like to be able to switch off all unnecessary scanning targets once and for all... Makes sense?
            I.e. general configuration should provide a possibility to select the subset of warnings your interested in for the whole server and then you can filter them out even more (from the given subset) for each particular job if needed?

            You can configure the parser in each job. You can also define filters based on regualar expressions.

            drulli Ulli Hafner added a comment - - edited well, it is probably not a good test but i have deployed the 1.399 and also removed warnings plugin from the server. The project that was at times taking 40 min to complete now took 4min 50 sec Another project that at some point took 1hr 30 min to complete now takes 9 min I'm taking it... Did you try with warnings plug-in and 1.399? as warnings plugin is scanning for many different types of warnings shouldn't this list be configurable on per server basis?? I.e. having about 450 jobs where 99,99% are java ant builds i would like to be able to switch off all unnecessary scanning targets once and for all... Makes sense? I.e. general configuration should provide a possibility to select the subset of warnings your interested in for the whole server and then you can filter them out even more (from the given subset) for each particular job if needed? You can configure the parser in each job. You can also define filters based on regualar expressions.
            eguess74 eguess74 added a comment -

            I have a monitoring plugin installed and here are my facts:

            the CPU usage mean on a scale 0-100%
            with warnings plugin - averaging 12% (this state causes slownesses and GUI is non-responsive at times)
            When we got the build with RecordReaper errors - mean average 25% (pretty much unusable most of the time - very slow)
            Installed 1.399, removed warnings plugin - mean average 2.5% (pretty snappy, very good)

            YMMV

            Thanks!

            eguess74 eguess74 added a comment - I have a monitoring plugin installed and here are my facts: the CPU usage mean on a scale 0-100% with warnings plugin - averaging 12% (this state causes slownesses and GUI is non-responsive at times) When we got the build with RecordReaper errors - mean average 25% (pretty much unusable most of the time - very slow) Installed 1.399, removed warnings plugin - mean average 2.5% (pretty snappy, very good) YMMV Thanks!
            drulli Ulli Hafner added a comment -

            And which parsers are you using? How big is the log file? How much warnings are found?

            drulli Ulli Hafner added a comment - And which parsers are you using? How big is the log file? How much warnings are found?
            drulli Ulli Hafner made changes -
            Field Original Value New Value
            Link This issue duplicates JENKINS-9090 [ JENKINS-9090 ]
            domcc Dom c added a comment -

            This problem is also plaguingme at the moment. My scenario is:
            1. MSBuild that has zero warnings
            2. Scan for compiler warnings checked
            3. Scan console log checked
            4. File pattern, empty

            When running the build i see:
            1:51:53 [WARNINGS] Parsing warnings in console log...
            11:51:53 [WARNINGS] Java Compiler : Found 0 warnings.
            11:51:53 [WARNINGS] Java Compiler : Found 0 warnings.
            11:51:53 [WARNINGS] JavaDoc : Found 0 warnings.
            11:51:53 [WARNINGS] Eclipse Java Compiler : Found 0 warnings.
            11:51:53 [WARNINGS] MSBuild : Found 0 warnings.
            11:51:53 [WARNINGS] GNU compiler (gcc) : Found 0 warnings.
            11:51:53 [WARNINGS] GNU compiler 4 (gcc) : Found 0 warnings.
            11:51:53 [WARNINGS] GNU compiler 4 (ld) : Found 0 warnings.
            11:51:53 [WARNINGS] Oracle Invalids : Found 0 warnings.
            11:51:53 [WARNINGS] SUN C++ Compiler : Found 0 warnings.
            11:51:53 [WARNINGS] Ada Compiler (gnat) : Found 0 warnings.
            11:51:54 [WARNINGS] Erlang Compiler : Found 0 warnings.
            11:51:54 [WARNINGS] Intel compiler : Found 0 warnings.
            11:51:54 [WARNINGS] IAR compiler (C/C++) : Found 0 warnings.
            11:51:54 [WARNINGS] PC-Lint : Found 0 warnings.
            11:51:54 [WARNINGS] Buckminster Compiler : Found 0 warnings.
            11:51:54 [WARNINGS] Texas Instruments Code Composer Studio (C/C++) : Found 0 warnings.
            11:51:54 [WARNINGS] AcuCobol Compiler : Found 0 warnings.
            11:51:54 [WARNINGS] Flex SDK Compilers (compc & mxmlc) : Found 0 warnings.
            11:51:54 [WARNINGS] PHP Runtime Warning : Found 0 warnings.
            11:51:54 [WARNINGS] Coolflux DSP Compiler : Found 0 warnings.
            11:51:54 [WARNINGS] Perforce Compiler : Found 0 warnings.
            11:51:54 [WARNINGS] Robocopy (please use /V in your commands!) : Found 0 warnings.
            11:51:54 [WARNINGS] Doxygen : Found 0 warnings.
            11:51:54 [WARNINGS] TNSDL Translator : Found 0 warnings.
            11:51:54 [WARNINGS] GHS Multi Compiler : Found 0 warnings.
            11:55:31 Archiving artifacts

            So the last WARNING step is taking over 3 minutes. This is a long time as it is only a subset of my repo that i have been testing this problem on. I can see that my workspace is being recursivly scanned which kills the performance. I do not understand why the file system is being scanned when i only checked the "Scan console log" option and i have zero warnings?

            Unfortunatley i will have to disable the warning plugin because of this

            domcc Dom c added a comment - This problem is also plaguingme at the moment. My scenario is: 1. MSBuild that has zero warnings 2. Scan for compiler warnings checked 3. Scan console log checked 4. File pattern, empty When running the build i see: 1:51:53 [WARNINGS] Parsing warnings in console log... 11:51:53 [WARNINGS] Java Compiler : Found 0 warnings. 11:51:53 [WARNINGS] Java Compiler : Found 0 warnings. 11:51:53 [WARNINGS] JavaDoc : Found 0 warnings. 11:51:53 [WARNINGS] Eclipse Java Compiler : Found 0 warnings. 11:51:53 [WARNINGS] MSBuild : Found 0 warnings. 11:51:53 [WARNINGS] GNU compiler (gcc) : Found 0 warnings. 11:51:53 [WARNINGS] GNU compiler 4 (gcc) : Found 0 warnings. 11:51:53 [WARNINGS] GNU compiler 4 (ld) : Found 0 warnings. 11:51:53 [WARNINGS] Oracle Invalids : Found 0 warnings. 11:51:53 [WARNINGS] SUN C++ Compiler : Found 0 warnings. 11:51:53 [WARNINGS] Ada Compiler (gnat) : Found 0 warnings. 11:51:54 [WARNINGS] Erlang Compiler : Found 0 warnings. 11:51:54 [WARNINGS] Intel compiler : Found 0 warnings. 11:51:54 [WARNINGS] IAR compiler (C/C++) : Found 0 warnings. 11:51:54 [WARNINGS] PC-Lint : Found 0 warnings. 11:51:54 [WARNINGS] Buckminster Compiler : Found 0 warnings. 11:51:54 [WARNINGS] Texas Instruments Code Composer Studio (C/C++) : Found 0 warnings. 11:51:54 [WARNINGS] AcuCobol Compiler : Found 0 warnings. 11:51:54 [WARNINGS] Flex SDK Compilers (compc & mxmlc) : Found 0 warnings. 11:51:54 [WARNINGS] PHP Runtime Warning : Found 0 warnings. 11:51:54 [WARNINGS] Coolflux DSP Compiler : Found 0 warnings. 11:51:54 [WARNINGS] Perforce Compiler : Found 0 warnings. 11:51:54 [WARNINGS] Robocopy (please use /V in your commands!) : Found 0 warnings. 11:51:54 [WARNINGS] Doxygen : Found 0 warnings. 11:51:54 [WARNINGS] TNSDL Translator : Found 0 warnings. 11:51:54 [WARNINGS] GHS Multi Compiler : Found 0 warnings. 11:55:31 Archiving artifacts So the last WARNING step is taking over 3 minutes. This is a long time as it is only a subset of my repo that i have been testing this problem on. I can see that my workspace is being recursivly scanned which kills the performance. I do not understand why the file system is being scanned when i only checked the "Scan console log" option and i have zero warnings? Unfortunatley i will have to disable the warning plugin because of this
            drulli Ulli Hafner added a comment -

            Why did you enable all parsers in your build step? How many warnings did you actually have?

            drulli Ulli Hafner added a comment - Why did you enable all parsers in your build step? How many warnings did you actually have?
            domcc Dom c added a comment -

            All parsers are enabled for demonstration. As said previously there are zero warnings. I do not know why the workspace is scanned when there are zero warnings. Let me know if you need any further information.

            domcc Dom c added a comment - All parsers are enabled for demonstration. As said previously there are zero warnings. I do not know why the workspace is scanned when there are zero warnings. Let me know if you need any further information.
            drulli Ulli Hafner added a comment -

            See fix for Jenkins-9090.

            drulli Ulli Hafner added a comment - See fix for Jenkins-9090.
            drulli Ulli Hafner made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 139059 ] JNJira + In-Review [ 188316 ]

            People

              drulli Ulli Hafner
              eguess74 eguess74
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: