Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26545

Persist build(s) log(s) in MySQL (or similar)

    XMLWordPrintable

Details

    Description

      Persisting build(s) log(s) in MySQL (or similar) will make Jenkins updates much easier to manage.

      In fact, when build logs change their format we sometimes end up with a broken system which it is often unable to read logs of old builds. Software changes even break stored builds info from time to time

      Attachments

        Issue Links

          Activity

            danielbeck Daniel Beck added a comment -

            for more than 200 maven modules per single build run per job.

            Don't use the Maven job type. It has some well known, severe performance problems. Problem solved.

            danielbeck Daniel Beck added a comment - for more than 200 maven modules per single build run per job. Don't use the Maven job type. It has some well known, severe performance problems. Problem solved.

            Thanks for the info danielbeck, is this documented somewhere?

            dserodio Daniel Serodio added a comment - Thanks for the info danielbeck , is this documented somewhere?
            danielbeck Daniel Beck added a comment -

            More learned through experience by a number of community members. I doubt it's explicitly documented somewhere. From my limited experience with Maven I'd also consider 200 modules in a single build to be rather unusual, so the performance issues you experience likely aren't experienced by most users. When loading a single build involves parsing hundreds of XML files (compared to one in the case of freestyle), performance problems are to be expected.

            Note that there are other issues with the job type, which were written up by stephenconnolly, a contributor to Jenkins and Maven, here:
            http://javaadventure.blogspot.de/2013/11/jenkins-maven-job-type-considered-evil.html

            danielbeck Daniel Beck added a comment - More learned through experience by a number of community members. I doubt it's explicitly documented somewhere. From my limited experience with Maven I'd also consider 200 modules in a single build to be rather unusual, so the performance issues you experience likely aren't experienced by most users. When loading a single build involves parsing hundreds of XML files (compared to one in the case of freestyle), performance problems are to be expected. Note that there are other issues with the job type, which were written up by stephenconnolly , a contributor to Jenkins and Maven, here: http://javaadventure.blogspot.de/2013/11/jenkins-maven-job-type-considered-evil.html

            Part of the issue in start-up performance for the evil job type is the mutability of build results issue...

            You can see this in the other side-effect issues: JENKINS-20731, JENKINS-25075

            Basically, back from when this job type was truly completely and utterly evil, it would run each module as a separate "job" in parallel and basically re-implement the whole Maven Reactor with a bunch of hacks... You can still enable this mode of using the evil job type, but it is at least no longer the default.

            So what you have is that you can re-run a single module only... and the build result for an evil job is the aggregate build result of the most recent build of all the modules...

            Thus if you get a foo-project build #5 broken due to a flakey test in foo-manchu-module, you just re-trigger the failing module and presto! your foo-project build #5 is now overall fixed because the foo-manchu-module build #6 was a success (note that the original build of foo-project #5 resulted in foo-manchu-module build #5)

            So when Jenkins loads an evil job build record, not only does it have to parse all latest build child module descriptors, but it has to re-evaluate all the latest build results of all modules in the current build...

            And then the weather column wants to evaluate the build stability... so it goes asking for the build result of the previous 4 builds...

            So actually I would contend (and teilo would agree) that the root cause of the performance issues is actually the mutability of the build result in the evil job type... fix that and a lot of the evil job's performance issues can be resolved... there are other performance issues with the evil job... and it will still be evil

            (FYI: I call it evil from my PoV as a committer to the Apache Maven project because it is a job type that expressly goes completely against the core principles of Maven... one of which being that the build can be completely reproducible using an identical environment variables, user permissions pom and command-line... the evil one doesn't do this because it modifies the effective pom in ways you cannot see or deterministically determine without injecting some additional inspection code into your Maven binaries)

            stephenconnolly Stephen Connolly added a comment - Part of the issue in start-up performance for the evil job type is the mutability of build results issue... You can see this in the other side-effect issues: JENKINS-20731 , JENKINS-25075 Basically, back from when this job type was truly completely and utterly evil, it would run each module as a separate "job" in parallel and basically re-implement the whole Maven Reactor with a bunch of hacks... You can still enable this mode of using the evil job type, but it is at least no longer the default. So what you have is that you can re-run a single module only... and the build result for an evil job is the aggregate build result of the most recent build of all the modules... Thus if you get a foo-project build #5 broken due to a flakey test in foo-manchu-module, you just re-trigger the failing module and presto! your foo-project build #5 is now overall fixed because the foo-manchu-module build #6 was a success (note that the original build of foo-project #5 resulted in foo-manchu-module build #5) So when Jenkins loads an evil job build record, not only does it have to parse all latest build child module descriptors, but it has to re-evaluate all the latest build results of all modules in the current build... And then the weather column wants to evaluate the build stability... so it goes asking for the build result of the previous 4 builds... So actually I would contend (and teilo would agree) that the root cause of the performance issues is actually the mutability of the build result in the evil job type... fix that and a lot of the evil job's performance issues can be resolved... there are other performance issues with the evil job... and it will still be evil (FYI: I call it evil from my PoV as a committer to the Apache Maven project because it is a job type that expressly goes completely against the core principles of Maven... one of which being that the build can be completely reproducible using an identical environment variables, user permissions pom and command-line... the evil one doesn't do this because it modifies the effective pom in ways you cannot see or deterministically determine without injecting some additional inspection code into your Maven binaries)
            oleg_nenashev Oleg Nenashev added a comment -

            FTR there was some design work and prototyping in JENKINS-38313

            oleg_nenashev Oleg Nenashev added a comment - FTR there was some design work and prototyping in JENKINS-38313

            People

              Unassigned Unassigned
              czerny Libero Scarcelli
              Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: