Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38313

External Build Log storage for Jenkins

    XMLWordPrintable

    Details

    • Epic Name:
      External Build Log storage for Jenkins
    • Similar Issues:

      Description

      Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

      It would be great to have such solution. Top-level requirements:

      • All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
      • Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
      • External Build Log Storage has an extensible and pluggable architecture
      • At least 2 reference implementations:
        • File-system-based storage (current implementation)
        • One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
      • Jenkins Pipeline is supported out of the box
      • Old logs should be browseable after changing the storage engine (Migration or parallel engines)

      Design document:

        Attachments

          Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-49327 Please show TOP of build log in console output - even when trimming

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is blocked by

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. JENKINS-38314 External Build Log storage for Jenkins. Prototype #1

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-32444 Proposal: Move build logs off of Jenkins master

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-45150 large console logging can take Jenkins down or affecting performance

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-26545 Persist build(s) log(s) in MySQL (or similar)

          • Critical - Crashes, loss of data, severe memory leak.
          • Reopened
          relates to

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-51229 Log Discarder to delete log files generated by Jenkins pipeline scripts, but keep build information

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-50291 Customize where AsyncPeriodicWork are logged

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link Core API: https://github.com/jenkinsci/jenkins/pull/3557/

          Web Link Design Document draft

          Web Link PoC Demo Slides

            Activity

            Ascending order - Click to sort in descending order
            oleg_nenashev Oleg Nenashev created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Epic Child JENKINS-38381 [ 174556 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is blocked by JENKINS-38314 [ JENKINS-38314 ]
            oleg_nenashev Oleg Nenashev made changes -
            Description Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution             		Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution.
            oleg_nenashev Oleg Nenashev made changes -
            Description Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution.


            		 Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution. Top-level requirements:

            * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
            * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
            * External Build Log Storage has an extensible and pluggable architecture
            * At least 2 reference implementations:
             ** File-system-based storage (current implementation)
             ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
            * Jenkins Pipeline is supported out of the box
            * Old logs should be browseable after changing the storage engine (Migration or parallel engines)
            oleg_nenashev Oleg Nenashev made changes -
            Description Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution. Top-level requirements:

            * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
            * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
            * External Build Log Storage has an extensible and pluggable architecture
            * At least 2 reference implementations:
             ** File-system-based storage (current implementation)
             ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
            * Jenkins Pipeline is supported out of the box
            * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

            		 Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

            It would be great to have such solution. Top-level requirements:

            * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
            * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
            * External Build Log Storage has an extensible and pluggable architecture
            * At least 2 reference implementations:
             ** File-system-based storage (current implementation)
             ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
            * Jenkins Pipeline is supported out of the box
            * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

            Design document:
            * https://docs.google.com/document/d/1_bquSeA_lC7zJhQoWhxlSKJAg6b8duNbyQ15zCz-e4Y/edit#
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "Design Document draft (Web Link)" [ 15820 ]
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "PoC Demo Slides (Web Link)" [ 15821 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-26545 [ JENKINS-26545 ]
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-45150 [ JENKINS-45150 ]
            jamesdumay James Dumay made changes -
            Component/s pipeline [ 21692 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels pipeline
            oleg_nenashev Oleg Nenashev made changes -
            Component/s pipeline [ 21692 ]
            svanoort Sam Van Oort made changes -
            Epic Child JENKINS-38381 [ 174556 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-38381 [ JENKINS-38381 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-32444 [ JENKINS-32444 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue blocks JENKINS-49327 [ JENKINS-49327 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-50291 [ JENKINS-50291 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-51778 [ 191226 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52691 [ 192576 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52692 [ 192577 ]
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "Core API: https://github.com/jenkinsci/jenkins/pull/3557/ (Web Link)" [ 21216 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52693 [ 192578 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52695 [ 192580 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52696 [ 192581 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52697 [ 192583 ]
            jglick Jesse Glick made changes -
            Epic Child JENKINS-52729 [ 192620 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-51229 [ JENKINS-51229 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52867 [ 192803 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52873 [ 192810 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52894 [ 192847 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52914 [ 192873 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52696 [ 192581 ]
            oleg_nenashev Oleg Nenashev made changes -
            Epic Child JENKINS-52697 [ 192583 ]
            ctran Cuong Tran made changes -
            Rank Ranked higher
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              oleg_nenashev Oleg Nenashev
              Votes:
              15 Vote for this issue
              Watchers:
              42 Start watching this issue

                Dates

                Created:
                Updated: