• External Build Log storage for Jenkins

      Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

      It would be great to have such solution. Top-level requirements:

      • All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
      • Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
      • External Build Log Storage has an extensible and pluggable architecture
      • At least 2 reference implementations:
        • File-system-based storage (current implementation)
        • One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
      • Jenkins Pipeline is supported out of the box
      • Old logs should be browseable after changing the storage engine (Migration or parallel engines)

      Design document:

          [JENKINS-38313] External Build Log storage for Jenkins

          Oleg Nenashev created issue -
          Jesse Glick made changes -
          Epic Child New: JENKINS-38381 [ 174556 ]
          Oleg Nenashev made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Oleg Nenashev made changes -
          Link New: This issue is blocked by JENKINS-38314 [ JENKINS-38314 ]
          Oleg Nenashev made changes -
          Description Original: Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

          It would be great to have such solution
          New: Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

          It would be great to have such solution.


          Oleg Nenashev made changes -
          Description Original: Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

          It would be great to have such solution.


          New: Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

          It would be great to have such solution. Top-level requirements:

          * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
          * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
          * External Build Log Storage has an extensible and pluggable architecture
          * At least 2 reference implementations:
           ** File-system-based storage (current implementation)
           ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
          * Jenkins Pipeline is supported out of the box
          * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

          Oleg Nenashev made changes -
          Description Original: Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

          It would be great to have such solution. Top-level requirements:

          * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
          * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
          * External Build Log Storage has an extensible and pluggable architecture
          * At least 2 reference implementations:
           ** File-system-based storage (current implementation)
           ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
          * Jenkins Pipeline is supported out of the box
          * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

          New: Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.

          It would be great to have such solution. Top-level requirements:

          * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
          * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
          * External Build Log Storage has an extensible and pluggable architecture
          * At least 2 reference implementations:
           ** File-system-based storage (current implementation)
           ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
          * Jenkins Pipeline is supported out of the box
          * Old logs should be browseable after changing the storage engine (Migration or parallel engines)

          Design document:
          * https://docs.google.com/document/d/1_bquSeA_lC7zJhQoWhxlSKJAg6b8duNbyQ15zCz-e4Y/edit#
          Oleg Nenashev made changes -
          Remote Link New: This issue links to "Design Document draft (Web Link)" [ 15820 ]
          Oleg Nenashev made changes -
          Remote Link New: This issue links to "PoC Demo Slides (Web Link)" [ 15821 ]
          Oleg Nenashev made changes -
          Status Original: In Progress [ 3 ] New: Open [ 1 ]
          Oleg Nenashev made changes -
          Assignee Original: Oleg Nenashev [ oleg_nenashev ]

            Unassigned Unassigned
            oleg_nenashev Oleg Nenashev
            Votes:
            16 Vote for this issue
            Watchers:
            43 Start watching this issue

              Created:
              Updated: