Details
-
Epic
-
Status: In Progress (View Workflow)
-
Major
-
Resolution: Unresolved
-
External Build Log storage for Jenkins
-
Description
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs.
It would be great to have such solution. Top-level requirements:
- All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage
- Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible
- External Build Log Storage has an extensible and pluggable architecture
- At least 2 reference implementations:
- File-system-based storage (current implementation)
- One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana
- Jenkins Pipeline is supported out of the box
- Old logs should be browseable after changing the storage engine (Migration or parallel engines)
Design document:
Attachments
Issue Links
- blocks
-
JENKINS-49327 Please show TOP of build log in console output - even when trimming
-
- Open
-
- is blocked by
-
JENKINS-38314
External Build Log storage for Jenkins. Prototype #1
-
- Resolved
-
-
JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task
-
- Resolved
-
- is duplicated by
-
JENKINS-32444 Proposal: Move build logs off of Jenkins master
-
- Resolved
-
- is related to
-
JENKINS-45150 large console logging can take Jenkins down or affecting performance
-
- Open
-
-
-
- Reopened
-
- relates to
-
-
- Open
-
-
-
- Resolved
-
- links to
(1 is related to, 2 relates to, 3 links to)
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Child |
|
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Link |
This issue is blocked by |
| Description |
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs. It would be great to have such solution |
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs. It would be great to have such solution. |
| Description |
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs. It would be great to have such solution. |
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs. It would be great to have such solution. Top-level requirements: * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible * External Build Log Storage has an extensible and pluggable architecture * At least 2 reference implementations: ** File-system-based storage (current implementation) ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana * Jenkins Pipeline is supported out of the box * Old logs should be browseable after changing the storage engine (Migration or parallel engines) |
| Description |
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs. It would be great to have such solution. Top-level requirements: * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible * External Build Log Storage has an extensible and pluggable architecture * At least 2 reference implementations: ** File-system-based storage (current implementation) ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana * Jenkins Pipeline is supported out of the box * Old logs should be browseable after changing the storage engine (Migration or parallel engines) |
Jenkins' logging system is one of potential scalability bottlenecks on big Jenkins instances. System logging can be redirected via JUL log appenders (or other such solutions), but there is no OOTB solution for build logs. It would be great to have such solution. Top-level requirements: * All or almost all build/task logging is moved from Jenkins filesystem to External Build Log Storage * Minimize log traffic between Jenkins agents and the master, logs should be reported from slaves directly when it is possible * External Build Log Storage has an extensible and pluggable architecture * At least 2 reference implementations: ** File-system-based storage (current implementation) ** One external build log storage based on the open-source stack. Preference - Elasticsearch-Logstash-Kibana * Jenkins Pipeline is supported out of the box * Old logs should be browseable after changing the storage engine (Migration or parallel engines) Design document: * https://docs.google.com/document/d/1_bquSeA_lC7zJhQoWhxlSKJAg6b8duNbyQ15zCz-e4Y/edit# |
| Remote Link | This issue links to "Design Document draft (Web Link)" [ 15820 ] |
| Remote Link | This issue links to "PoC Demo Slides (Web Link)" [ 15821 ] |
| Status | In Progress [ 3 ] | Open [ 1 ] |
| Assignee | Oleg Nenashev [ oleg_nenashev ] |
| Link | This issue is related to JENKINS-26545 [ JENKINS-26545 ] |
| Link | This issue is related to JENKINS-45150 [ JENKINS-45150 ] |
| Component/s | pipeline [ 21692 ] |
| Labels | pipeline |
| Component/s | pipeline [ 21692 ] |
Sam Van Oort
made changes -
| Epic Child |
|
| Link |
This issue is blocked by |
| Link |
This issue is duplicated by |
| Assignee | Oleg Nenashev [ oleg_nenashev ] |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Link | This issue blocks JENKINS-49327 [ JENKINS-49327 ] |
| Link |
This issue relates to |
| Epic Child |
|
| Epic Child |
|
| Epic Child | JENKINS-52692 [ 192577 ] |
| Remote Link | This issue links to "Core API: https://github.com/jenkinsci/jenkins/pull/3557/ (Web Link)" [ 21216 ] |
| Epic Child |
|
| Epic Child |
|
| Epic Child |
|
| Epic Child |
|
| Epic Child |
|
| Link | This issue relates to JENKINS-51229 [ JENKINS-51229 ] |
| Epic Child | JENKINS-52867 [ 192803 ] |
| Epic Child | JENKINS-52873 [ 192810 ] |
| Epic Child | JENKINS-52894 [ 192847 ] |
| Epic Child | JENKINS-52914 [ 192873 ] |
| Epic Child |
|
| Epic Child |
|
| Rank | Ranked higher |
| Assignee | Oleg Nenashev [ oleg_nenashev ] |