-
Bug
-
Resolution: Done
-
Blocker
-
Jenkins Version: 2.332.3
OS: CentOS Linux release 7.3.1611 (Core)
JDK Version: oracle-jdk 1.8.0_282
Run directly in physical machine
Plugins:
PAM Authentication plugin:1.8,
Pipeline implementation for Blue Ocean:1.25.5,
Web for Blue Ocean:1.25.5,
Dashboard for Blue Ocean:1.25.5,
Pipeline: Supporting APIs:820.vd1a_6cc65ef33,
Pipeline: Job:1186.v8def1a_5f3944,
OkHttp Plugin:4.9.3-105.vb96869f8ac3a,
Build Timeout:1.21,
Jersey 2 API:2.36-2,
Pipeline: API:1164.v760c223ddb_32,
Localization Support Plugin:1.1,
GitHub Branch Source Plugin:1637.vd833b_7ca_7654,
Pipeline: Stage Step:293.v200037eefcd5,
JavaScript GUI Lib: Moment.js bundle plugin:1.1.1,
Kubernetes Credentials Plugin:0.9.0,
Caffeine API Plugin:2.9.3-65.v6a_47d0f4d1fe,
Pipeline: Milestone Step:101.vd572fef9d926,
Pipeline: Stage View Plugin:2.24,
Mina SSHD API :: Core:2.8.0-18.vd98674ecd652,
Personalization for Blue Ocean:1.25.5,
Trilead API Plugin:1.57.v6e90e07157e1,
Durable Task Plugin:496.va67c6f9eefa7,
HTML Publisher plugin:1.30,
Log Parser Plugin:2.2,
Timestamper:1.18,
REST API for Blue Ocean:1.25.5,
CAS Plugin:1.6.2,
Server Sent Events (SSE) Gateway Plugin:1.25,
Display URL API:2.3.6,
JWT for Blue Ocean:1.25.5,
Events API for Blue Ocean:1.25.5,
GitHub plugin:1.34.3,
Jenkins Apache HttpComponents Client 4.x API Plugin:4.5.13-138.v4e7d9a_7b_a_e61,
Rebuilder:1.34,
Pipeline: Declarative Extension Points API:2.2097.v33db_b_de764b_e,
Pipeline: Build Step:2.18,
Common API for Blue Ocean:1.25.5,
Plain Credentials Plugin:1.8,
Font Awesome API Plugin:6.1.1-1,
Token Macro Plugin:293.v283932a_0a_b_49,
Pipeline:581.v0c46fa_697ffd,
SCM API Plugin:608.vfa_f971c5a_a_e9,
Pipeline: Nodes and Processes:1146.v1a_d2e603f929,
Resource Disposer Plugin:0.19,
Jenkins Design Language:1.25.5,
GitHub API Plugin:1.303-400.v35c2d8258028,
JavaScript GUI Lib: ACE Editor bundle plugin:1.1,
Checks API plugin:1.7.4,
Java JSON Web Token (JJWT) Plugin:0.11.5-77.v646c772fddb_0,
Script Security Plugin:1175.v4b_d517d6db_f0,
Pipeline: REST API Plugin:2.24,
Jenkins Mailer Plugin:414.vcc4c33714601,
OWASP Markup Formatter Plugin:2.7,
Email Extension Plugin:2.89,
Jenkins Workspace Cleanup Plugin:0.42,
Pipeline: Groovy Libraries:593.va_a_fc25d520e9,
SSH Build Agents plugin:1.821.vd834f8a_c390e,
Pipeline Graph Analysis Plugin:195.v5812d95a_a_2f9,
Git plugin:4.11.5,
Branch API Plugin:2.1046.v0ca_37783ecc5,
SnakeYAML API Plugin:1.30.1,
Jackson 2 API Plugin:2.13.3-285.vc03c0256d517,
REST Implementation for Blue Ocean:1.25.5,
Gradle Plugin:1.39.1,
SSH Credentials Plugin:277.v95c2fec1c047,
JavaScript GUI Lib: Handlebars bundle plugin:3.0.8,
Kubernetes Client API Plugin:5.12.2-193.v26a_6078f65a_9,
Popper.js 2 API Plugin:2.11.5-2,
Pipeline: SCM Step:400.v6b_89a_1317c9a_,
Blue Ocean Core JS:1.25.5,
SSH server:3.242.va_db_9da_b_26a_c3,
Pipeline: Input Step:448.v37cea_9a_10a_70,
Prometheus metrics plugin:2.0.11,
Jenkins Job Configuration History Plugin:1139.v888b_656ca_f6d,
Bootstrap 5 API Plugin:5.1.3-7,
Display URL for Blue Ocean:2.4.1,
JUnit Plugin:1119.va_a_5e9068da_d7,
Job Import Plugin:3.5,
Matrix Project Plugin:771.v574584b_39e60,
Jenkins Extended Read Permission Plugin:3.2,
Summary Display Plugin:1.15,
Localization: Chinese (Simplified):1.0.24,
Pipeline: GitHub Groovy Libraries:38.v445716ea_edda_,
GitHub Pipeline for Blue Ocean:1.25.5,
Oracle Java SE Development Kit Installer Plugin:1.5,
Command Agent Launcher Plugin:84.v4a_97f2027398,
Favorite:2.4.1,
Pipeline: Declarative:2.2097.v33db_b_de764b_e,
Pipeline: Step API:625.vd896b_f445a_f8,
Jenkins Pub-Sub \light\ Bus:1.16,
Folders Plugin:6.729.v2b_9d1a_74d673,
Bitbucket Pipeline for Blue Ocean:1.25.5,
Mask Passwords Plugin:3.3,
bouncycastle API Plugin:2.26,
JAXB plugin:2.3.6-1,
Credentials Binding Plugin:523.vd859a_4b_122e6,
Handy Uri Templates 2.x API Plugin:2.1.8-22.v77d5b_75e6953,
Pipeline: Groovy:2725.v7b_c717eb_12ce,
Pipeline: Multibranch:716.vc692a_e52371b_,
JavaBeans Activation Framework (JAF) API:1.2.0-5,
Authentication Tokens API Plugin:1.4,
JQuery3 API Plugin:3.6.0-4,
Matrix Authorization Strategy Plugin:3.1.3,
LDAP Plugin:2.10,
Mina SSHD API :: Common:2.8.0-18.vd98674ecd652,
Git Pipeline for Blue Ocean:1.25.5,
Config API for Blue Ocean:1.25.5,
Bitbucket Branch Source Plugin:773.v4b_9b_005b_562b_,
ECharts API Plugin:5.3.3-1,
Jenkins Git client plugin:3.11.0,
Metrics Plugin:4.1.6.2,
JavaMail API:1.6.2-6,
GitLab Plugin:1.5.34,
Pipeline: Model API:2.2097.v33db_b_de764b_e,
Ant Plugin:475.vf34069fef73c,
Pipeline SCM API for Blue Ocean:1.25.5,
Blue Ocean:1.25.5,
Structs Plugin:318.va_f3ccb_729b_71,
Pipeline: Basic Steps:948.v2c72a_091b_b_68,
Blue Ocean Pipeline Editor:1.25.5,
Jenkins JSch dependency plugin:0.1.55.2,
Variant Plugin:1.4,
Autofavorite for Blue Ocean:1.2.5,
i18n for Blue Ocean:1.25.5,
Build Name and Description Setter:2.2.0,
Plugin Utilities API Plugin:2.17.0,
Credentials Plugin:1087.1089.v2f1b_9a_b_040e4,
Pipeline: Stage Tags Metadata:2.2097.v33db_b_de764b_e,
Kubernetes plugin:3651.v908e7db_10d06Jenkins Version: 2.332.3 OS: CentOS Linux release 7.3.1611 (Core) JDK Version: oracle-jdk 1.8.0_282 Run directly in physical machine Plugins: PAM Authentication plugin:1.8, Pipeline implementation for Blue Ocean:1.25.5, Web for Blue Ocean:1.25.5, Dashboard for Blue Ocean:1.25.5, Pipeline: Supporting APIs:820.vd1a_6cc65ef33, Pipeline: Job:1186.v8def1a_5f3944, OkHttp Plugin:4.9.3-105.vb96869f8ac3a, Build Timeout:1.21, Jersey 2 API:2.36-2, Pipeline: API:1164.v760c223ddb_32, Localization Support Plugin:1.1, GitHub Branch Source Plugin:1637.vd833b_7ca_7654, Pipeline: Stage Step:293.v200037eefcd5, JavaScript GUI Lib: Moment.js bundle plugin:1.1.1, Kubernetes Credentials Plugin:0.9.0, Caffeine API Plugin:2.9.3-65.v6a_47d0f4d1fe, Pipeline: Milestone Step:101.vd572fef9d926, Pipeline: Stage View Plugin:2.24, Mina SSHD API :: Core:2.8.0-18.vd98674ecd652, Personalization for Blue Ocean:1.25.5, Trilead API Plugin:1.57.v6e90e07157e1, Durable Task Plugin:496.va67c6f9eefa7, HTML Publisher plugin:1.30, Log Parser Plugin:2.2, Timestamper:1.18, REST API for Blue Ocean:1.25.5, CAS Plugin:1.6.2, Server Sent Events (SSE) Gateway Plugin:1.25, Display URL API:2.3.6, JWT for Blue Ocean:1.25.5, Events API for Blue Ocean:1.25.5, GitHub plugin:1.34.3, Jenkins Apache HttpComponents Client 4.x API Plugin:4.5.13-138.v4e7d9a_7b_a_e61, Rebuilder:1.34, Pipeline: Declarative Extension Points API:2.2097.v33db_b_de764b_e, Pipeline: Build Step:2.18, Common API for Blue Ocean:1.25.5, Plain Credentials Plugin:1.8, Font Awesome API Plugin:6.1.1-1, Token Macro Plugin:293.v283932a_0a_b_49, Pipeline:581.v0c46fa_697ffd, SCM API Plugin:608.vfa_f971c5a_a_e9, Pipeline: Nodes and Processes:1146.v1a_d2e603f929, Resource Disposer Plugin:0.19, Jenkins Design Language:1.25.5, GitHub API Plugin:1.303-400.v35c2d8258028, JavaScript GUI Lib: ACE Editor bundle plugin:1.1, Checks API plugin:1.7.4, Java JSON Web Token (JJWT) Plugin:0.11.5-77.v646c772fddb_0, Script Security Plugin:1175.v4b_d517d6db_f0, Pipeline: REST API Plugin:2.24, Jenkins Mailer Plugin:414.vcc4c33714601, OWASP Markup Formatter Plugin:2.7, Email Extension Plugin:2.89, Jenkins Workspace Cleanup Plugin:0.42, Pipeline: Groovy Libraries:593.va_a_fc25d520e9, SSH Build Agents plugin:1.821.vd834f8a_c390e, Pipeline Graph Analysis Plugin:195.v5812d95a_a_2f9, Git plugin:4.11.5, Branch API Plugin:2.1046.v0ca_37783ecc5, SnakeYAML API Plugin:1.30.1, Jackson 2 API Plugin:2.13.3-285.vc03c0256d517, REST Implementation for Blue Ocean:1.25.5, Gradle Plugin:1.39.1, SSH Credentials Plugin:277.v95c2fec1c047, JavaScript GUI Lib: Handlebars bundle plugin:3.0.8, Kubernetes Client API Plugin:5.12.2-193.v26a_6078f65a_9, Popper.js 2 API Plugin:2.11.5-2, Pipeline: SCM Step:400.v6b_89a_1317c9a_, Blue Ocean Core JS:1.25.5, SSH server:3.242.va_db_9da_b_26a_c3, Pipeline: Input Step:448.v37cea_9a_10a_70, Prometheus metrics plugin:2.0.11, Jenkins Job Configuration History Plugin:1139.v888b_656ca_f6d, Bootstrap 5 API Plugin:5.1.3-7, Display URL for Blue Ocean:2.4.1, JUnit Plugin:1119.va_a_5e9068da_d7, Job Import Plugin:3.5, Matrix Project Plugin:771.v574584b_39e60, Jenkins Extended Read Permission Plugin:3.2, Summary Display Plugin:1.15, Localization: Chinese (Simplified):1.0.24, Pipeline: GitHub Groovy Libraries:38.v445716ea_edda_, GitHub Pipeline for Blue Ocean:1.25.5, Oracle Java SE Development Kit Installer Plugin:1.5, Command Agent Launcher Plugin:84.v4a_97f2027398, Favorite:2.4.1, Pipeline: Declarative:2.2097.v33db_b_de764b_e, Pipeline: Step API:625.vd896b_f445a_f8, Jenkins Pub-Sub \light\ Bus:1.16, Folders Plugin:6.729.v2b_9d1a_74d673, Bitbucket Pipeline for Blue Ocean:1.25.5, Mask Passwords Plugin:3.3, bouncycastle API Plugin:2.26, JAXB plugin:2.3.6-1, Credentials Binding Plugin:523.vd859a_4b_122e6, Handy Uri Templates 2.x API Plugin:2.1.8-22.v77d5b_75e6953, Pipeline: Groovy:2725.v7b_c717eb_12ce, Pipeline: Multibranch:716.vc692a_e52371b_, JavaBeans Activation Framework (JAF) API:1.2.0-5, Authentication Tokens API Plugin:1.4, JQuery3 API Plugin:3.6.0-4, Matrix Authorization Strategy Plugin:3.1.3, LDAP Plugin:2.10, Mina SSHD API :: Common:2.8.0-18.vd98674ecd652, Git Pipeline for Blue Ocean:1.25.5, Config API for Blue Ocean:1.25.5, Bitbucket Branch Source Plugin:773.v4b_9b_005b_562b_, ECharts API Plugin:5.3.3-1, Jenkins Git client plugin:3.11.0, Metrics Plugin:4.1.6.2, JavaMail API:1.6.2-6, GitLab Plugin:1.5.34, Pipeline: Model API:2.2097.v33db_b_de764b_e, Ant Plugin:475.vf34069fef73c, Pipeline SCM API for Blue Ocean:1.25.5, Blue Ocean:1.25.5, Structs Plugin:318.va_f3ccb_729b_71, Pipeline: Basic Steps:948.v2c72a_091b_b_68, Blue Ocean Pipeline Editor:1.25.5, Jenkins JSch dependency plugin:0.1.55.2, Variant Plugin:1.4, Autofavorite for Blue Ocean:1.2.5, i18n for Blue Ocean:1.25.5, Build Name and Description Setter:2.2.0, Plugin Utilities API Plugin:2.17.0, Credentials Plugin:1087.1089.v2f1b_9a_b_040e4, Pipeline: Stage Tags Metadata:2.2097.v33db_b_de764b_e, Kubernetes plugin:3651.v908e7db_10d06
Description
We've suffered from a periodically crush of jenkins in past a few months. After some investigation, we found out that the jenkins config.xml file size has been increasing periodically and finally lead to a OOME.
What happened
There are tons of hudson.slaves.EnvironmentVariablesNodeProperty declaring MAVEN_OPTS in our config.xml file, and there's a screenshot showing how big this config.xml file is (we restart jenkins and reset this file periodically, and the first time we found out this problem, this file is up to 300mb big).
What did we try
We made config.xml file read-only by `chattr +i config.xml`, and found out that gradle plugin is trying to add MAVEN_OPTS every-time we start a pipeline job.
What is weird
Our jenkins is working with a kubernetes backend, and it seems like that it only happens when new pods are delivered by built-in node, but we've configured number of executors to 0 for built-in node.
What i want to know
1. why built-in node still taking job builds when its "number of executors" is zero?
2. we've disabled gradle plugin to fix this problem, but what should we do if we need this plugin?
