Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72564

OOM periodically with a growing size of config.xml

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • gradle-plugin, kubernetes-plugin

      Description

      We've suffered from a periodically crush of jenkins in past a few months. After some investigation, we found out that the jenkins config.xml file size has been increasing periodically and finally lead to a OOME.

      What happened

      There are tons of hudson.slaves.EnvironmentVariablesNodeProperty declaring MAVEN_OPTS in our config.xml file,  and there's a screenshot showing how big this config.xml file is (we restart jenkins and reset this file periodically, and the first time we found out this problem, this file is up to 300mb big).

      What did we try

      We made config.xml file read-only by `chattr +i config.xml`, and found out that gradle plugin is trying to add MAVEN_OPTS every-time we start a pipeline job.

      What is weird

      Our jenkins is working with a kubernetes backend, and it seems like that it only happens when new pods are delivered by built-in node, but we've configured number of executors to 0 for built-in node.

      What i want to know

      1. why built-in node still taking job builds when its "number of executors" is zero?

      2. we've disabled gradle plugin to fix this problem, but what should we do if we need this plugin?

        1. image-2024-01-16-13-04-39-493.png
          image-2024-01-16-13-04-39-493.png
          48 kB
        2. image-2024-01-16-12-57-55-968.png
          image-2024-01-16-12-57-55-968.png
          14 kB
        3. image-2024-01-16-12-52-02-459.png
          image-2024-01-16-12-52-02-459.png
          487 kB

          [JENKINS-72564] OOM periodically with a growing size of config.xml

          wang created issue -
          wang made changes -
          Environment Original: Jenkins Version: 2.332.3
          OS: CentOS Linux release 7.3.1611 (Core)
          JDK Version: oracle-jdk 1.8.0_282
          Plugins:
              PAM Authentication plugin:1.8,
              Pipeline implementation for Blue Ocean:1.25.5,
              Web for Blue Ocean:1.25.5,
              Dashboard for Blue Ocean:1.25.5,
              Pipeline: Supporting APIs:820.vd1a_6cc65ef33,
              Pipeline: Job:1186.v8def1a_5f3944,
              OkHttp Plugin:4.9.3-105.vb96869f8ac3a,
              Build Timeout:1.21,
              Jersey 2 API:2.36-2,
              Pipeline: API:1164.v760c223ddb_32,
              Localization Support Plugin:1.1,
              GitHub Branch Source Plugin:1637.vd833b_7ca_7654,
              Pipeline: Stage Step:293.v200037eefcd5,
              JavaScript GUI Lib: Moment.js bundle plugin:1.1.1,
              Kubernetes Credentials Plugin:0.9.0,
              Caffeine API Plugin:2.9.3-65.v6a_47d0f4d1fe,
              Pipeline: Milestone Step:101.vd572fef9d926,
              Pipeline: Stage View Plugin:2.24,
              Mina SSHD API :: Core:2.8.0-18.vd98674ecd652,
              Personalization for Blue Ocean:1.25.5,
              Trilead API Plugin:1.57.v6e90e07157e1,
              Durable Task Plugin:496.va67c6f9eefa7,
              HTML Publisher plugin:1.30,
              Log Parser Plugin:2.2,
              Timestamper:1.18,
              REST API for Blue Ocean:1.25.5,
              CAS Plugin:1.6.2,
              Server Sent Events (SSE) Gateway Plugin:1.25,
              Display URL API:2.3.6,
              JWT for Blue Ocean:1.25.5,
              Events API for Blue Ocean:1.25.5,
              GitHub plugin:1.34.3,
              Jenkins Apache HttpComponents Client 4.x API Plugin:4.5.13-138.v4e7d9a_7b_a_e61,
              Rebuilder:1.34,
              Pipeline: Declarative Extension Points API:2.2097.v33db_b_de764b_e,
              Pipeline: Build Step:2.18,
              Common API for Blue Ocean:1.25.5,
              Plain Credentials Plugin:1.8,
              Font Awesome API Plugin:6.1.1-1,
              Token Macro Plugin:293.v283932a_0a_b_49,
              Pipeline:581.v0c46fa_697ffd,
              SCM API Plugin:608.vfa_f971c5a_a_e9,
              Pipeline: Nodes and Processes:1146.v1a_d2e603f929,
              Resource Disposer Plugin:0.19,
              Jenkins Design Language:1.25.5,
              GitHub API Plugin:1.303-400.v35c2d8258028,
              JavaScript GUI Lib: ACE Editor bundle plugin:1.1,
              Checks API plugin:1.7.4,
              Java JSON Web Token (JJWT) Plugin:0.11.5-77.v646c772fddb_0,
              Script Security Plugin:1175.v4b_d517d6db_f0,
              Pipeline: REST API Plugin:2.24,
              Jenkins Mailer Plugin:414.vcc4c33714601,
              OWASP Markup Formatter Plugin:2.7,
              Email Extension Plugin:2.89,
              Jenkins Workspace Cleanup Plugin:0.42,
              Pipeline: Groovy Libraries:593.va_a_fc25d520e9,
              SSH Build Agents plugin:1.821.vd834f8a_c390e,
              Pipeline Graph Analysis Plugin:195.v5812d95a_a_2f9,
              Git plugin:4.11.5,
              Branch API Plugin:2.1046.v0ca_37783ecc5,
              SnakeYAML API Plugin:1.30.1,
              Jackson 2 API Plugin:2.13.3-285.vc03c0256d517,
              REST Implementation for Blue Ocean:1.25.5,
              Gradle Plugin:1.39.1,
              SSH Credentials Plugin:277.v95c2fec1c047,
              JavaScript GUI Lib: Handlebars bundle plugin:3.0.8,
              Kubernetes Client API Plugin:5.12.2-193.v26a_6078f65a_9,
              Popper.js 2 API Plugin:2.11.5-2,
              Pipeline: SCM Step:400.v6b_89a_1317c9a_,
              Blue Ocean Core JS:1.25.5,
              SSH server:3.242.va_db_9da_b_26a_c3,
              Pipeline: Input Step:448.v37cea_9a_10a_70,
              Prometheus metrics plugin:2.0.11,
              Jenkins Job Configuration History Plugin:1139.v888b_656ca_f6d,
              Bootstrap 5 API Plugin:5.1.3-7,
              Display URL for Blue Ocean:2.4.1,
              JUnit Plugin:1119.va_a_5e9068da_d7,
              Job Import Plugin:3.5,
              Matrix Project Plugin:771.v574584b_39e60,
              Jenkins Extended Read Permission Plugin:3.2,
              Summary Display Plugin:1.15,
              Localization: Chinese (Simplified):1.0.24,
              Pipeline: GitHub Groovy Libraries:38.v445716ea_edda_,
              GitHub Pipeline for Blue Ocean:1.25.5,
              Oracle Java SE Development Kit Installer Plugin:1.5,
              Command Agent Launcher Plugin:84.v4a_97f2027398,
              Favorite:2.4.1,
              Pipeline: Declarative:2.2097.v33db_b_de764b_e,
              Pipeline: Step API:625.vd896b_f445a_f8,
              Jenkins Pub-Sub \light\ Bus:1.16,
              Folders Plugin:6.729.v2b_9d1a_74d673,
              Bitbucket Pipeline for Blue Ocean:1.25.5,
              Mask Passwords Plugin:3.3,
              bouncycastle API Plugin:2.26,
              JAXB plugin:2.3.6-1,
              Credentials Binding Plugin:523.vd859a_4b_122e6,
              Handy Uri Templates 2.x API Plugin:2.1.8-22.v77d5b_75e6953,
              Pipeline: Groovy:2725.v7b_c717eb_12ce,
              Pipeline: Multibranch:716.vc692a_e52371b_,
              JavaBeans Activation Framework (JAF) API:1.2.0-5,
              Authentication Tokens API Plugin:1.4,
              JQuery3 API Plugin:3.6.0-4,
              Matrix Authorization Strategy Plugin:3.1.3,
              LDAP Plugin:2.10,
              Mina SSHD API :: Common:2.8.0-18.vd98674ecd652,
              Git Pipeline for Blue Ocean:1.25.5,
              Config API for Blue Ocean:1.25.5,
              Bitbucket Branch Source Plugin:773.v4b_9b_005b_562b_,
              ECharts API Plugin:5.3.3-1,
              Jenkins Git client plugin:3.11.0,
              Metrics Plugin:4.1.6.2,
              JavaMail API:1.6.2-6,
              GitLab Plugin:1.5.34,
              Pipeline: Model API:2.2097.v33db_b_de764b_e,
              Ant Plugin:475.vf34069fef73c,
              Pipeline SCM API for Blue Ocean:1.25.5,
              Blue Ocean:1.25.5,
              Structs Plugin:318.va_f3ccb_729b_71,
              Pipeline: Basic Steps:948.v2c72a_091b_b_68,
              Blue Ocean Pipeline Editor:1.25.5,
              Jenkins JSch dependency plugin:0.1.55.2,
              Variant Plugin:1.4,
              Autofavorite for Blue Ocean:1.2.5,
              i18n for Blue Ocean:1.25.5,
              Build Name and Description Setter:2.2.0,
              Plugin Utilities API Plugin:2.17.0,
              Credentials Plugin:1087.1089.v2f1b_9a_b_040e4,
              Pipeline: Stage Tags Metadata:2.2097.v33db_b_de764b_e,
              Kubernetes plugin:3651.v908e7db_10d06           		New: Jenkins Version: 2.332.3
          OS: CentOS Linux release 7.3.1611 (Core)
          JDK Version: oracle-jdk 1.8.0_282

          Run directly in physical machine

          Plugins:
              PAM Authentication plugin:1.8,
              Pipeline implementation for Blue Ocean:1.25.5,
              Web for Blue Ocean:1.25.5,
              Dashboard for Blue Ocean:1.25.5,
              Pipeline: Supporting APIs:820.vd1a_6cc65ef33,
              Pipeline: Job:1186.v8def1a_5f3944,
              OkHttp Plugin:4.9.3-105.vb96869f8ac3a,
              Build Timeout:1.21,
              Jersey 2 API:2.36-2,
              Pipeline: API:1164.v760c223ddb_32,
              Localization Support Plugin:1.1,
              GitHub Branch Source Plugin:1637.vd833b_7ca_7654,
              Pipeline: Stage Step:293.v200037eefcd5,
              JavaScript GUI Lib: Moment.js bundle plugin:1.1.1,
              Kubernetes Credentials Plugin:0.9.0,
              Caffeine API Plugin:2.9.3-65.v6a_47d0f4d1fe,
              Pipeline: Milestone Step:101.vd572fef9d926,
              Pipeline: Stage View Plugin:2.24,
              Mina SSHD API :: Core:2.8.0-18.vd98674ecd652,
              Personalization for Blue Ocean:1.25.5,
              Trilead API Plugin:1.57.v6e90e07157e1,
              Durable Task Plugin:496.va67c6f9eefa7,
              HTML Publisher plugin:1.30,
              Log Parser Plugin:2.2,
              Timestamper:1.18,
              REST API for Blue Ocean:1.25.5,
              CAS Plugin:1.6.2,
              Server Sent Events (SSE) Gateway Plugin:1.25,
              Display URL API:2.3.6,
              JWT for Blue Ocean:1.25.5,
              Events API for Blue Ocean:1.25.5,
              GitHub plugin:1.34.3,
              Jenkins Apache HttpComponents Client 4.x API Plugin:4.5.13-138.v4e7d9a_7b_a_e61,
              Rebuilder:1.34,
              Pipeline: Declarative Extension Points API:2.2097.v33db_b_de764b_e,
              Pipeline: Build Step:2.18,
              Common API for Blue Ocean:1.25.5,
              Plain Credentials Plugin:1.8,
              Font Awesome API Plugin:6.1.1-1,
              Token Macro Plugin:293.v283932a_0a_b_49,
              Pipeline:581.v0c46fa_697ffd,
              SCM API Plugin:608.vfa_f971c5a_a_e9,
              Pipeline: Nodes and Processes:1146.v1a_d2e603f929,
              Resource Disposer Plugin:0.19,
              Jenkins Design Language:1.25.5,
              GitHub API Plugin:1.303-400.v35c2d8258028,
              JavaScript GUI Lib: ACE Editor bundle plugin:1.1,
              Checks API plugin:1.7.4,
              Java JSON Web Token (JJWT) Plugin:0.11.5-77.v646c772fddb_0,
              Script Security Plugin:1175.v4b_d517d6db_f0,
              Pipeline: REST API Plugin:2.24,
              Jenkins Mailer Plugin:414.vcc4c33714601,
              OWASP Markup Formatter Plugin:2.7,
              Email Extension Plugin:2.89,
              Jenkins Workspace Cleanup Plugin:0.42,
              Pipeline: Groovy Libraries:593.va_a_fc25d520e9,
              SSH Build Agents plugin:1.821.vd834f8a_c390e,
              Pipeline Graph Analysis Plugin:195.v5812d95a_a_2f9,
              Git plugin:4.11.5,
              Branch API Plugin:2.1046.v0ca_37783ecc5,
              SnakeYAML API Plugin:1.30.1,
              Jackson 2 API Plugin:2.13.3-285.vc03c0256d517,
              REST Implementation for Blue Ocean:1.25.5,
              Gradle Plugin:1.39.1,
              SSH Credentials Plugin:277.v95c2fec1c047,
              JavaScript GUI Lib: Handlebars bundle plugin:3.0.8,
              Kubernetes Client API Plugin:5.12.2-193.v26a_6078f65a_9,
              Popper.js 2 API Plugin:2.11.5-2,
              Pipeline: SCM Step:400.v6b_89a_1317c9a_,
              Blue Ocean Core JS:1.25.5,
              SSH server:3.242.va_db_9da_b_26a_c3,
              Pipeline: Input Step:448.v37cea_9a_10a_70,
              Prometheus metrics plugin:2.0.11,
              Jenkins Job Configuration History Plugin:1139.v888b_656ca_f6d,
              Bootstrap 5 API Plugin:5.1.3-7,
              Display URL for Blue Ocean:2.4.1,
              JUnit Plugin:1119.va_a_5e9068da_d7,
              Job Import Plugin:3.5,
              Matrix Project Plugin:771.v574584b_39e60,
              Jenkins Extended Read Permission Plugin:3.2,
              Summary Display Plugin:1.15,
              Localization: Chinese (Simplified):1.0.24,
              Pipeline: GitHub Groovy Libraries:38.v445716ea_edda_,
              GitHub Pipeline for Blue Ocean:1.25.5,
              Oracle Java SE Development Kit Installer Plugin:1.5,
              Command Agent Launcher Plugin:84.v4a_97f2027398,
              Favorite:2.4.1,
              Pipeline: Declarative:2.2097.v33db_b_de764b_e,
              Pipeline: Step API:625.vd896b_f445a_f8,
              Jenkins Pub-Sub \light\ Bus:1.16,
              Folders Plugin:6.729.v2b_9d1a_74d673,
              Bitbucket Pipeline for Blue Ocean:1.25.5,
              Mask Passwords Plugin:3.3,
              bouncycastle API Plugin:2.26,
              JAXB plugin:2.3.6-1,
              Credentials Binding Plugin:523.vd859a_4b_122e6,
              Handy Uri Templates 2.x API Plugin:2.1.8-22.v77d5b_75e6953,
              Pipeline: Groovy:2725.v7b_c717eb_12ce,
              Pipeline: Multibranch:716.vc692a_e52371b_,
              JavaBeans Activation Framework (JAF) API:1.2.0-5,
              Authentication Tokens API Plugin:1.4,
              JQuery3 API Plugin:3.6.0-4,
              Matrix Authorization Strategy Plugin:3.1.3,
              LDAP Plugin:2.10,
              Mina SSHD API :: Common:2.8.0-18.vd98674ecd652,
              Git Pipeline for Blue Ocean:1.25.5,
              Config API for Blue Ocean:1.25.5,
              Bitbucket Branch Source Plugin:773.v4b_9b_005b_562b_,
              ECharts API Plugin:5.3.3-1,
              Jenkins Git client plugin:3.11.0,
              Metrics Plugin:4.1.6.2,
              JavaMail API:1.6.2-6,
              GitLab Plugin:1.5.34,
              Pipeline: Model API:2.2097.v33db_b_de764b_e,
              Ant Plugin:475.vf34069fef73c,
              Pipeline SCM API for Blue Ocean:1.25.5,
              Blue Ocean:1.25.5,
              Structs Plugin:318.va_f3ccb_729b_71,
              Pipeline: Basic Steps:948.v2c72a_091b_b_68,
              Blue Ocean Pipeline Editor:1.25.5,
              Jenkins JSch dependency plugin:0.1.55.2,
              Variant Plugin:1.4,
              Autofavorite for Blue Ocean:1.2.5,
              i18n for Blue Ocean:1.25.5,
              Build Name and Description Setter:2.2.0,
              Plugin Utilities API Plugin:2.17.0,
              Credentials Plugin:1087.1089.v2f1b_9a_b_040e4,
              Pipeline: Stage Tags Metadata:2.2097.v33db_b_de764b_e,
              Kubernetes plugin:3651.v908e7db_10d06

          Iurii Ignatko added a comment - - edited

          Hi wangchen!
          I think the problem you're referring to has been fixed in this ticket - https://issues.jenkins.io/browse/JENKINS-70663.

          Based off of the ticket description, you're using Gradle Plugin 1.39.1 which is quite old. I would recommend you consider upgrading to latter versions, as they contain multiple improvements (this one in particular).

          In your case, version 2.3.2 should be enough, but feel free to check ones higher - https://github.com/jenkinsci/gradle-plugin/releases.

           

          Iurii Ignatko added a comment - - edited Hi wangchen ! I think the problem you're referring to has been fixed in this ticket - https://issues.jenkins.io/browse/JENKINS-70663 . Based off of the ticket description, you're using Gradle Plugin 1.39.1 which is quite old. I would recommend you consider upgrading to latter versions, as they contain multiple improvements (this one in particular). In your case, version 2.3.2 should be enough, but feel free to check ones higher - https://github.com/jenkinsci/gradle-plugin/releases .  
          Iurii Ignatko made changes -
          Assignee Original: Stefan Wolf [ wolfs ] New: Iurii Ignatko [ welandaz ]
          Iurii Ignatko made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          wang added a comment -

          Thanks welandaz! 

          After an upgrade of gradle plugin (to 2.3.2) in our staging environment, the config.xml file size stopped to grow with empty and worthless MAVEN_OPTS variable, thus will not occupy tons of memory during runtime.

          But built-in node still taking our pipeline jobs for a short duration, normally a couple of hundred milliseconds and these builds are then forwarded to Pods created by Kubernetes. I've searched over jenkins community and stackoverflow, but there's no clear answer to it.

          Like this from community: https://community.jenkins.io/t/why-are-builds-appearing-on-built-in-node-but-its-not-using-the-executors-assigned-causing-high-memory-issue/9666, explained that parts of our pipeline runs on the controller node regardless of jenkins file

          wang added a comment - Thanks welandaz !   After an upgrade of gradle plugin (to 2.3.2) in our staging environment, the config.xml file size stopped to grow with empty and worthless MAVEN_OPTS variable, thus will not occupy tons of memory during runtime. But built-in node still taking our pipeline jobs for a short duration, normally a couple of hundred milliseconds and these builds are then forwarded to Pods created by Kubernetes. I've searched over jenkins community and stackoverflow, but there's no clear answer to it. Like this from community: https://community.jenkins.io/t/why-are-builds-appearing-on-built-in-node-but-its-not-using-the-executors-assigned-causing-high-memory-issue/9666, explained that parts of our pipeline runs on the controller node regardless of jenkins file

          Iurii Ignatko added a comment -

          hi wangchen !
          I'm glad to hear the upgrade fixed the issue!

          Regarding another problem you've mentioned - it doesn't seem to me that it's a Gradle Plugin problem per se (but, please, correct me if I'm wrong). It sounds like a different issue related to scheduling in pipeline jobs (Jenkins specific, not plugin specific).

          I'd suggest you open another ticket specifically for that and we close this one as the upgrade of the plugin fixed the issue for you.

          Iurii Ignatko added a comment - hi wangchen ! I'm glad to hear the upgrade fixed the issue! Regarding another problem you've mentioned - it doesn't seem to me that it's a Gradle Plugin problem per se (but, please, correct me if I'm wrong). It sounds like a different issue related to scheduling in pipeline jobs (Jenkins specific, not plugin specific). I'd suggest you open another ticket specifically for that and we close this one as the upgrade of the plugin fixed the issue for you.

          wang added a comment - - edited

          welandaz okay, thank you for help

          wang added a comment - - edited welandaz okay, thank you for help
          Iurii Ignatko made changes -
          Resolution New: Done [ 10000 ]
          Status Original: In Progress [ 3 ] New: Closed [ 6 ]

            welandaz Iurii Ignatko
            wangchen wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: