-
Type:
Bug
-
Resolution: Cannot Reproduce
-
Priority:
Critical
-
Component/s: core
-
Environment:openjdk 11.0.6 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing)
Jenkins ver. 2.204.2
Plugins:
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Active Directory plugin (active-directory): 2.16
Analysis Model API Plug-in (analysis-model-api): 7.0.4
AnsiColor (ansicolor): 0.6.2
Ant Plugin (ant): 1.10
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.6
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.10-2.0
Authentication Tokens API Plugin (authentication-tokens): 1.3
Authorize Project (authorize-project): 1.3.0
bouncycastle API Plugin (bouncycastle-api): 2.17
Branch API Plugin (branch-api): 2.5.4
Build Blocker Plugin (build-blocker-plugin): 1.7.3
Build Name and Description Setter (build-name-setter): 2.0.3
Build Timeout (build-timeout): 1.19
Folders Plugin (cloudbees-folder): 6.9
Cobertura Plugin (cobertura): 1.15
Code Coverage API Plugin (code-coverage-api): 1.1.2
Command Agent Launcher Plugin (command-launcher): 1.3
Compact Columns (compact-columns): 1.10
Conditional BuildStep (conditional-buildstep): 1.3.6
Credentials Plugin (credentials): 2.3.0
Credentials Binding Plugin (credentials-binding): 1.20
Display URL API (display-url-api): 2.3.2
Docker Commons Plugin (docker-commons): 1.15
Docker Pipeline (docker-workflow): 1.21
DTKit 2 API. (dtkit-api): 2.1.1-1
Durable Task Plugin (durable-task): 1.33
Email Extension Plugin (email-ext): 2.68
External Monitor Job Type Plugin (external-monitor-job): 1.7
Extra Columns Plugin (extra-columns): 1.21
Forensics API Plugin (forensics-api): 0.6.1
Filesystem Trigger Plug-in (fstrigger): 0.39
Git client plugin (git-client): 3.0.0
GIT server Plugin (git-server): 1.8
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Jackson 2 API Plugin (jackson2-api): 2.10.0
Javadoc Plugin (javadoc): 1.5
JAXB plugin (jaxb): 2.3.0.1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
JSch dependency plugin (jsch): 0.1.55.1
JUnit Plugin (junit): 1.28
LDAP Plugin (ldap): 1.21
Lockable Resources plugin (lockable-resources): 2.7
Mailer Plugin (mailer): 1.29
Matrix Authorization Strategy Plugin (matrix-auth): 2.5
Matrix Project Plugin (matrix-project): 1.14
Maven Integration plugin (maven-plugin): 3.4
Mercurial plugin (mercurial): 2.8
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Monitoring (monitoring): 1.81.0
PAM Authentication plugin (pam-auth): 1.6
Parameterized Trigger plugin (parameterized-trigger): 2.35.2
Pipeline: Build Step (pipeline-build-step): 2.9
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10
Pipeline: Input Step (pipeline-input-step): 2.11
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
Pipeline: Model API (pipeline-model-api): 1.4.0
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
Pipeline: Declarative (pipeline-model-definition): 1.4.0
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.4.0
Pipeline: REST API Plugin (pipeline-rest-api): 2.12
Pipeline: Stage Step (pipeline-stage-step): 2.3
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.4.0
Pipeline: Stage View Plugin (pipeline-stage-view): 2.12
Plain Credentials Plugin (plain-credentials): 1.5
Resource Disposer Plugin (resource-disposer): 0.14
Robot Framework plugin (robot): 2.0.0
Role-based Authorization Strategy (role-strategy): 2.15
Run Condition Plugin (run-condition): 1.2
Safe Restart Plugin (saferestart): 0.3
SCM API Plugin (scm-api): 2.6.3
Script Security Plugin (script-security): 1.70
SSH Credentials Plugin (ssh-credentials): 1.18
SSH Slaves plugin (ssh-slaves): 1.31.0
Structs Plugin (structs): 1.20
Timestamper (timestamper): 1.10
Token Macro Plugin (token-macro): 2.10
Trilead API Plugin (trilead-api): 1.0.5
Validating String Parameter Plugin (validating-string-parameter): 2.4
Warnings Next Generation Plugin (warnings-ng): 7.3.0
WMI Windows Agents Plugin (windows-slaves): 1.5
Pipeline (workflow-aggregator): 2.6
Pipeline: API (workflow-api): 2.37
Pipeline: Basic Steps (workflow-basic-steps): 2.18
Pipeline: Groovy (workflow-cps): 2.80
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.15
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.35
Pipeline: Job (workflow-job): 2.36
Pipeline: Multibranch (workflow-multibranch): 2.21
Pipeline: SCM Step (workflow-scm-step): 2.9
Pipeline: Step API (workflow-step-api): 2.22
Pipeline: Supporting APIs (workflow-support): 3.3
Workspace Cleanup Plugin (ws-cleanup): 0.37
xUnit plugin (xunit): 2.3.6openjdk 11.0.6 2020-01-14 OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1) OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing) Jenkins ver. 2.204.2 Plugins: JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 Active Directory plugin (active-directory): 2.16 Analysis Model API Plug-in (analysis-model-api): 7.0.4 AnsiColor (ansicolor): 0.6.2 Ant Plugin (ant): 1.10 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.6 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.10-2.0 Authentication Tokens API Plugin (authentication-tokens): 1.3 Authorize Project (authorize-project): 1.3.0 bouncycastle API Plugin (bouncycastle-api): 2.17 Branch API Plugin (branch-api): 2.5.4 Build Blocker Plugin (build-blocker-plugin): 1.7.3 Build Name and Description Setter (build-name-setter): 2.0.3 Build Timeout (build-timeout): 1.19 Folders Plugin (cloudbees-folder): 6.9 Cobertura Plugin (cobertura): 1.15 Code Coverage API Plugin (code-coverage-api): 1.1.2 Command Agent Launcher Plugin (command-launcher): 1.3 Compact Columns (compact-columns): 1.10 Conditional BuildStep (conditional-buildstep): 1.3.6 Credentials Plugin (credentials): 2.3.0 Credentials Binding Plugin (credentials-binding): 1.20 Display URL API (display-url-api): 2.3.2 Docker Commons Plugin (docker-commons): 1.15 Docker Pipeline (docker-workflow): 1.21 DTKit 2 API. (dtkit-api): 2.1.1-1 Durable Task Plugin (durable-task): 1.33 Email Extension Plugin (email-ext): 2.68 External Monitor Job Type Plugin (external-monitor-job): 1.7 Extra Columns Plugin (extra-columns): 1.21 Forensics API Plugin (forensics-api): 0.6.1 Filesystem Trigger Plug-in (fstrigger): 0.39 Git client plugin (git-client): 3.0.0 GIT server Plugin (git-server): 1.8 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Jackson 2 API Plugin (jackson2-api): 2.10.0 Javadoc Plugin (javadoc): 1.5 JAXB plugin (jaxb): 2.3.0.1 Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 JSch dependency plugin (jsch): 0.1.55.1 JUnit Plugin (junit): 1.28 LDAP Plugin (ldap): 1.21 Lockable Resources plugin (lockable-resources): 2.7 Mailer Plugin (mailer): 1.29 Matrix Authorization Strategy Plugin (matrix-auth): 2.5 Matrix Project Plugin (matrix-project): 1.14 Maven Integration plugin (maven-plugin): 3.4 Mercurial plugin (mercurial): 2.8 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Monitoring (monitoring): 1.81.0 PAM Authentication plugin (pam-auth): 1.6 Parameterized Trigger plugin (parameterized-trigger): 2.35.2 Pipeline: Build Step (pipeline-build-step): 2.9 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10 Pipeline: Input Step (pipeline-input-step): 2.11 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Pipeline: Model API (pipeline-model-api): 1.4.0 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 Pipeline: Declarative (pipeline-model-definition): 1.4.0 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.4.0 Pipeline: REST API Plugin (pipeline-rest-api): 2.12 Pipeline: Stage Step (pipeline-stage-step): 2.3 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.4.0 Pipeline: Stage View Plugin (pipeline-stage-view): 2.12 Plain Credentials Plugin (plain-credentials): 1.5 Resource Disposer Plugin (resource-disposer): 0.14 Robot Framework plugin (robot): 2.0.0 Role-based Authorization Strategy (role-strategy): 2.15 Run Condition Plugin (run-condition): 1.2 Safe Restart Plugin (saferestart): 0.3 SCM API Plugin (scm-api): 2.6.3 Script Security Plugin (script-security): 1.70 SSH Credentials Plugin (ssh-credentials): 1.18 SSH Slaves plugin (ssh-slaves): 1.31.0 Structs Plugin (structs): 1.20 Timestamper (timestamper): 1.10 Token Macro Plugin (token-macro): 2.10 Trilead API Plugin (trilead-api): 1.0.5 Validating String Parameter Plugin (validating-string-parameter): 2.4 Warnings Next Generation Plugin (warnings-ng): 7.3.0 WMI Windows Agents Plugin (windows-slaves): 1.5 Pipeline (workflow-aggregator): 2.6 Pipeline: API (workflow-api): 2.37 Pipeline: Basic Steps (workflow-basic-steps): 2.18 Pipeline: Groovy (workflow-cps): 2.80 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.15 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.35 Pipeline: Job (workflow-job): 2.36 Pipeline: Multibranch (workflow-multibranch): 2.21 Pipeline: SCM Step (workflow-scm-step): 2.9 Pipeline: Step API (workflow-step-api): 2.22 Pipeline: Supporting APIs (workflow-support): 3.3 Workspace Cleanup Plugin (ws-cleanup): 0.37 xUnit plugin (xunit): 2.3.6
Hi,
we've started to get an Out of memory issues on our Jenkins instance last week. There was no notable change in the code (1 month ago, we've installed warnings-ng for pylint reports and Cobertura), but the problems started last week.
Â
Right now, we are seeing odd behavior with our builds and such that something is probably leaking memory and then GC utilizes a CPU for 100%. This is like one build looks like:
Â
Start: !1.png|thumbnail|
During: !2.png|thumbnail|
During: !3.png|thumbnail|
After some time: !4.png|thumbnail|
My guess is that the memory should be released, but something is holding it. I was trying to go through a your help page and tried to analyze a Heap Dump. The notable is Java.Util.HashMap class has almost all allocated bytes.
I'm providing a 2 heap dumps, one during a build link (~2.93GB) and one after the Jenkins instance is unresponsible due to the Out Of Memory link (~3.57GB).
They suggest that the problem is org.codehaus.groovy.ast.ClassNode, but to say the truth, I'm not a Java developer so I'm not getting what can be an issue.
Also, I'm providing a /var/log/jenkins/jenkins.log which contains lots of warnings, which may help: link (~30MB)
I've already spent a great of time debugging what is happening, but I can't get my head around.
Could you take a look or point where the problem should be?
Thanks
Andrej
Â
