-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
openjdk 11.0.6 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing)
Jenkins ver. 2.204.2
Plugins:
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Active Directory plugin (active-directory): 2.16
Analysis Model API Plug-in (analysis-model-api): 7.0.4
AnsiColor (ansicolor): 0.6.2
Ant Plugin (ant): 1.10
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.6
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.10-2.0
Authentication Tokens API Plugin (authentication-tokens): 1.3
Authorize Project (authorize-project): 1.3.0
bouncycastle API Plugin (bouncycastle-api): 2.17
Branch API Plugin (branch-api): 2.5.4
Build Blocker Plugin (build-blocker-plugin): 1.7.3
Build Name and Description Setter (build-name-setter): 2.0.3
Build Timeout (build-timeout): 1.19
Folders Plugin (cloudbees-folder): 6.9
Cobertura Plugin (cobertura): 1.15
Code Coverage API Plugin (code-coverage-api): 1.1.2
Command Agent Launcher Plugin (command-launcher): 1.3
Compact Columns (compact-columns): 1.10
Conditional BuildStep (conditional-buildstep): 1.3.6
Credentials Plugin (credentials): 2.3.0
Credentials Binding Plugin (credentials-binding): 1.20
Display URL API (display-url-api): 2.3.2
Docker Commons Plugin (docker-commons): 1.15
Docker Pipeline (docker-workflow): 1.21
DTKit 2 API. (dtkit-api): 2.1.1-1
Durable Task Plugin (durable-task): 1.33
Email Extension Plugin (email-ext): 2.68
External Monitor Job Type Plugin (external-monitor-job): 1.7
Extra Columns Plugin (extra-columns): 1.21
Forensics API Plugin (forensics-api): 0.6.1
Filesystem Trigger Plug-in (fstrigger): 0.39
Git client plugin (git-client): 3.0.0
GIT server Plugin (git-server): 1.8
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Jackson 2 API Plugin (jackson2-api): 2.10.0
Javadoc Plugin (javadoc): 1.5
JAXB plugin (jaxb): 2.3.0.1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
JSch dependency plugin (jsch): 0.1.55.1
JUnit Plugin (junit): 1.28
LDAP Plugin (ldap): 1.21
Lockable Resources plugin (lockable-resources): 2.7
Mailer Plugin (mailer): 1.29
Matrix Authorization Strategy Plugin (matrix-auth): 2.5
Matrix Project Plugin (matrix-project): 1.14
Maven Integration plugin (maven-plugin): 3.4
Mercurial plugin (mercurial): 2.8
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Monitoring (monitoring): 1.81.0
PAM Authentication plugin (pam-auth): 1.6
Parameterized Trigger plugin (parameterized-trigger): 2.35.2
Pipeline: Build Step (pipeline-build-step): 2.9
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10
Pipeline: Input Step (pipeline-input-step): 2.11
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
Pipeline: Model API (pipeline-model-api): 1.4.0
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
Pipeline: Declarative (pipeline-model-definition): 1.4.0
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.4.0
Pipeline: REST API Plugin (pipeline-rest-api): 2.12
Pipeline: Stage Step (pipeline-stage-step): 2.3
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.4.0
Pipeline: Stage View Plugin (pipeline-stage-view): 2.12
Plain Credentials Plugin (plain-credentials): 1.5
Resource Disposer Plugin (resource-disposer): 0.14
Robot Framework plugin (robot): 2.0.0
Role-based Authorization Strategy (role-strategy): 2.15
Run Condition Plugin (run-condition): 1.2
Safe Restart Plugin (saferestart): 0.3
SCM API Plugin (scm-api): 2.6.3
Script Security Plugin (script-security): 1.70
SSH Credentials Plugin (ssh-credentials): 1.18
SSH Slaves plugin (ssh-slaves): 1.31.0
Structs Plugin (structs): 1.20
Timestamper (timestamper): 1.10
Token Macro Plugin (token-macro): 2.10
Trilead API Plugin (trilead-api): 1.0.5
Validating String Parameter Plugin (validating-string-parameter): 2.4
Warnings Next Generation Plugin (warnings-ng): 7.3.0
WMI Windows Agents Plugin (windows-slaves): 1.5
Pipeline (workflow-aggregator): 2.6
Pipeline: API (workflow-api): 2.37
Pipeline: Basic Steps (workflow-basic-steps): 2.18
Pipeline: Groovy (workflow-cps): 2.80
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.15
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.35
Pipeline: Job (workflow-job): 2.36
Pipeline: Multibranch (workflow-multibranch): 2.21
Pipeline: SCM Step (workflow-scm-step): 2.9
Pipeline: Step API (workflow-step-api): 2.22
Pipeline: Supporting APIs (workflow-support): 3.3
Workspace Cleanup Plugin (ws-cleanup): 0.37
xUnit plugin (xunit): 2.3.6openjdk 11.0.6 2020-01-14 OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1) OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing) Jenkins ver. 2.204.2 Plugins: JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 Active Directory plugin (active-directory): 2.16 Analysis Model API Plug-in (analysis-model-api): 7.0.4 AnsiColor (ansicolor): 0.6.2 Ant Plugin (ant): 1.10 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.6 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.10-2.0 Authentication Tokens API Plugin (authentication-tokens): 1.3 Authorize Project (authorize-project): 1.3.0 bouncycastle API Plugin (bouncycastle-api): 2.17 Branch API Plugin (branch-api): 2.5.4 Build Blocker Plugin (build-blocker-plugin): 1.7.3 Build Name and Description Setter (build-name-setter): 2.0.3 Build Timeout (build-timeout): 1.19 Folders Plugin (cloudbees-folder): 6.9 Cobertura Plugin (cobertura): 1.15 Code Coverage API Plugin (code-coverage-api): 1.1.2 Command Agent Launcher Plugin (command-launcher): 1.3 Compact Columns (compact-columns): 1.10 Conditional BuildStep (conditional-buildstep): 1.3.6 Credentials Plugin (credentials): 2.3.0 Credentials Binding Plugin (credentials-binding): 1.20 Display URL API (display-url-api): 2.3.2 Docker Commons Plugin (docker-commons): 1.15 Docker Pipeline (docker-workflow): 1.21 DTKit 2 API. (dtkit-api): 2.1.1-1 Durable Task Plugin (durable-task): 1.33 Email Extension Plugin (email-ext): 2.68 External Monitor Job Type Plugin (external-monitor-job): 1.7 Extra Columns Plugin (extra-columns): 1.21 Forensics API Plugin (forensics-api): 0.6.1 Filesystem Trigger Plug-in (fstrigger): 0.39 Git client plugin (git-client): 3.0.0 GIT server Plugin (git-server): 1.8 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Jackson 2 API Plugin (jackson2-api): 2.10.0 Javadoc Plugin (javadoc): 1.5 JAXB plugin (jaxb): 2.3.0.1 Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 JSch dependency plugin (jsch): 0.1.55.1 JUnit Plugin (junit): 1.28 LDAP Plugin (ldap): 1.21 Lockable Resources plugin (lockable-resources): 2.7 Mailer Plugin (mailer): 1.29 Matrix Authorization Strategy Plugin (matrix-auth): 2.5 Matrix Project Plugin (matrix-project): 1.14 Maven Integration plugin (maven-plugin): 3.4 Mercurial plugin (mercurial): 2.8 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Monitoring (monitoring): 1.81.0 PAM Authentication plugin (pam-auth): 1.6 Parameterized Trigger plugin (parameterized-trigger): 2.35.2 Pipeline: Build Step (pipeline-build-step): 2.9 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10 Pipeline: Input Step (pipeline-input-step): 2.11 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Pipeline: Model API (pipeline-model-api): 1.4.0 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 Pipeline: Declarative (pipeline-model-definition): 1.4.0 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.4.0 Pipeline: REST API Plugin (pipeline-rest-api): 2.12 Pipeline: Stage Step (pipeline-stage-step): 2.3 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.4.0 Pipeline: Stage View Plugin (pipeline-stage-view): 2.12 Plain Credentials Plugin (plain-credentials): 1.5 Resource Disposer Plugin (resource-disposer): 0.14 Robot Framework plugin (robot): 2.0.0 Role-based Authorization Strategy (role-strategy): 2.15 Run Condition Plugin (run-condition): 1.2 Safe Restart Plugin (saferestart): 0.3 SCM API Plugin (scm-api): 2.6.3 Script Security Plugin (script-security): 1.70 SSH Credentials Plugin (ssh-credentials): 1.18 SSH Slaves plugin (ssh-slaves): 1.31.0 Structs Plugin (structs): 1.20 Timestamper (timestamper): 1.10 Token Macro Plugin (token-macro): 2.10 Trilead API Plugin (trilead-api): 1.0.5 Validating String Parameter Plugin (validating-string-parameter): 2.4 Warnings Next Generation Plugin (warnings-ng): 7.3.0 WMI Windows Agents Plugin (windows-slaves): 1.5 Pipeline (workflow-aggregator): 2.6 Pipeline: API (workflow-api): 2.37 Pipeline: Basic Steps (workflow-basic-steps): 2.18 Pipeline: Groovy (workflow-cps): 2.80 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.15 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.35 Pipeline: Job (workflow-job): 2.36 Pipeline: Multibranch (workflow-multibranch): 2.21 Pipeline: SCM Step (workflow-scm-step): 2.9 Pipeline: Step API (workflow-step-api): 2.22 Pipeline: Supporting APIs (workflow-support): 3.3 Workspace Cleanup Plugin (ws-cleanup): 0.37 xUnit plugin (xunit): 2.3.6
Hi,
we've started to get an Out of memory issues on our Jenkins instance last week. There was no notable change in the code (1 month ago, we've installed warnings-ng for pylint reports and Cobertura), but the problems started last week.
Right now, we are seeing odd behavior with our builds and such that something is probably leaking memory and then GC utilizes a CPU for 100%. This is like one build looks like:
Start: !1.png|thumbnail|
During: !2.png|thumbnail|
During: !3.png|thumbnail|
After some time: !4.png|thumbnail|
My guess is that the memory should be released, but something is holding it. I was trying to go through a your help page and tried to analyze a Heap Dump. The notable is Java.Util.HashMap class has almost all allocated bytes.
I'm providing a 2 heap dumps, one during a build link (~2.93GB) and one after the Jenkins instance is unresponsible due to the Out Of Memory link (~3.57GB).
They suggest that the problem is org.codehaus.groovy.ast.ClassNode, but to say the truth, I'm not a Java developer so I'm not getting what can be an issue.
Also, I'm providing a /var/log/jenkins/jenkins.log which contains lots of warnings, which may help: link (~30MB)
I've already spent a great of time debugging what is happening, but I can't get my head around.
Could you take a look or point where the problem should be?
Thanks
Andrej
