-
Bug
-
Resolution: Fixed
-
Critical
-
Jenkins ver. 1.480.3
Java memory used: 170 Mb / 910 Mb
Nb of http sessions: 7
Nb of active threads
(current http requests): 0
Details
OS: Windows Server 2008 R2 , amd64/64 (16 cores)
Java: Java(TM) SE Runtime Environment, 1.6.0_23-b05
JVM: Java HotSpot(TM) 64-Bit Server VM, 19.0-b09, mixed mode
PID of process: 1416
Server: Winstone Servlet Engine v0.9.10
Webapp context:
Start: 3/2/13 4:51 AM
JVM arguments: -Xrs
-Xmx1024m
-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle
Mean age of http sessions (min): 194
Memory: Non heap memory = 104 Mb (Perm Gen, Code Cache),
Loaded classes = 12,501,
Garbage collection time = 530,283 ms,
Process cpu time = 4,929,846,038 ms,
Committed virtual memory = 967 Mb,
Free physical memory = 11,554 Mb,
Total physical memory = 16,371 Mb,
Free swap space = 28,954 Mb,
Total swap space = 32,740 Mb
Perm Gen memory: 81 Mb / 82 Mb
Free disk space: 1,762,773 Mb
Dependencies: 110 dependencies ;Jenkins ver. 1.480.3 Java memory used: 170 Mb / 910 Mb Nb of http sessions: 7 Nb of active threads (current http requests): 0 Details OS: Windows Server 2008 R2 , amd64/64 (16 cores) Java: Java(TM) SE Runtime Environment, 1.6.0_23-b05 JVM: Java HotSpot(TM) 64-Bit Server VM, 19.0-b09, mixed mode PID of process: 1416 Server: Winstone Servlet Engine v0.9.10 Webapp context: Start: 3/2/13 4:51 AM JVM arguments: -Xrs -Xmx1024m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle Mean age of http sessions (min): 194 Memory: Non heap memory = 104 Mb (Perm Gen, Code Cache), Loaded classes = 12,501, Garbage collection time = 530,283 ms, Process cpu time = 4,929,846,038 ms, Committed virtual memory = 967 Mb, Free physical memory = 11,554 Mb, Total physical memory = 16,371 Mb, Free swap space = 28,954 Mb, Total swap space = 32,740 Mb Perm Gen memory: 81 Mb / 82 Mb Free disk space: 1,762,773 Mb Dependencies: 110 dependencies ;
We are using Jenkins ver. 1.480.3, upgraded on Mar 1st.
During the past month, the CPU usage continuously creasing...
The 1st week: <5%
The 2ed week: ~=17%
The 3ed week: ~= 25%,
And today it increasd to ~= 44%!!!
- is related to
-
-
- Resolved
-
[JENKINS-17349] CPU on master node keeps increasing and never come down
sharon xia
created issue -
| Attachment | New: percent http errors.png [ 23340 ] |
sharon xia
added a comment - Good catch!
Today I found another thing: There is a slave node(used to do our daily build), in which we have many branch build jobs connected to it.
I found the sudden increases were caused when a branch build started, and the CPU didn't decreased even if the branch build finished. However, during this week we might have >20 branch builds and only 1 of 2 builds caused the cpu sudden increase.
If I disconnected the slave build node, then the cup will decrease from the plug-in, althought from task manager the java.exe still keeps high. And after I reconnect the slave build node back, the cpu went back to high.
sharon xia
added a comment - Good catch!
Today I found another thing: There is a slave node(used to do our daily build), in which we have many branch build jobs connected to it.
I found the sudden increases were caused when a branch build started, and the CPU didn't decreased even if the branch build finished. However, during this week we might have >20 branch builds and only 1 of 2 builds caused the cpu sudden increase.
If I disconnected the slave build node, then the cup will decrease from the plug-in, althought from task manager the java.exe still keeps high. And after I reconnect the slave build node back, the cpu went back to high. sharon xia
added a comment - Perm gen memory : 80 Mb / 82 Mb, which is quite full.
This should not be the direct reason that CPU incresed, correct? I checked it will changes from 80Mb to 81Mb, 82Mb and turned to be 80Mb again...
sharon xia
added a comment - Perm gen memory : 80 Mb / 82 Mb, which is quite full.
This should not be the direct reason that CPU incresed, correct? I checked it will changes from 80Mb to 81Mb, 82Mb and turned to be 80Mb again...
The limited perm gen may cause some cascading problems, for example in GC and in class loading, and perhaps introduce an overhead in CPU.
If your large artifacts are copied between node and master or sent from master to browsers, this may also be a cause of high cpu usage. In fact, there are some other Jenkins issues, with some threads currently executing "java.util.zip.Deflater.deflateBytes", like your threads.
The recent fix of JENKINS-7813, in the future Jenkins v1.509, may help perhaps for communication between node and master.
sharon xia
added a comment - Thanks evernat!
I've modified the configuration file and restart the server(restart Jenkins service won't take effect), and the CPU goes back to <5%, let's wait for longer time to check whether it will address the CPU increase issue.
JVM arguments: -Xrs
-Xmx2048m
-XX:MaxPermSize=512m
-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle
And the large artifact download are expected users' behavior. Currently the download speed is fast so I don't think there is an issue.
sharon xia
added a comment - Thanks evernat!
I've modified the configuration file and restart the server(restart Jenkins service won't take effect), and the CPU goes back to <5%, let's wait for longer time to check whether it will address the CPU increase issue.
JVM arguments: -Xrs
-Xmx2048m
-XX:MaxPermSize=512m
-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle
And the large artifact download are expected users' behavior. Currently the download speed is fast so I don't think there is an issue.
sharon xia
added a comment - JavaMelody pdf after memory inceased. sharon xia
made changes -
| Attachment | New: JavaMelody__CNRDGPS_4_8_13.pdf [ 23413 ] |
I have joined 2 interesting graphics from the PDF: % CPU for 1 month and % http errors for 1 month.
It is strange to me that % CPU increases suddenly, about each week. (Note: this graphic is the %CPU of the Jenkins process, and not of the OS.)
And in the second graph, it seems that you have many http errors lately.
From the joined PDF:
It can be seen in the http statistics, that you have some downloads of very large artifacts (100 to 400 Mb) taking a long time.
And most importantly, in the "Detailed system information" from the PDF, it says:
Perm gen memory : 80 Mb / 82 Mb, which is quite full.
I suggest first to add a jvm parameter, probably in JENKINS_HOME/jenkins.xml: -XX:MaxPermSize=256m
And restart Jenkins.