-
Bug
-
Resolution: Unresolved
-
Critical
-
Jenkins 2.12, Linux RHEL7, OpenJDK 1.8
$ uname -a
Linux socvm459 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
$ java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)Jenkins 2.12, Linux RHEL7, OpenJDK 1.8 $ uname -a Linux socvm459 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux $ java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
Every few days, the Jenkins UI hangs and becomes totally unresponsive, i.e. "Jenkins is not picking up HTTP requests, so all attempts to connect hang and no page is ever rendered."
It doens't seem to be connected with a certain job getting started or anything.
I couldn't get a thread dump from the UI, nor from jstack, only from jstack -F. I could get the jmap -heap, but not the full heap dump. Both outputs are attached. If I understand that right, we are in a GC spiral.
Is there anything I can do to help debugging this?
- is related to
-
-
- Open
-
-
JENKINS-7615 remote file operation failed due to OutOfMemory on slave
-
- Open
-
-
-
- Open
-
[JENKINS-36559] File pattern (e.g. archive) field validation can run forever
| Environment |
Original:
Jenkins 2.12 $ uname -a Linux socvm459 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux $ java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) |
New:
Jenkins 2.12, Linux RHEL7, OpenJDK 1.8 $ uname -a Linux socvm459 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Aug 11 13:37:49 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux $ java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) |
It looks as though the memory overflow happens when I start editing a job. The attached screenshot shows the jconsole memory tab.
Jenkins had already run for about an hour without any trouble.
At 11:34, I started editing one of the job configurations. After about 2 minutes, memory was full, and GC couldn't recover any memory, so Jenkins hangs.
| Attachment | New: jconsole.png [ 33265 ] |
More information: it looks as though the problem only happens when I want to edit one particular job. I have attached the job's XML.
| Attachment | New: config.xml [ 33266 ] |
I have managed to create a heap dump shortly before jenkins hangs. Unfortunately, the heap dump is rather big (4.6 GB), so I better do not attach it here.
| Link | New: This issue is related to JENKINS-31981 [ JENKINS-31981 ] |
| Link | New: This issue is related to JENKINS-7615 [ JENKINS-7615 ] |
Aparently, it looks like a heap memory overflow.
Within a few minutes after start, when looking at jmap --heap, I see that the Eden space becomes completely filled.
I have updated to 2.13, but that didn't help.