Aparently, it looks like a heap memory overflow.
Within a few minutes after start, when looking at jmap --heap, I see that the Eden space becomes completely filled.

I have updated to 2.13, but that didn't help.

It looks as though the memory overflow happens when I start editing a job. The attached screenshot shows the jconsole memory tab.
Jenkins had already run for about an hour without any trouble.
At 11:34, I started editing one of the job configurations. After about 2 minutes, memory was full, and GC couldn't recover any memory, so Jenkins hangs.

More information: it looks as though the problem only happens when I want to edit one particular job. I have attached the job's XML.

I have managed to create a heap dump shortly before jenkins hangs. Unfortunately, the heap dump is rather big (4.6 GB), so I better do not attach it here.

Probably the recursive validation of a file name pattern in the config that results in DirectoryScanner.scandir invocations.

Should be bounded by hudson.FilePath.VALIDATE_ANT_FILE_MASK_BOUND to 10k by default, but may not work as expected – Does this issue disappear with a suitably low value configured as described here? https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

I will test that. Now that you mention it, I have seen hundreds of instances of this message in the jenkins log:

skipping symbolic link /workspaces/socvm459/coverity/ws-r8.0-coverity/opt/93000/src/hp83000/hp83000/hp83000/hp83000/hp83000/test/devices/offline/recursiveLink1/recursiveLink1/recursiveLink1/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2/recursiveLink2/recursiveLink1/recursiveLink1/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2/recursiveLinkDir/recursiveLink3/recursiveLink2 -- too many levels of symbolic links.

Setting the property on the command line doesn't help, even when setting it to 100.
I also tested it with Jenkins 2.7.1, but it has the same problem.

I have created a simple job that reproduces the behaviour and attached the config.xml. It creates a subdir that contains a recursive link, plus 1000 empty files, and archives the files. This leads to a memory overflow within a few minutes, apparently independent of hudson.FilePath.VALIDATE_ANT_FILE_MASK_BOUND.

It looks as though it is crucial that the directory also contains a lot of files. My diagnosis would be that Jenkins tries to create a catalogue of files recursively. If there are very few files in the dir, the memory won't overflow as the limit of scanDir-calls is reached soon, but if there are files, each scanDir call is large.

Note that the config that I have attached doesn't exactly reproduce the issue, as this only crashes jenkins when building the job. The actual problem is worse insofar as it crashes jenkins when trying to configure the job.

Should be enough to remove the archiving, build once to get the files, then config to get the archiving again – pattern validation will then result in OOM