Edit: Root cause was the performance plugin running perfReport on ~240mb worth of test data.

We're experiencing a similar issue for the past 2 days.

Attached is a thread dump. ganthore-threads.dump

We're running on core 2.344 and latest pipelines plugins.

Version: Jenkins 2.355 with latest plugins.

Problem:

1. Some job cannot finish even if it's actually done, and keeps logging same output
2. Nodes of those jobs report unresponsive for 5 sec/ 10 sec... ，and the time can reset to 5 sec（means it's not a dead lock）
3. Other job on those node cannot start or finish
4. jstack jenkins, jobs are waiting for a thread. And there is tee in this thread backtrace. (I forgot to snapshot/save it...)

Reason:

1. We use tee in pipeline
2. Maybe cmd in tee does not close fd properly, or unstable network(packet loss for EOF), or tee self has bugs
3. Such huge and infinite outputs occupy the lock for too long.
4. The node start unresponsive, and the unresponsive time will reset when lock is get.
5. Jobs on those node cannot start or finish.

After we remove all tee in all jobs, the problem disappeared. But the reason might be different in other user's reports.

If you got tee in your backtrace, might be the tee problem, you should try to remove it.

Report by: Aliyun PolarDB Testing team.

We are having this same problem, I'm not sure if it's related to the CPS issue of the original subject or not. Several times a day, jobs will just stop, at various spots in their pipeline. These are all declarative pipelines, Linux controller, windows agent. There is plenty of disk space.

Jenkins 2.346.2
mostly all updated plugins

that -Xmx256m looks suspiciously low

This is from the jenkins.xml:

<executable>/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin</executable> -->
  <arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djsse.enableSNIExtension=false -jar "%BASE%\jenkins.war" --httpPort=8080</arguments>

memory:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        3.6G        1.2G        256M        2.9G        3.5G
Swap:          1.0G        345M        678M

Should we bump up the -Xmx to 4g?

mcascone , Not sure whether you have already fixed it. Try updating to openjdk 11

touseef We're experiencing the same issue. Do you mean updating the jdk version will help us fix the issue?

Which specific version helps? The latest one?