-
Bug
-
Resolution: Duplicate
-
Major
-
VM, installed w. yum
Jenkins 2.263.2 (LTS)
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.8.0_275-b01
CentOS Linux release 7.9.2009 (Core)
----------------
Docker - jenkins/jenkins:lts
Jenkins 2.263.2 (LTS)
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.8.0_242-b08VM, installed w. yum Jenkins 2.263.2 (LTS) java.runtime.name OpenJDK Runtime Environment java.runtime.version 1.8.0_275-b01 CentOS Linux release 7.9.2009 (Core) ---------------- Docker - jenkins/jenkins:lts Jenkins 2.263.2 (LTS) java.runtime.name OpenJDK Runtime Environment java.runtime.version 1.8.0_242-b08
-
-
2.276 and 2.263.3
The Jenkins master does not close files / release file descriptors after archived artifacts have been downloaded. When the OS hard limit for open files have been reached, the Jenkins master starts to fail, giving 404-responses to several components in the UI, and eventually becomes unresponsive.
In the Jenkins system log, an increasing number "Too many open files" messages can be observed in various contexts:
2021-01-17 03:12:56.755+0000 [id=9] WARNING o.e.j.server.AbstractConnector#handleAcceptFailure:
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:702)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.lang.Thread.run(Thread.java:748)
2021-01-17 03:12:57.888+0000 [id=104] WARNING o.j.p.s.EventHistoryStore$DeleteStaleHistoryTask#run: Error deleting stale/expired events from EventHistoryStore.
java.nio.file.FileSystemException: /var/lib/jenkins/logs/sse-events: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at org.jenkinsci.plugins.ssegateway.EventHistoryStore.createDirectoryStream(EventHistoryStore.java:176)
at org.jenkinsci.plugins.ssegateway.EventHistoryStore.deleteAllFilesInDir(EventHistoryStore.java:221)
at org.jenkinsci.plugins.ssegateway.EventHistoryStore.deleteStaleHistory(EventHistoryStore.java:196)
at org.jenkinsci.plugins.ssegateway.EventHistoryStore$DeleteStaleHistoryTask.run(EventHistoryStore.java:270)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
The bug have been observed in multiple production masters with slightly different configurations (major components are all of the same version as specified in environment above). The bug have been reproduced in a newly installed VM and Jenkins with default plugins, as well as the official Jenkins Docker image jenkins/jenkins:lts.
The open files are not affected by a forced (.../gc) garbage collect. A full .../safeRestart is needed to close the files and release the file descriptors. Archived artifacts can be viewed via the "view" link without leaking resources.
- duplicates
-
-
- Closed
-