Oleg Nenashev added a comment - 2017-04-10 17:19 released in 2.53

Daniel Trebbien added a comment - 2017-12-28 03:05 Regarding this paragraph: The recommended solution is to switch to Files.newInputStream and Files.newOutputStream respectively as these use a reference queue to clean up unclosed streams that are eligible for GC rather than waiting for a finalizer always. I tried looking through the OpenJDK source code, but I am not seeing where this reference queue is. It looks like one cannot rely on an unclosed stream created by Files.newInputStream , etc. being closed automatically. java.nio.file.Files has this: private static FileSystemProvider provider(Path path) { return path.getFileSystem().provider(); } //... public static InputStream newInputStream(Path path, OpenOption... options) throws IOException { return provider(path).newInputStream(path, options); } //... public static SeekableByteChannel newByteChannel(Path path, Set<? extends OpenOption> options, FileAttribute<?>... attrs) throws IOException { return provider(path).newByteChannel(path, options, attrs); } //... public static SeekableByteChannel newByteChannel(Path path, OpenOption... options) throws IOException { Set<OpenOption> set = new HashSet<OpenOption>(options.length); Collections.addAll(set, options); return newByteChannel(path, set); } java.nio.file.spi.FileSystemProvider has this: public InputStream newInputStream(Path path, OpenOption... options) throws IOException { if (options.length > 0) { for (OpenOption opt: options) { // All OpenOption values except for APPEND and WRITE are allowed if (opt == StandardOpenOption.APPEND || opt == StandardOpenOption.WRITE) throw new UnsupportedOperationException( " '" + opt + "' not allowed" ); } } return Channels.newInputStream(Files.newByteChannel(path, options)); } sun.nio.fs.UnixFileSystemProvider has this: @Override public SeekableByteChannel newByteChannel(Path obj, Set<? extends OpenOption> options, FileAttribute<?>... attrs) throws IOException { UnixPath file = UnixPath.toUnixPath(obj); int mode = UnixFileModeAttribute .toUnixMode(UnixFileModeAttribute.ALL_READWRITE, attrs); try { return UnixChannelFactory.newFileChannel(file, options, mode); } catch (UnixException x) { x.rethrowAsIOException(file); return null ; // keep compiler happy } } sun.nio.fs.UnixChannelFactory has this: static FileChannel newFileChannel( int dfd, UnixPath path, String pathForPermissionCheck, Set<? extends OpenOption> options, int mode) throws UnixException { Flags flags = Flags.toFlags(options); // default is reading; append => writing if (!flags.read && !flags.write) { if (flags.append) { flags.write = true ; } else { flags.read = true ; } } // validation if (flags.read && flags.append) throw new IllegalArgumentException( "READ + APPEND not allowed" ); if (flags.append && flags.truncateExisting) throw new IllegalArgumentException( "APPEND + TRUNCATE_EXISTING not allowed" ); FileDescriptor fdObj = open(dfd, path, pathForPermissionCheck, flags, mode); return FileChannelImpl.open(fdObj, flags.read, flags.write, flags.append, null ); } //... static FileChannel newFileChannel(UnixPath path, Set<? extends OpenOption> options, int mode) throws UnixException { return newFileChannel(-1, path, null , options, mode); } sun.nio.ch.FileChannelImpl has this: package sun.nio.ch; import java.io.FileDescriptor; import java.io.IOException; import java.nio.ByteBuffer; import java.nio.MappedByteBuffer; import java.nio.channels.*; import java.util.ArrayList; import java.util.List; import java.security.AccessController; import sun.misc.Cleaner; import sun.security.action.GetPropertyAction; public class FileChannelImpl extends FileChannel { //... private FileChannelImpl(FileDescriptor fd, boolean readable, boolean writable, boolean append, Object parent) { this .fd = fd; this .readable = readable; this .writable = writable; this .append = append; this .parent = parent; this .nd = new FileDispatcherImpl(append); } //... public static FileChannel open(FileDescriptor fd, boolean readable, boolean writable, boolean append, Object parent) { return new FileChannelImpl(fd, readable, writable, append, parent); } //... protected void implCloseChannel() throws IOException { // Release and invalidate any locks that we still hold if (fileLockTable != null ) { for (FileLock fl: fileLockTable.removeAll()) { synchronized (fl) { if (fl.isValid()) { nd.release(fd, fl.position(), fl.size()); ((FileLockImpl)fl).invalidate(); } } } } nd.preClose(fd); threads.signalAndWait(); if (parent != null ) { // Close the fd via the parent stream's close method. The parent // will reinvoke our close method, which is defined in the // superclass AbstractInterruptibleChannel, but the isOpen logic in // that method will prevent this method from being reinvoked. // ((java.io.Closeable)parent).close(); } else { nd.close(fd); } } //... It looks to me like the FileDescriptor is not closed unless close() (defined by java.nio.channels.spi.AbstractInterruptibleChannel , which calls implCloseChannel() to do the actual closing) is called. What am I missing?

Stephen Connolly added a comment - 2018-01-03 10:35 So the issue with `FileInputStream` and `FileOutputStream` is that they do not release the file handle until finalization even if you close them. What we have done here is ensure that almost all cases will close the stream immediately (there are some cases where we have to pass the stream on to another method / thread and hence we could still leak if analysis of the transfer of ownership is incorrect) thus the switch should fix the case of dangling unclosed file handles. The reports I saw from Hadoop indicated that these new methods used a reference queue to perform tidy up rather than relying on a finalizer... if that proves to be a rumour then so be it, and we would have to assume that leaks from the new code paths should therefore be even easier to catch (as they should accumulate faster) and therefore fixing should be easier too

Daniel Trebbien added a comment - 2018-01-13 20:46 There might be an issue with using Files.newOutputStream() in hudson.util.StreamTaskListener(File out, Charset charset) because the operating system file handle might not be closed. For example, hudson.model.TaskThread.ListenerAndText.forFile(File f, TaskAction context) creates a StreamTaskListener for the file, but there is no way to close the StreamTaskListener . I did a "Find Usages" in NetBeans for the ListenerAndText.forFile() APIs, and it appears that they are not being used.