Files.newOutputStream().write()使用direct memory

(Updated: )

Files.newOutputStream()的优点

如果你使用IDEA写代码的话, 当你写下new FileOutputStream(file)时, IDE会提示你修改为Files.newOutputStream(file.path), 具体的描述在Settings-Editor-Inspections-Java-Performance中

Reports new FileInputStream() or new FileOutputStream() expressions that can be replaced with Files.newInputStream() or Files.newOutputStream() calls respectively. The streams created using Files methods are usually more efficient than those created by stream constructors.

那么, usually more efficient的点在哪里呢?

java - Files.newOutputStream vs FileOutputStream - Stack Overflow

  1. 旧的方法有finalize(), 在GC后、finalize()调用前, 对象会处于未回收状态

Is java.nio.file.Files.newInputStream(myfile.toPath()) better than new FileInputStream(file)? - Stack Overflow

FileInputStream / FileOutputStream Considered Harmful

  1. windows系统下, 新方法打开文件会添加FILE_SHARE_DELETE选项

Consider replacing FileOutputStream with Files.newOutputStream · Issue #2117 · apache/logging-log4j2

查看源码

以下均基于Java8, 省略读写选项等逻辑

1
2
3
4
// Files
public static OutputStream newOutputStream(Path path, OpenOption... options) throws IOException {
    return provider(path).newOutputStream(path, options);
}

不同平台会提供不同的FileSystemProvider, MacOS是MacOSXFileSystemProvider, Linux是LinuxFileSystemProvider, 对于newOutputStream方法均没有单独的实现, 生效的是FileSystemProvider#newOutputStream

1
2
3
4
// FileSystemProvider
public OutputStream newOutputStream(Path path, OpenOption... options) throws IOException {
    return Channels.newOutputStream(newByteChannel(path, opts));
}

newByteChannel的实现是UnixFileSystemProvider#newByteChannel, 最终返回一个FileChannelImpl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// UnixFileSystemProvider
public SeekableByteChannel newByteChannel(Path obj, Set<? extends OpenOption> options, FileAttribute<?>... attrs) throws IOException {
    UnixPath file = UnixPath.toUnixPath(obj);
    return UnixChannelFactory.newFileChannel(file, options, mode);
}

// UnixChannelFactory
static FileChannel newFileChannel(UnixPath path, Set<? extends OpenOption> options, int mode) throws UnixException {
    return newFileChannel(-1, path, null, options, mode);
}

// UnixChannelFactory
static FileChannel newFileChannel(int dfd, UnixPath path, String pathForPermissionCheck, Set<? extends OpenOption> options, int mode) throws UnixException {
    FileDescriptor fdObj = open(dfd, path, pathForPermissionCheck, flags, mode);
    return FileChannelImpl.open(fdObj, path.toString(), flags.read, flags.write, flags.append, null);
}

newOutputStream的实现是Channels#newOutputStream, 返回了一个匿名内部类OutputStream, write时将byte[]转为ByteBuffer, 使用channel的锁

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
// Channels
public static OutputStream newOutputStream(final WritableByteChannel ch) {
    checkNotNull(ch, "ch");

    return new OutputStream() {

            private ByteBuffer bb = null;
            private byte[] bs = null;       // Invoker's previous array
            private byte[] b1 = null;

            public synchronized void write(int b) throws IOException {
                if (b1 == null)
                    b1 = new byte[1];
                b1[0] = (byte)b;
                this.write(b1);
            }

            public synchronized void write(byte[] bs, int off, int len)
                throws IOException
            {
                if ((off < 0) || (off > bs.length) || (len < 0) ||
                    ((off + len) > bs.length) || ((off + len) < 0)) {
                    throw new IndexOutOfBoundsException();
                } else if (len == 0) {
                    return;
                }
                ByteBuffer bb = ((this.bs == bs)
                                    ? this.bb
                                    : ByteBuffer.wrap(bs));
                bb.limit(Math.min(off + len, bb.capacity()));
                bb.position(off);
                this.bb = bb;
                this.bs = bs;
                Channels.writeFully(ch, bb);
            }

            public void close() throws IOException {
                ch.close();
            }

        };
}

// Channels
private static void writeFully(WritableByteChannel ch, ByteBuffer bb) throws IOException {
    if (ch instanceof SelectableChannel) {
        SelectableChannel sc = (SelectableChannel)ch;
        synchronized (sc.blockingLock()) {
            if (!sc.isBlocking())
                throw new IllegalBlockingModeException();
            writeFullyImpl(ch, bb);
        }
    } else {
        writeFullyImpl(ch, bb);
    }
}

// Channels
private static void writeFullyImpl(WritableByteChannel ch, ByteBuffer bb) throws IOException {
    while (bb.remaining() > 0) {
        int n = ch.write(bb);
        if (n <= 0)
            throw new RuntimeException("no bytes written");
    }
}

实际调用的是FileChannelImpl#write, 最终使用IOUtil#write, 即direct memory进行写入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
// FileChannelImpl
public int write(ByteBuffer src) throws IOException {
    ensureOpen();
    synchronized (positionLock) {
        int n = 0;
        int ti = -1;
        try {
            begin();
            ti = threads.add();
            if (!isOpen())
                return 0;
            do {
                n = IOUtil.write(fd, src, -1, nd);
            } while ((n == IOStatus.INTERRUPTED) && isOpen());
            return IOStatus.normalize(n);
        } finally {
            threads.remove(ti);
            end(n > 0);
            assert IOStatus.check(n);
        }
    }
}

// IOUtil
static int write(FileDescriptor fd, ByteBuffer src, long position, NativeDispatcher nd) throws IOException {
    if (src instanceof DirectBuffer)
        return writeFromNativeBuffer(fd, src, position, nd);

    // Substitute a native buffer
    int pos = src.position();
    int lim = src.limit();
    assert (pos <= lim);
    int rem = (pos <= lim ? lim - pos : 0);
    ByteBuffer bb = Util.getTemporaryDirectBuffer(rem);
    try {
        bb.put(src);
        bb.flip();
        // Do not update src until we see how many bytes were written
        src.position(pos);

        int n = writeFromNativeBuffer(fd, bb, position, nd);
        if (n > 0) {
            // now update src
            src.position(pos + n);
        }
        return n;
    } finally {
        Util.offerFirstTemporaryDirectBuffer(bb);
    }
}

Direct Memory

对于Java堆内存的介绍Java memory management - Azure Spring Apps | Microsoft Learn, 注意只有FullGC时Direct Memory才会被回收

配置

Java8的文档中搜索MaxDirectMemorySize

-XX:MaxDirectMemorySize=size

java - Default for XX:MaxDirectMemorySize - Stack Overflow不设置默认值的话direct memory的大小与-Xmx=size有关

查看

Is there a way to measure direct memory usage in Java? - Stack Overflow

使用ManagementFactory获取BufferPoolMXBean, 读取BufferPoolMXBean的内存信息

LLM的回答

1
long maxDirectMemory = sun.misc.VM.maxDirectMemory();