从一个文件读取一个字节块并写入另一个文件直到读取所有块？

Question

我正在做一个项目，我必须在其中完成一些文件读写任务。我必须一次从一个文件中读取 8 个字节并对该块执行一些操作，然后将该块写入第二个文件，然后重复循环，直到第一个文件每次都以 8 个字节的块完全读取，然后处理数据应该添加/附加到第二个。但是，在这样做的过程中，我遇到了一些问题。以下是我正在尝试的：

private File readFromFile1(File file1) {

    int offset = 0;
    long message= 0;

    try {
        FileInputStream fis = new FileInputStream(file1);
        byte[] data = new byte[8];
        file2 = new File("file2.txt");
        FileOutputStream fos = new FileOutputStream(file2.getAbsolutePath(), true);
        DataOutputStream dos = new DataOutputStream(fos);

        while(fis.read(data, offset, 8) != -1)
        {
            message = someOperation(data); // operation according to business logic
            dos.writeLong(message);
        }
        fos.close();
        dos.close();
        fis.close(); 
    } catch (IOException e) {
        System.out.println("Some error occurred while reading from File:" + e);
    }
    return file2;
}

我没有以这种方式获得所需的输出。任何帮助表示赞赏。

Answer 1

考虑以下代码：

private File readFromFile1(File file1) {

    int offset = 0;
    long message = 0;
    File file2 = null;

    try {
        FileInputStream fis = new FileInputStream(file1);

        byte[] data = new byte[8]; //Read buffer
        byte[] tmpbuf = new byte[8]; //Temporary chunk buffer

        file2 = new File("file2.txt");
        FileOutputStream fos = new FileOutputStream(file2.getAbsolutePath(), true);
        DataOutputStream dos = new DataOutputStream(fos);

        int readcnt; //Read count
        int chunk; //Chunk size to write to tmpbuf

        while ((readcnt = fis.read(data, 0, 8)) != -1) {

            //// POINT A ////
            //Skip chunking system if an 8 byte octet is read directly.
            if(readcnt == 8 && offset == 0){
                message = someOperation(tmpbuf); // operation according to business logic
                dos.writeLong(message);
                continue;
            }

            //// POINT B ////
            chunk = Math.min(tmpbuf.length - offset, readcnt); //Determine how much to add to the temp buf.

            System.arraycopy(data, 0, tmpbuf, offset, chunk); //Copy bytes to temp buf

            offset = offset + chunk; //Sets the offset to temp buf

            if (offset == 8) {
                message = someOperation(tmpbuf); // operation according to business logic
                dos.writeLong(message);

                if (chunk < readcnt) {
                    System.arraycopy(data, chunk, tmpbuf, 0, readcnt - chunk);
                    offset = readcnt - chunk;
                } else {
                    offset = 0;
                }
            }
        }

        //// POINT C ////
        //Process remaining bytes here...
        //message = foo(tmpbuf);
        //dos.writeLong(message);

        fos.close();
        dos.close();
        fis.close(); 
    } catch (IOException e) {
        System.out.println("Some error occurred while reading from File:" + e);
    }

    return file2;
}

在这段代码摘录中，我所做的是：

1. 修改您的读取代码以包含从 read() 方法实际读取的字节数（记为 readcnt）。
2. 添加了字节分块系统（在分块缓冲区中至少有 8 个字节之前不会进行处理）。
3. 允许单独处理最终字节（不构成 8 字节八位字节）。

从代码中可以看出，正在读取的数据首先存储在分块缓冲区（表示为 tmpbuf）中，直到至少有 8 个字节可用。仅当 8 个字节不总是可用时才会发生这种情况（如果 8 个字节直接可用并且没有分块，则直接处理。参见代码中的“点 A”）。这是作为一种优化形式来完成的，以防止过多的数组副本。

分块系统使用偏移量，每次将字节写入 tmpbuf 时，偏移量都会增加，直到达到 8 值（它不会继续，因为分配“块”时使用的 Math.min() 方法会限制该值）。在offset == 8时，继续执行处理代码。

如果该特定读取产生的字节数比实际处理的多，则继续将它们从头开始写入 tmpbuf，同时适当地设置偏移量，否则将偏移量设置为 0。

重复循环。

代码会将最后几个字节的数据保留在数组 tmpbuf 中的八位字节中，其中偏移变量指示实际写入了多少。然后可以在 C 点单独处理这些数据。

似乎比它应该的要复杂得多，并且可能有更好的解决方案（可能使用现有的 java 库方法），但在我的脑海中，这就是我得到的。希望这足够清楚，您可以理解。

Answer 2

您可以使用以下内容，它使用NIO，尤其是ByteBuffer 类来处理long。你当然可以用标准的 java 方式来实现它，但由于我是NIO 的粉丝，这里有一个可能的解决方案。

您的代码中的主要问题是while(fis.read(data, offset, 8) != -1) 将读取最多 8 个字节，并不总是 8 个字节，而且读取这么小的部分效率不高。

我在我的代码中添加了一些 cmets，如果有不清楚的地方请发表评论。我的someOperation(...) 函数只是从缓冲区复制下一个long 值。

更新：

添加 finally 块来关闭文件。

import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;

public class TestFile {

  static final int IN_BUFFER_SIZE = 1024 * 8;
  static final int OUT_BUFFER_SIZE = 1024 *9; // make the out-buffer > in-buffer, i am lazy and don't want to check for overruns
  static final int MIN_READ_BYTES = 8;
  static final int MIN_WRITE_BYTES = 8;

  private File readFromFile1(File inFile) {

    final File outFile = new File("file2.txt");

    final ByteBuffer inBuffer = ByteBuffer.allocate(IN_BUFFER_SIZE);
    final ByteBuffer outBuffer = ByteBuffer.allocate(OUT_BUFFER_SIZE);

    FileChannel readChannel = null;
    FileChannel writeChannel = null;
    try {
      // open a file channel for reading and writing
      readChannel = FileChannel.open(inFile.toPath(), StandardOpenOption.READ);
      writeChannel = FileChannel.open(outFile.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);

      long totalReadByteCount = 0L;
      long totalWriteByteCount = 0L;

      boolean readMore = true;
      while (readMore) {

        // read some bytes into the in-buffer
        int readOp = 0;
        while ((readOp = readChannel.read(inBuffer)) != -1) {
          totalReadByteCount += readOp;
        } // while

        // prepare the in-buffer to be consumed
        inBuffer.flip();

        // check if there where errors
        if (readOp == -1) {
          // end of file reached, read no more
          readMore = false;
        } // if

        // now consume the in-buffer until there are at least MIN_READ_BYTES in the buffer
        while (inBuffer.remaining() >= MIN_READ_BYTES) {
          // add data to the write buffer
          outBuffer.putLong(someOperation(inBuffer));
        } // while

        // compact the in-buffer and prepare for the next read, if we need to read more.
        // that way the possible remaining bytes of the in-buffer can be consumed after leaving the loop
        if (readMore) inBuffer.compact();

        // prepare the out-buffer to be consumed
        outBuffer.flip();

        // write the out-buffer until the buffer is empty
        while (outBuffer.hasRemaining())
          totalWriteByteCount += writeChannel.write(outBuffer);

        // prepare the out-buffer for writing again
        outBuffer.flip();
      } // while

      // error handling
      if (inBuffer.hasRemaining()) {
        System.err.println("Truncated data! Not a long value! bytes remaining: " + inBuffer.remaining());
      } // if

      System.out.println("read total: " + totalReadByteCount + " bytes.");
      System.out.println("write total: " + totalWriteByteCount + " bytes.");

    } catch (IOException e) {
      System.out.println("Some error occurred while reading from File: " + e);
    } finally {
      if (readChannel != null) {
        try {
          readChannel.close();
        } catch (IOException e) {
          System.out.println("Could not close read channel: " + e);
        } // catch
      } // if

      if (writeChannel != null) {
        try {
          writeChannel.close();
        } catch (IOException e) {
          System.out.println("Could not close write channel: " + e);
        } // catch
      } // if
    } // finally

    return outFile;
  }

  private long someOperation(ByteBuffer bb) {
    // consume the buffer, do whatever you want with the buffer.
    return bb.getLong(); // consumes 8 bytes of the buffer.
  }


  public static void main(String[] args) {
    TestFile testFile = new TestFile();
    File source = new File("input.txt");
    testFile.readFromFile1(source);
  }

}