通过 GPU 读取多个 .dat 文件

Question

我了解通过 GPU 读取文件是一项低效的任务，因为它面对的是系统中最慢的部分，即 IO。但是，我想出了另一种方法，即使用 CPU 读取文件并让 GPU 处理处理负担。我用 C++ 编写了以下代码，但我被困在集成点，即如何让 GPU 在 CPU 读取这些文件后处理这些文件。换句话说，C++-amp 与代码添加和集成的出发点是什么？还是我应该从头开始重写整个代码？

{/* this code to read multiple .dat files from the directory that contains the implementation (from my account of stackoverflow) */

#include <Windows.h>
#include <ctime>
#include <stdint.h>
#include <iostream>
using std::cout;
using std::endl;
#include <fstream>
using std::ifstream;

#include <cstring>

/* Returns the amount of milliseconds elapsed since the UNIX epoch. Works on both
 * windows and linux. */

uint64_t GetTimeMs64()
{


 FILETIME ft;
 LARGE_INTEGER li;

 /* Get the amount of 100 nano seconds intervals elapsed since January 1, 1601 (UTC) and copy it
  * to a LARGE_INTEGER structure. */
 GetSystemTimeAsFileTime(&ft);
 li.LowPart = ft.dwLowDateTime;
 li.HighPart = ft.dwHighDateTime;

 uint64_t ret;
 ret = li.QuadPart;
 ret -= 116444736000000000LL; /* Convert from file time to UNIX epoch time. */
 ret /= 10000; /* From 100 nano seconds (10^-7) to 1 millisecond (10^-3) intervals */

 return ret;

}


const int MAX_CHARS_PER_LINE = 512;
const int MAX_TOKENS_PER_LINE = 20;
const char* const DELIMITER = "|";

int main()
{
  // create a file-reading object
  uint64_t a = GetTimeMs64();
  cout << a << endl;
  HANDLE h;
WIN32_FIND_DATA find_data;
h = FindFirstFile( "*.dat", & find_data );
if( h == INVALID_HANDLE_VALUE ) {
    cout<<"error"<<endl;

}
do {
    char * s = find_data.cFileName;

    ifstream fin;
  fin.open(s); // open a file
  if (!fin.good()) 
    return 1; // exit if file not found

  // read each line of the file
  while (!fin.eof())
  {
    // read an entire line into memory
    char buf[MAX_CHARS_PER_LINE];
    fin.getline(buf, MAX_CHARS_PER_LINE);

    // parse the line into blank-delimited tokens
    int n = 0; // a for-loop index

    // array to store memory addresses of the tokens in buf
    const char* token[MAX_TOKENS_PER_LINE] = {}; // initialize to 0

    // parse the line
    token[0] = strtok(buf, DELIMITER); // first token
    if (token[0]) // zero if line is blank
    {
      for (n = 1; n < MAX_TOKENS_PER_LINE; n++)
      {
    token[n] = strtok(0, DELIMITER); // subsequent tokens
        if (!token[n]) break; // no more tokens
  }
}

    // process (print) the tokens
    for (int i = 0; i < n; i++) // n = #of tokens
      cout << "Token[" << i << "] = " << token[i] << endl;
    cout << endl;
  }
            // Your code here
} while( FindNextFile( h, & find_data ) );
FindClose( h );



  uint64_t b = GetTimeMs64();
  cout << a << endl;
  cout << b << endl;
  uint64_t c = b - a;
  cout << c << endl;

  system("pause");
}

Answer 1

没有办法为 GPU 处理文件。正如您假设的 CPU 处理 IO。 因此，您需要将读取的信息存储在内存中，将其发送到 GPU，在那里进行计算等等。

处理文件的一种好方法是归档（使用 GPU）您的信息。

所以你用 CPU 读取文件，用 GPU 提取 > 计算 > 存档，然后用 CPU 存储它。

更新。

(CPU IO READ from file (should be already archived information))  to -> main memory
(CPU SEND) to -> GPU global memory from main memory

(GPU EXTRACT (if archived))
(GPU COMPUTE (your work here))
(GPU ARCHIVE)

(CPU RETRIEVE) to -> main memory from GPU global memory
(CPU IO WRITE to file)