如何使用 Node.js SDK 将 Azure Blob Storage 中的 Append Blob 读取到字符串中？

Question

我正在按照https://github.com/Azure/azure-storage-js/blob/master/blob/samples/basic.sample.js 此处的示例使用 Node.js SDK 从 Azure Blob 存储读取 Blob 到字符串。

我正在尝试读取的 blob 是 Append Blob。

首先将流读入字符串需要很长时间，最后我得到一个 HTTP 412 错误。

我也在这里问过这个问题：https://github.com/Azure/azure-storage-js/issues/51

我正在使用 Node.js v10.14.1 执行此操作，我使用的 SDK 是 @azure/storage-blob@10.3.0。

我的代码在这里：

const {
  Aborter,
  BlobURL,
  ContainerURL,
  SharedKeyCredential,
  ServiceURL,
  StorageURL,
} = require('@azure/storage-blob');
const format = require('date-fns/format');

async function streamToString(readableStream) {
  return new Promise((resolve, reject) => {
    const chunks = [];
    readableStream.on('data', (data) => {
      chunks.push(data.toString());
    });
    readableStream.on('end', () => {
      resolve(chunks.join(''));
    });
    readableStream.on('error', reject);
  });
}

async function run() {
  const accountName = 'xxxstor';
  const accountKey = 'omitted';
  const credential = new SharedKeyCredential(accountName, accountKey);
  const pipeline = StorageURL.newPipeline(credential);
  const serviceURL = new ServiceURL(
    `https://${accountName}.blob.core.windows.net`,
    pipeline
  );
  const containerName = 'request-logs';
  const containerURL = ContainerURL.fromServiceURL(serviceURL, containerName);
  const blobName = `${format(new Date(), 'YYYY-MM-DD[.txt]')}`;
  const blobURL = BlobURL.fromContainerURL(containerURL, blobName);
  console.log('Downloading blob...');
  const response = await blobURL.download(Aborter.none, 0);
  console.log('Reading response to string...');
  const body = await streamToString(response.);
  console.log(body.length);
}

run().catch((err) => {
  console.error(err);
});

我得到的错误是这样的：

{ Error: Unexpected status code: 412
    at new RestError (C:\projects\xxx\RequestLogViewer\node_modules\@azure\ms-rest-js\dist\msRest.node.js:1397:28)
    at C:\projects\xxx\RequestLogViewer\node_modules\@azure\ms-rest-js\dist\msRest.node.js:1849:37
    at process._tickCallback (internal/process/next_tick.js:68:7)
  code: undefined,
  statusCode: 412,
  request:
  WebResource {
    streamResponseBody: true,
    url:
      'https://xxxstor.blob.core.windows.net/request-logs/2019-01-04.txt',
    method: 'GET',
    headers: HttpHeaders { _headersMap: [Object] },
    body: undefined,
    query: undefined,
    formData: undefined,
    withCredentials: false,
    abortSignal:
      a {
        _aborted: false,
        children: [],
        abortEventListeners: [Array],
        parent: undefined,
        key: undefined,
        value: undefined },
    timeout: 0,
    onUploadProgress: undefined,
    onDownloadProgress: undefined,
    operationSpec:
      { httpMethod: 'GET',
        path: '{containerName}/{blob}',
        urlParameters: [Array],
        queryParameters: [Array],
        headerParameters: [Array],
        responses: [Object],
        isXML: true,
        serializer: [Serializer] } },
  response:
  { body: undefined,
    headers: HttpHeaders { _headersMap: [Object] },
    status: 412 },
  body: undefined }

Answer 1

此问题已在 GitHub 问题 https://github.com/Azure/azure-storage-js/issues/51 将解决方案从 GitHub 问题复制到此处解决。

blobURL.download() 将尝试使用 HTTP Get 请求将 blob 下载到流中。当由于网络中断等导致流意外结束时，重试将使用新的 HTTP Get 请求恢复从断点读取的流。

第二个 HTTP 请求将使用条件标头 IfMatch 和在第一个请求中返回的 blob 的 ETag，以确保在第二次重试发生时 blob 不会更改。否则，将返回 412 条件头不匹配错误。这种严格的策略用于避免数据完整性问题，例如 blob 可能被其他人完全覆盖。但是，这种策略似乎可以避免您在重试发生时读取不断更新的日志文件。

虽然我不认为这是错误，但我们需要让这个场景适合您。请尝试以下解决方案：先对附加 blob 进行快照，然后从快照 blob 中读取

const blobURL = BlobURL.fromContainerURL(containerURL, blobName);
console.log('Downloading blob...');
const snapshotResponse = await blobURL.createSnapshot(Aborter.none);
const snapshotURL = blobURL.withSnapshot(snapshotResponse.snapshot);
const response = await snapshotURL.download(Aborter.none, 0);
console.log('Reading response to string...', snapshotURL.blobContext.length);
const body = await streamToString(response.readableStreamBody);