使用 DropZone.js 通过 signedUrl 将 MP3 上传到 Google Cloud Storage 已损坏答案

【问题标题】：Uploading an MP3 using DropZone.js to Google Cloud Storage via signedUrl is corrupted使用 DropZone.js 通过 signedUrl 将 MP3 上传到 Google Cloud Storage 已损坏
【发布时间】：2020-06-26 20:06:54
【问题描述】：

我通过 Google 的 getSignedUrl 调用创建“签名上传网址”，将 mp3 上传到存储桶：

const { Storage } = require('@google-cloud/storage');
const bucketName = 'uploads';
const storage = new Storage();
const uploadBucket = storage.bucket(bucketName);
const config = require('config');

const EXPIRES_SECONDS = 600;

exports.generateUploadLink = async (req, res) => {
  let contentId = req.query.contentId || req.body.contentId;

  let file = uploadBucket.file(`${contentId}.mp3`);

  try {
    let signedUrl = await file.getSignedUrl({
      action: 'write',
      expires: Date.now() + (EXPIRES_SECONDS * 1000),
      contentType: 'audio/mpeg'
    });
    res.status(200).json({meta: { status: 'OK'});
  } catch(err) {
    console.log(`Error while getting signed url: ${err}`);
    res.status(500).json({meta: { status: 'FAIL', message: `Could not generate signed upload url for: ${contentId}`, error: err}});
  }
  return;
};

然后我们使用Dropzone 像这样上传它：

    $('div#mp3DropzoneArea').dropzone({
      url: uploadUrl,
      previewTemplate: template,
      autoProcessQueue: false,
      createImageThumbnails: false,
      method: 'put',
      clickable: this.get('clickElementSelector'),
      filesizeBase: 1024,
      maxFilesize: 250,  // MB
      acceptedFiles: '.mp3',
      dictDefaultMessage: this.get('defaultMessage'),
      dictInvalidFileType: 'Invalid file type.  Only .mp3 files can be imported.',
      parallelUploads: 1,
      maxFiles: this.get('maxFiles'),
      headers: {
        'Content-Type': 'audio/mpeg'
      },
...
   })

我面临的问题是，一旦文件上传，它往往会被损坏。现在，它仍然会播放，但会发生奇怪的事情，例如，如果我尝试搜索 10 秒，它会从头开始播放文件，并且所有内容都会提前 10 秒。

我对源文件和使用ffmpeg 上传后的文件进行了比较。结果如下：

源 MP3

$ ffmpeg -i uncover-iUpa1OtA-20200317.mp3
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.1 (GCC) 20200122
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mp3, from 'uncover-iUpa1OtA-20200317.mp3':
  Metadata:
    comment         : ∩╗┐From the '60s to the '90s, parents worried messages hidden in rock albums would make their children do drugs and worship the devil. The truth could only be revealed if these records were played backwards. Twenty Thousand Hertz ΓÇö a podcast about the w
                    :
                    :
    album           : Uncover
    title           : Bonus: Hidden Messages, Backmasking and the Satanic Panic
    artist          : Canadian Broadcasting Corporation
    track           : 1
    date            : 2020
  Duration: 00:29:17.86, start: 0.000000, bitrate: 129 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, fltp, 128 kb/s
    Stream #0:1: Video: mjpeg (Progressive), yuvj420p(pc, bt470bg/unknown/unknown), 1400x1400 [SAR 1:1 DAR 1:1], 90k tbr, 90k tbn, 90k tbc (attached pic)
    Metadata:
      comment         : Other

上传的 MP3

$ ffmpeg -i uncover-uploaded-51ebf5bb-e4ea-4372-b410-d90b04abec6a.mp3
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.1 (GCC) 20200122
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[mp3float @ 0000013aa370d800] Header missing
    Last message repeated 1 times
[mp3 @ 0000013aa370bdc0] Estimating duration from bitrate, this may be inaccurate
Input #0, mp3, from 'uncover-uploaded-51ebf5bb-e4ea-4372-b410-d90b04abec6a.mp3':
  Duration: 00:29:39.25, start: 0.000000, bitrate: 127 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, fltp, 128 kb/s
At least one output file must be specified

请注意“缺少标题”和持续时间警告。原版中的所有 id3 信息都消失了。

我检查了 Dropzone 向 uploadUrl 发出的请求（PUT），其格式为：

-----------------------------84271646821941633862998010702
Content-Disposition: form-data; name="file"; filename="uncover-iUpa1OtA-20200317.mp3"
Content-Type: audio/mpeg

<File data...>

谁能想到会发生这种情况的原因？

【问题讨论】：

我注意到您针对两个不同的文件名运行 ffmpeg。您确定要比较已上传然后重新下载的文件吗？另外，如果您对这两个文件运行 md5sum（或其他一些校验和程序），校验和是否一致？
它们确实是相同的，因为它们是相同的音频（第一个是“上传之前”，第二个是“上传之后”。我感觉校验和会有所不同，因为上传文件。

标签： google-api google-cloud-storage dropzone.js

【解决方案1】：

好的，想通了！ DropZone 正在发送包含 MP3 作为正文的序列化 FormData blob。但是，GCS signedUrl 只需要文件的内容。

例如，当我检查上传的 MP3 文件的内容时，我看到了：

解决此问题的方法是向我正在使用的名为 gcsUpload 的 Dropzone 实例添加一个新的 option。每当我上传到 GCS，并且我要设置文件的 url 时，我也设置了gcsUpload = true。

在我为 Dropzone 设置的 sending 事件处理程序中，我执行以下操作：

       dropzone.on('sending', function(file, xhr,/* formData*/) {
          // for Google Cloud Storage we don't want to send formData
          // as that information will be serialized into the file itself.
          // Instead, we only want to send the file to the signedUrl
          if (dropzone.options.gcsUpload) {
            let _send = xhr.send;
            xhr.send = function() {
              _send.call(xhr, file);
            }
          }

        });

这样做只会将file 数据发送到signedUrl，而不是formdata。

【讨论】：