将大文件上传到 Amazon S3 时出现问题答案

【问题标题】：Problems when uploading large files to Amazon S3将大文件上传到 Amazon S3 时出现问题
【发布时间】：2011-06-09 14:10:40
【问题描述】：

我尝试使用 Amazon-SDK(Java) 示例代码 S3TransferProgressSample.java 将大文件上传到 Amazon-S3 存储 (also posted here on AWS docs)。

但是当我尝试上传 11 GB 的文件时，上传卡在不同的点并显示错误消息：

Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot).

在 IOException 发生后，SDK 似乎无法重试请求（见下文）。

有人遇到过吗？解决此问题的最佳做法是什么？任何代码表示赞赏。

 INFO: Received successful response: 200, AWS Request ID:
 2B66E7669E24DA75<br> Jan 15, 2011 6:44:46 AM
 com.amazonaws.http.HttpClient execute<br> INFO: Sending Request: PUT
 s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
 (uploadId:
     m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )<br> Jan 15, 2011 6:45:10 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
     Jan 15, 2011 6:45:10 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     INFO: Retrying request<br> Jan 15, 2011 6:45:12 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:12 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
     Jan 15, 2011 6:45:12 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
     Jan 15, 2011 6:45:13 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
     com.amazonaws.http.HttpClient execute<br>
     **WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.**<br> Jan 15, 2011 6:45:14 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:14 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:16 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:16 AM

 com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:17 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:19 AM
     com.amazonaws.http.HttpClient execute<br> ....<br> Jan 15, 2011
     6:45:21 AM com.amazonaws.http.HttpClient handleResponse<br>
     **INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007**<br> Jan 15, 2011 6:45:21 AM
     com.amazonaws.http.HttpClient execute<br> ...<br> Jan 15, 2011 6:45:19
     AM com.amazonaws.http.HttpClient execute<br> INFO: Sending Request:
     DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt
     Parameters:<br> ...<br> Jan 15, 2011 6:47:01 AM
     com.amazonaws.http.HttpClient handleErrorResponse<br> INFO: Received
     error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595,
     AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload
     does not exist. The upload ID may be invalid, or the upload may have
     been aborted or completed.<br>

【问题讨论】：

标签： java service amazon-web-services amazon-s3

【解决方案1】：

尝试使用low level API。

这将使您在出现问题时获得更多控制权，因为它们很可能与 11GB 文件有关。

与 S3 之间的请求有时会失败。使用低级 API，如果上传失败，您将能够重试部分上传。

稍微重构一下亚马逊文档中的示例：

// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
    // Last part can be less than 5 MB. Adjust part size.
    partSize = Math.min(partSize, (contentLength - filePosition));

    // Create request to upload a part.
    UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(existingBucketName).withKey(keyName)
                .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                .withFileOffset(filePosition)
                .withFile(file)
                .withPartSize(partSize);

    // repeat the upload until it succeeds.
    boolean anotherPass;  
        do {
              anotherPass = false;  // assume everythings ok
              try {
                  // Upload part and add response to our list.
                  partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
              } catch (Exception e) {
                    anotherPass = true; // repeat
              }
        } while (anotherPass);

     filePosition += partSize;
}

   // Step 3: complete.
   CompleteMultipartUploadRequest compRequest = new 
                     CompleteMultipartUploadRequest(
                                existingBucketName, 
                                keyName, 
                                initResponse.getUploadId(), 
                                partETags);

   s3Client.completeMultipartUpload(compRequest);

注意：我不是 Java 开发人员，所以我可能在语法上搞砸了，但希望这能让你朝着正确的方向前进。此外，您还需要添加一个“重试计数器”，以防止在上传反复失败时出现无限循环。

【讨论】：

好的，您能否提供一段代码如何在失败时重试部分上传？
@Alexey - 我添加了一个示例，希望能让您走上正轨。基本思想是在失败时重试一部分，而不是仅仅取消整个上传。
谢谢，这解决了我的问题。一个好主意是同时使用多个线程来加快上传速度
@Alexey - 太好了。乐于助人：）是的，现在能够使用多个线程将是一个巨大的胜利。如您所见，以前上传到 S3 的速度很慢且不可靠。

【解决方案2】：

附带说明，如果您尝试对已在分段上传下的密钥进行分段上传，则可能会引发 404 错误。

【讨论】：

【解决方案3】：

我认为您应该尝试 AWS 支持的 Multipart API。

看看这个：http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html

【讨论】：

【解决方案4】：

Geoff Appleford 的答案对我有用。但是，我会在 while 循环控制语句中添加一个 && retryCount

航空

【讨论】：

【解决方案5】：

我想对 Geoff Appleford 的回答添加评论，但我不允许这样做。一般来说，他对使用低级 API 的回答很好，但即使我们现在有一个 do-while 循环，for 循环的设计方式也有内置的重试逻辑。在他的代码 sn-p 中，文件位置仅在成功时才会增加，否则您将再次上传相同的部分。

【讨论】：