Java AmazonS3 putObject 静默失败

Question

其他人在没有收到答案的情况下发布了这个问题，现在我遇到了同样的问题。实际上它已经持续了 9 个月，但我现在才注意到它。

此序列不抛出异常，并记录最后的消息：

AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
String bucket = "...";
String key = "...";
File f = new File("...");
PutObjectResult r = s3.putObject(bucket, key, f);
String etag = r.getETag();
LOGGER.info("file ... saved with etag = "+etag);

当我查看时，该文件不在存储桶中。

在发布的数千个文件中，每天都会失败几十次。有 25 个活动线程使用此代码序列。 aws-java-sdk 线程安全吗？其他想法？

这是在亚马逊云中的一个 ec2 实例上运行的。

详情：

aws-java-sdk-s3-1.11.693.jar
java: 1.8.0_201-b09
ubuntu: 4.4.0-1077-aws

Answer 1

我在使用 AmazonS3.putObject 上传 27MB 文件时看到了这个问题。该命令返回了PutObjectResult，但预期位置没有 S3 对象，结果元数据 (result.getMetadata().getContentLength()) 显示 0 字节。我通过使用 this link 的分段上传解决了这个问题。

private static final Logger LOGGER = LoggerFactory.getLogger(S3Handler.class);
private static final long MAX_SINGLE_PART_UPLOAD_BYTES = 5 * 1024 * 1024;

private final AmazonS3 amazonS3;

public S3Handler(AmazonS3 amazonS3) {
    this.amazonS3 = amazonS3;
}

public void putS3Object(String bucket, String objectKey, File file) {
    if (file.length() <= MAX_SINGLE_PART_UPLOAD_BYTES) {
        putS3ObjectSinglePart(bucket, objectKey, file);
    } else {
        putS3ObjectMultiPart(bucket, objectKey, file);
    }
}

private void putS3ObjectSinglePart(String bucket, String objectKey, File file) {
    PutObjectRequest request = new PutObjectRequest(bucket, objectKey, file);
    PutObjectResult result = amazonS3.putObject(request);
    long bytesPushed = result.getMetadata().getContentLength();
    LOGGER.info("Pushed {} bytes to s3://{}/{}", bytesPushed, bucket, objectKey);
}

private void putS3ObjectMultiPart(String bucket, String objectKey, File file) {
    long contentLength = file.length();
    long partSize = MAX_SINGLE_PART_UPLOAD_BYTES;
    List<PartETag> partETags = new ArrayList<>();

    // Initiate the multipart upload.
    InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucket, objectKey);
    InitiateMultipartUploadResult initResponse = amazonS3.initiateMultipartUpload(initRequest);

    // Upload the file parts.
    long fileOffset = 0;
    for (int partNumber = 1; fileOffset < contentLength; ++partNumber) {
        // Because the last part could be less than 5 MB, adjust the part size as needed.
        partSize = Math.min(partSize, (contentLength - fileOffset));

        // Create the request to upload a part.
        UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(bucket)
                .withKey(objectKey)
                .withUploadId(initResponse.getUploadId())
                .withPartNumber(partNumber)
                .withFileOffset(fileOffset)
                .withFile(file)
                .withPartSize(partSize);

        // Upload the part and add the response's ETag to our list.
        UploadPartResult uploadResult = amazonS3.uploadPart(uploadRequest);
        LOGGER.info("Uploading part {} of Object s3://{}/{}", partNumber, bucket, objectKey);
        partETags.add(uploadResult.getPartETag());

        fileOffset += partSize;
    }

    // Complete the multipart upload.
    CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucket, objectKey, initResponse.getUploadId(), partETags);
    amazonS3.completeMultipartUpload(compRequest);
}