Amazon S3 预签名 URL 转义密钥中的斜杠

Question

我正在使用 Java Amazon SDK 与 S3 一起使用来存储上传的文件。我想保留原始文件名，并将其放在密钥的末尾，但我也在使用虚拟目录结构 - 类似于<dirname>/<uuid>/<originalFilename>。

问题是，当我想使用 api 生成预签名 URL 进行下载时：

URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
return url.toExternalForm();

sdk url 转义了整个密钥，包括斜杠。虽然它仍然有效，但这意味着下载的文件的名称包括整个密钥，而不是最后的原始文件名位。我知道在不转义斜杠的情况下应该可以做到这一点，但我试图避免重写 SDK 中已有的大量代码。有一个通用的解决方案吗？我知道我使用的网络应用程序遵循相同的模式并且没有斜线转义问题。

Answer 1

这是当前 Java SDK 中的一个错误：

如果你看https://github.com/aws/aws-sdk-java/blob/master/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L2820

内部调用的presignRequest方法代码如下：

    String resourcePath = "/" +
        ((bucketName != null) ? bucketName + "/" : "") +
        ((key != null) ? ServiceUtils.urlEncode(key) : "") +
        ((subResource != null) ? "?" + subResource : "");

密钥是在签名之前在这里编码的 URL，我认为这是错误的。

您也许可以从 AmazonS3Client 继承并覆盖该函数来解决此问题。

在某些地方，建议使用 url.getQuery() 并在其前面加上您的原始 awsURL (https://forums.aws.amazon.com/thread.jspa?messageID=356271)。但是，正如您自己所说，这会产生错误，因为资源密钥与签名不匹配。

以下问题也可能与此有关，我没有查看建议的解决方法：

How to generate pre-signed Amazon S3 url for a vanity domain, using amazon sdk?

亚马逊之前发现并修复了一个类似的错误： https://forums.aws.amazon.com/thread.jspa?messageID=418537

所以我希望它会在下一个版本中修复。

Answer 2

我仍然希望有一个比这更好的解决方案，但是看到@aKzenT 已经证实了我的结论，即我写了一个没有现成的解决方案。它只是 AmazonS3Client 的一个简单子类。我担心它很脆弱，因为我不得不从我覆盖的方法中复制很多代码，但这似乎是最简单的解决方案。我可以确认它在我自己的代码库中运行良好。我将代码发布在gist，但为了完整的答案：

import com.amazonaws.AmazonWebServiceRequest;
import com.amazonaws.HttpMethod;
import com.amazonaws.Request;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.handlers.RequestHandler;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.Headers;
import com.amazonaws.services.s3.internal.S3QueryStringSigner;
import com.amazonaws.services.s3.internal.ServiceUtils;

import java.util.Date;

/**
 * This class should be a drop in replacement for AmazonS3Client as long as you use the single credential
 * constructor. It could probably be modified to add additional constructors if needed, but this is the one we use.
 * Supporting all of them didn't seem trivial because of some dependencies in the original presignRequest method.
 *
 * The only real purpose of this class is to change the behavior of generating presigned URLs. The original version
 * escaped slashes in the key and this one does not. Pretty url paths are kept intact.
 *
 * @author Russell Leggett
 */
public class PrettyUrlS3Client extends AmazonS3Client{
    private AWSCredentials awsCredentials;

    /**
     * This constructor is the only one provided because it is only one I needed, and it
     * retains awsCredentials which might be needed in the presignRequest method
     *
     * @param awsCredentials
     */
    public PrettyUrlS3Client(AWSCredentials awsCredentials) {
        super(awsCredentials);
        this.awsCredentials = awsCredentials;
    }

    /**
     * WARNING: This method is an override of the AmazonS3Client presignRequest
     * and copies most of the code. Should be careful of updates to the original.
     *
     * @param request
     * @param methodName
     * @param bucketName
     * @param key
     * @param expiration
     * @param subResource
     * @param <T>
     */
    @Override
    protected <T> void presignRequest(Request<T> request, HttpMethod methodName, String bucketName, String key, Date expiration, String subResource) {

        // Run any additional request handlers if present
        if (requestHandlers != null) {
            for (RequestHandler requestHandler : requestHandlers) {
                requestHandler.beforeRequest(request);
            }
        }
        String resourcePath = "/" +
                ((bucketName != null) ? bucketName + "/" : "") +
                ((key != null) ? keyToEscapedPath(key)/* CHANGED: this is the primary change */ : "") +
                ((subResource != null) ? "?" + subResource : "");

        //the request apparently needs the resource path without a starting '/'
        request.setResourcePath(resourcePath.substring(1));//CHANGED: needed to match the signature with the URL generated from the request
        AWSCredentials credentials = awsCredentials;
        AmazonWebServiceRequest originalRequest = request.getOriginalRequest();
        if (originalRequest != null && originalRequest.getRequestCredentials() != null) {
            credentials = originalRequest.getRequestCredentials();
        }

        new S3QueryStringSigner<T>(methodName.toString(), resourcePath, expiration).sign(request, credentials);

        // The Amazon S3 DevPay token header is a special exception and can be safely moved
        // from the request's headers into the query string to ensure that it travels along
        // with the pre-signed URL when it's sent back to Amazon S3.
        if (request.getHeaders().containsKey(Headers.SECURITY_TOKEN)) {
            String value = request.getHeaders().get(Headers.SECURITY_TOKEN);
            request.addParameter(Headers.SECURITY_TOKEN, value);
            request.getHeaders().remove(Headers.SECURITY_TOKEN);
        }
    }

    /**
     * A simple utility method which url escapes an S3 key, but leaves the
     * slashes (/) unescaped so they can stay part of the url.
     * @param key
     * @return
     */
    public static String keyToEscapedPath(String key){
        String[] keyParts = key.split("/");
        StringBuilder result = new StringBuilder();
        for(String part : keyParts){
            if(result.length()>0){
                result.append("/");
            }
            result.append(ServiceUtils.urlEncode(part));
        }
        return result.toString().replaceAll("%7E","~");
    }
}

更新我更新了要点和这段代码来解决我在使用 ~ 时遇到的问题。即使使用标准客户端也会发生这种情况，但是取消转义 ~ 修复了它。请参阅要点了解更多详细信息/跟踪我可能做出的任何进一步更改。

Answer 3

Java SDK 1.4.3 版似乎已经解决了这个问题。也许，它早先已修复，但我可以确认它在 1.4.3 中正常工作。