AWS S3 预签名请求缓存

Question

我想将用户个人资料图片存储在 S3 存储桶中，但将这些图片保密。为了做到这一点，每当需要图像时，我都会创建一个预签名的 url。但是，这每次都会创建一个唯一的 url，这意味着图像永远不会被浏览器缓存，我最终会在 GET 请求中支付更多费用。

这是我生成 url 的代码示例，我使用的是 Laravel：

$s3 = \Storage::disk('s3');
$client = $s3->getDriver()->getAdapter()->getClient();
$expiry = new \DateTime('2017-07-25');

$command = $client->getCommand('GetObject', [
    'Bucket' => \Config::get('filesystems.disks.s3.bucket'),
    'Key'    => $key
]);

$request = $client->createPresignedRequest($command, $expiry);

return (string) $request->getUri();

我认为通过指定一个日期时间而不是一个时间单位，它会创建相同的 url，但实际上它会将剩余的秒数添加到 url，这是一个示例：

xxxx.s3.eu-west-2.amazonaws.com/profile-pics/92323.png?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential =AXXXXXXXXXXXX%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20170720T112123Z&X-Amz-SignedHeaders=host&X-Amz-Expires=391117&X-Amz-Signature=XXXXXXXXX

是否可以生成可重复的预签名请求 url，以便用户浏览器缓存图像？

Answer 1

也许回复晚了，但我会添加我的方法以供将来阅读本文的人使用。

要强制浏览器缓存启动，每次生成完全相同的 URL 非常重要，直到您特别希望浏览器从服务器重新加载内容。 不幸的是，sdk 中提供的 presigner 每次都依赖于当前时间戳导致一个新的 url。

这个例子是用 Java 编写的，但它可以很容易地扩展到其他语言

GetObjectRequest 构建器（用于创建预签名的 url）允许覆盖配置。我们可以提供一个自定义签名者来修改其行为

AwsRequestOverrideConfiguration.builder()
    .signer(new CustomAwsS3V4Signer())
    .credentialsProvider(<You may need to provide a custom credential provider 
here>)))
.build())

GetObjectRequest getObjectRequest =
    GetObjectRequest.builder()
            .bucket(getUserBucket())
            .key(key)
            .responseCacheControl("max-age="+(TimeUnit.DAYS.toSeconds(7)+ defaultIfNull(version,0L)))
            .overrideConfiguration(overrideConfig)
            .build();

public class CustomAwsS3V4Signer implements Presigner, Signer
{
    private final AwsS3V4Signer awsSigner;

    public CustomAwsS3V4Signer()
    {
        awsSigner = AwsS3V4Signer.create();
    }

@Override
public SdkHttpFullRequest presign(SdkHttpFullRequest request, ExecutionAttributes executionAttributes)
{
    Instant baselineInstant = Instant.now().truncatedTo(ChronoUnit.DAYS);

    executionAttributes.putAttribute(AwsSignerExecutionAttribute.PRESIGNER_EXPIRATION,
            baselineInstant.plus(3, ChronoUnit.DAYS));

在这里，我们覆盖签名时钟以模拟一个固定时间，最终导致网址中的到期和签名一致，直到未来的某个日期：

    Aws4PresignerParams.Builder builder = Aws4PresignerParams.builder()
            .signingClockOverride(Clock.fixed(baselineInstant, ZoneId.of("UTC")));

    Aws4PresignerParams signingParams =
            extractPresignerParams(builder, executionAttributes).build();

    return awsSigner.presign(request, signingParams);
    }
}

更多详情请点击此处：

https://murf.ai/resources/creating-cache-friendly-presigned-s3-urls-using-v4signer-q1bbqgk

Answer 2

也许您可以向您的应用程序添加一个经过身份验证的端点并在该端点中检索图像，而不是使用预签名的 URL 机制？在您的 img 标签等中使用此 URL。该端点可以缓存图像并为浏览器提供适当的响应头来缓存图像。

Answer 3

类似于@Aragorn 的概念，但这是更完整的代码。不过，这又是 Java。此外，由于我的应用程序是多区域的，因此我必须输入区域属性。

import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.signer.Signer;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;

import javax.annotation.PostConstruct;
import javax.validation.constraints.NotNull;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Path;
import java.time.Duration;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;

@Component
@Slf4j
public class S3Operations {

    @Autowired
    private Signer awsSigner;

    private final Map<Region, S3Presigner> presignerMap = new ConcurrentHashMap<>();

    private S3Presigner buildPresignerForRegion(
      AwsCredentialsProvider credentialsProvider,
      Region region) {

        return S3Presigner.builder()
            .credentialsProvider(credentialsProvider)
            .region(region)
            .build();

    }


    /**
     * Convert an S3 URI to a normal HTTPS URI that expires.
     *
     * @param s3Uri S3 URI (e.g. s3://bucketname/ArchieTest/フェニックス.jpg)
     * @return https URI
     */
    @SneakyThrows
    public URI getExpiringUri(final URI s3Uri) {

        final GetObjectRequest getObjectRequest =
            GetObjectRequest.builder()
                .bucket(s3Uri.getHost())
                .key(s3Uri.getPath().substring(1))
                .overrideConfiguration(builder -> builder.signer(awsSigner))
                .build();

        final Region bucketRegion = bucketRegionMap.computeIfAbsent(s3Uri.getHost(),
            bucketName -> {
                final GetBucketLocationRequest getBucketLocationRequest = GetBucketLocationRequest.builder()
                    .bucket(bucketName)
                    .build();

                return Region.of(s3Client.getBucketLocation(getBucketLocationRequest).locationConstraint().toString());
            });

        final GetObjectPresignRequest getObjectPresignRequest = GetObjectPresignRequest.builder()
            .signatureDuration(Duration.ofSeconds(0)) // required, but ignored
            .getObjectRequest(getObjectRequest)
            .build();

        return presignerMap.computeIfAbsent(bucketRegion, this::buildPresignerForRegion).presignGetObject(getObjectPresignRequest).url().toURI();

    }

对于上面注入的CustomAwsSigner。主要区别在于我抛出了一个不受支持的操作异常。

import org.jetbrains.annotations.TestOnly;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import software.amazon.awssdk.auth.signer.AwsS3V4Signer;
import software.amazon.awssdk.auth.signer.AwsSignerExecutionAttribute;
import software.amazon.awssdk.auth.signer.params.Aws4PresignerParams;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.signer.Presigner;
import software.amazon.awssdk.core.signer.Signer;
import software.amazon.awssdk.http.SdkHttpFullRequest;

import java.time.Clock;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.temporal.ChronoField;
import java.time.temporal.ChronoUnit;

/**
 * This is a custom signer where the expiration is preset to a 5 minute block within an hour.
 * This must only be used for presigning.
 */
@Component
public class CustomAwsSigner implements Signer, Presigner {
    private final AwsS3V4Signer theSigner = AwsS3V4Signer.create();

    /**
     * This is the clip time for the expiration.  This should be divisible into 60.
     */
    @Value("${aws.s3.clipTimeInMinutes:5}")
    private long clipTimeInMinutes;

    @Value("${aws.s3.expirationInSeconds:3600}")
    private long expirationInSeconds;

    /**
     * Computes the base time as the processing time to the floor of nearest clip block.
     *
     * @param processingDateTime processing date time
     * @return base time
     */
    @TestOnly
    public ZonedDateTime computeBaseTime(final ZonedDateTime processingDateTime) {

        return processingDateTime
            .truncatedTo(ChronoUnit.MINUTES)
            .with(temporal -> temporal.with(ChronoField.MINUTE_OF_HOUR, temporal.get(ChronoField.MINUTE_OF_HOUR) / clipTimeInMinutes * clipTimeInMinutes));

    }

    @Override
    public SdkHttpFullRequest presign(final SdkHttpFullRequest request, final ExecutionAttributes executionAttributes) {

        final Instant baselineInstant = computeBaseTime(ZonedDateTime.now()).toInstant();

        final Aws4PresignerParams signingParams = Aws4PresignerParams.builder()
            .awsCredentials(executionAttributes.getAttribute(AwsSignerExecutionAttribute.AWS_CREDENTIALS))
            .signingName(executionAttributes.getAttribute(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME))
            .signingRegion(executionAttributes.getAttribute(AwsSignerExecutionAttribute.SIGNING_REGION))
            .signingClockOverride(Clock.fixed(baselineInstant, ZoneId.of("UTC")))
            .expirationTime(baselineInstant.plus(expirationInSeconds, ChronoUnit.SECONDS))
            .build();
        return theSigner.presign(request, signingParams);

    }

    @Override
    public SdkHttpFullRequest sign(final SdkHttpFullRequest request, final ExecutionAttributes executionAttributes) {

        throw new UnsupportedOperationException("this class is only used for presigning");

    }
}

Answer 4

这是我在关注这篇文章后提出的一个 Python 解决方案。 它使用freezegun 库来操纵时间，以使签名在给定时间段内保持不变。

import time
import datetime

import boto3
from freezegun import freezetime


S3_CLIENT = boto3.client("s3")

SEVEN_DAYS_IN_SECONDS = 604800
MAX_EXPIRES_SECONDS = SEVEN_DAYS_IN_SECONDS



def get_presigned_get_url(bucket: str, key: str, expires_in_seconds: int = MAX_EXPIRES_SECONDS) -> str:
        current_timestamp = int(time.time())
        truncated_timestamp = current_timestamp - (current_timestamp % expires_in_seconds)
        with freeze_time(datetime.datetime.fromtimestamp(truncated_timestamp)):
            presigned_url = S3_CLIENT.generate_presigned_url(
                ClientMethod="get_object",
                Params={
                    "Bucket": bucket,
                    "Key": key,
                    "ResponseCacheControl": f"private, max-age={expires_in_seconds}, immutable",
                },
                ExpiresIn=expires_in_seconds,
                HttpMethod="GET",
            )
        return presigned_url

Answer 5

如果有人在使用 golang 为具有缓存可能性的 url 进行预签名而苦苦挣扎，您可以创建一个自定义签名处理程序并将命名的处理程序替换为您自己的处理程序以更改签名时间并使时间段的 url 相同：

import (
    "time"

    "github.com/aws/aws-sdk-go/aws/request"
    v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
)

// Will create same url if in the same 15 minutes time bucket
const presignPeriod = 15 * time.Minute

// TimeInterface implements an interface that
// has the a time variable (Now) and a function
// to retrieve the time variable
type TimeInterface struct {
    Now time.Time
}

func (t *TimeInterface) NowFunc() time.Time {
    return t.Now
}

// getSignTime function returns the signing time
// (initial time) for the time bucket
func getSignTime() time.Time {
    now := time.Now().UTC()
    signTime := now.Round(presignPeriod)
    if signTime.After(now) {
        signTime.Add(-presignPeriod)
    }

    return signTime
}

// CustomSignSDKRequest Implements a custom aws signing
// handler that sets signing time on buckets of 
// <presignPeriod> minutes.
// It is used so browsers can cache the result of the
// url for get requests, instead of downloading the resource everytime.
func CustomSignSDKRequest(req *request.Request) {
    t := TimeInterface{
        Now: getSignTime(),
    }
    v4.SignSDKRequestWithCurrentTime(req, t.NowFunc)
}