使用 boto3 连接到 S3 加速端点

Question

我想从已激活加速的 S3 存储桶中将文件下载到 Python 文件对象中。我遇到了一些建议是否将endpoint_url 覆盖为“s3-accelerate.amazonaws.com”和/或使用use_accelerate_endpoint 属性的资源。

我已经尝试了这两种方法，并且有几种变体，但每次都返回相同的错误。我试过的脚本之一是：

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "path"}))
   input = BytesIO() 
   s3.download_fileobj(<MY_BUCKET>,<MY_KEY>, input)

返回以下错误：

            ---------------------------------------------------------------------------
            ClientError                               Traceback (most recent call last)
            <ipython-input-61-92b89b45f215> in <module>()
                 11      "addressing_style": "path"}))
                 12 input = BytesIO()
            ---> 13 s3.download_fileobj(bucket, filename, input)
                 14 
                 15 

            ~/Project/venv/lib/python3.5/site-packages/boto3/s3/inject.py in download_fileobj(self, Bucket, Key, Fileobj, ExtraArgs, Callback, Config)
                568             bucket=Bucket, key=Key, fileobj=Fileobj,
                569             extra_args=ExtraArgs, subscribers=subscribers)
            --> 570         return future.result()
                571 
                572 

            ~/Project//venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                 71             # however if a KeyboardInterrupt is raised we want want to exit
                 72             # out of this and propogate the exception.
            ---> 73             return self._coordinator.result()
                 74         except KeyboardInterrupt as e:
                 75             self.cancel()

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                231         # final result.
                232         if self._exception:
            --> 233             raise self._exception
                234         return self._result
                235 

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/tasks.py in _main(self, transfer_future, **kwargs)
                253             # Call the submit method to start submitting tasks to execute the
                254             # transfer.
            --> 255             self._submit(transfer_future=transfer_future, **kwargs)
                256         except BaseException as e:
                257             # If there was an exception raised during the submission of task

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/download.py in _submit(self, client, config, osutil, request_executor, io_executor, transfer_future)
                347                 Bucket=transfer_future.meta.call_args.bucket,
                348                 Key=transfer_future.meta.call_args.key,
            --> 349                 **transfer_future.meta.call_args.extra_args
                350             )
                351             transfer_future.meta.provide_transfer_size(

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
                310                     "%s() only accepts keyword arguments." % py_operation_name)
                311             # The "self" in this scope is referring to the BaseClient.
            --> 312             return self._make_api_call(operation_name, kwargs)
                313 
                314         _api_call.__name__ = str(py_operation_name)

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
                603             error_code = parsed_response.get("Error", {}).get("Code")
                604             error_class = self.exceptions.from_code(error_code)
            --> 605             raise error_class(parsed_response, operation_name)
                606         else:
                607             return parsed_response

            ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

当我使用"use_accelerate_endpoint": False 运行相同的脚本时，它运行良好。 但是，它在以下情况下返回了相同的错误：

• 我用“s3-accelerate.amazonaws.com”覆盖endpoint_url
• 我定义"addressing_style": "virtual"

跑步时 s3.get_bucket_accelerate_configuration(Bucket=<MY_BUCKET>)

我按预期得到了{..., 'Status': 'Enabled'}。

知道该代码有什么问题以及我应该更改什么以正确查询该存储桶的加速端点吗？

在 Ubuntu 17.04 上使用带有 boto3==1.4.7、botocore==1.7.43 的 python3.5。

编辑： 我也尝试过类似的上传脚本：

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "virtual"}))
   output = BytesIO()
   output.seek(0)
   s3.upload_fileobj(output, <MY_BUCKET>,<MY_KEY>)

如果没有 use_accelerate_endpoint 选项（所以我的键很好），但在 True 时返回此错误：

   ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

我在这里也尝试了addressing_style 选项（虚拟和路径）

Answer 1

使用boto3==1.4.7 和botocore==1.7.43。 这是从启用了传输加速的存储桶中检索对象的一种方法。

import boto3
from botocore.config import Config
from io import BytesIO

config = Config(s3={"use_accelerate_endpoint": True})
s3_resource = boto3.resource("s3",
                             aws_access_key_id=<MY_KEY_ID>,               
                             aws_secret_access_key=<MY_KEY>,
                             region_name="us-west-2",
                             config=config)

s3_client = s3_resource.meta.client

file_object = BytesIO()
s3_client.download_fileobj(<MY_BUCKET>, <MY_KEY>, file_object)

注意客户端在GET之前向加速端点发送HEAD请求。

其中的规范请求如下所示：

CanonicalRequest:
HEAD
/<MY_KEY>

host:<MY_BUCKET>.s3-accelerate.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200520T204128Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

HEAD 请求可能失败的一些原因包括：

• 具有给定键的对象不存在或启用了严格的访问控制
• 凭据无效
• 传输加速未启用