AWS s3 对象上传到谷歌云存储

Question

我们正在尝试将数据从 aws s3 迁移到 gcp 存储。我们尝试在 gcp 中转移工作，并且工作正常。所以我们想通过 aws lambda 以编程方式实现这一点，因为我们依赖于 aws。 当我尝试导入 google.cloud 模块时出现此错误 lambda cloudwatch logs 这是我的代码：

import os
import logging
import boto3
#from StringIO import StringIO
from google.cloud import storage
#import google-cloud-storage

# Setup logging
LOG = logging.getLogger(__name__)
LOG.setLevel(os.environ.get('LOG_LEVEL', 'INFO'))

GCS_BUCKET_NAME=os.environ['GCS_BUCKET_NAME']
S3 = boto3.client('s3')


def lambda_handler(event, context):
    try:
        l_t_bucketKey = _getKeys(event)

        # Create google client
        storage_client = storage.Client()
        gcs_bucket = storage_client.get_bucket(os.environ['GCS_BUCKET_NAME'])

        LOG.debug('About to copy %d files', len(l_t_bucketKey))
        for bucket, key in l_t_bucketKey:
            try:
                inFileObj = StringIO()
                S3.download_fileobj(
                    Bucket=bucket,
                    Key=key,
                    Fileobj=inFileObj
                )
                blob = gcs_bucket.blob(key)
                blob.upload_from_file(inFileObj, rewind=True)  # seek(0) before reading file obj

                LOG.info('Copied s3://%s/%s to gcs://%s/%s', bucket, key, GCS_BUCKET_NAME, key)
            except:
                LOG.exception('Error copying file: {k}'.format(k=key))
        return 'SUCCESS'
    except Exception as e:
        LOG.exception("Lambda function failed:")
        return 'ERROR'


def _getKeys(d_event):
    """
    Extracts (bucket, key) from event
    :param d_event: Event dict
    :return: List of tuples (bucket, key)
    """
    l_t_bucketKey = []
    if d_event:
        if 'Records' in d_event and d_event['Records']:
            for d_record in d_event['Records']:
                try:
                    bucket = d_record['s3']['bucket']['name']
                    key = d_record['s3']['object']['key']
                    l_t_bucketKey.append((bucket, key))
                except:
                    LOG.warn('Error extracting bucket and key from event')
    return l_t_bucketKey

我从 pypi 网站下载了 google-cloud-storage 模块并将其导入到 aws lambda 层。请帮助为我提供下载此模块的最佳链接。

Answer 1

Google Storage Bucket 可以与 S3 API 一起使用，因此您可以在 Lambda 函数中使用它，而无需任何额外的 GCP 库。

    source_client = boto3.client(
        's3',
        endpoint_url='https://storage.googleapis.com',
        aws_access_key_id=os.environ['GCP_KEY'],
        aws_secret_access_key=os.environ['GCP_SECRET']

要获取 access_key 和 secret - 转到 GS 存储桶设置 -> 互操作性 -> 您的用户帐户的访问密钥 -> 创建密钥