【问题标题】:Writing file to S3 in a invoked AWS lambda function from another function在从另一个函数调用的 AWS lambda 函数中将文件写入 S3
【发布时间】:2020-08-20 23:04:23
【问题描述】:

这个问题与从另一个 AWS lambda 函数调用的 AWS lambda 函数有关,然后在调用的函数中我想读取一个文件并将该文件写回 S3。但是,不知何故它不起作用。详情如下:

我使用一个主要的 AWS Lambda 函数来调用另一个具有此代码的函数:

import json
import boto3

def lambda_handler(event, context):
    invokerLam = boto3.client("lambda") # Defining an object for the lambda service
    BlastFoldername = '1.047. 2020 S3C 5009 P1'
    inputsForInvocation = {'BlastFoldername' : BlastFoldername }


    response = invokerLam.invoke(FunctionName = 'arn:aws:lambda:us-east- 
                         1:131394402205:function:DestinationLambda', InvocationType = 'Event',
                                 Payload = json.dumps(inputsForInvocation))

然后调用的 AWS lambda 函数会导入有效负载,从存储桶中读取文件并将相同的文件写入存储桶中,代码如下:

import json
import boto3
import io
import pandas as pd

def lambda_handler(event, context):

    # Reading the payload
    BlastFolder = event['BlastFoldername']
    NewBlastFolder = BlastFolder + "_transformed"
    FileName = "Collar Coordiantes 2820 S3C 5007 P1"
    LocationKey1 = NewBlastFolder + "/" + FileName



    #Read from the s3 Bucket with the same code
    s3 = boto3.client('s3')
    bucket_name = 'shivay.aws108'
    file_name = 'Transformed/Blast/Collar Coordinates.xls'
    file_cor = s3.get_object(Bucket = bucket_name, Key = file_name)
    file_content = file_cor['Body'].read()
    read_excel_data = io.BytesIO(file_content)
    df = pd.read_excel(read_excel_data)
    print(df)



    #Writing the data frame as CSV in the same S3 bucket
    csv_buffer = io.StringIO()
    df.to_csv(csv_buffer)
    s3_resource = boto3.resource('s3')
    HoleFile_save1 = "/Transformed/WritingBackCheck.csv"
    s3_path = HoleFile_save1
    s3_resource.Object(bucket_name, s3_path).put(Body = csv_buffer.getvalue())

将文件写入 s3 的第三步不起作用。文件的读取和数据框的创建正在工作。 此函数在其他普通 AWS Lambda 函数中工作,其中 Lambda 由对象创建触发器触发,但当它由另一个 lambda 触发时,它不会将文件写入 s3。

供参考,第一个函数的作用是:


{
  "permissionsBoundary": {},
  "roleName": "trackore-lambda",
  "policies": [
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
          }
        ]
      },
      "name": "AmazonS3FullAccess",
      "id": "ANPAIFIR6V6BVTRAHWINE",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Action": [
              "autoscaling:Describe*",
              "cloudwatch:*",
              "logs:*",
              "sns:*",
              "iam:GetPolicy",
              "iam:GetPolicyVersion",
              "iam:GetRole"
            ],
            "Effect": "Allow",
            "Resource": "*"
          },
          {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents*",
            "Condition": {
              "StringLike": {
                "iam:AWSServiceName": "events.amazonaws.com"
              }
            }
          }
        ]
      },
      "name": "CloudWatchFullAccess",
      "id": "ANPAIKEABORKUXN6DEAZU",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/CloudWatchFullAccess"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:PutLogEvents"
            ],
            "Resource": "*"
          }
        ]
      },
      "name": "AWSLambdaBasicExecutionRole",
      "id": "ANPAJNCQGXC42545SKXIK",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "lambda:InvokeFunction"
            ],
            "Resource": [
              "*"
            ]
          }
        ]
      },
      "name": "AWSLambdaRole",
      "id": "ANPAJX4DPCRGTC4NFDUXI",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaRole"
    }
  ],
  "trustedEntities": [
    "lambda.amazonaws.com"
  ]
}

而第二个功能的作用是:

{
  "permissionsBoundary": {},
  "roleName": "trackore-lambda",
  "policies": [
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
          }
        ]
      },
      "name": "AmazonS3FullAccess",
      "id": "ANPAIFIR6V6BVTRAHWINE",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Action": [
              "autoscaling:Describe*",
              "cloudwatch:*",
              "logs:*",
              "sns:*",
              "iam:GetPolicy",
              "iam:GetPolicyVersion",
              "iam:GetRole"
            ],
            "Effect": "Allow",
            "Resource": "*"
          },
          {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::*:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents*",
            "Condition": {
              "StringLike": {
                "iam:AWSServiceName": "events.amazonaws.com"
              }
            }
          }
        ]
      },
      "name": "CloudWatchFullAccess",
      "id": "ANPAIKEABORKUXN6DEAZU",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/CloudWatchFullAccess"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:PutLogEvents"
            ],
            "Resource": "*"
          }
        ]
      },
      "name": "AWSLambdaBasicExecutionRole",
      "id": "ANPAJNCQGXC42545SKXIK",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "lambda:InvokeFunction"
            ],
            "Resource": [
              "*"
            ]
          }
        ]
      },
      "name": "AWSLambdaRole",
      "id": "ANPAJX4DPCRGTC4NFDUXI",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaRole"
    }
  ],
  "trustedEntities": [
    "lambda.amazonaws.com"
  ]
}

请帮忙。

谢谢

【问题讨论】:

  • CloudWatch 日志中是否有任何错误?
  • @Marcin - CloudWatch 日志中没有报告错误,当它到达回写部分时它没有做任何事情,我尝试在写入部分打印一些东西以了解它的工作,它没有也打印这些行。
  • 第一个函数呢?如果您异步调用第二个函数,您是否为第二个函数设置了死信队列?
  • 抱歉什么是死信队列?我刚刚在角色文档中添加了权限为“Lambda:InvokeAsync”
  • lambda:InvokeAsync 权限已弃用。请使用lambda:InvokeFunction。由于您使用异步调用(事件类型),dlq 可能很有用。

标签: python amazon-web-services amazon-s3 lambda aws-lambda


【解决方案1】:

根据聊天对话,发现问题超时

第二个函数的默认超时时间是 3 秒。它太短了。解决方案是增加超时时间。

【讨论】:

  • 在本地机器上使用 AWS SAM 进行开发。您可以在 SAM 终端中看到此类错误。
猜你喜欢
  • 2017-03-04
  • 2018-05-19
  • 2019-06-28
  • 2018-08-21
  • 2021-09-23
  • 2018-06-09
  • 1970-01-01
  • 1970-01-01
  • 2022-12-05
相关资源
最近更新 更多