【问题标题】:Intermittent behaviour of a lambda connecting to DAX连接到 DAX 的 lambda 的间歇性行为
【发布时间】:2023-02-14 19:40:31
【问题描述】:

我正在尝试将 lambda 连接到 DAX(DynamoDB 内存缓存)。我正在通过 CDK 进行设置。经过一番努力,我今天让它工作了,然后几个小时后它就停止工作了。

它所做的只是根据模式将记录写入 DynamoDB 或 DAX。它可以可靠地写入 DynamoDB,但我似乎在使用 DAX 时犯了一个基本错误,而且似乎没有什么韵律或理由说明它为什么能工作或不能工作。

首先,这是 lambda 的简化版本:

import { DynamoDB } from 'aws-sdk';
import AmazonDaxClient from 'amazon-dax-client';

function instantiateDrivers(daxClusterEndpoint: string) {
    const options = {
        endpoint: 'http://dynamodb.eu-west-1.amazonaws.com',
    };

    // See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.concepts.cluster.html
    if (daxClusterEndpoint && daxClusterEndpoint.length > 0) {
        console.log(`Using DAX cluster: ${daxClusterEndpoint}`);
        const dax = new AmazonDaxClient({
            endpoints: [`daxs://${daxClusterEndpoint}`],
            region: 'eu-west-1',
            maxRetries: 1,
        });
        // @ts-ignore
        options.service = dax;
    }

    const dynamoDbServiceClient = new DynamoDB(options);
    console.log(`Instantiated DynamoDB`);
    const dynamoDbDocumentClient = new DynamoDB.DocumentClient(options);
    console.log(`Instantiated DocumentClient`);

    return { dynamoDbServiceClient, dynamoDbDocumentClient };
}

async function writeRecord(daxClusterEndpoint: string, voucherTableName: string, record: any) {
    // Create the drivers
    let { dynamoDbDocumentClient } = instantiateDrivers(daxClusterEndpoint);

    console.log(`Instantiated drivers OK`);

    // Do the write
    await dynamoDbDocumentClient.
    put({ TableName: voucherTableName, Item: record}).
    promise();
}

function validateArguments(event: any) {
    const { VOUCHER_TABLE_NAME, DAX_CLUSTER_ENDPOINT } = event;

    if (!VOUCHER_TABLE_NAME) {
        throw new Error(`A VOUCHER_TABLE_NAME must be supplied`);
    }

    return {
        voucherTableName: VOUCHER_TABLE_NAME,
        daxClusterEndpoint: DAX_CLUSTER_ENDPOINT,
    };
}

export const handler =  async (event: any, context: any) => {
    console.log("Received lambda call: " + JSON.stringify(event, null, 2));

    // Try to get args
    const { voucherTableName, daxClusterEndpoint } = validateArguments(event);

    // This is a record we want to write
    await writeRecord(
        daxClusterEndpoint,
        voucherTableName,
        // Note the object is in v2 format, not v1 (with the 'S' 'M' and 'BOOL' type keys)
        {
            "code": "TEST-VOUCHER-1234",
            "siteId": "3",
            "endDate": "2022-08-31T13:39:56.000Z",
            "name": "permanent",
            "startDate": "2022-07-20T13:39:56.000Z",
            "type": "XPercentOffProductPromotion",
            "used": false,
        }
    );

    return context.logStreamName;
};

接下来,这是 CDK:

import {CfnOutput, Stack, StackProps, Duration} from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as path from "path";
import {NodejsFunction} from "aws-cdk-lib/aws-lambda-nodejs";
import {Runtime} from "aws-cdk-lib/aws-lambda";
import {PolicyStatement, Role, ServicePrincipal, Effect} from "aws-cdk-lib/aws-iam"
import {Vpc, SecurityGroup, SubnetType} from "aws-cdk-lib/aws-ec2";

export class SupportScriptStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const importedVpc = Vpc.fromLookup(this, 'imported-vpc', { vpcName: `${process.env.STAGE}-eu-west-1`})

    const voucherStatement = new PolicyStatement({
      resources: [
        'arn:aws:dynamodb:eu-west-1:*:table/voucher-promotion-*',
        'arn:aws:dax:eu-west-1:*:cache/dax-cluster-preprod'
      ],
      actions: [
        "dynamodb:BatchGetItem",
        "dynamodb:BatchWriteItem",
        "dynamodb:PutItem",
        "dynamodb:DescribeTable",
        "dynamodb:DeleteItem",
        "dynamodb:GetItem",
        "dynamodb:Scan",
        "dynamodb:Query",
        "dynamodb:UpdateItem",
        "dynamodb:DescribeTimeToLive",
        "dynamodb:ListTables",
        "dynamodb:DescribeLimits",
        "dax:BatchGetItem",
        "dax:BatchWriteItem",
        "dax:PutItem",
        "dax:DescribeTable",
        "dax:DeleteItem",
        "dax:GetItem",
        "dax:Scan",
        "dax:Query",
        "dax:UpdateItem",
        "dax:DescribeTimeToLive",
        "dax:ListTables",
        "dax:DescribeLimits"
      ]
    });

    const securityGroup = new SecurityGroup(
      this,
      "LambdaSecurityGroup",
      {
        vpc: importedVpc,
        description: "Demo API Security Group",
        allowAllOutbound: true,
      }
    );

    const role = new Role(this, "DemoApiLambdaRole", {
      assumedBy: new ServicePrincipal("lambda.amazonaws.com"),
    });
    role.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        actions: [
          // VPC
          "ec2:DescribeNetworkInterfaces",
          "ec2:CreateNetworkInterface",
          "ec2:DeleteNetworkInterface",
          "ec2:DescribeInstances",
          "ec2:AttachNetworkInterface",
          // DAX
          "dax:*",
          // Need Dynamo as well
          "dynamo:*",
        ],
        resources: ["*"],
      })
    );

    const voucherCodeFunction = new NodejsFunction(this, `VC-Create`, {
      runtime: Runtime.NODEJS_16_X,
      functionName: `voucher-code-importer`,
      timeout: Duration.minutes(3), // Long timeout for now during debug phase
      memorySize: 512,
      bundling: {
        minify: true,
      },
      handler: "handler",
      entry: path.join(__dirname, `../src/create-voucher/handler.ts`),
      vpc: importedVpc,
      // See https://stackoverflow.com/a/72159511
      securityGroups: [securityGroup],
      role
    });

    voucherCodeFunction.addToRolePolicy(voucherStatement);

    new CfnOutput(this, 'voucher-code-function-arn', {value: voucherCodeFunction.functionArn})
  }
}

CDK 经历了一系列令人沮丧的混乱迭代——它以一种格式工作,我整理它或收紧一些权限,它停止工作,我恢复到旧的工作版本,现在它也停止工作了。

我确信这在没有 const voucherStatement = new PolicyStatement 部分的情况下工作了一段时间 - 我恢复了它以防它是新的一组失败的原因。我很确定这会重复 lambda 单独拥有的权限。

可能有用的一件事是将 lambda 的子网与 lambda 的子网同步。在这个项目中,一位同事在名为cdk.context.json 的文件中设置了可用子网的声明。

我相信我已经正确地同步了这一点,但此时我几乎抓住了救命稻草。稍微编辑了一下,但应该足够清楚了:

{
  "vpc-provider:account=9015xxxxxxxx:filter.tag:Name=preprod-eu-west-1:region=eu-west-1:returnAsymmetricSubnets=true": {
    "vpcId": "vpc-0d891xxxxxxxxxxxx",
    "vpcCidrBlock": "172.35.0.0/16",
    "availabilityZones": [],
    "subnetGroups": [
      {
        "name": "Private",
        "type": "Private",
        "subnets": [
          {
            "subnetId": "subnet-0ad04xxxxxxxxxxxx",
            "cidr": "172.35.a.0/22",
            "availabilityZone": "eu-west-1b",
            "routeTableId": "rtb-0fee4xxxxxxxxxxxx"
          },
          {
            "subnetId": "subnet-08598xxxxxxxxxxxx",
            "cidr": "172.35.z.0/22",
            "availabilityZone": "eu-west-1c",
            "routeTableId": "rtb-0f477xxxxxxxxxxxx"
          }
        ]
      },
      {
        "name": "Public",
        "type": "Public",
        "subnets": [
          {
            "subnetId": "subnet-0fba3xxxxxxxxxxxx",
            "cidr": "172.35.y.0/22",
            "availabilityZone": "eu-west-1b",
            "routeTableId": "rtb-02dfbxxxxxxxxxxxx"
          },
          {
            "subnetId": "subnet-0a3b8xxxxxxxxxxxx",
            "cidr": "172.35.x.0/22",
            "availabilityZone": "eu-west-1c",
            "routeTableId": "rtb-02dfbxxxxxxxxxxxx"
          }
        ]
      }
    ]
  }
}

如果我将记录写入 Dynamo(通过不在有效负载中提供 DAX_CLUSTER_ENDPOINT 键)那么就可以了。但是,如果我为此 (dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com) 提供一个值,那么它有时会工作几个小时,有时不会。

我注意到在错误中系统无法解析 daxs:// 地址,但后来给出了它解析到的 IP 地址(并​​且“无法从中提取”):

2022-08-03T12:01:58.698+01:00   START RequestId: 08510000-0000-0000-0000-dc6255000000 Version: $LATEST
    2022-08-03T12:01:58.700+01:00   2022-08-03T11:01:58.700Z 08510000-0000-0000-0000-dc6255000000 INFO Received lambda call: { "VOUCHER_TABLE_NAME": "voucher-promotion-preprod", "DAX_CLUSTER_ENDPOINT": "dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com" }
    2022-08-03T12:01:58.701+01:00   2022-08-03T11:01:58.700Z 08510000-0000-0000-0000-dc6255000000 INFO Using DAX cluster: dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com
    2022-08-03T12:01:58.738+01:00   2022-08-03T11:01:58.738Z 08510000-0000-0000-0000-dc6255000000 INFO Instantiated DynamoDB
    2022-08-03T12:01:58.740+01:00   2022-08-03T11:01:58.740Z 08510000-0000-0000-0000-dc6255000000 INFO Instantiated DocumentClient
    2022-08-03T12:01:58.740+01:00   2022-08-03T11:01:58.740Z 08510000-0000-0000-0000-dc6255000000 INFO Instantiated drivers OK
    2022-08-03T12:02:28.754+01:00   2022-08-03T11:02:28.754Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.222.11,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524538749, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:28.762+01:00   2022-08-03T11:02:28.762Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.222.11,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524538761, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:32.734+01:00   2022-08-03T11:02:32.734Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.222.11,172.35.111.222,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524542732, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:32.740+01:00   2022-08-03T11:02:32.740Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.212,172.35.111.222,172.35.222.11): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) { time: 1659524542739, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:36.735+01:00   2022-08-03T11:02:36.735Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.212,172.35.222.11,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) { time: 1659524546732, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:36.741+01:00   2022-08-03T11:02:36.741Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.222.11,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524546740, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:40.736+01:00   2022-08-03T11:02:40.735Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.111.212,172.35.222.11): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) { time: 1659524550732, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:40.740+01:00   2022-08-03T11:02:40.740Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.222.11,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524550740, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:56.740+01:00   2022-08-03T11:02:56.740Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.212,172.35.222.11,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524566738, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:56.743+01:00   2022-08-03T11:02:56.743Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.111.212,172.35.222.11): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at processTimers (node:internal/timers:499:9) { time: 1659524566743, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:58.756+01:00   2022-08-03T11:02:58.756Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.212,172.35.222.11,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524568755, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:58.756+01:00   2022-08-03T11:02:58.756Z 08510000-0000-0000-0000-dc6255000000 ERROR as [Error]: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 { time: 1659524578756, code: 'NoRouteException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:58.764+01:00   2022-08-03T11:02:58.764Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.212,172.35.222.11,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524568763, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:02:58.764+01:00   2022-08-03T11:02:58.764Z 08510000-0000-0000-0000-dc6255000000 ERROR as [Error]: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 { time: 1659524578764, code: 'NoRouteException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:00.740+01:00   2022-08-03T11:03:00.740Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.212,172.35.222.11,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524570738, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:02.743+01:00   2022-08-03T11:03:02.743Z 08510000-0000-0000-0000-dc6255000000 ERROR caught exception during cluster refresh: as [Error]: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524582743, code: 'NoRouteException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:02.743+01:00   2022-08-03T11:03:02.743Z 08510000-0000-0000-0000-dc6255000000 ERROR Error: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7)
    2022-08-03T12:03:02.744+01:00   2022-08-03T11:03:02.744Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.222.11,172.35.111.222,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524572742, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:02.744+01:00   2022-08-03T11:03:02.744Z 08510000-0000-0000-0000-dc6255000000 ERROR caught exception during cluster refresh: as [Error]: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 { time: 1659524582744, code: 'NoRouteException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:02.744+01:00   2022-08-03T11:03:02.744Z 08510000-0000-0000-0000-dc6255000000 ERROR Error: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645
    2022-08-03T12:03:04.741+01:00   2022-08-03T11:03:04.741Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.222.11,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524574739, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:04.743+01:00   2022-08-03T11:03:04.743Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.222.11,172.35.111.212,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524574742, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:06.743+01:00   2022-08-03T11:03:06.743Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.222.11,172.35.111.212,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524576740, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:06.745+01:00   2022-08-03T11:03:06.743Z 08510000-0000-0000-0000-dc6255000000 ERROR caught exception during cluster refresh: as [Error]: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524586743, code: 'NoRouteException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:06.745+01:00   2022-08-03T11:03:06.745Z 08510000-0000-0000-0000-dc6255000000 ERROR Error: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7)
    2022-08-03T12:03:06.746+01:00   2022-08-03T11:03:06.746Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.222.11,172.35.111.212,172.35.111.222): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 { time: 1659524576742, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:06.746+01:00   2022-08-03T11:03:06.746Z 08510000-0000-0000-0000-dc6255000000 ERROR caught exception during cluster refresh: as [Error]: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645 { time: 1659524586746, code: 'NoRouteException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:06.746+01:00   2022-08-03T11:03:06.746Z 08510000-0000-0000-0000-dc6255000000 ERROR Error: NoRouteException: not able to resolve address: [{"host":"dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com","port":9111,"scheme":"daxs"}] at yu._resolveAddr (/var/task/index.js:1:7887) at /var/task/index.js:1:8298 at /var/task/index.js:1:8645
    2022-08-03T12:03:08.746+01:00   2022-08-03T11:03:08.746Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.222.11,172.35.111.212): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524578744, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:08.747+01:00   2022-08-03T11:03:08.747Z 08510000-0000-0000-0000-dc6255000000 ERROR Failed to pull from dax-cluster-preprod.xxxxxx.dax-clusters.eu-west-1.amazonaws.com (172.35.111.222,172.35.111.212,172.35.222.11): so [Error]: ConnectionException: Connection timeout after 10000ms at Tu.alloc (/var/task/index.js:10:7418) at /var/task/index.js:66:89369 at runNextTicks (node:internal/process/task_queues:61:5) at listOnTimeout (node:internal/timers:528:9) at processTimers (node:internal/timers:502:7) { time: 1659524578745, code: 'ConnectionException', retryable: true, requestId: null, statusCode: -1, _tubeInvalid: false, waitForRecoveryBeforeRetrying: false }
    2022-08-03T12:03:08.753+01:00   2022-08-03T11:03:08.752Z 08510000-0000-0000-0000-dc6255000000 ERROR Invoke Error {"errorType":"Error","errorMessage":"NoRouteException: No endpoints available","code":"NoRouteException","time":1659524588752,"retryable":true,"requestId":null,"statusCode":-1,"_tubeInvalid":false,"waitForRecoveryBeforeRetrying":false,"stack":["Error: NoRouteException: No endpoints available"," at vu.leaderClient (/var/task/index.js:10:11634)"," at Object.getClient (/var/task/index.js:66:116939)"," at /var/task/index.js:66:120803"," at new Promise (<anonymous>)"," at G0.makeRequestWithRetries (/var/task/index.js:66:120779)"," at /var/task/index.js:66:121294"]}
    2022-08-03T12:03:08.757+01:00   END RequestId: 08510000-0000-0000-0000-dc6255000000
    2022-08-03T12:03:08.757+01:00
REPORT RequestId: 08510000-0000-0000-0000-dc6255000000  Duration: 70055.20 ms   Billed Duration: 70056 ms   Memory Size: 512 MB Max Memory Used: 89 MB  Init Duration: 463.81 ms    
    REPORT RequestId: 08510000-0000-0000-0000-dc6255000000 Duration: 70055.20 ms Billed Duration: 70056 ms Memory Size: 512 MB Max Memory Used: 89 MB Init Duration: 463.81 ms 

我已经在这方面四处走动,网上几乎没有关于 DAX 的相关故障排除信息。我做这项工作只是因为 AWS 控制台中没有支持 DAX 的记录编辑器,并且担心我正在使用的系统已经有效地致力于预发布产品。

我可以尝试解决这个问题吗?

【问题讨论】:

  • (我从日志中删除了一些相当相同的行,因为它超出了问题的允许长度)。
  • 如果相同的 lambda 有时有效而有时无效,则这是一个很大的暗示,表明子网路由已关闭。你有多少个子网?它们的配置相同吗?您(至少出于测试目的)是否考虑过仅将 lambda 部署到一个子网中并检查这是否会导致它持续工作或失败(两个结果都会告诉您一些事情)?
  • @luk2302:好主意,谢谢。我想我会尝试的一件事是削减上下文文件,以便它只包含私有子网。我偷偷怀疑 DAX 只附加到私人的,我想知道这个文件中有两个公共的,两个私人的是否是不可靠的原因。
  • 恼人的是,我个人没有运行aws dax describe-clusters的权限,但我想知道我的部署管道是否有!我会在这里更新。
  • 好的,从上下文文件中删除公共子网也没有用。我会看看我是否可以直接确认有关集群的一些信息(我有一些来自同事的屏幕截图,我确信他们是正确的,但也许走过场会激发一个新想法)。

标签: typescript amazon-web-services amazon-dynamodb aws-cdk amazon-dynamodb-dax


【解决方案1】:

我真的不会猜到这个问题的答案。我们有一些其他的 lambda(在 Serverless 而不是 SDK 中创建),有人向我指出它们在安全组上有入站规则。但是我正在创建自己的,没有入站规则:

const securityGroup = new SecurityGroup(
  this,
  "LambdaSecurityGroup",
  {
    vpc: importedVpc,
    description: "Demo API Security Group",
    allowAllOutbound: true,
  }
);

相反,我应该重用现有的:

// This looks like the SG that web-based lambdas use to talk to DAX
const securityGroup = SecurityGroup.fromLookupByName(
  this,
  'VoucherImporterSecurityGroup',
  'default',
  importedVpc
);

我不知道为什么需要将某些东西伸入 SG。 SG 复制了我现有的 allowAllOutbound: true 规则。然而,我确信我之前没有使用过这个规则,并且我的 lambda 能够在几个小时内多次连接到 DAX。

时间会证明这是否是一个可靠的修复。

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2019-11-02
    • 1970-01-01
    • 2011-01-11
    • 2018-09-25
    • 2014-07-16
    • 2018-09-24
    • 1970-01-01
    • 2023-02-26
    相关资源
    最近更新 更多