将数据从 AWS Lambda 发送到 SQS 队列时连接重置

Question

我正在使用 AWS SDK for Java 将数据从 AWS Lambda 发送到 SQS。

我们遇到了异常：

Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:120)
at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:167)
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)

代码：

 List<SendMessageBatchRequestEntry> sqsList= new LinkedList<SendMessageBatchRequestEntry>();
    int batchId = 0; //To send a unique batchId for each msg in a batch
    for (Metadata metadata: metadataList) {
        String jsonString = new Gson().toJson(metadata);
        sqsList.add(new SendMessageBatchRequestEntry(batchId + "", jsonString));
        batchId++;
    }
    amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));

背景我们正在尝试做的事情：

我们有一个主要的 Lambda 函数，它创建和初始化一个 SQS 队列，并包含应处理的每条记录的详细信息。 现在需要设置 SQS 队列以从队列中创建 X 条消息的批次，并为每个批次自动调用另一个 SQS Lambda 函数。

Answer 1

每个批次的最大消息数为 10。您不能一次用 20k 填充 SQS 队列并发送该请求。试着把它分成 10 个。

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-limits.html#limits-queues

Answer 2

我们可以批量发送 10 个。 工作代码：

List<SendMessageBatchRequestEntry> sqsList= new LinkedList<SendMessageBatchRequestEntry>();
    int batchId = 1; //To send a unique batchId for each msg in a batch
    for (Metadata metadata: metadataList) {
        String jsonString = new Gson().toJson(metadata);
        if (sqsList.size() == 10) {
            amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
            sqsList.clear();
        } 
        sqsList.add(new SendMessageBatchRequestEntry(batchId + "", jsonString)); 
        batchId++;
    }
    if(sqsList.size()>0) {
        amazonSqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl, sqsList));
    }

Answer 3

似乎您的代码很好，据我记得（我自己已经多次看到此错误），由于 SDK 如何重用 HTTP 连接，在使用 SDK 时会不时发生这种情况。此错误仅告诉您您的 Lambda 重置了 HTTP 连接，但 SDK 内置了重试失败请求的功能，因此如果您在每个请求上都没有看到此错误，您应该没问题。