在 AWS CodeDeploy 期间从 HAProxy 中删除实例

Question

我们的应用程序需要使用 HAProxy 来负载平衡和路由流量（每个 AZ 一个），ALB 和 ELB 的可配置性不足以满足我们的目的。通过 AWS CodeDeploy 部署新代码时，我们希望被修补的实例置于维护模式（从负载平衡中移除，连接耗尽）。我们修改了默认的 CodeDeploy 生命周期 bash 脚本，通过从相关实例向 HAProxy 发送 SSM 运行命令来从其各自的 HAProxy 实例中删除这些实例。目前此修改不起作用，失败原因未知。该脚本在逐步手动执行时起作用（至少到当前的故障点）。失败的部分要么是返回“$INSTANCE_ID 似乎不在具有 HAProxy 实例的 AZ 中，跳过注销。”的测试，要么是上述测试所依赖的 $HAPROXY_ID 的设置。该脚本在此之前一直运行良好，但此时由于找不到 HAProxy 实例 ID 而退出。

我检查了 IAM 角色权限/凭据、环境变量和文件权限，这些似乎都是正确的。通常我会在脚本中放置更多日志以进行调试，但部署太少而且距离我们太远，无法实现。

我的问题：有没有更好的方法来做到这一点？我只能猜测我们不是唯一将 HAProxy 与 CodeDeploy 一起使用的人，而且必须有一种可靠的方法来做到这一点。以下是当前正在使用但不起作用的代码。

#!/bin/bash
#
# Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#  http://aws.amazon.com/apache2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

. $(dirname $0)/common_functions.sh

if [[ "$DEPLOYMENT_GROUP_NAME" != "redacted" ]]; then
  msg "ELB Deregistration doesn't need to happen when not on redacted."
  exit
fi

msg "Running AWS CLI with region: $(get_instance_region)"

# get this instance's ID
INSTANCE_ID=$(get_instance_id)
if [ $? != 0 -o -z "$INSTANCE_ID" ]; then
    error_exit "Unable to get this instance's ID; cannot continue."
fi

# Get current time
msg "Started $(basename $0) at $(/bin/date "+%F %T")"
start_sec=$(/bin/date +%s.%N)

msg "Checking if instance $INSTANCE_ID is part of an AutoScaling group"
asg=$(autoscaling_group_name $INSTANCE_ID)
if [ $? == 0 -a -n "${asg}" ]; then
    msg "Found AutoScaling group for instance $INSTANCE_ID: ${asg}"

    msg "Checking that installed CLI version is at least at version required for AutoScaling Standby"
    check_cli_version
    if [ $? != 0 ]; then
        error_exit "CLI must be at least version ${MIN_CLI_X}.${MIN_CLI_Y}.${MIN_CLI_Z} to work with AutoScaling Standby"
    fi

    msg "Attempting to put instance into Standby"
    autoscaling_enter_standby $INSTANCE_ID "${asg}"
    if [ $? != 0 ]; then
        error_exit "Failed to move instance into standby"
    else
        msg "Instance is in standby"
    fi
fi

msg "Instance is not part of an ASG, continuing..."

## Get the instanceID of the HAProxy instance in this AZ and ENVIRONMENT - Will there ever be more than one???

HAPROXY_ID=$(/usr/local/bin/aws ec2 describe-instances --region us-east-1 --filters "Name=availability-zone,Values=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)" "Name=tag:deployment_group,Values=haproxy.$ENVIRONMENT" --output text  | \
grep INSTANCES | \
awk '{print $8}' )

HAPROXY_IP=$(/usr/local/bin/aws ec2 describe-instances --region us-east-1 --filters "Name=availability-zone,Values=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)" "Name=tag:deployment_group,Values=haproxy.$ENVIRONMENT" --output text  | \
grep INSTANCES | \
awk '{print $13}' )

if test -z "$HAPROXY_ID"; then
    msg "$INSTANCE_ID doesn't seem to be in an AZ with a HAProxy instance, skipping deregistration."
    exit
fi

## Put the current instance into MAINT mode with the HAProxy instance via SSM

msg "Deregistering $INSTANCE_ID from HAProxy $HAPROXY_ID"

DEREGCMD="{\"commands\":[\"haproxyctl disable server bk_app_servers/$INSTANCEID\"],\"executionTimeout\":[\"3600\"]}"

/usr/local/bin/aws ssm send-command \
--document-name "AWS-RunShellScript" \
--instance-ids "$HAPROXY_ID" \
--parameters "$DEREGCMD" \
--timeout-seconds 600 \
--output-s3-bucket-name "redacted" \
--output-s3-key-prefix "haproxy-codedeploy/deregister" \
--region us-east-1

if [ $? != 0 ]; then
    error_exit "Failed to send SSM command to deregister instance $INSTANCE_ID from HAProxy $HAPROXY_ID"
fi

## Wait for all connections to drain from instance

SESS_COUNT=$(/usr/bin/curl -s "http://$HAPROXY_IP:<portredacted>/<urlredacted>" | grep $INSTANCEID | awk -F "," '{print $5}')
DRAIN_TIME=60

msg "Initial session count: $SESS_COUNT"

while [[ "$SESS_COUNT" -gt 0 ]]; do
    if [[ "$COUNTER" -gt "$DRAIN_TIME" ]]; then
        msg "Instance failed to drain all connections within $DRAIN_TIME seconds. Continuing to deploy anyway."
        break
    fi
    msg $SESS_COUNT
    sleep 1
    COUNTER=$(($COUNTER + 1))
    SESS_COUNT=$(/usr/bin/curl -s "http://$HAPROXY_IP:<portredacted>/<urlredacted>" | grep $INSTANCEID | awk -F "," '{print $5}')
done

msg "Finished $(basename $0) at $(/bin/date "+%F %T")"

end_sec=$(/bin/date +%s.%N)
elapsed_seconds=$(echo "$end_sec - $start_sec" | /usr/bin/bc)

msg "Elapsed time: $elapsed_seconds"

Answer 1

目前，您唯一的选择是添加更多日志记录并发布部署以测试此脚本，然后查看您的部署日志。听起来你不知道它为什么会失败，只有日志才能告诉你。

尝试添加日志并查看会发生什么。我们应该按原样执行您的脚本，所以它不应该有任何不同，但如果不查看日志就很难判断。

祝你好运， -阿萨夫