Set-S3Acl Powershell cmdlet 在 20-30k 个对象后不起作用

Question

我的客户遇到了一个问题，即他们不小心将 1300 万个对象（文件）复制到具有错误权限的 S3 存储桶。他们已经要求我的团队修复它。我们必须使用正确的 ACL 更新 S3 存储桶中的每 1300 万个文件。我们正在使用下面的 powershell 脚本来修复它。但是，当脚本在包含超过 20-30k 个对象的文件夹上运行时，它无法设置 ACL。 [它通过循环进行迭代，但它不会在 20-30k 个对象后设置权限，也不例外]

我怀疑请求可能会受到限制。有没有人遇到过这样的问题。请帮助我了解如何进行。

我正在寻找以下问题的答案： 1. 如果 API 调用在 20-30k 个对象时受到限制，我该如何修改我的脚本来克服它。 2. 为数百万个对象“修改”AWS 资源（例如为 S3 对象设置 ACL 权限）的最佳实践是什么

（我不是在寻找“BucketPolicy”方法，因为我们必须使用脚本来完成，并将 ACL 应用于每个 S3 对象）

Param (
    [Parameter(Position=0,Mandatory=$true)]
    [string]$profile,
    [Parameter(Position=1,Mandatory=$true)]
    [string]$switchToAccount,
    [Parameter(Position=2,Mandatory=$true)]
    [string]$roleName,
    [Parameter(Position=3,Mandatory=$true)]
    [string]$keyPrefix
)

#Set base AWS credentials
Set-AWSCredentials -ProfileName $profile
Set-DefaultAWSRegion -Region $region

#Get and set MFA device ARN
$userName = (Get-IAMUser).UserName
$mfaArn = "arn:aws:iam::xxxxxxxxx:mfa/" + "$userName"

#Configure CAA roles
$roleArn = "arn:aws:iam::" + "$switchToAccount" + ":role/" + "$roleName"
$roleSessionName = "xxxxxxxxxxxx"

#Prompt for MFA token and perform CAA request
$tokenCode = Read-Host -Prompt "Enter MFA token for $accountNumber"
$switchRole = Use-STSRole -RoleSessionName $roleSessionName -RoleArn $roleArn -TokenCode $tokenCode -SerialNumber $mfaArn

#Set new role for CAA
Set-AWSCredentials -Credential $switchRole.Credentials

#Declare access level for S3 Object ACL grantees
$FULL_CONTROL = [Amazon.S3.S3Permission]::FULL_CONTROL
$grants = @();

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxxx
$grantee1 = New-Object -TypeName Amazon.S3.Model.S3Grantee
$grantee1.EmailAddress = "xxxxxxxxxxxxxxxxxxx"

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxx
$grantee2 = New-Object -TypeName Amazon.S3.Model.S3Grantee
$grantee2.EmailAddress = "xxxxxxxxxxxxxxxxxxx"

#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxx
$grantee3 = New-Object -TypeName Amazon.S3.Model.S3Grantee
$grantee3.EmailAddress = "xxxxxxxxxxxxxxxxxxxxx"

#Create grant and add to grant list
$grant1 = New-Object -TypeName Amazon.S3.Model.S3Grant
$grant1.Grantee = $grantee1
$grant1.Permission = $FULL_CONTROL
$grants += $grant1

#Create grant and add to grant list
$grant2 = New-Object -TypeName Amazon.S3.Model.S3Grant
$grant2.Grantee = $grantee2
$grant2.Permission = $FULL_CONTROL
$grants += $grant2

#Create grant and add to grant list
$grant3 = New-Object -TypeName Amazon.S3.Model.S3Grant
$grant3.Grantee = $grantee3
$grant3.Permission = $FULL_CONTROL
$grants += $grant3



#Set bucket name for S3 objects
$bucketName = "xxxxxxxxxxxxxxxxxxxxxxxxx"
#Get all S3 Objects in specified bucket
$s3Objects = Get-S3Object -BucketName $bucketName -KeyPrefix $keyPrefix




#Count for progress bar
$totalObjects = $s3Objects.length
$i = 1
$fail_count = 0
$current_count = 0
$file_path = "C:\Users\Administrator\Desktop\Failed_Objects_new\" + $keyPrefix.Replace("/","_") + ".txt"
$file_path_retry = "C:\Users\Administrator\Desktop\Failed_Objects_new_retry\" + $keyPrefix.Replace("/","_") + ".txt"

        new-item $file_path -ItemType file
        new-item $file_path_retry -ItemType file

"Total Object Count:" + $totalObjects + "`n" | Out-File $file_path -Append


foreach($s3Object in $s3Objects){

    $owner = $s3Object.owner.id 
    $s3Object.name | Write-Output
    $current_count++
    #Extracts Key for each S3 object in bucket
    $key = $s3Object.Key

    #Logging
    Write-Host "Setting $bucketName | $key | $grants"

    # Pick objects that were modified on or before July 15th

    try {

        if (($s3Object.LastModified.month -lt 7)) {
            Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner
            $owner | Write-Host
        }

        elseif(($s3Object.LastModified.month -eq 7) -and ($s3Object.LastModified.day -le 15)) {

            Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner
            $owner | Write-Host
        }

    }catch{

        "Failed $bucketName | $key | $grants" | out-file $file_path -Append
        $key | Out-File $file_path_retry -Append
        $fail_count++
    }

    Write-Host "progress: " $current_count "/"  $totalObjects
    #Update progress bar
    $percentComplete = $i/$totalObjects
    Write-Progress -Activity "Setting S3 Object ACL's" -Status "$i% complete" -PercentComplete $percentComplete
    $i++

}



"`n`n Total Fail Count:" + $fail_count | Out-File $file_path -Append

Answer 1

调试问题的步骤：

1. 确定是否是节流问题。在for循环中；在 10k 个对象后中断，看看是否一切正常。

2. 另外，将 print 语句放在 try 块中 if 和 else.. 以确保它是否到达那里；什么时候失败。