使用 Powershell 重试 BITS 文件传输失败的下载

Question

我正在从网络服务器进行每日图像的 BITS 传输，并且在传输过程中我不断收到随机掉落。

在循环下载时，我偶尔会收到“连接过早关闭”或“安全通道支持出现错误”。每个文件夹中大约有 180 张图像，其中可能有 5-10% 发生这种情况。对于那些没有完成的，我需要重试下载。

我的代码如下 - 我不完美的解决方法是运行循环两次，但我希望找到更好的解决方案。

# Set the URL where the images are located
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'

# Set the local path where the images will be stored
$path = 'C:\images\Wind_Waves\latest\'

# Create a list of all assets returned from $url
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | Where-Object{ $_.tagName -eq 'A' -and $_.href.ToLower().EndsWith("jpg") }

# Create a list of all href items from the table & call it $images
$images = $table.href 

# Enumerate all of the images - for troubleshooting purposes - can be removed
$images

# Check to make sure there are images available for download - arbitrarily picked more than 2 $images
if($images.count -gt 2){

    # Delete all of the files in the "latest" folder
    Remove-Item ($path + "*.*") -Force
# For loop to check to see if we already have the image and, if not, download it
ForEach ($image in $images)
{
if(![System.IO.File]::Exists($path + $image)){
    Write-Output "Downloading: " $image
    Start-BitsTransfer -Source ($url + $image) -Destination $path -TransferType Download -RetryInterval 60
    Start-Sleep 2
    }
}
Get-BitsTransfer | Where-Object {$_.JobState -eq "Transferred"} | Complete-BitsTransfer
} else {
Write-Output "No images to download"}

Answer 1

我在您的代码中没有看到任何错误处理以在失败时恢复/重试/重新启动。

1. 意思是为什么循环或Get中没有try/catch？
2. 如果 Get 在循环中的每个下载作业中打开，为什么它在循环之外？

TransferType默认为Download，无需指定，一般会报错。

所以，像这样。我确实对此进行了测试，但从未失败过。然而，我有一个非常高速的互联网连接。如果您在企业内部执行此操作，边缘设备（过滤器、代理也可能会减慢速度，可能会导致超时。）

$url  = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object{
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)

<#
Check to make sure there are images available for download - arbitrarily 
picked more than 2 $images
#>
if($images.count -gt 2)
{
    Remove-Item ($path + '*.*') -Force
    ForEach ($image in $images)
    {
        Try
        {
            Write-Verbose -Message "Downloading: $image" -Verbose

            if(![System.IO.File]::Exists($path + $image))
            {
                $StartBitsTransferSplat = @{
                    Source              = ($url + $image) 
                    Destination         = $path 
                    RetryInterval       = 60
                }
                Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                Start-Sleep 2
            }

            Get-BitsTransfer | 
            Where-Object {$PSItem.JobState -eq 'Transferred'} | 
            Complete-BitsTransfer
        }
        Catch
        {
            $PSItem.Exception.Message
            Write-Warning -Message "Download of $image not complete or failed. Attempting a resume/retry" -Verbose
            Get-BitsTransfer -Name $image | Resume-BitsTransfer

        }
    }
} 
else 
{
    Write-Warning -Message 'No images to download'
    $PSItem.Exception.Message
}

查看帮助文件

Resume-BitsTransfer 模块：bitstransfer 恢复 BITS 传输作业。

# Example 1: Resume all BITS transfer jobs owned by the current user
Get-BitsTransfer | Resume-BitsTransfer

# Example 2: Resume a new BITS transfer job that was initially suspended
$Bits = Start-BitsTransfer -DisplayName "MyJob" -Suspended
Add-BitsTransfer -BitsJob $Bits  -ClientFileName C:\myFile -ServerFileName http://www.SomeSiteName.com/file1
Resume-BitsTransfer -BitsJob $Bits -Asynchronous

# Example 3: Resume the BITS transfer by the specified display name
Get-BitsTransfer -Name "TestJob01" | Resume-BitsTransfer

Answer 2

这是上述代码的稍微修改后的版本。发生错误时，BITS 传输作业对象似乎消失了，因此尝试查找/恢复该作业是没有用的。相反，我将整个 Try-Catch 块包装在一个 while 循环中，并在下载文件时退出。

$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
$MaxRetries = 3     # Initialize the maximum number of retry attempts.

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object {
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)

<#
Check to make sure there are images available for download - arbitrarily 
picked more than 2 $images
#>
if ($images.count -gt 2) {
    Remove-Item ($path + '*.*') -Force
    ForEach ($image in $images) {
        # Due to occasional failures to transfer, wrap the BITS transfer in a while loop
        # re-initialize the exit counter for each new image
        $retryCount = 0
        while ($retryCount -le $MaxRetries){
            Try {
                Write-Verbose -Message "Downloading: $image" -Verbose

                if (![System.IO.File]::Exists($path + $image)) {
                    $StartBitsTransferSplat = @{
                        Source        = ($url + $image) 
                        Destination   = $path 
                        RetryInterval = 60
                    }
                    Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                    Start-Sleep 2
                }

                # To get here, the transfer must have finished, so set the counter
                # greater than the max value to exit the loop
                $retryCount = $MaxRetries + 1
            } # End Try block

            Catch {
                $PSItem.Exception.Message
                $retryCount += 1
                Write-Warning -Message "Download of $image not complete or failed. Attempting retry #: $retryCount" -Verbose
            } # End Catch Block
            
        } # End While loop for retries

    } # End of loop over images

} # End of test for new images
else {
    Write-Warning -Message 'No images to download'
    $PSItem.Exception.Message
} # End of result for no new images

Answer 3

这里是 postanote 提供的代码和一个 Do-While 循环的组合，如果抛出错误，可以重试下载最多 5 次。

$url  = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object{
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>

($images = $table.href)

 <# Check to make sure there are images available for download - arbitrarily 
    picked more than 2 $images #>

    if($images.count -gt 2)
    {
        Remove-Item ($path + '*.*') -Force
        ForEach ($image in $images)
        {
            # Create a Do-While loop to retry downloads up to 5 times if they fail
            $Stoploop = $false
            [int]$Retrycount = "0"
            do{
            Try
            {
                Write-Verbose -Message "Downloading: $image" -Verbose
    
                if(![System.IO.File]::Exists($path + $image))
                {
                    $StartBitsTransferSplat = @{
                        Source              = ($url + $image) 
                        Destination         = $path 
                        RetryInterval       = 60
                    }
                    Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                    Start-Sleep 10
                    $Stoploop = $true
                }
    
                Get-BitsTransfer | 
                Where-Object {$PSItem.JobState -eq 'Transferred'} | 
                Complete-BitsTransfer
            }
            Catch
            {
                if ($Retrycount -gt 5){
                $PSItem.Exception.Message
                Write-Warning -Message "Download of $image not complete or failed." -Verbose
                $Stoploop = $true
                }
                else {
                Write-Host "Could not download the image, retrying..."
                Start-Sleep 10
                $Retrycount = $Retrycount + 1
                }
            }
        }
        While ($Stoploop -eq $false)
        }
    }
    else 
    {
        Write-Warning -Message 'No images to download'
        $PSItem.Exception.Message
    }