在 Powershell 中使用 Start-ThreadJob 复制项目

Question

关闭此线程的背面：Copy-item using invoke-async in Powershell 我有以下内容：

@mklement0 的方法（复制自 here 并由 here 修改）有效，但因为它为每个文件创建一个线程异常缓慢，并且在我的测试系统上使用约 14,000 个文件消耗 > 4GB 内存：

# This works but is INCREDIBLY SLOW because it creates a thread per file
 Create sample CSV file with 10 rows.
 $FileList = Join-Path ([IO.Path]::GetTempPath()) "tmp.$PID.csv"
 @'
 Foo,SrcFileName,DestFileName,Bar
 1,c:\tmp\a,\\server\share\a,baz
 2,c:\tmp\b,\\server\share\b,baz
 3,c:\tmp\c,\\server\share\c,baz
 4,c:\tmp\d,\\server\share\d,baz
 5,c:\tmp\e,\\server\share\e,baz
 6,c:\tmp\f,\\server\share\f,baz
 7,c:\tmp\g,\\server\share\g,baz
 8,c:\tmp\h,\\server\share\h,baz
 9,c:\tmp\i,\\server\share\i,baz
 10,c:\tmp\j,\\server\share\j,baz
 '@ | Set-Content $FileList

# How many threads at most to run concurrently.
 $NumCopyThreads = 8

Write-Host 'Creating jobs...'
$dtStart = [datetime]::UtcNow

# Import the CSV data and transform it to [pscustomobject] instances
# with only .SrcFileName and .DestFileName properties - they take
# the place of your original [fileToCopy] instances.
$jobs = Import-Csv $FileList | Select-Object SrcFileName, DestFileName | 
  ForEach-Object {
    # Start the thread job for the file pair at hand.
    Start-ThreadJob -ThrottleLimit $NumCopyThreads -ArgumentList $_ { 
        param($f) 
        [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
        [String]$DestinationDir = $DestinationFilePath.DirectoryName
        if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
            new-item -Path $DestinationDir -ItemType Directory #-Verbose
        }
        copy-item -path $f.srcFileName -Destination $f.destFilename
        "Copied $($f.SrcFileName) to $($f.DestFileName)"
    }
  }

Write-Host "Waiting for $($jobs.Count) jobs to complete..."

# Synchronously wait for all jobs (threads) to finish and output their results
# *as they become available*, then remove the jobs.
# NOTE: Output will typically NOT be in input order.
Receive-Job -Job $jobs -Wait -AutoRemoveJob
Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"

# Clean up the temp. file
Remove-Item $FileList

This article（特别是 PowerShell 作业部分）给了我将完整列表拆分为 1000 个文件批次的想法，当它在我的测试用例中运行时，我得到 15 个线程（因为我有大约 14,500 个文件）但线程只处理每个“块”中的第一个文件，然后停止：

<#
.SYNOPSIS
<Brief description>
For examples type:
Get-Help .\<filename>.ps1 -examples
.DESCRIPTION
Copys files from one path to another
.PARAMETER FileList
e.g. C:\path\to\list\of\files\to\copy.txt
.PARAMETER NumCopyThreads
default is 8 (but can be 100 if you want to stress the machine to maximum!)
.PARAMETER LogName
default is output.csv located in the same path as the Filelist
.EXAMPLE
to run using defaults just call this file:
.\CopyFilesToBackup
to run using anything else use this syntax:
.\CopyFilesToBackup -filelist C:\path\to\list\of\files\to\copy.txt -NumCopyThreads 20 -LogName C:\temp\backup.log -CopyMethod Runspace
.\CopyFilesToBackup -FileList .\copytest.csv -NumCopyThreads 30 -Verbose
.NOTES
#>

[CmdletBinding()] 
Param( 
    [String] $FileList = "C:\temp\copytest.csv", 
    [int] $NumCopyThreads = 8,
    [String] $LogName
) 

$filesPerBatch = 1000

$files = Import-Csv $FileList | Select-Object SrcFileName, DestFileName

$i = 0
$j = $filesPerBatch - 1
$batch = 1

Write-Host 'Creating jobs...'
$dtStart = [datetime]::UtcNow

$jobs = while ($i -lt $files.Count) {
    $fileBatch = $files[$i..$j]

    $jobName = "Batch$batch"
    Start-ThreadJob -Name $jobName -ThrottleLimit $NumCopyThreads -ArgumentList ($fileBatch) -ScriptBlock {
        param($filesInBatch)
        foreach ($f in $filesInBatch) {
            [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
            [String]$DestinationDir = $DestinationFilePath.DirectoryName
            if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
                new-item -Path $DestinationDir -ItemType Directory -Verbose
            }
            copy-item -path $f.srcFileName -Destination $f.DestFileName -Verbose
        }
    } 

    $batch += 1
    $i = $j + 1
    $j += $filesPerBatch

    if ($i -gt $files.Count) {$i = $files.Count}
    if ($j -gt $files.Count) {$j = $files.Count}
}

Write-Host "Waiting for $($jobs.Count) jobs to complete..."

Receive-Job -Job $jobs -Wait -AutoRemoveJob
Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"

我觉得我遗漏了一些明显的东西，但我不知道是什么。

谁能帮忙？

Answer 1

变化：

Start-ThreadJob -Name $jobName -ThrottleLimit $NumCopyThreads -ArgumentList ($fileBatch) -ScriptBlock {

到

Start-ThreadJob -Name $jobName -ThrottleLimit $NumCopyThreads -ArgumentList (,$fileBatch) -ScriptBlock {

注意参数列表中$fileBatch 前面的逗号。

修复它的原因是因为ArgumentList 需要一个数组并将每个元素提供给参数。您试图将整个数组传递给第一个参数，这意味着您必须将数组放入数组中。

显然（这对我来说是个新闻），Powershell 很乐意将您的字符串视为foreach 循环中的单个项目数组，这就是为什么在每批中处理第一个项目的原因。

Answer 2

因此，我们经过一周的反复试验才得出这一点，总的来说，我对结果非常满意。我将在下面分享的脚本负责处理我正在使用的文件的大约 3 个步骤：

1. 创建文件夹
2. 将文件复制到新文件夹
3. 验证文件复制没有错误

在 Excel 中执行步骤 1) 和 2) 的时间只有

.SYNOPSIS
<Brief description>
For examples type:
Get-Help .\<filename>.ps1 -examples
.DESCRIPTION
Copys files from one path to another
.PARAMETER FileList
e.g. C:\path\to\list\of\files\to\copy.txt
.PARAMETER NumCopyThreads
default is 8 (but can be 100 if you want to stress the machine to maximum!)
.PARAMETER FilesPerBatch
default is 1000 this can be tweaked if performance becomes an issue because the Threading will HAMMER any network you run it on.
.PARAMETER LogName
Desired log file output. Must include full or relative (.\blah) path. If blank, location of FileList is used.
.PARAMETER DryRun
Boolean value denoting whether we're testing this thing or not. (Default is $false)
.PARAMETER DryRunNum
The number of files to Dry Run. (Default is 100)
.EXAMPLE
to run using defaults just call this file:
.\CopyFilesToBackup
to run using anything else use this syntax:
.\CopyFilesToBackup -filelist C:\path\to\list\of\files\to\copy.txt -NumCopyThreads 20 -LogName C:\temp\backup.log -CopyMethod Runspace
.\CopyFilesToBackup -FileList .\copytest.csv -NumCopyThreads 30 -Verbose
.NOTES
#>

[CmdletBinding()] 
Param( 
    [String] $FileList = "C:\temp\copytest.csv", 
    [int] $NumCopyThreads =75,
    [String] $JobName,
    [int] $FilesPerBatch = 1000,
    [String] $LogName,
    [Boolean] $DryRun = $false, #$true,
    [int] $DryRunNum = 100
) 



Write-Host 'Creating log file if it does not exist...'

function CreateFile([string]$filepath) {
    if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($filepath)))) {
        new-item -Path $filepath -ItemType File
    }
    if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($filepath)))) {
        return $false
    } else {
        return $true
    }
}

$dtStart = [datetime]::UtcNow

if ($LogName -eq "") {
    [System.IO.Fileinfo]$CsvPath = $FileList
    [String]$LogDirectory = $CsvPath.DirectoryName
    [string]$LognameBaseName = $CsvPath.BaseName
    $LogName = $LogDirectory + "\" + $LognameBaseName + ".log"
    if (-not (CreateFile($LogName)) ) { 
        write-host "Unable to create log, exiting now!"
        Break
    }
}
else {
    if (-not (CreateFile($LogName)) ) { 
        write-host "Unable to create log, exiting now!"
        Break
    }
}

Add-Content -Path $LogName -Value "[INFO],[Src Filename],[Src Hash],[Dest Filename],[Dest Hash]"

Write-Host 'Loading CSV data into memory...'

$files = Import-Csv $FileList | Select-Object SrcFileName, DestFileName

Write-Host 'CSV Data loaded...'

Write-Host 'Collecting unique Directory Names...'

$allFolders = New-Object "System.Collections.Generic.List[PSCustomObject]"

ForEach ($f in $files) {
    [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
    [String]$DestinationDir = $DestinationFilePath.DirectoryName
    $allFolders.add($DestinationDir)
}

$folders = $allFolders | get-unique

Write-Host 'Creating Directories...'
foreach($DestinationDir in $folders) {
    if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
        new-item -Path $DestinationDir -ItemType Directory | Out-Null #-Verbose
    }
}
Write-Host 'Finished Creating Directories...'
$scriptBlock = {
    param(
        [PSCustomObject]$filesInBatch, 
        [String]$LogFileName)
        function ProcessFileAndHashToLog {
            param( [String]$LogFileName, [PSCustomObject]$FileColl)
            foreach ($f in $FileColl) {
                $mutex = New-object -typename 'Threading.Mutex' -ArgumentList $false, 'MyInterProcMutex'
                # [System.IO.Fileinfo]$DestinationFilePath = $f.DestFileName
                # [String]$DestinationDir = $DestinationFilePath.DirectoryName
                # if (-not (Test-path([Management.Automation.WildcardPattern]::Escape($DestinationDir)))) {
                #     new-item -Path $DestinationDir -ItemType Directory | Out-Null #-Verbose
                # }
                copy-item -path $f.srcFileName -Destination $f.DestFileName | Out-Null #-Verbose

                $srcHash = (Get-FileHash -Path $f.srcFileName -Algorithm SHA1).Hash #| Out-Null #could also use MD5 here but it needs testing
                if (Test-path([Management.Automation.WildcardPattern]::Escape($f.destFileName))) {
                    $destHash = (Get-FileHash -Path $f.destFileName -Algorithm SHA1).Hash #| Out-Null #could also use MD5 here but it needs testing
                } else {
                    $destHash = $f.destFileName + " not found at location."
                }
                if (-not ($null -eq $destHash) -and -not ($null -eq $srcHash)) {
                    $info = $f.srcFileName + "," + $srcHash + "," + $f.destFileName + "," + $destHash
                }
                $mutex.WaitOne() | Out-Null
                $DateTime = Get-date -Format "yyyy-MM-dd HH:mm:ss:fff"
                if ($DryRun) { Write-Host 'Writing to log file: '$LogFileName'...' }
                Add-Content -Path $LogFileName -Value "$DateTime,$Info"
                $mutex.ReleaseMutex() | Out-Null
            }
        }
        ProcessFileAndHashToLog -LogFileName $LogFileName -FileColl $filesInBatch
}

$i = 0
$j = $filesPerBatch - 1
$batch = 1
Write-Host 'Creating jobs...'
if (-not ($DryRun)) {
    $jobs = while ($i -lt $files.Count) {
        $fileBatch = $files[$i..$j]
        Start-ThreadJob -Name $jobName -ArgumentList $fileBatch, $LogName -ScriptBlock $scriptBlock #-ThrottleLimit $NumCopyThreads -ArgumentList $fileBatch, $LogName -ScriptBlock $scriptBlock
        $batch += 1
        $i = $j + 1
        $j += $filesPerBatch
        if ($i -gt $files.Count) {$i = $files.Count}
        if ($j -gt $files.Count) {$j = $files.Count}
    }
    Write-Host "Waiting for $($jobs.Count) jobs to complete..."
    Receive-Job -Job $jobs -Wait -AutoRemoveJob
} else {
    Write-Host 'Going in Dry...'
    $DummyFileBatch = $files[$i..$DryRunNum]
    & $scriptBlock -filesInBatch $DummyFileBatch -LogFileName $LogName
    Write-Host 'That wasn''t so bad was it..?'
}

Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"

（我很乐意接受改进上述解决方案的建议。）