大文件复制过程中的进度（复制项目和写入进度？）

Question

有没有办法在 PowerShell 中复制一个非常大的文件（从一台服务器到另一台服务器）并显示其进度？

有一些解决方案可以将 Write-Progress 与循环结合使用来复制许多文件并显示进度。但是我似乎找不到任何可以显示单个文件进度的内容。

有什么想法吗？

Answer 1

仅使用 BitsTransfer 似乎是一个更好的解决方案，它似乎在大多数具有 PowerShell 2.0 或更高版本的 Windows 机器上都提供了 OOTB。

Import-Module BitsTransfer
Start-BitsTransfer -Source $Source -Destination $Destination -Description "Backup" -DisplayName "Backup"

Answer 2

我还没有听说Copy-Item 的进展。如果您不想使用任何外部工具，可以尝试使用流。缓冲区大小不同，您可以尝试不同的值（从 2kb 到 64kb）。

function Copy-File {
    param( [string]$from, [string]$to)
    $ffile = [io.file]::OpenRead($from)
    $tofile = [io.file]::OpenWrite($to)
    Write-Progress -Activity "Copying file" -status "$from -> $to" -PercentComplete 0
    try {
        [byte[]]$buff = new-object byte[] 4096
        [long]$total = [int]$count = 0
        do {
            $count = $ffile.Read($buff, 0, $buff.Length)
            $tofile.Write($buff, 0, $count)
            $total += $count
            if ($total % 1mb -eq 0) {
                Write-Progress -Activity "Copying file" -status "$from -> $to" `
                   -PercentComplete ([long]($total * 100 / $ffile.Length))
            }
        } while ($count -gt 0)
    }
    finally {
        $ffile.Dispose()
        $tofile.Dispose()
        Write-Progress -Activity "Copying file" -Status "Ready" -Completed
    }
}

Answer 3

或者这个选项使用本机 Windows 进度条...

$FOF_CREATEPROGRESSDLG = "&H0&"

$objShell = New-Object -ComObject "Shell.Application"

$objFolder = $objShell.NameSpace($DestLocation) 

$objFolder.CopyHere($srcFile, $FOF_CREATEPROGRESSDLG)

Answer 4

cmd /c copy /z src dest

不是纯 PowerShell，而是在 PowerShell 中可执行，并以百分比显示进度

Answer 5

我修改了 stej 中的代码（这很棒，正是我需要的！）以使用更大的缓冲区，[long] 用于更大的文件，并使用 System.Diagnostics.Stopwatch 类来跟踪经过的时间并估计剩余时间。

还增加了传输过程中传输速率的报告，并输出总经过时间和总传输速率。

使用 4MB（4096*1024 字节）缓冲区，通过 wifi 从 NAS 复制到笔记本电脑上的 USB 记忆棒，比 Win7 本机吞吐量更好。

待办事项列表：

• 添加错误处理（catch）
• 处理 get-childitem 文件列表作为输入
• 复制多个文件时嵌套进度条（文件 x of y，% if 复制的总数据等）
• 缓冲区大小的输入参数

随意使用/改进:-)

function Copy-File {
param( [string]$from, [string]$to)
$ffile = [io.file]::OpenRead($from)
$tofile = [io.file]::OpenWrite($to)
Write-Progress `
    -Activity "Copying file" `
    -status ($from.Split("\")|select -last 1) `
    -PercentComplete 0
try {
    $sw = [System.Diagnostics.Stopwatch]::StartNew();
    [byte[]]$buff = new-object byte[] (4096*1024)
    [long]$total = [long]$count = 0
    do {
        $count = $ffile.Read($buff, 0, $buff.Length)
        $tofile.Write($buff, 0, $count)
        $total += $count
        [int]$pctcomp = ([int]($total/$ffile.Length* 100));
        [int]$secselapsed = [int]($sw.elapsedmilliseconds.ToString())/1000;
        if ( $secselapsed -ne 0 ) {
            [single]$xferrate = (($total/$secselapsed)/1mb);
        } else {
            [single]$xferrate = 0.0
        }
        if ($total % 1mb -eq 0) {
            if($pctcomp -gt 0)`
                {[int]$secsleft = ((($secselapsed/$pctcomp)* 100)-$secselapsed);
                } else {
                [int]$secsleft = 0};
            Write-Progress `
                -Activity ($pctcomp.ToString() + "% Copying file @ " + "{0:n2}" -f $xferrate + " MB/s")`
                -status ($from.Split("\")|select -last 1) `
                -PercentComplete $pctcomp `
                -SecondsRemaining $secsleft;
        }
    } while ($count -gt 0)
$sw.Stop();
$sw.Reset();
}
finally {
    write-host (($from.Split("\")|select -last 1) + `
     " copied in " + $secselapsed + " seconds at " + `
     "{0:n2}" -f [int](($ffile.length/$secselapsed)/1mb) + " MB/s.");
     $ffile.Close();
     $tofile.Close();
    }
}

Answer 6

我不知道。无论如何，我不建议为此使用 copy-item 。我不认为它被设计为像 robocopy.exe 那样健壮，以支持重试，您希望通过网络复制非常大的文件。

Answer 7

我发现上面的例子都没有满足我的需求，我想复制一个带有子目录的目录，问题是我的源目录有太多文件，所以我很快达到了 BITS 文件限制（我有 > 1500 个文件）总目录大小相当大。

我发现了一个使用 robocopy 的函数，它是 https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress/ 的一个很好的起点，但是我发现它不够健壮，它不能优雅地处理斜杠、空格，并且在脚本执行时没有停止复制被暂停了。

这是我的改进版：

function Copy-ItemWithProgress
{
    <#
    .SYNOPSIS
    RoboCopy with PowerShell progress.

    .DESCRIPTION
    Performs file copy with RoboCopy. Output from RoboCopy is captured,
    parsed, and returned as Powershell native status and progress.

    .PARAMETER Source
    Directory to copy files from, this should not contain trailing slashes

    .PARAMETER Destination
    DIrectory to copy files to, this should not contain trailing slahes

    .PARAMETER FilesToCopy
    A wildcard expresion of which files to copy, defaults to *.*

    .PARAMETER RobocopyArgs
    List of arguments passed directly to Robocopy.
    Must not conflict with defaults: /ndl /TEE /Bytes /NC /nfl /Log

    .PARAMETER ProgressID
    When specified (>=0) will use this identifier for the progress bar

    .PARAMETER ParentProgressID
    When specified (>= 0) will use this identifier as the parent ID for progress bars
    so that they appear nested which allows for usage in more complex scripts.

    .OUTPUTS
    Returns an object with the status of final copy.
    REMINDER: Any error level below 8 can be considered a success by RoboCopy.

    .EXAMPLE
    C:\PS> .\Copy-ItemWithProgress c:\Src d:\Dest

    Copy the contents of the c:\Src directory to a directory d:\Dest
    Without the /e or /mir switch, only files from the root of c:\src are copied.

    .EXAMPLE
    C:\PS> .\Copy-ItemWithProgress '"c:\Src Files"' d:\Dest /mir /xf *.log -Verbose

    Copy the contents of the 'c:\Name with Space' directory to a directory d:\Dest
    /mir and /XF parameters are passed to robocopy, and script is run verbose

    .LINK
    https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress

    .NOTES
    By Keith S. Garner (KeithGa@KeithGa.com) - 6/23/2014
    With inspiration by Trevor Sullivan @pcgeek86
    Tweaked by Justin Marshall - 02/20/2020

    #>

    [CmdletBinding()]
    param(
        [Parameter(Mandatory=$true)]
        [string]$Source,
        [Parameter(Mandatory=$true)]
        [string]$Destination,
        [Parameter(Mandatory=$false)]
        [string]$FilesToCopy="*.*",
        [Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)] 
        [string[]] $RobocopyArgs,
        [int]$ParentProgressID=-1,
        [int]$ProgressID=-1
    )

    #handle spaces and trailing slashes
    $SourceDir = '"{0}"' -f ($Source -replace "\\+$","")
    $TargetDir = '"{0}"' -f ($Destination -replace "\\+$","")


    $ScanLog  = [IO.Path]::GetTempFileName()
    $RoboLog  = [IO.Path]::GetTempFileName()
    $ScanArgs = @($SourceDir,$TargetDir,$FilesToCopy) + $RobocopyArgs + "/ndl /TEE /bytes /Log:$ScanLog /nfl /L".Split(" ")
    $RoboArgs = @($SourceDir,$TargetDir,$FilesToCopy) + $RobocopyArgs + "/ndl /TEE /bytes /Log:$RoboLog /NC".Split(" ")

    # Launch Robocopy Processes
    write-verbose ("Robocopy Scan:`n" + ($ScanArgs -join " "))
    write-verbose ("Robocopy Full:`n" + ($RoboArgs -join " "))
    $ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs
    try
    {
        $RoboRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $RoboArgs
        try
        {
            # Parse Robocopy "Scan" pass
            $ScanRun.WaitForExit()
            $LogData = get-content $ScanLog
            if ($ScanRun.ExitCode -ge 8)
            {
                $LogData|out-string|Write-Error
                throw "Robocopy $($ScanRun.ExitCode)"
            }
            $FileSize = [regex]::Match($LogData[-4],".+:\s+(\d+)\s+(\d+)").Groups[2].Value
            write-verbose ("Robocopy Bytes: $FileSize `n" +($LogData -join "`n"))
            #determine progress parameters
            $ProgressParms=@{}
            if ($ParentProgressID -ge 0) {
                $ProgressParms['ParentID']=$ParentProgressID
            }
            if ($ProgressID -ge 0) {
                $ProgressParms['ID']=$ProgressID
            } else {
                $ProgressParms['ID']=$RoboRun.Id
            }
            # Monitor Full RoboCopy
            while (!$RoboRun.HasExited)
            {
                $LogData = get-content $RoboLog
                $Files = $LogData -match "^\s*(\d+)\s+(\S+)"
                if ($null -ne $Files )
                {
                    $copied = ($Files[0..($Files.Length-2)] | ForEach-Object {$_.Split("`t")[-2]} | Measure-Object -sum).Sum
                    if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
                    {
                        write-progress Copy -ParentID $ProgressParms['ID'] -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
                        $Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
                    }
                    else
                    {
                        write-progress Copy -ParentID $ProgressParms['ID'] -Complete
                    }
                    write-progress ROBOCOPY  -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1] @ProgressParms
                }
            }
        } finally {
            if (!$RoboRun.HasExited) {Write-Warning "Terminating copy process with ID $($RoboRun.Id)..."; $RoboRun.Kill() ; }
            $RoboRun.WaitForExit()
            # Parse full RoboCopy pass results, and cleanup
            (get-content $RoboLog)[-11..-2] | out-string | Write-Verbose
            remove-item $RoboLog
            write-output ([PSCustomObject]@{ ExitCode = $RoboRun.ExitCode })

        }
    } finally {
        if (!$ScanRun.HasExited) {Write-Warning "Terminating scan process with ID $($ScanRun.Id)..."; $ScanRun.Kill() }
        $ScanRun.WaitForExit()

        remove-item $ScanLog
    }
}

Answer 8

讨厌成为老话题的人，但我发现这篇文章非常有用。在 stej 对 sn-ps 进行了性能测试，Graham Gold 对其进行了改进，再加上 Nacht 的 BITS 建议，我认为：

1. 我真的喜欢 Graham 的时间估计和速度读数命令。
2. 我也真的喜欢使用 BITS 作为我的传输方法显着提高的速度。

面对两者之间的抉择……我发现 Start-BitsTransfer 支持异步模式。所以这是我合并两者的结果。

function Copy-File {
    # ref: https://stackoverflow.com/a/55527732/3626361
    param([string]$From, [string]$To)

    try {
        $job = Start-BitsTransfer -Source $From -Destination $To `
            -Description "Moving: $From => $To" `
            -DisplayName "Backup" -Asynchronous

        # Start stopwatch
        $sw = [System.Diagnostics.Stopwatch]::StartNew()
        Write-Progress -Activity "Connecting..."

        while ($job.JobState.ToString() -ne "Transferred") {
            switch ($job.JobState.ToString()) {
                "Connecting" {
                    break
                }
                "Transferring" {
                    $pctcomp = ($job.BytesTransferred / $job.BytesTotal) * 100
                    $elapsed = ($sw.elapsedmilliseconds.ToString()) / 1000

                    if ($elapsed -eq 0) {
                        $xferrate = 0.0
                    }
                    else {
                        $xferrate = (($job.BytesTransferred / $elapsed) / 1mb);
                    }

                    if ($job.BytesTransferred % 1mb -eq 0) {
                        if ($pctcomp -gt 0) {
                            $secsleft = ((($elapsed / $pctcomp) * 100) - $elapsed)
                        }
                        else {
                            $secsleft = 0
                        }

                        Write-Progress -Activity ("Copying file '" + ($From.Split("\") | Select-Object -last 1) + "' @ " + "{0:n2}" -f $xferrate + "MB/s") `
                            -PercentComplete $pctcomp `
                            -SecondsRemaining $secsleft
                    }
                    break
                }
                "Transferred" {
                    break
                }
                Default {
                    throw $job.JobState.ToString() + " unexpected BITS state."
                }
            }
        }

        $sw.Stop()
        $sw.Reset()
    }
    finally {
        Complete-BitsTransfer -BitsJob $job
        Write-Progress -Activity "Completed" -Completed
    }
}

Answer 9

此递归函数将文件和目录从源路径递归复制到目标路径

如果目标路径上已存在文件，则仅使用较新的文件复制它们。

Function Copy-FilesBitsTransfer(
        [Parameter(Mandatory=$true)][String]$sourcePath, 
        [Parameter(Mandatory=$true)][String]$destinationPath, 
        [Parameter(Mandatory=$false)][bool]$createRootDirectory = $true)
{
    $item = Get-Item $sourcePath
    $itemName = Split-Path $sourcePath -leaf
    if (!$item.PSIsContainer){ #Item Is a file

        $clientFileTime = Get-Item $sourcePath | select LastWriteTime -ExpandProperty LastWriteTime

        if (!(Test-Path -Path $destinationPath\$itemName)){
            Start-BitsTransfer -Source $sourcePath -Destination $destinationPath -Description "$sourcePath >> $destinationPath" -DisplayName "Copy Template file" -Confirm:$false
            if (!$?){
                return $false
            }
        }
        else{
            $serverFileTime = Get-Item $destinationPath\$itemName | select LastWriteTime -ExpandProperty LastWriteTime

            if ($serverFileTime -lt $clientFileTime)
            {
                Start-BitsTransfer -Source $sourcePath -Destination $destinationPath -Description "$sourcePath >> $destinationPath" -DisplayName "Copy Template file" -Confirm:$false
                if (!$?){
                    return $false
                }
            }
        }
    }
    else{ #Item Is a directory
        if ($createRootDirectory){
            $destinationPath = "$destinationPath\$itemName"
            if (!(Test-Path -Path $destinationPath -PathType Container)){
                if (Test-Path -Path $destinationPath -PathType Leaf){ #In case item is a file, delete it.
                    Remove-Item -Path $destinationPath
                }

                New-Item -ItemType Directory $destinationPath | Out-Null
                if (!$?){
                    return $false
                }

            }
        }
        Foreach ($fileOrDirectory in (Get-Item -Path "$sourcePath\*"))
        {
            $status = Copy-FilesBitsTransfer $fileOrDirectory $destinationPath $true
            if (!$status){
                return $false
            }
        }
    }

    return $true
}

Answer 10

Hey, Scripting Guy! Blog 的 Sean Kearney 有一个我发现效果很好的解决方案。

Function Copy-WithProgress
{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory=$true,
            ValueFromPipelineByPropertyName=$true,
            Position=0)]
        $Source,
        [Parameter(Mandatory=$true,
            ValueFromPipelineByPropertyName=$true,
            Position=0)]
        $Destination
    )

    $Source=$Source.tolower()
    $Filelist=Get-Childitem "$Source" –Recurse
    $Total=$Filelist.count
    $Position=0

    foreach ($File in $Filelist)
    {
        $Filename=$File.Fullname.tolower().replace($Source,'')
        $DestinationFile=($Destination+$Filename)
        Write-Progress -Activity "Copying data from '$source' to '$Destination'" -Status "Copying File $Filename" -PercentComplete (($Position/$total)*100)
        Copy-Item $File.FullName -Destination $DestinationFile
        $Position++
    }
}

然后使用它：

Copy-WithProgress -Source $src -Destination $dest

Answer 11

Trevor Sullivan 撰写了一篇关于如何在 Robocopy 上将名为 Copy-ItemWithProgress 的命令添加到 PowerShell 的文章。

Answer 12

在复制单个大文件时获取更新进度

在复制文件时获取进度条的一种方法是在单独的进程中启动复制操作，然后从父进程轮询目标文件的大小，直到子进程完成。

function Copy-LargeFileWithProgress {
    param(
        [string] $SourcePath,
        [string] $DestPath
    )
    
    $source = Get-Item $SourcePath
    $name   = $source.Name
    $size   = $source.Size
    $argv   = @('-Command', "Copy-Item '$SourcePath' '$DestPath'")

    $proc   = Start-Process 'pwsh' $argv -PassThru 
 
    while (-not $proc.HasExited) {
        Start-Sleep -Seconds 2
        if (Test-Path $DestPath) {
            $destSize = (Get-Item $DestPath).Size
            $status   = '{0:N0}/{1:N0} bytes' -f $destSize, $size
            $complete = [Math]::Max($destSize / $size * 100, 100)
    
            Write-Progress -Activity        "Copying $name" `
                           -Status          $status `
                           -PercentComplete $complete
        }
    }
}

上面的代码在 Linux 上运行，但在 Windows 上没有试过。在 Windows 上，Start-Process 可能需要-WindowStyle Minimized 选项。

上面的实现是稀疏的，以简洁地演示该方法。如果有人想使用它，它可以使用一些润色，例如以更好的格式显示状态消息（即，1.51G/210.00G，而不是原始字节数）。此外，最初由(Get-Item $DestPath).Size 报告的大小可能在几次迭代中不正确（因此使用了 max 操作）。这可以更优雅地解释。