通过 Powershell 拆分导出大型 CSV 文件

Question

就像每个人都知道 excel 的最大行数为 1048574。我的问题是：当达到最大行数时，然后移动到一个新的 CSV 文件我尝试了一些没有运气的东西，我无法弄清楚。

$RootFolder = Get-Content "c:\DRIVERS\myfile.txt"

foreach ($arrayOfPaths in $RootFolder){

  $csv  = $arrayofPaths -replace '^\\\\[^\\]+\\([^\\]+)\\([^\\]+).*', 'C:\output\Company_name_${1}_${2}.csv'
  Get-ChildItem $arrayOfPaths -Recurse | Where-Object {$_.mode -match "d"} | ForEach-Object {
$path = $_.FullName
  Get-Acl $path | Select-Object -Expand Access |
    Select-Object @{n='Path';e={$path}}, IdentityReference, AccessControlType,
                  FileSystemRights |
    Export-Csv $csv -Append -NoType
  }

}

最后更新：

$RootFolder = Get-Content "c:\DRIVERS\myfile.txt"

foreach ($arrayOfPaths in $RootFolder){

  $csv  = $arrayofPaths -replace '^\\\\[^\\]+\\([^\\]+)\\([^\\]+).*', 'C:\output\Company_name_${1}_${2}.csv'
  $csvIndex = 1
  $maxRows = 1000000
  $rowsLeft = $maxRows

  Get-ChildItem $arrayOfPaths -Recurse | Where-Object {$_.mode -match "d"} | ForEach-Object {
  #$csv  = $_.FullName -replace '^\\\\[^\\]+\\([^\\]+)\\([^\\]+).*', 'C:\output\Company_name_${1}_${2}.csv'# <- construct CSV path here
  $path = $_.FullName
  $thisCSV = Get-Acl $path | Select-Object -Expand Access |
    Select-Object @{n='Path';e={$path}}, IdentityReference, AccessControlType,
                  FileSystemRights |
    ConvertTo-Csv
if ($thisCSV.count -lt $rowsLeft) {
    $thisCSV | Export-Csv $csv -append -noType
    $rowsLeft -= $thisCSV.count
} else {
    $thisCSV[0..($rowsLeft - 1)] | Export-Csv $csv -append -noType
    $csvIndex++
    $csv = $csv -replace '\.csv$', "$csvIndex.csv"
    if ($thisCSV.count -gt $rowsLeft) {
        $thisCSV[$rowsLeft..($thisCSV.count - 1)] | Export-Csv $csv -append -noType
    }
    $rowsLeft = $maxRows - ($thisCSV.count - $rowsLeft)
}


  }

}

Answer 1

在临时变量中执行ConvertTo-Csv，将其.Count 属性相加，直到达到拆分点，然后将 CSV 数组拆分为一个新文件，该文件具有从此时起使用的新名称。

下面是一个简化示例，假设单个 CSV 不能超过 1000000 行。

$csv = .............
$csvIndex = 1
$maxRows = 1000000
$rowsLeft = $maxRows

...........
$thisCSV = Select-Object ........ |
           ConvertTo-Csv
if ($thisCSV.count -lt $rowsLeft) {
    $thisCSV | Export-Csv $csv -append -noType
    $rowsLeft -= $thisCSV.count
} else {
    $thisCSV[0..($rowsLeft - 1)] | Export-Csv $csv -append -noType
    $csvIndex++
    $csv = $csv -replace '\.csv$', "$csvIndex.csv"
    if ($thisCSV.count -gt $rowsLeft) {
        $thisCSV[$rowsLeft..($thisCSV.count - 1)] | Export-Csv $csv -append -noType
    }
    $rowsLeft = $maxRows - ($thisCSV.count - $rowsLeft)
}