Microsoft Excel - 压缩行数答案

【问题标题】：Microsoft Excel - condense number of rowsMicrosoft Excel - 压缩行数
【发布时间】：2021-12-05 20:47:08
【问题描述】：

我有一个包含超过 400,000 个 eircode 的 CSV，并使用 ArcGIS Pro Generate Near Table 函数来计算最近的 3 个空气质量监测器。

反过来，这又生成了超过 100 万行的新 csv，因为每个 eircode 都被一式三份并为每个监视器分配了一行。（见下例）

ObjectID	Eircode	Easting	Northing	Monitor Name	Monitor Location	Monitor Easting	Monitor Northing
1	K67AH57	717387.8	748192.3	Swords	Dublin	718046.6	747362
2	K67AH57	717387.8	748192.3	Dublin Airport	Dublin	.....	....
3	K67AH57	717387.8	748192.3	Finglass	Dublin	.....	....
4	H54W283	544468	752503	Claremorris	Mayo	.....	.....
5	H54W283	544468	752503	Rahoon	Galway	.....	.....
6	H54W283	544468	752503	Roscommon	Roscommon	.....	.....

有没有办法在 Microsoft Excel 中实现以下输出？因为我只想要一个单行的 eircode 并包含所有相关信息。

ObjectID	Eircode	Easting	Northing	M1_Name	M1_Location	M1_Easting	M1_Northing	M2_Name	M2_Location	M2_Easting	M2_Northing	M3_Name	M3_Location	M3_Easting	M3_Northing
1	K67AH57	717387.8	748192.3	Swords	Dublin	718046.6	747362	Dublin Airport	Dublin	.....	....	Finglass	Dublin	.....	....
2	H54W283	544468	752503	Claremorris	Mayo	.....	.....	Rahoon	Galway	.....	.....	Roscommon	Roscommon	.....	.....

【问题讨论】：

您使用了错误的工具。 Excel 的行数刚刚超过 100 万行，数据量如此之大真的很糟糕。
@BigBen 我不知道还能用什么，因为文件以 csv 格式保存
Python、R，任君挑选。请注意，Excel 不是 CSV。它可以处理 CSV 文件，但肯定不是 CSV。
或者，如果您坚持这条路线，您可以考虑在 Excel 中使用 Power Query。
@BigBen 我熟悉 python，但不知道如何编写我想要实现的确切更改

标签： excel csv

【解决方案1】：

这是一个使用 Python 及其 DictReader/Writer 类的解决方案，它可以让您非常简单地处理您需要的标题/列。

#!/usr/bin/env python3

import csv
import sys

COMBINED_ROW_TMPL = dict(
    ObjectID=0,
    Eircode='',
    Easting='',
    Northing='',
    M1_Name='',
    M1_Location='',
    M1_Easting='',
    M1_Northing='',
    M2_Name='',
    M2_Location='',
    M2_Easting='',
    M2_Northing='',
    M3_Name='',
    M3_Location='',
    M3_Easting='',
    M3_Northing=''
)

combined_rows = []
with open('input.csv', newline='') as f:
    reader = csv.DictReader(f)

    object_id = 0
    last_eircode = ''
    combined_row = {}
    m_i = 0
    for row in reader:
        this_eircode = row['Eircode']

        if this_eircode != last_eircode:
            # A new Eircode, set up combined_row with data that doesn't change
            object_id += 1
            combined_row = dict(COMBINED_ROW_TMPL)  # a clean copy of the template

            combined_row['ObjectID'] = object_id
            combined_row['Eircode']  = this_eircode
            combined_row['Easting']  = row['Easting']
            combined_row['Northing'] = row['Northing']

            m_i = 1  # reset monitor group to first instance (i.e., "M1")
            last_eircode = this_eircode

        # Monitor group
        pfx = f'M{m_i}'
        combined_row[pfx + '_Name']     = row['Monitor Name']
        combined_row[pfx + '_Location'] = row['Monitor Location']
        combined_row[pfx + '_Easting']  = row['Monitor Easting']
        combined_row[pfx + '_Northing'] = row['Monitor Northing']

        if m_i == 3:
            combined_rows.append(combined_row)  # only append once per monitor group

        m_i += 1


with open('output.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, COMBINED_ROW_TMPL.keys())
    writer.writeheader()
    writer.writerows(combined_rows)

当我将您的示例作为 input.csv

运行时

ObjectID,Eircode,Easting,Northing,Monitor Name,Monitor Location,Monitor Easting,Monitor Northing
1,K67AH57,717387.8,748192.3,Swords,Dublin,718046.6,747362
2,K67AH57,717387.8,748192.3,Dublin Airport,Dublin,.....,....
3,K67AH57,717387.8,748192.3,Finglass,Dublin,.....,....
4,H54W283,544468,752503,Claremorris,Mayo,.....,.....
5,H54W283,544468,752503,Rahoon,Galway,.....,.....
6,H54W283,544468,752503,Roscommon,Roscommon,.....,.....

我得到以下 output.csv：

ObjectID,Eircode,Easting,Northing,M1_Name,M1_Location,M1_Easting,M1_Northing,M2_Name,M2_Location,M2_Easting,M2_Northing,M3_Name,M3_Location,M3_Easting,M3_Northing
1,K67AH57,717387.8,748192.3,Swords,Dublin,718046.6,747362,Dublin Airport,Dublin,.....,....,Finglass,Dublin,.....,....
2,H54W283,544468,752503,Claremorris,Mayo,.....,.....,Rahoon,Galway,.....,.....,Roscommon,Roscommon,.....,.....

【讨论】：

太棒了！！！您是救生员 - 这完全按照我的需要完成，并且很容易进行任何修改！

【解决方案2】：

BigBen 对 Excel 的看法是正确的，但这激起了我的兴趣，如果您只是偶尔这样做……所以我只是在午餐时间敲出了这个极其简单（而且可能充满错误）的代码。将以下内容粘贴到 Excel 中 VBA 的代码模块中（使用 F8 进行步骤）。将数据复制到 Sheet1，它将处理到 Sheet2。从几百行开始，看看需要多长时间（观看 Excel 窗口左下角的状态栏进度）......我猜对于 1M 行来说太长了。（可以加快 10 倍以上如果可行的话，完全在数组/内存中工作）。所以，为了它的价值：

Option Explicit

Sub ToOneLine()

Dim CurrEir As String
Dim PreviousEir As String
Dim EirIndex As Integer
Dim RowIndex As Long
Dim WriteRow As Long
Dim LastRow As Long

Worksheets("Sheet1").Select

'Debug.Print Worksheets("Sheet1").Rows.Count 'result of this put in as Maximum Array size in the line below
Const MaxArraySize = 1048576
Dim ArrayEir(1 To MaxArraySize, 1 To 16)

LastRow = Worksheets("Sheet1").UsedRange.Rows.Count

        WriteRow = 0
        PreviousEir = ""
        RowIndex = 1
        
    Do While EirIndex <= 3

        CurrEir = Range(Cells(RowIndex, 2), Cells(RowIndex, 2))
        
        If PreviousEir <> CurrEir Then
            EirIndex = 1
            WriteRow = WriteRow + 1
        ElseIf PreviousEir = CurrEir Then
            EirIndex = EirIndex + 1
        End If
        PreviousEir = CurrEir
        
        ArrayEir(WriteRow, 1) = WriteRow                                        '"Line Number"
        ArrayEir(WriteRow, 2) = Range(Cells(RowIndex, 2), Cells(RowIndex, 2))  'Eircode
        ArrayEir(WriteRow, 3) = Range(Cells(RowIndex, 3), Cells(RowIndex, 3))  'Easting
        ArrayEir(WriteRow, 4) = Range(Cells(RowIndex, 4), Cells(RowIndex, 4))  'Northing
        If EirIndex = 1 Then
            ArrayEir(WriteRow, 5) = Range(Cells(RowIndex, 5), Cells(RowIndex, 5))  'M1_Name
            ArrayEir(WriteRow, 6) = Range(Cells(RowIndex, 6), Cells(RowIndex, 6))  'M1_Location
            ArrayEir(WriteRow, 7) = Range(Cells(RowIndex, 7), Cells(RowIndex, 7))  'M1_Easting
            ArrayEir(WriteRow, 8) = Range(Cells(RowIndex, 8), Cells(RowIndex, 8))  'M1_Northing
        ElseIf EirIndex = 2 Then
            ArrayEir(WriteRow, 9) = Range(Cells(RowIndex, 5), Cells(RowIndex, 5))  'M2_Name
            ArrayEir(WriteRow, 10) = Range(Cells(RowIndex, 6), Cells(RowIndex, 6))  'M2_Location
            ArrayEir(WriteRow, 11) = Range(Cells(RowIndex, 7), Cells(RowIndex, 7))  'M2_Easting
            ArrayEir(WriteRow, 12) = Range(Cells(RowIndex, 8), Cells(RowIndex, 8))  'M2_Northing
        ElseIf EirIndex = 3 Then
            ArrayEir(WriteRow, 13) = Range(Cells(RowIndex, 5), Cells(RowIndex, 5))  'M3_Name
            ArrayEir(WriteRow, 14) = Range(Cells(RowIndex, 6), Cells(RowIndex, 6))  'M3_Location
            ArrayEir(WriteRow, 15) = Range(Cells(RowIndex, 7), Cells(RowIndex, 7))  'M3_Easting
            ArrayEir(WriteRow, 16) = Range(Cells(RowIndex, 8), Cells(RowIndex, 8))  'M3_Northing
        End If

        Application.StatusBar = "Progress: " & RowIndex & " in " & LastRow & " of possibly " & MaxArraySize
        
        
        RowIndex = RowIndex + 1
    Loop
    
        Application.StatusBar = False

Worksheets("Sheet2").Activate
Worksheets("Sheet2").Range(Cells(1, 1), Cells(UBound(ArrayEir, 1), UBound(ArrayEir, 2))).Value = ArrayEir

End Sub

【讨论】：