从具有不同列数的 csv 文件中读取和选择项目答案

【问题标题】：Reading and selecting items from a csv file with different number of columns从具有不同列数的 csv 文件中读取和选择项目
【发布时间】：2019-04-15 23:28:51
【问题描述】：

我正在尝试从 csv 文件中获取一些项目，但是有一个问题，它有不同的列数，所以我不能使用 pandas.read_csv(filepath) 函数来读取它。我需要打开它，然后我可以选择一些显示的项目。 csv文件如下（每行之间加了一个空行，方便大家阅读）：

“路径”、“文件”、“获取日期”、“样本”、“其他”

"C:\msdchem\2\DATA\AlbertVirgili\DaniGM\","DGM_CPTIS003 1h.D","25-Mar-19, 11:55:48","DGM_CPTIS003 1h",""

“INT FID1A.CH”

“2019 年 3 月 25 日星期一 17:48:31”

"峰值","R.T.","开始","结束","PK TY","高度","面积","Pct Max","Pct Total"

1, 2.082, 2.063, 2.189,"BB",223849319,4951058782,100.00, 46.349

2, 2.317, 2.281, 2.386,"BB",73209942,1093871144, 22.09, 10.240

3, 3.343, 3.224, 3.403,"BB",93165657,2220621038, 44.85, 20.788

4, 5.538, 5.409, 5.598,"BB",51783798,1975386485, 39.90, 18.492

5, 5.744, 5.693, 5.803,"BB",24084957,360235490, 7.28, 3.372

6, 8.716, 8.676, 8.776,"BB",8566883, 80973220, 1.64, 0.758

“路径”、“文件”、“获取日期”、“样本”、“其他”

"C:\msdchem\2\DATA\AlbertVirgili\DaniGM\","DGM_CPTIS003 2h.D","25-Mar-19, 12:15:42","DGM_CPTIS003 2h",""

“INT FID1A.CH”

“2019 年 3 月 25 日星期一 12:31:45”

"峰值","R.T.","开始","结束","PK TY","高度","面积","Pct Max","Pct Total"

1, 2.083, 2.064, 2.194,"BB",232382153,5255486688,100.00, 59.673

2, 2.318, 2.282, 2.384,"BB",37916041,587535474, 11.18, 6.671

3, 3.322, 3.241, 3.381,"BB",67715293,1373898201, 26.14, 15.600

4, 5.509, 5.406, 5.569,"BB",39502747,1227609422, 23.36, 13.939

5, 5.731, 5.689, 5.791,"BB",17799521,230201751, 4.38, 2.614

6, 8.717, 8.674, 8.776,"BB",12367646,132409300, 2.52, 1.503

我需要做的是阅读标题下的项目：Peak, R.T., Start, End, PK TY,... 但我不能这样做，因为它们与前几行的长度不同（带有标题路径、文件、获取日期...）。我不能使用 skiprows 函数来消除 0-3 和 11-14 中的行，因为我要读取的部分的行数并不总是一致的（这种类型的文件是由外部程序生成的，我不能修改其结构）。有什么方法可以让我只读取 csv 代码中位于所需标题下的部分，以便我可以使用它从这些值中选择所需的数据？

提前感谢您的帮助。

【问题讨论】：

使用标准 python 库阅读它。查找要拆分的区域和特定的跳行功能。然后，使用StringIO 分别读取每个 CSV。

标签： python pandas csv

【解决方案1】：

您需要进行一些预处理。如果您使用来自外部系统的数据，考虑这些集成点是很常见的。

外部文件包含结构化数据。一系列 CSV 行，每个项目有 5 个标题行。最后一个标题行包含 CSV 列标签。

从外部文件中读取内容。根据您的需要调整以下代码。

external_file_content = r'''
"Path","File","Date Acquired","Sample","Misc"
"C:\msdchem\2\DATA\AlbertVirgili\DaniGM\","DGM_CPTIS003 1h.D","25-Mar-19, 11:55:48","DGM_CPTIS003 1h"," "
"INT FID1A.CH"
"Mon Mar 25 17:48:31 2019"
"Peak","R.T.","Start","End","PK TY","Height","Area","Pct Max","Pct Total"
1, 2.082, 2.063, 2.189,"BB ",223849319,4951058782,100.00, 46.349
2, 2.317, 2.281, 2.386,"BB ",73209942,1093871144, 22.09, 10.240
3, 3.343, 3.224, 3.403,"BB ",93165657,2220621038, 44.85, 20.788
4, 5.538, 5.409, 5.598,"BB ",51783798,1975386485, 39.90, 18.492
5, 5.744, 5.693, 5.803,"BB ",24084957,360235490, 7.28, 3.372
6, 8.716, 8.676, 8.776,"BB ",8566883, 80973220, 1.64, 0.758
"Path","File","Date Acquired","Sample","Misc"
"C:\msdchem\2\DATA\AlbertVirgili\DaniGM\","DGM_CPTIS003 2h.D","25-Mar-19, 12:15:42","DGM_CPTIS003 2h"," "
"INT FID1A.CH"
"Mon Mar 25 12:31:45 2019"
"Peak","R.T.","Start","End","PK TY","Height","Area","Pct Max","Pct Total"
1, 2.083, 2.064, 2.194,"BB ",232382153,5255486688,100.00, 59.673
2, 2.318, 2.282, 2.384,"BB ",37916041,587535474, 11.18, 6.671
3, 3.322, 3.241, 3.381,"BB ",67715293,1373898201, 26.14, 15.600
4, 5.509, 5.406, 5.569,"BB ",39502747,1227609422, 23.36, 13.939
5, 5.731, 5.689, 5.791,"BB ",17799521,230201751, 4.38, 2.614
6, 8.717, 8.674, 8.776,"BB ",12367646,132409300, 2.52, 1.503
'''

使用定义明确的分隔符将序列拆分为不同的部分

parts = external_file_content.split('"Path","File","Date Acquired","Sample","Misc"')

选择单个部分以进一步处理成 pandas DataFrame。配置pd.read_csv 跳过4行。

df = pd.read_csv(StringIO(parts[1]), skiprows=4);

显示DataFrame的第一行

df.head(5)


    Peak    R.T.    Start   End     PK TY   Height  Area    Pct Max     Pct Total
0   1   2.082   2.063   2.189   BB  223849319   4951058782  100.00  46.349
1   2   2.317   2.281   2.386   BB  73209942    1093871144  22.09   10.240
2   3   3.343   3.224   3.403   BB  93165657    2220621038  44.85   20.788
3   4   5.538   5.409   5.598   BB  51783798    1975386485  39.90   18.492
4   5   5.744   5.693   5.803   BB  24084957    360235490   7.28    3.372

【讨论】：

您好，感谢您的建议。我应该如何从 csv 文件中获取初始数据，以便我可以应用拆分功能？通过使用 with open (filepath) as csvFile 获取行并从那里获取行？
用 Python 读取文件是互联网上一个很好的话题。 “反复试验”是探索和学习的有力方式。

【解决方案2】：

过滤掉非数字行

def gen_rows(stream):
    for row in csv.reader(stream):             
        if row.pop(0).isdigit(): # check that value is a number  
            yield row

with open('data.csv') as fo:
    df = pd.DataFrame.from_records(gen_rows(fo), 
    columns = ["Peak","R.T.","Start","End","PKTY",
                    "Height","Area","Pct Max","Pct Total"])

【讨论】：