python 3.X 将压缩的 csv 文件连接到一个非压缩的 csv 文件

Question

这是我的 python 3 代码：

import zipfile
import os
import time
from timeit import default_timer as timer
import re
import glob
import pandas as pd


# local variabless
# pc version
# the_dir = r'c:\ImpExpData'
# linux version
the_dir = '/home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95'


def main():
    """
    this is the function that controls the processing
    """
    start_time = timer()
    for root, dirs, files in os.walk(the_dir):
        for file in files:
            if file.endswith(".zip"):
                print("working dir is ...", the_dir)
                zipPath = os.path.join(root, file)
                z = zipfile.ZipFile(zipPath, "r")
                for filename in z.namelist():
                    if filename.endswith(".csv"):
                        # print filename
                        if re.match(r'^Trade-Geo.*\.csv$', filename):
                            pass  #  do somethin with geo file
                        # print " Geo data:  " , filename
                        elif re.match(r'^Trade-Metadata.*\.csv$', filename):
                            pass  # do something with metadata file
                        # print "Metadata:    ", filename
                        else:
                            try:
                                with zipfile.ZipFile(zipPath) as z:
                                    with z.open(filename) as f:
                                        # print("send to test def...", filename)
                                        # print(zipPath)
                                        with zipfile.ZipFile(zipPath) as z:
                                            with z.open(filename) as f:
                                                frame = pd.DataFrame()
                                                # EmptyDataError: No columns to parse from file -- how to deal with this error
                                                train_df = read_csv(f, index_col=None, header=0, skiprows=1, encoding="cp1252")
                                                # train_df = pd.read_csv(f, header=0, skiprows=1, delimiter=",", encoding="cp1252")
                                                list_ = []
                                                list_.append(train_df)
                                                # print(list_)
                                                frame = pd.concat(list_, ignore_index=True)
                                                frame.to_csv('/home/ralph/Documents/lulumcusb/ImpExpData/Exports/concat_test.csv', encoding='cp1252')   # works
                            except:  # catches EmptyDataError: No columns to parse from file
                                print("EmptyDataError...." ,filename, "...", zipPath)

#    GetSubDirList(the_dir)
    end_time = timer()
    print("Elapsed time was %g seconds" % (end_time - start_time))


if __name__ == '__main__':
    main()

它最有效——只是它不会将所有压缩的 csv 文件连接成一个。有一个空文件，所有 csv 文件具有相同的字段结构，所有 csv 文件的行数各不相同。

这是我运行 spyder 时报告的内容：

runfile('/home/ralph/Documents/lulumcusb/Sep15_cocncatCSV.py', wdir='/home/ralph/Documents/lulumcusb')

working dir is ... /home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95

EmptyDataError.... Trade-Exports-Chp-77.csv ... /home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95/Trade-Exports-Yr1992-1995.zip

/home/ralph/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py:688: DtypeWarning: Columns (1) have mixed types. Specify dtype option on import or set low_memory=False.
  execfile(filename, namespace)

Elapsed time was 104.857 seconds

最终的 csvfile 是最后处理的压缩 csv 文件；处理文件时 csv 文件的大小会发生变化

压缩文件中有 99 个 csv 文件，我希望将它们合并为一个非压缩 csv 文件

字段或列名是： colmNames = ["hs_code", "uom", "country", "state", "prov", "value", "quatity", "year", "month"]

csv 文件被标记：chp01.csv、cht02.csv 等到 chp99.csv，其中“uom”（计量单位）为空，或者为整数或字符串，具体取决于 hs_code

问题：如何将压缩后的 csv 文件连接成一个大的（估计 100 mb 未压缩的）csv 文件？

添加细节： 我试图不解压缩 csv 文件，然后我将不得不删除它们。我需要连接文件，因为我有额外的处理要做。提取压缩的 csv 文件是一个可行的选择，我希望不必这样做

Answer 1

你有什么理由不想用你的 shell 做这个吗？

假设您连接的顺序无关紧要：

cd "/home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95"
unzip "Trade-Exports-Yr1992-1995.zip" -d unzipped && cd unzipped
for f in Trade-Exports-Chp*.csv; do tail --lines=+2 "$f" >> concat.csv; done

这会在附加到concat.csv 之前从每个 csv 文件中删除第一行（列名）。

如果你刚刚这样做：

tail --lines=+2 "Trade-Exports-Chp*.csv" > concat.csv

你最终会得到：

==> Trade-Exports-Chp-1.csv <==
...

==> Trade-Exports-Chp-10.csv <==
...

==> Trade-Exports-Chp-2.csv <==
...

etc.

如果您关心订单，请将Trade-Exports-Chp-1.csv .. Trade-Exports-Chp-9.csv 更改为Trade-Exports-Chp-01.csv .. Trade-Exports-Chp-09.csv。

虽然它在 Python 中是可行的，但在这种情况下，我认为它不是适合这项工作的工具。

---

如果您想在不实际提取 zip 文件的情况下就地完成工作：

for i in {1..99}; do
  unzip -p "Trade-Exports-Yr1992-1995.zip" "Trade-Exports-Chp$i.csv" | tail --lines=+2 >> concat.csv
done