寻找“清理”要放入 Excel 的 csv 文件的理想方法

Question

这个问题有点涉及设置，但请耐心等待！

（将以下块复制并粘贴到编辑器中效果很好）

我正在使用 smartcsv 从金融网站的 csv 文件中加载我的数据。 每行都作为一个项目存储在列表中。

data = clevercsv.wrappers.read_csv(in_file_name)

在一些帐户信息行之后，股票数据开始：

stock_data = data[8:]

我希望删除数据：市场、贷款价值 - 一直到 - 日高（包括 0

并保留符号、描述 -> 持仓百分比（含）、52 周低位、52 周高位

每只股票的相关行上都有与之关联的数据。 删除此数据的任何最佳做法？我一直在尝试，但似乎有逻辑错误。

截至日期，2020-04-29 18:44:29

账户，道明自管投资 - 哈哈哈

现金，123.12

投资，1234.12

总值，12345.12

保证金,123456.12,

,

符号、市场、描述、数量、平均成本、价格、账面成本、市场价值、未实现的美元、未实现的百分比、头寸的百分比、贷款价值、今日变动美元、今日变动百分比、出价、出价手数、要价、询价手数、成交量、日低、日高、52 周低点、52 周高点

AFL,US,"AFLAC INC",500,43.79,39.23,21895.79,19615.00,-2280.79,-10.42,7.26,,1.4399986,3.81,39.19,1,40.2,1,3001288,38.31,39.48,23.07 ,57.18

AKTS,US,"AKOUSTIS TECHNOLOGIES INC",2500,5.04,8.94,12609.87,22350.00,9740.13,77.24,8.27,,0.35999966,4.20,8.68,1,9.2,10,1161566,8.65,9.25,3.76, 9.25

到目前为止，这是我的代码：

import clevercsv
data = clevercsv.wrappers.read_csv(in_file_name)

# store the earlier lines for later use, all rows 8 and later are stock data
cash = data[2]
investments = data[3]
tot_value = data[4]
margin = data[5]
full_header = data[7]
stock_data = data[8:]

new_header = []
new_stock_data = []

# I have found the index positions I wish to save, append their data to the new_ lists:
for i in range(len(full_header)):
    if i == 0:
        new_header.append(full_header[i])
    if (i >= 2 and i <= 10):
        new_header.append(full_header[i])
    if i == 21:
        new_header.append(full_header[i])
    if i == 22: 
        new_header.append(full_header[i])
# I have found the index positions I wish to save, append their data to the new_ lists:
for i in range(len(stock_data)):
    if i == 0:
        new_stock_data.append(stock_data[i])
    if (i >= 2 and i <= 10):
        new_stock_data.append(stock_data[i])
    if i == 21:
        new_stock_data.append(stock_data[i])
    if i == 22: 
        new_stock_data.append(stock_data[i])

with open(os.path.join(folder_path,out_file_name),'w') as out_file:
    writer = clevercsv.writer(out_file)
    writer.writerow(cash)
    writer.writerow(investments)
    writer.writerow(tot_value)
    writer.writerow(margin)
    writer.writerow(new_header)
    for row in new_stock_data:
         writer.writerow(row)

如果这太复杂了，我理解，如果有人有更好的库可以使用，或者有更好的方法来使用 csv 库，那么它本身就会有很多帮助。

Answer 1

如果您已经知道列索引和标题长度，您可以这样做：

import csv

with open('input.csv', 'r', newline='') as input_file, open('output.csv','w', newline='') as output_file:
    reader = csv.reader(input_file)
    writer = csv.writer(output_file)

    for line_number, row in enumerate(reader, start=0):  # Avoid range(len(x))
        if line_number < 7:
            writer.writerow(row)  # Write cash, investments, etc
        else:
            shortened_row = row[0:1] + row[2:11] + row[21:] # Slice only the columns you need
            writer.writerow(shortened_row)

每当您发现自己在编写 range(len(something)) 时，这就是您可能想要使用 enumerate() 的好兆头，它会遍历您的数据并自动跟踪当前索引。

为了解析标题之后的每一行，您可以使用切片符号row[start:end] 并将切片添加在一起以获得一个新列表，然后您可以将其写入文件。请记住，row[start:end] 将不会返回索引 end 处的项目，这可能违反直觉。

最后，在使用 CSV 时，我总是添加 newline=''，否则您可能会遇到意外的换行符，但这可能是 clevercsv 为您处理的。

Answer 2

在 Python 中，我建议使用 Pandas 进行此类操作。

首先隔离 CSV 数据。然后将其视为流。我将您的部分样本放入x：

# This is python3 code
# first treat string as though it is a file
import io
x = io.StringIO("""Symbol,Market,Description,Quantity,Average Cost,Price,Book Cost,Market Value,Unrealized $,Unrealized %,% of Positions,Loan Value,Change Today $,Change Today %,Bid,Bid Lots,Ask,Ask Lots,Volume,Day Low,Day High,52-wk Low,52-wk High
AFL,US,"AFLAC INC",500,43.79,39.23,21895.79,19615.00,-2280.79,-10.42,7.26,,1.4399986,3.81,39.19,1,40.2,1,3001288,38.31,39.48,23.07,57.18
AKTS,US,"AKOUSTIS TECHNOLOGIES INC",2500,5.04,8.94,12609.87,22350.00,9740.13,77.24,8.27,,0.35999966,4.20,8.68,1,9.2,10,1161566,8.65,9.25,3.76,9.25""")

然后使用pandas将字符串读取为CSV，默认将第一行作为标题：

import pandas as pd
df = pd.read_csv(x)

然后通过将列名列表传递给数据框来选择所需的列：

new_df = df[['Book Cost', 'Market Value', 'Unrealized $', 'Unrealized %','% of Positions','52-wk Low', '52-wk High']]

   Book Cost  Market Value  Unrealized $  Unrealized %  % of Positions  \
0   21895.79       19615.0      -2280.79        -10.42            7.26   
1   12609.87       22350.0       9740.13         77.24            8.27   

   52-wk Low  52-wk High  
0      23.07       57.18  
1       3.76        9.25 

终于可以保存了：

new_df.to_csv('test.csv', index=False)  # Turn off indexing

你已经准备好了：

Book Cost,Market Value,Unrealized $,Unrealized %,% of Positions,52-wk Low,52-wk High
21895.79,19615.0,-2280.79,-10.42,7.26,23.07,57.18
12609.87,22350.0,9740.13,77.24,8.27,3.76,9.25

Answer 3

（完全披露，我是 CleverCSV 的作者。）

如果您想在此任务中使用 CleverCSV，并且您的数据足够小以适合内存，您可以使用 clevercsv.read_csv 加载数据并使用 clevercsv.write_table 保存数据。通过使用这些功能，您不必担心 CSV 方言等。您还可以自动找到标题行的索引。它可能是这样的：

from clevercsv import read_csv, write_table

# Load the table with CleverCSV
table = read_csv(in_file_name)

# Find the index of the header row and get the header
header_idx = next((i for i, r in enumerate(table) if r[0] == 'Symbol'), None)
header = table[header_idx]

# Extract the data as a separate table
data = table[header_idx+1:]

# Create a list of header names that you want to keep
keep = ["Symbol", "Description", "Quantity","Average Cost","Price","Book Cost","Market Value","Unrealized $","Unrealized %","% of Positions","52-wk Low", "52-wk High"]

# Turn that list into column indices (and ensure all exist)
keep_idx = [header.index(k) for k in keep]

# Then create a new table by adding the header and the sliced rows
new_table = [keep]
for row in data:
  new_row = [row[i] for i in keep_idx]
  new_table.append(new_row)

# Finally, write the table to a new csv file
write_table(new_table, out_file_name)