附加到来自不同文件目录的数据框，从 .tsv 文件中读取答案

【问题标题】：Append to dataframe from different file directories, reading from .tsv files附加到来自不同文件目录的数据框，从 .tsv 文件中读取
【发布时间】：2020-05-15 16:17:14
【问题描述】：

我正在尝试从 .tsv 文件中读取 text_A 和 text_B 并将它们附加到数据框中。我正在改编 Tensorflow 教程中的代码。

这是我改编的代码：

from absl import logging
import tensorflow as tf
import os
import pandas as pd
import csv

def load_directory_data(directory):
data = {}
data["text_A"] = []
data["text_B"] = []
for file_path in os.listdir(directory):                                 
   with open(os.path.join(directory, file_path), "r", encoding='utf-8') as csvfile: 
   texts = 0
   texts = csv.reader(csvfile, delimiter="\t", quotechar='"')
   for text in texts:
      print(text[0])
      #I want to apppend here
return pd.DataFrame.from_dict(data)

# Merge examples, add similiarity 
def load_dataset(directory):
  sa_df = load_directory_data(os.path.join(directory, "sa"))
  s_df = load_directory_data(os.path.join(directory, "s"))
  ns_df = load_directory_data(os.path.join(directory, "ns"))
  sa_df["similarity"] = 2
  s_df["similarity"] = 1
  ns_df["similarity"] = 0
  return pd.concat([sa_df, s_df, ns_df]).sample(frac=1).reset_index(drop=True)

# Download and process the dataset files.
def download_and_load_datasets(force_download=False):
   dataset = tf.keras.utils.get_file(
      fname="tfm_dataset.tar.gz", 
      origin="file:///mypath/tfm_dataset.tar.gz", 
      extract=True)

   train_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                   "tfm_dataset", "train"))
   test_df = load_dataset(os.path.join(os.path.dirname(dataset), 
                                  "tfm_dataset", "test"))

   return train_df, test_df

# Reduce logging output.
logging.set_verbosity(logging.ERROR)

train_df, test_df = download_and_load_datasets()
train_df.head()

我正在读取的目录具有以下结构：

test/sa:
   01_02.tsv
   03_04.tsv
   .
   .
   11_12.tsv
test/s:
   13_14.tsv
   .
   .
   17_18.tsv
test/ns:
   19_20.tsv
   .
   .
   29_30.tsv

train 目录具有类似的结构。

tsv 文件的格式类似于下面的示例：

"Text A, it could be the story about a black dog." "Text B, it could be a story about a bee."

当我打印 (text[0]) 时，它会打印出 "A" 位置的所有文本，直到 test/sa 中的最后一个文件。然后我得到了错误：

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-49-f5de8939de2e> in <module>
     43 logging.set_verbosity(logging.ERROR)
     44 
---> 45 train_df, test_df = download_and_load_datasets()
     46 train_df.head(30)

<ipython-input-49-f5de8939de2e> in download_and_load_datasets(force_download)
     34 
     35   train_df = load_dataset(os.path.join(os.path.dirname(dataset), 
---> 36                                        "tfm_dataset", "train"))
     37   test_df = load_dataset(os.path.join(os.path.dirname(dataset), 
     38                                       "tfm_dataset", "test"))

<ipython-input-49-f5de8939de2e> in load_dataset(directory)
     18 # Merge positive and negative examples, add a polarity column and shuffle.
     19 def load_dataset(directory):
---> 20   sa_df = load_directory_data(os.path.join(directory, "sa"))
     21   s_df = load_directory_data(os.path.join(directory, "s"))
     22   ns_df = load_directory_data(os.path.join(directory, "ns"))

<ipython-input-49-f5de8939de2e> in load_directory_data(directory)
     11        texts = csv.reader(csvfile, delimiter="\t", quotechar='"')
     12        for text in texts:
---> 13           print(text[0])
     14 
     15 

IndexError: list index out of range

我需要遍历来自不同目录的所有文件而不会出错，因此我将能够在其中附加文本以构建 pandas 数据框。

我的代码中的“#I need append here”行将被这两个命令替换：

data["text_A"].append(text[0])
data["text_B"].append(text[1])

有什么建议吗？

非常感谢

【问题讨论】：

标签： python pandas tensorflow

【解决方案1】：

我发现某些文件的末尾有制表符空格。我使用了代码：

 with open(os.path.join(directory, file_path), "r", encoding='utf-8') as csvfile: 
     texts = 0
     texts = csv.reader(csvfile, delimiter="\t", quotechar='"')
     for text in texts:
       print(directory,file_path,text)

然后修复坏文件，删除多余的空格。

【讨论】：