LSTM 项目与 CSV 格式不兼容

Question

我正在尝试复制 Chevalier 的 LSTM Human Activity Recognition 算法，但在尝试以 CSV 格式实现自己的数据时遇到了问题。 git中使用的格式是txt。我的 CSV 数据格式如下：

0.000995,8
0.020801,8
0.040977,8
0.060786,8
0.080970,8
...            ...

可以在here找到原始文件。 x 值（时间）位于第 0 列（-80.060003 等），y 值（值）位于第 1 列（8、8 等）。我尝试使用熊猫

pandas.read_csv(DATASET_PATH + TRAIN + "data_train.csv", skiprows=1, header=None, sep=',', usecols=[0, 1])

但它似乎与“准备数据集”部分（可能还有其他）中的数据格式不兼容：

TRAIN = "train/"
TEST = "test/"

# Load "X" (the neural network's training and testing inputs)

def load_X(X_signals_paths):
    X_signals = []

    for signal_type_path in X_signals_paths:
        file = open(signal_type_path, 'r')
        # Read dataset from disk, dealing with text files' syntax
        X_signals.append(
            [np.array(serie, dtype=np.float32) for serie in [
                row.replace('  ', ' ').strip().split(' ') for row in file
            ]]
        )
        file.close()

    return np.transpose(np.array(X_signals), (1, 2, 0))

X_train_signals_paths = [
    DATASET_PATH + TRAIN + "Inertial Signals/" + signal + "train.txt" for signal in INPUT_SIGNAL_TYPES
]
X_test_signals_paths = [
    DATASET_PATH + TEST + "Inertial Signals/" + signal + "test.txt" for signal in INPUT_SIGNAL_TYPES
]

X_train = load_X(X_train_signals_paths)
X_test = load_X(X_test_signals_paths)


# Load "y" (the neural network's training and testing outputs)

def load_y(y_path):
    file = open(y_path, 'r')
    # Read dataset from disk, dealing with text file's syntax
    y_ = np.array(
        [elem for elem in [
            row.replace('  ', ' ').strip().split(' ') for row in file
        ]], 
        dtype=np.int32
    )
    file.close()

    # Substract 1 to each output class for friendly 0-based indexing 
    return y_ - 1

y_train_path = DATASET_PATH + TRAIN + "y_train.txt"
y_test_path = DATASET_PATH + TEST + "y_test.txt"

y_train = load_y(y_train_path)
y_test = load_y(y_test_path)

这就是我通过 iPython3 实现的情况：

在[0]：

TRAIN = "train/"
TEST = "test/"

def load_X(X_signals_paths):
    X_signals = []
    for signal_type_path in X_signals_paths:
        file = pandas.read_csv(DATASET_PATH + TRAIN + "data_train.csv", skiprows=1, header=None, sep=',', usecols=[0])
        X_signals.append(
            [np.array(serie, dtype=np.float32) for serie in [
                str(row).replace('  ', ' ').strip().split(' ') for row in file
            ]]
        )

    return np.transpose(np.array(X_signals), (1, 2, 0))

_train_signals_paths = [
    DATASET_PATH + TRAIN + signal + "train.csv" for signal in INPUT_SIGNAL_TYPES
]
X_test_signals_paths = [
    DATASET_PATH + TEST + signal + "test.csv" for signal in INPUT_SIGNAL_TYPES
]

X_train = load_X(X_train_signals_paths)
X_test = load_X(X_test_signals_paths)
print(X_train, X_test)

输出[0]：

[[[ 0.]]] [[[ 0.]]]

我希望我能在正确格式化我的数据以与此算法无缝协作方面获得一些帮助。如果有任何问题，请告诉我。

Answer 1

跟踪中的代码与您在问题中实际发布的代码不同 - 工作代码在裸文件句柄上运行，而不是 Pandas 数据框。

作为参考，这里是您再次引用的项目中的代码：

def load_X(X_signals_paths):
    X_signals = []

    for signal_type_path in X_signals_paths:
        file = open(signal_type_path, 'r')
        # ^ the error comes where you have file = pandas.read_csv(...)

        # Read dataset from disk, dealing with text files' syntax
        X_signals.append(
            [np.array(serie, dtype=np.float32) for serie in [
                row.replace('  ', ' ').strip().split(' ') for row in file
            ]]
        )
        file.close()

file 只是一个迭代器，它返回一个以换行符结尾的原始行（字符序列）；在这个输入上，去掉换行符和压缩空格是有意义的。但是您的代码已经打开、解析并将文件内容重新格式化为 Pandas 数据框，该数据框没有换行符或空格，只有已经解析的数字。也许回退到上游代码；或者，如果您想在其中更改某些内容，请弄清楚如何询问该更改。 CSV 本身没有任何问题。

Python 有一个非常强大的 csv module，所以也许可以简单地使用它而不是手动解析 CSV 中的各个字段。

    for signal_type_path in X_signals_paths:
        with open(signal_type_path, 'r') as csvfile:
            reader = csv.reader(csvfile)
            X_signals.append([np.array(row[0:2], dtype=np.float32) for row in reader])

或者作为最小的更改，用逗号而不是空格分隔。 （您的数据看起来实际上不需要删除空格。）

此外，您的代码对它读取的文件进行硬编码。最好将DATASET_PATH 和TRAIN 参数完全保留在调用代码中，并让load_X 简单地接受完整文件路径的列表，它无需以任何方式修改它们即可接受。