将数据集（CONLL 格式）拆分为开发、训练和测试答案

【问题标题】：Split dataset (CONLL format) into dev, train and test将数据集（CONLL 格式）拆分为开发、训练和测试
【发布时间】：2021-09-28 05:17:47
【问题描述】：

我有一个遵循 CONLL 格式并带有令牌级注释的数据集。

token   label
Also    O
,   O
outdoor B-claim
activities  I-claim
enable  I-claim
me  I-claim
to  I-claim
socialize   I-claim
with    I-claim
other   I-claim
people  I-claim
and I-claim
enjoy   I-claim
natural I-claim
beauty  I-claim
.   O
                    
There   O
are O
strong  O
advantages  O
to  O
spend   O
leisure O
time    O
outdoors    O
.   O

空行分隔文档的句子。每个句子都被视为机器学习模型中的实例。我想将数据集拆分为训练、测试和开发，但要确保数据集之间没有拆分任何句子。 python中是否有任何库可用于拆分此类数据集，还是我必须手动执行此操作？

提前致谢！

【问题讨论】：

标签： python machine-learning dataset conll

【解决方案1】：

我没有找到一种方法自动拆分它，所以我建立了一个函数来做。
希望对您有所帮助。

def read_conll_data(filepath = "/path/to/ner_data.txt"):

    count = 0
    num_examples = 0

    first_line = True

    # Change the extension from conll to txt to read the file properly
    with open(filepath, 'r') as f:
        data = f.read()

    # Count number of examples in the file
    for line in data.splitlines():
        if line == "":
            num_examples += 1

    for line in data.splitlines():

        # Collect 80% of the data as training examples
        if count < int(num_examples * 0.8):
            with open("train_set.txt", "a") as train_set:
                if line != "":
                    train_set.write(line)
                    train_set.write("\n")
                else:
                    count += 1
                    train_set.write("\n")

        else:
            with open("test_set.txt", "a") as test_set:
                # Make sure that -DOCSTART- -X- O is included at the top of the file
                if first_line:
                    test_set.write(data.splitlines()[0])
                    test_set.write("\n")
                    first_line = False
                
                if line != "":
                    test_set.write(line)
                    test_set.write("\n")
                else:
                    test_set.write("\n")

【讨论】：