在torchtext bucketiterator中获取填充前每个句子的长度答案

【问题标题】：Get the length of every sentence before padding in torchtext bucketiterator在torchtext bucketiterator中获取填充前每个句子的长度
【发布时间】：2021-04-10 02:53:07
【问题描述】：

是否可以在torchtext bucketiterator中填充之前获取每个句子的长度：

train_loader = torchtext.legacy.data.BucketIterator(train_data, batch_size = 64, repeat=True, shuffle=True, sort_key = lambda x: len(x.text), sort=False, sort_within_batch=True, device = device)

bucketiterator 数据加载器：

inputs: tensor([[  34,   87,    2,  ...,  227,  239,  263],
        [ 138,    7, 1006,  ...,  840,  142,  665],
        [ 549,    4, 1028,  ...,   11,   14,    4],
        ...,
        [   1,    1,    5,  ...,   66,   23,   13],
        [   1,    1, 1062,  ...,  177,  252, 1587],
        [   1,    1,   66,  ...,  553,   52,   73]]), shape: torch.Size([64, 91])

就像使用 pytorch 数据加载器时一样：

train_loader = data.DataLoader(train_data, batch_size = 64, shuffle=True, collate_fn=padding)

def padding(batch):
 doc = [doc['input'] for doc in batch]
 len_doc = [len(doc['input']) for doc in batch]
 doc_pad = pad_sequence(doc, batch_first=True, padding_value=0)
 return doc_pad, len_doc

pytorch 数据加载器：

inputs: tensor([[   2, 1396, 2686,  ...,    0,    0,    0],
        [   2, 1391, 1396,  ...,    0,    0,    0],
        [   2, 2018, 2597,  ...,    0,    0,    0],
        ...,
        [   2, 1546, 1623,  ...,    0,    0,    0],
        [   2, 1435, 1396,  ...,    0,    0,    0],
        [   2, 1391, 1396,  ...,    0,    0,    0]]), shape: torch.Size([64, 40])

inputs_len_before_padding: tensor([18,  8, 21, 16, 16, 12, 40, 12,  9, 12, 17, 12, 17, 15, 16, 12,  8, 24,
        25, 10, 22,  8,  8, 13, 12, 22, 17, 14, 21, 14, 19, 13, 21,  8, 28, 16,
        31, 24, 23, 19, 10,  7, 16, 12, 16, 12, 17, 12, 18, 11,  8, 13, 17, 14,
        11, 13, 13, 20,  8, 12, 22,  7,  9, 11]), shape: torch.Size([64])

【问题讨论】：

如果你使用torchtext.data.Field来定义你的数据字段，你可以将include_lengths切换到true，这将“返回一个填充的小批量的元组和一个包含每个长度的列表示例”而不是“只是一个填充的小批量”

标签： pytorch torchtext pytorch-dataloader

【解决方案1】：

这是一个使用torchtext.data.Field 和torchtext.data.BucketIterator 的最小示例：

import torchtext.data as data

# sample data
text = [
    'This is sentence 1.',
    'This sentence is a bit longer than the previous sentence.'
]

# define field -- notice include_lengths is set to True
text_field = data.Field(include_lengths=True, tokenize=lambda x: x.split())
fields = [('text', text_field)]

# create dataset and build vocabulary
examples = [data.Example.fromlist([t], fields) for t in text]
dataset = data.Dataset(examples, fields)
text_field.build_vocab(dataset)

# create iterator
data_iter = data.BucketIterator(dataset, batch_size=2, shuffle=False)

# the text field will now return both the data tensor and the length of the input text
for x in data_iter:
    print('Data:', x.text[0])
    print('Lengths:', x.text[1])

这应该打印（为简洁起见，缩短了数据张量）：

Data: tensor([[ 2,  2],
        ...
        [ 1, 10]])
Lengths: tensor([ 4, 10])

【讨论】：