如何迭代 keras 数据集和编辑内容

Question

我正在处理这个电影分类问题 https://www.tensorflow.org/tutorials/keras/text_classification

在这个例子中，文本文件（12500 个带有电影评论的文件）被读取并准备了一个批处理数据集，如下所示

raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/train', 
    batch_size=batch_size, 
    validation_split=0.2, 
    subset='training', 
    seed=seed)

标准化时

def custom_standardization(input_data):
  lowercase = tf.strings.lower(input_data)
  stripped_html = tf.strings.regex_replace(lowercase, '<br />', ' ')
#I WANT TO REMOVE STOP WORDS HERE, CAN I DO
  return tf.strings.regex_replace(stripped_html,'[%s]' % re.escape(string.punctuation),'')

问题：我了解到我的训练数据集带有变量“raw_train_ds”中的标签。现在我想遍历这个数据集并从电影评论文本中删除停用词并存储回同一个变量，我尝试在函数“custom_standardization”中执行此操作，但它给出了类型错误，

我也尝试使用tf.strings.as_strings，但它返回错误 InvalidArgumentError: attr 'T' of string 的值不在允许值列表中：int8、int16、int32、int64

有人可以帮忙吗，或者只是请帮助如何从批处理数据集中删除停用词

Answer 1

现在 TensorFlow 似乎没有内置对停用词删除的支持，只是基本的标准化（小写和标点符号剥离）。教程中使用的TextVectorization 支持自定义标准化回调，但我找不到任何停用词示例。

由于本教程下载了 imdb 数据集并从光盘读取文本文件，因此您可以在读取它们之前使用 python 手动进行标准化。这将修改文本文件本身，但随后您可以使用 tf.keras.preprocessing.text_dataset_from_directory 正常读取文件，并且条目将已经删除了停用词。

#!/usr/bin/env python3

import pathlib
import re

from bs4 import BeautifulSoup
from nltk.corpus import stopwords

stop_words = set(stopwords.words("english"))


def cleanup_text_files_in_folder(folder_name):
    text_files = []

    for file_path in pathlib.Path(folder_name).glob('*.txt'):
        text_files.append(str(file_path))

    print(f'Found {len(text_files)} files in {folder_name}')

    # Give some kind of status
    i = 0
    for text_file in text_files:
        replace_file_contents(text_file)
        i += 1

        if i % 1000 == 0:
            print("No of files processed =", i)

    return text_files


def replace_file_contents(input_file):
    """
    This will read in the contents of the text file, process it (clean up, remove stop words)
    and overwrite the new 'processed' output to that same file

    """
    with open(input_file, 'r') as file:
        file_data = file.read()

    file_data = process_text_adv(file_data)

    with open(input_file, 'w') as file:
        file.write(file_data)


def process_text_adv(text):
    # review without HTML tags
    text = BeautifulSoup(text, features="html.parser").get_text()

    # review without punctuation and numbers
    text = re.sub(r'[^\w\s]','',text, re.UNICODE)

    # lowercase
    text = text.lower()

    # simple split
    text = text.split()

    swords = set(stopwords.words("english"))  # conversion into set for fast searching
    text = [w for w in text if w not in swords]

    # joining of splitted paragraph by spaces and return
    return " ".join(text)


if __name__ == "__main__":
    # Download & untar dataset beforehand, then running this would modify the text files
    # in place. Back up the originals if that's a concern.
    cleanup_text_files_in_folder('aclImdb/train/pos/')
    cleanup_text_files_in_folder('aclImdb/train/neg/')
    cleanup_text_files_in_folder('aclImdb/test/pos/')
    cleanup_text_files_in_folder('aclImdb/test/neg/')