从“gibber”中分离出好的文本并不是一件容易的事，尤其是当您处理短信/聊天时（这对我来说就是这样）。
拼写错误的单词不会使样本无法使用，即使是句法错误的句子也不应该取消整个文本的资格。这是可以用于报纸文本的标准，但不适用于用户生成的原始内容。
我会注释一个语料库，您可以在其中将好样本与坏样本分开，并在其中训练一个简单的分类器。注释不必费很大力气，因为这些乱码文本比好的文本短，应该是易于识别（至少是一些）。此外，您可以尝试从大约 100 个数据点（50 个好/50 个坏）的语料库大小开始，并在第一个模型或多或少工作时对其进行扩展。
这是我一直用于文本分类的示例代码。您需要安装 scikit-learn 和 numpy：
import re
import random
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
# Prepare data
def prepare_data(data):
"""
data is expected to be a list of tuples of category and texts.
Returns a tuple of a list of lables and a list of texts
"""
random.shuffle(data)
return zip(*data)
# Format training data
training_data = [
("good", "rain a lot the packs maybe damage."),
("good", "15107 Lane Pflugerville, TX customer called me and his phone number and my phone numbers were not masked. thank you customer has had a stroke and items were missing from his delivery the cleaning supplies for his wet vacuum steam cleaner. he needs a call back from customer support "),
("gibber", "wh. screen"),
("gibber", "How will I know if I")
]
training_labels, training_texts = prepare_data(training_data)
# Format test set
test_data = [
("gibber", "an quality"),
("good", "<datapoint with valid text>",
# ...
]
test_labels, test_texts = prepare_data(test_data)
# Create feature vectors
"""
Convert a collection of text documents to a matrix of token counts.
See: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
"""
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(training_texts)
y = training_labels
# Train the classifier
clf = LogisticRegression()
clf.fit(X, y)
# Test performance
X_test = vectorizer.transform(test_texts)
y_test = test_labels
# Generates a list of labels corresponding to the samples
test_predictions = clf.predict(X_test)
# Convert back to the usual format
annotated_test_data = list(zip(test_predictions, test_texts))
# evaluate predictions
y_test = np.array(test_labels)
print(metrics.classification_report(y_test, test_predictions))
print("Accuracy: %0.4f" % metrics.accuracy_score(y_test, test_predictions))
# predict labels for unknown texts
data = ["text1", "text2",]
# Important: use the same vectorizer you used for the training.
# When saving the model (e.g. via pickle) always serialize
# classifier & vectorizer
X = vectorizer.transform(data)
# Now predict the labels for the texts in 'data'
labels = clf.predict(X)
# And put them back together
result = list(zip(labels, data))
# result = [("good", "text1"), ("gibber", "text2")]
简单介绍一下它的工作原理：计数向量器对文本进行标记并创建包含语料库中所有单词计数的向量。基于这些向量，分类器尝试识别模式以区分这两个类别。只有少数不常见（b/c 拼写错误）单词的文本宁愿属于“gibber”类别，而包含很多常见句子典型单词的文本（想想这里的所有停用词：“我', 'you', 'is'... ) 更容易成为好文本。
如果此方法适合您，您还应该尝试其他分类器并使用第一个模型来半自动注释更大的训练语料库。