从 pandas 中的文本中删除 unicode

Question

对于一个字符串，下面的代码会删除 unicode 字符和换行符/回车符：

t = "We've\xe5\xcabeen invited to attend TEDxTeen, an independently organized TED event focused on encouraging youth to find \x89\xdb\xcfsimply irresistible\x89\xdb\x9d solutions to the complex issues we face every day.,"

t2 = t.decode('unicode_escape').encode('ascii', 'ignore').strip()
import sys
sys.stdout.write(t2.strip('\n\r'))

但是当我尝试在 pandas 中编写一个函数以将其应用于列的每个单元格时，它要么因属性错误而失败，要么我收到一条警告说试图在切片的副本上设置一个值来自数据框

def clean_text(row):
    row= row["text"].decode('unicode_escape').encode('ascii', 'ignore')#.strip()
    import sys
    sys.stdout.write(row.strip('\n\r'))
    return row

应用于我的数据框：

df["text"] = df.apply(clean_text, axis=1)

如何将此代码应用于系列的每个元素？

Answer 1

问题似乎是您在执行应用功能时尝试访问和更改row['text'] 并返回行本身，当您在DataFrame 上执行apply 时，它适用于每个系列，所以如果更改为这应该会有所帮助：

import pandas as pd

df = pd.DataFrame([t for _ in range(5)], columns=['text'])

df 
                                                text
0  We've������been invited to attend TEDxTeen, an ind...
1  We've������been invited to attend TEDxTeen, an ind...
2  We've������been invited to attend TEDxTeen, an ind...
3  We've������been invited to attend TEDxTeen, an ind...
4  We've������been invited to attend TEDxTeen, an ind...

---

def clean_text(row):
    # return the list of decoded cell in the Series instead 
    return [r.decode('unicode_escape').encode('ascii', 'ignore') for r in row]

df['text'] = df.apply(clean_text)

df
                                                text
0  We'vebeen invited to attend TEDxTeen, an indep...
1  We'vebeen invited to attend TEDxTeen, an indep...
2  We'vebeen invited to attend TEDxTeen, an indep...
3  We'vebeen invited to attend TEDxTeen, an indep...
4  We'vebeen invited to attend TEDxTeen, an indep...

您也可以使用lambda，如下所示，直接应用于text 列：

df['text'] = df['text'].apply(lambda x: x.decode('unicode_escape').\
                                          encode('ascii', 'ignore').\
                                          strip())

Answer 2

我实际上无法重现您的错误：以下代码为我运行而没有错误或警告。

df = pd.DataFrame([t,t,t],columns = ['text'])
df["text"] = df.apply(clean_text, axis=1)

如果有帮助，我认为解决此类问题的更“熊猫”方式可能是使用带有 DataFrame.str 方法之一的正则表达式，例如：

df["text"] =  df.text.str.replace('[^\x00-\x7F]','')

Answer 3

类似这样，其中 column_to_convert 是您要转换的列：

series = df['column_to_convert']
df["text"] =  [s.encode('ascii', 'ignore').strip()
               for s in series.str.decode('unicode_escape')]