我们如何检测文本多语言的单独文本？ [关闭]

Question

我在输入中有一个文本（英文和法文）：我想将文本分成每种语言的两个子文本：所以，我们将检测文本中的语言（> 2 种语言），然后剪切每个用他自己的语言写的文字：

输入：

You will then discover galleries in which 25-million bottles rest in the cellars, waiting for the 
perfect moment to be tasted. From the bottle to the salmanazar, from the youngest wines to the oldest   
vintages. - Vous trouverez alors des galeries parmi lesquelles 25 000 000 bouteilles reposent dans les   
caves, attendant le parfait moment pour être dégustées.

期望的输出：

This text contains two languages : "fr" and "en"

text_in_english= "You will then discover galleries in which 25-million bottles rest in the cellars,  
waiting for the perfect moment to be tasted. From the bottle to the salmanazar, from the youngest  
wines to the oldest vintages."

text_in_frensh= "- Vous trouverez alors des galeries parmi lesquelles 25 000 000 bouteilles reposent 
dans les caves, attendant le parfait moment pour être dégustées."

请问我们该怎么做？

Answer 1

我建议使用 nltk（如果需要，请取消注释以下载 punkt）和 langid。步骤是：

1. Split the text into sentences  
2. Predict the language of each sentence  
3. Add the predicted sentences to a dictionary to group them by language, in order.

python3：

from langid import classify
from nltk import tokenize
import nltk
from collections import defaultdict
#nltk.download('punkt')

mytext = """
You will then discover galleries in which 25-million bottles rest in the cellars, waiting for the 
perfect moment to be tasted. From the bottle to the salmanazar, from the youngest wines to the oldest 
vintages. - Vous trouverez alors des galeries parmi lesquelles 25 000 000 bouteilles reposent dans les 
caves, attendant le parfait moment pour être dégustées.
"""
mytext = mytext.replace('\n', ' ').replace('\r', '')
sentences = tokenize.sent_tokenize(mytext)
languages = defaultdict(list)

for sentence in sentences:
    languages[str(classify(sentence)[0])].append(sentence)

for k,v in languages.items():
    print(k,v)
#en [' You will then discover galleries in which 25-million bottles rest in the cellars, waiting for the  perfect moment to be tasted.', 'From the bottle to the salmanazar, from the youngest wines to the oldest  vintages.']   
#fr ['- Vous trouverez alors des galeries parmi lesquelles 25 000 000 bouteilles reposent dans les  caves, attendant le parfait moment pour être dégustées.']