【发布时间】：2018-11-28 16:47:47
【问题描述】：
我有如下的话语（字符串）/语料库列表，
List<string> allUtterances = new List<string>
{
"c2's are above the hierarchy than c1's",
"c2's are better than c1's",
"get me a group of 10 c1's",
"he is a c2",
"he was a c two",
"hey i am a c1",
"jdsaxkjhasx",
"khndsmcsdfcs",
"my competency is c2",
"none intent",
"she is still a c 1",
"this is a none intent, please ignore",
"we are hiring fresh c1's"
};
这是类架构：
public class ListEntity
{
public string name { get; set; }
public List<Sublist> subLists { get; set; }
}
public class Sublist
{
public string canonicalForm { get; set; }
public List<string> list { get; set; }
}
这是一个示例 POCO：
List<ListEntity> listEntities = new List<ListEntity>
{
new ListEntity
{
name = "Competency",
subLists = new List<Sublist>
{
new Sublist
{
canonicalForm = "C1",
list = new List<string>
{
"c1",
"c one",
"c 1",
"C 1",
"C1",
"C one",
"C ONE"
}
},
new Sublist
{
canonicalForm = "C2",
list = new List<string>
{
"c2",
"c two",
"c 2",
"C 2",
"C2",
"C two",
"C TWO"
}
}
}
}
};
var canonicalForms = listEntities.Select(x => x.subLists.Select(y => y.list).ToList()).ToList();
假设如果我从上面提到的列表 allUtterances 中有以下话语：
"query": "C2's are better than C1's"
我想为上述话语获得以下输出：
{
"entity": "c2",
"type": "Competency",
"startIndex": 0,
"endIndex": 1,
"resolution": {
"values": [
"C2"
]
}
},
{
"entity": "c1",
"type": "Competency",
"startIndex": 21,
"endIndex": 22,
"resolution": {
"values": [
"C1"
]
}
}
我要匹配的规则如下：
对于allUtterances list 中的每个话语，如果话语文本包含来自类子列表的属性list 的值，我想提取start 和end 定位并用适当的键标记它们，在这种情况下是 canonicalForm 用 ListEntityClass 中的 name 属性更新我的 JSON 负载中的类型键。
我尝试了以下方法：
using System;
using System.Linq;
using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
namespace ListEntityProblem
{
class Program
{
static void Main(string[] args)
{
List<string> allUtterances = new List<string>
{
"c2's are above the hierarchy than c1's",
"c2's are better than c1's",
"get me a group of 10 c1's",
"he is a c2",
"he was a c two",
"hey i am a c1",
"jdsaxkjhasx",
"khndsmcsdfcs",
"my competency is c2",
"none intent",
"she is still a c 1",
"this is a none intent, please ignore",
"we are hiring fresh c1's"
};
List<ListEntity> listEntities = new List<ListEntity>
{
new ListEntity
{
name = "Competency",
subLists = new List<Sublist>
{
new Sublist
{
canonicalForm = "C1",
list = new List<string>
{
"c1",
"c one",
"c 1",
"C 1",
"C1",
"C one",
"C ONE"
}
},
new Sublist
{
canonicalForm = "C2",
list = new List<string>
{
"c2",
"c two",
"c 2",
"C 2",
"C2",
"C two",
"C TWO"
}
}
}
}
};
List<Tuple<string, string, List<string>>> ListEntityLookup = new List<Tuple<string, string, List<string>>>();
//n^2, construct lookup for list entities
foreach (var item in listEntities)
{
string listEntityName = item.name;
foreach (var innerItem in item.subLists)
{
string normalizedValue = innerItem.canonicalForm;
List<string> synonymValues = innerItem.list;
ListEntityLookup.Add(Tuple.Create<string, string, List<string>>(listEntityName, normalizedValue, synonymValues));
}
}
List<JObject> parsedEntities = new List<JObject>();
//n^3, populate the parsed payload with start and end indices
foreach (var item in allUtterances)
{
foreach (var ll in ListEntityLookup)
{
foreach (var cf in ll.Item3)
{
int start = 0, end = 0;
if (item.Contains(cf))
{
start = item.IndexOf(cf);
end = start + cf.Length;
parsedEntities.Add(new JObject
{
new JProperty("Start", start),
new JProperty("End", end),
new JProperty("Query", item),
new JProperty("CanonicalForm", ll.Item2),
new JProperty("ListEntity", ll.Item1)
});
}
}
}
}
//Group by query
var groupedParsedEntities = parsedEntities.GroupBy(x => x["Query"]).ToList();
}
}
}
编辑：
我曾尝试重新编写 for-each 循环，但这会导致更多的嵌套。
foreach (var item in allUtterances)
{
foreach (var listEntity in listEntities)
{
foreach (var canonicalForm in listEntity.subLists)
{
foreach(var synonym in canonicalForm.list)
{
int start = item.IndexOf(synonym);
if(start != -1)
{
parsedEntities.Add(new JObject
{
new JProperty("Start", start),
new JProperty("End", start + synonym.Length),
new JProperty("Query", item),
new JProperty("CanonicalForm", canonicalForm.canonicalForm),
new JProperty("ListEntity", listEntity.name)
});
}
}
}
}
}
但是这种方法对于大量的话语来说似乎很慢并且不能很好地扩展。由于主循环运行 n^3 次。我们的服务器每秒必须进行太多计算。
我一直在思考是否应该使用 Regex 是否会给我带来一些性能优势。
请帮我优化这个算法。
任何帮助将不胜感激。
【问题讨论】：
-
这对Code Review来说可能也是一个好问题
-
绝对不要使用正则表达式。字符串方法比正则表达式更有效。
-
@jdweng 我主要关心的是运行 n^3 次的循环，它会很重，我怀疑
.IndexOf()是否健壮/防呆？ -
正则表达式会差 10 倍。
标签： c# arrays linq optimization substring