可能有不止一种方法可以做到这一点,但我建议使用FastVectorHighlighter,因为它可以让您访问位置和偏移数据。
索引要求
要使用这种方法,您需要确保您的索引数据在创建索引时使用存储术语向量数据的字段:
final String fieldName = "body";
// a shorter version of the input data in the question, for testing:
final String content = "State WEAPONRY 123 War WORD1 Force And Book4 Book WEAPONRY";
FieldType fieldType = new FieldType();
fieldType.setStored(true);
fieldType.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
fieldType.setStoreTermVectors(true);
fieldType.setStoreTermVectorPositions(true);
fieldType.setStoreTermVectorOffsets(true);
doc.add(new Field(fieldName, content, fieldType));
writer.addDocument(doc);
(如果您尚未捕获术语向量,这可能会显着增加索引数据的大小。)
库要求
快速矢量荧光笔是lucene-highlighter 库的一部分:
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-highlighter</artifactId>
<version>8.9.0</version>
</dependency>
搜索示例
假设以下查询:
final String searchTerm = "\"War Force\"~1";
我们希望这会从我们的测试数据中找到War WORD1 Force。
流程的第一部分使用经典查询解析器执行标准查询:
Directory dir = FSDirectory.open(Paths.get(indexPath));
try ( DirectoryReader dirReader = DirectoryReader.open(dir)) {
IndexSearcher indexSearcher = new IndexSearcher(dirReader);
Analyzer analyzer = new StandardAnalyzer();
QueryParser parser = new QueryParser(fieldName, analyzer);
Query query = parser.parse(searchTerm);
TopDocs topDocs = indexSearcher.search(query, 100);
ScoreDoc[] hits = topDocs.scoreDocs;
for (ScoreDoc hit : hits) {
handleHit(hit, query, dirReader, indexSearcher);
}
handleHit() 方法(如下所示)是我们使用FastVectorHighlighter 的地方。
如果您只想执行突出显示(并且不需要位置/偏移数据),您可以使用:
FastVectorHighlighter fvh = new FastVectorHighlighter();
fvh.getBestFragment(fieldQuery, dirReader, docId, fieldName, fragCharSize)
但要访问我们需要的额外数据,您可以执行以下操作:
FieldTermStack fieldTermStack = new FieldTermStack(dirReader, hit.doc, fieldName, fieldQuery);
FieldPhraseList fieldPhraseList = new FieldPhraseList(fieldTermStack, fieldQuery);
FragListBuilder fragListBuilder = new SimpleFragListBuilder();
FragmentsBuilder fragmentsBuilder = new SimpleFragmentsBuilder();
FastVectorHighlighter fvh = new FastVectorHighlighter(phraseHighlight, fieldMatch,
fragListBuilder, fragmentsBuilder);
这会构建一个FastVectorHighlighter,其中包含一个FieldPhraseList,它将由荧光笔填充。
getBestFragment 方法现在变为:
// use whatever you want for these settings:
int fragCharSize = 100;
int maxNumFragments = 100;
String[] preTags = new String[]{"-->"};
String[] postTags = new String[]{"<--"};
Encoder encoder = new DefaultEncoder();
// the fragments string array contains the highlighted results:
String[] fragments = fvh.getBestFragments(fieldQuery, dirReader, hit.doc,
fieldName, fragCharSize, maxNumFragments, fragListBuilder,
fragmentsBuilder, preTags, postTags, encoder);
最后我们可以使用fieldPhraseList 来访问我们需要的数据:
// the following gives you access to positions and offsets:
fieldPhraseList.getPhraseList().forEach(weightedPhraseInfo -> {
int phraseStartOffset = weightedPhraseInfo.getStartOffset(); // 19
int phraseEndOffset = weightedPhraseInfo.getEndOffset(); // 34
weightedPhraseInfo.getTermsInfos().forEach(termInfo -> {
String term = termInfo.getText(); // "war" "force"
int termPosition = termInfo.getPosition() + 1; // 4 6
int termStartOffset = termInfo.getStartOffset(); // 19 29
int termEndOffset = termInfo.getEndOffset(); // 22 34
});
});
phraseStartOffset 和 phraseEndOffset 是字符计数,告诉我们可以在源文档中的何处找到整个短语:
State WEAPONRY 123 War WORD1 Force And Book4 Book WEAPONRY
因此,在我们的例子中,这是从偏移量 19 到 34 的字符串(偏移量 0 是第一个“S”左侧的位置)。
然后,对于搜索查询中的每个特定术语(“war”和“force”),我们可以访问它们的偏移量以及它们的单词位置 (termPosition)。位置 0 是最前面的词,所以我在这个索引上加 1 以在原始文档中的位置 4 给出“war”,在位置 6 给出“force”:
1 2 3 4 5 6 7 8 9 10
State WEAPONRY 123 War WORD1 Force And Book4 Book WEAPONRY
完整代码供参考:
import java.io.IOException;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.nio.file.Paths;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.queryparser.classic.ParseException;
import org.apache.lucene.queryparser.classic.QueryParser;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.search.highlight.DefaultEncoder;
import org.apache.lucene.search.highlight.Encoder;
import org.apache.lucene.search.vectorhighlight.FastVectorHighlighter;
import org.apache.lucene.search.vectorhighlight.FieldPhraseList;
import org.apache.lucene.search.vectorhighlight.FieldQuery;
import org.apache.lucene.search.vectorhighlight.FieldTermStack;
import org.apache.lucene.search.vectorhighlight.FragListBuilder;
import org.apache.lucene.search.vectorhighlight.FragmentsBuilder;
import org.apache.lucene.search.vectorhighlight.SimpleFragListBuilder;
import org.apache.lucene.search.vectorhighlight.SimpleFragmentsBuilder;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
public class VectorIndexHighlighterDemo {
final String indexPath = "./index";
final String fieldName = "body";
final String searchTerm = "\"War Force\"~1";
public void doDemo() throws IOException, ParseException {
Directory dir = FSDirectory.open(Paths.get(indexPath));
try ( DirectoryReader dirReader = DirectoryReader.open(dir)) {
IndexSearcher indexSearcher = new IndexSearcher(dirReader);
Analyzer analyzer = new StandardAnalyzer();
QueryParser parser = new QueryParser(fieldName, analyzer);
Query query = parser.parse(searchTerm);
System.out.println();
System.out.println("Search term: [" + searchTerm + "]");
System.out.println("Parsed query: [" + query.toString() + "]");
TopDocs topDocs = indexSearcher.search(query, 100);
ScoreDoc[] hits = topDocs.scoreDocs;
for (ScoreDoc hit : hits) {
handleHit(hit, query, dirReader, indexSearcher);
}
}
}
private void handleHit(ScoreDoc hit, Query query, DirectoryReader dirReader,
IndexSearcher indexSearcher) throws IOException {
boolean phraseHighlight = Boolean.TRUE;
boolean fieldMatch = Boolean.TRUE;
FieldQuery fieldQuery = new FieldQuery(query, dirReader, phraseHighlight, fieldMatch);
FieldTermStack fieldTermStack = new FieldTermStack(dirReader, hit.doc, fieldName, fieldQuery);
FieldPhraseList fieldPhraseList = new FieldPhraseList(fieldTermStack, fieldQuery);
FragListBuilder fragListBuilder = new SimpleFragListBuilder();
FragmentsBuilder fragmentsBuilder = new SimpleFragmentsBuilder();
FastVectorHighlighter fvh = new FastVectorHighlighter(phraseHighlight, fieldMatch,
fragListBuilder, fragmentsBuilder);
// use whatever you want for these settings:
int fragCharSize = 100;
int maxNumFragments = 100;
String[] preTags = new String[]{"-->"};
String[] postTags = new String[]{"<--"};
Encoder encoder = new DefaultEncoder();
// the fragments string array contains the highlighted results:
String[] fragments = fvh.getBestFragments(fieldQuery, dirReader, hit.doc,
fieldName, fragCharSize, maxNumFragments, fragListBuilder,
fragmentsBuilder, preTags, postTags, encoder);
// the following gives you access to positions and offsets:
fieldPhraseList.getPhraseList().forEach(weightedPhraseInfo -> {
int phraseStartOffset = weightedPhraseInfo.getStartOffset(); // 19
int phraseEndOffset = weightedPhraseInfo.getEndOffset(); // 34
weightedPhraseInfo.getTermsInfos().forEach(termInfo -> {
String term = termInfo.getText(); // "war" "force"
int termPosition = termInfo.getPosition() + 1; // 4 6
int termStartOffset = termInfo.getStartOffset(); // 19 29
int termEndOffset = termInfo.getEndOffset(); // 22 34
});
});
// get the scores, also, if needed:
BigDecimal score = new BigDecimal(String.valueOf(hit.score))
.setScale(3, RoundingMode.HALF_EVEN);
Document hitDoc = indexSearcher.doc(hit.doc);
}
}