使用分析器的错误索引弹性搜索答案

【问题标题】：Wrong indexation elasticsearch using the analyser使用分析器的错误索引弹性搜索
【发布时间】：2016-05-13 20:04:22
【问题描述】：

我做了一个非常简单的测试。我建立一个学生索引和一个类型，然后定义一个映射：

POST student
{

    "mappings" : {
        "ing3" : {
            "properties" : {
                "quote": {
                  "type": "string",
                  "analyzer": "english"
                }
            }
        }
    }
}

之后我将 3 名学生添加到该索引中：

POST /student/ing3/1
{
  "name": "Smith",
  "first_name" : "John",
  "quote" : "Learning is so cool!!"
}

POST /student/ing3/2
{
  "name": "Roosevelt",
  "first_name" : "Franklin",
  "quote" : "I learn everyday"
}

POST /student/ing3/3
{
  "name": "Black",
  "first_name" : "Mike",
  "quote" : "I learned a lot at school"
}

此时我认为英语标记器会标记我引号中的所有单词，所以如果我进行如下搜索：

GET /etudiant/ing3/_search
{
    "query" : {
        "term" : { "quote" : "learn" }
    }
}

结果我将拥有所有文档，因为我的标记器将做出平等的“学习、学习、学习”，我是对的。但是当我尝试这个请求时：

GET /student/ing3/_search
{
    "query" : {
        "term" : { "quote" : "learned" }
    }
}

我的命中率为零，我认为我应该拥有第三份文件（至少？）。但对我来说，Elasticsearch 也应该索引 learned 和 learning 而不仅仅是 learn。我错了吗？我的要求有错吗？

【问题讨论】：

标签： elasticsearch indexing lucene

【解决方案1】：

如果你检查：

GET 'index/_analyze?field=quote' -d "I learned a lot at school"

你会看到你的句子被分析为：

{
   "tokens":[
      {
         "token":"i",
         "start_offset":0,
         "end_offset":1,
         "type":"<ALPHANUM>",
         "position":0
      },
      {
         "token":"learn",
         "start_offset":2,
         "end_offset":9,
         "type":"<ALPHANUM>",
         "position":1
      },
      {
         "token":"lot",
         "start_offset":12,
         "end_offset":15,
         "type":"<ALPHANUM>",
         "position":3
      },
      {
         "token":"school",
         "start_offset":19,
         "end_offset":25,
         "type":"<ALPHANUM>",
         "position":5
      }
   ]
}

因此，英语分析器会删除标点符号和停用词，并以词根形式标记词。

https://www.elastic.co/guide/en/elasticsearch/guide/current/using-language-analyzers.html

您可以使用match 查询，它还将分析您的搜索文本，以便匹配：

GET /etudiant/ing3/_search
{
    "query" : {
        "match" : { "quote" : "learned" }
    }
}

【讨论】：

【解决方案2】：

还有另一种方法。您可以both 词干（english 分析器确实有词干分析器），但也可以保留原始词，方法是使用 keyword_repeat token filter，然后使用 unique token filter 和 "only_on_same_position": true 来删除词干后不必要的重复：

PUT student
{
  "settings": {
    "analysis": {
      "analyzer": {
        "myAnalyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "english_possessive_stemmer",
            "lowercase",
            "english_stop",
            "keyword_repeat",
            "english_stemmer",
            "unique_stem"
          ]
        }
      },
      "filter": {
        "unique_stem": {
          "type": "unique",
          "only_on_same_position": true
        },
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "english_possessive_stemmer": {
          "type": "stemmer",
          "language": "possessive_english"
        }
      }
    }
  },
  "mappings": {
    "ing3": {
      "properties": {
        "quote": {
          "type": "string",
          "analyzer": "myAnalyzer"
        }
      }
    }
  }
}

在这种情况下，term 查询也将起作用。如果您查看实际索引了哪些术语：

GET /student/_search
{
  "fielddata_fields": ["quote"]
}

现在它匹配的原因会很清楚：

  "hits": [
     {
        "_index": "student",
        "_type": "ing3",
        "_id": "2",
        "_score": 1,
        "_source": {
           "name": "Roosevelt",
           "first_name": "Franklin",
           "quote": "I learn everyday"
        },
        "fields": {
           "quote": [
              "everydai",
              "everyday",
              "i",
              "learn"
           ]
        }
     },
     {
        "_index": "student",
        "_type": "ing3",
        "_id": "1",
        "_score": 1,
        "_source": {
           "name": "Smith",
           "first_name": "John",
           "quote": "Learning is so cool!!"
        },
        "fields": {
           "quote": [
              "cool",
              "learn",
              "learning",
              "so"
           ]
        }
     },
     {
        "_index": "student",
        "_type": "ing3",
        "_id": "3",
        "_score": 1,
        "_source": {
           "name": "Black",
           "first_name": "Mike",
           "quote": "I learned a lot at school"
        },
        "fields": {
           "quote": [
              "i",
              "learn",
              "learned",
              "lot",
              "school"
           ]
        }
     }
  ]

【讨论】：