solr 评分 - fieldnorm

【问题标题】：solr scoring - fieldnormsolr 评分 - fieldnorm
【发布时间】：2011-12-24 15:38:03
【问题描述】：

当我搜索“iphone”时，我有以下记录和分数 -

记录1： 字段名称 - 显示名称：“Iphone” 字段名称 - 名称：“Iphone”

11.654595 = (MATCH) sum of:
  11.654595 = (MATCH) max plus 0.01 times others of:
    7.718274 = (MATCH) weight(DisplayName:iphone^10.0 in 915195), product of:
      0.6654692 = queryWeight(DisplayName:iphone^10.0), product of:
        10.0 = boost
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.0057376726 = queryNorm
      11.598244 = (MATCH) fieldWeight(DisplayName:iphone in 915195), product of:
        1.0 = tf(termFreq(DisplayName:iphone)=1)
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        1.0 = fieldNorm(field=DisplayName, doc=915195)
    11.577413 = (MATCH) weight(Name:iphone^15.0 in 915195), product of:
      0.99820393 = queryWeight(Name:iphone^15.0), product of:
        15.0 = boost
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.0057376726 = queryNorm
      11.598244 = (MATCH) fieldWeight(Name:iphone in 915195), product of:
        1.0 = tf(termFreq(Name:iphone)=1)
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        1.0 = fieldNorm(field=Name, doc=915195)

记录2： FieldName - DisplayName : "The Iphone Book" FieldName - 名称：“The Iphone Book”

7.284122 = (MATCH) sum of:
  7.284122 = (MATCH) max plus 0.01 times others of:
    4.823921 = (MATCH) weight(DisplayName:iphone^10.0 in 453681), product of:
      0.6654692 = queryWeight(DisplayName:iphone^10.0), product of:
        10.0 = boost
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.0057376726 = queryNorm
      7.2489023 = (MATCH) fieldWeight(DisplayName:iphone in 453681), product of:
        1.0 = tf(termFreq(DisplayName:iphone)=1)
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.625 = fieldNorm(field=DisplayName, doc=453681)
    7.2358828 = (MATCH) weight(Name:iphone^15.0 in 453681), product of:
      0.99820393 = queryWeight(Name:iphone^15.0), product of:
        15.0 = boost
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.0057376726 = queryNorm
      7.2489023 = (MATCH) fieldWeight(Name:iphone in 453681), product of:
        1.0 = tf(termFreq(Name:iphone)=1)
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.625 = fieldNorm(field=Name, doc=453681)

记录3： 字段名称 - 显示名称：“iPhone” 字段名称 - 名称：“iPhone”

7.284122 = (MATCH) sum of:
  7.284122 = (MATCH) max plus 0.01 times others of:
    4.823921 = (MATCH) weight(DisplayName:iphone^10.0 in 5737775), product of:
      0.6654692 = queryWeight(DisplayName:iphone^10.0), product of:
        10.0 = boost
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.0057376726 = queryNorm
      7.2489023 = (MATCH) fieldWeight(DisplayName:iphone in 5737775), product of:
        1.0 = tf(termFreq(DisplayName:iphone)=1)
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.625 = fieldNorm(field=DisplayName, doc=5737775)
    7.2358828 = (MATCH) weight(Name:iphone^15.0 in 5737775), product of:
      0.99820393 = queryWeight(Name:iphone^15.0), product of:
        15.0 = boost
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.0057376726 = queryNorm
      7.2489023 = (MATCH) fieldWeight(Name:iphone in 5737775), product of:
        1.0 = tf(termFreq(Name:iphone)=1)
        11.598244 = idf(docFreq=484, maxDocs=19431244)
        0.625 = fieldNorm(field=Name, doc=5737775)

为什么 Record2 有 3 个词而 Record3 只有一个词时，Record2 和 Record3 的分数相同。所以 Record3 应该比 Record 2 具有更高的相关性。为什么 Record2 和 Record3 的 fieldNorm 相同？

QueryParser: Dismax FieldType：solrconfig.xml 中默认的文本字段类型

添加数据源：

记录 1：Iphone

{
        "ListPrice":1184.526,
        "ShipsTo":1,
        "OID":"190502",
        "EAN":"9780596804299",
        "ISBN":"0596804296",
        "Author":"Pogue, David",
        "product_type_fq":"Books",
        "ShipmentDurationDays":"21",
        "CurrencyValue":"24.9900",
        "ShipmentDurationText":"NORMALLY SHIPS IN 21 BUSINESS DAYS",
        "Availability":0,
        "COD":0,
        "PublicationDate":"2009-08-07 00:00:00.0",
        "Discount":"25",
        "SubCategory_fq":"Hardware",
        "Binding":"Paperback",
        "Category_fq":"Non Classifiable",
        "ShippingCharges":"0",
        "OIDType":8,
        "Pages":"397",
        "CallOrder":"0",
        "TrackInventory":"Ingram",
        "Author_fq":"Pogue, David",
        "DisplayName":"Iphone",
        "url":"/iphone-pogue-david/books/9780596804299.htm",
        "CurrencyType":"USD",
        "SubSubCategory":"Handheld Devices",
        "Mask":0,
        "Publisher":"Oreilly & Associates Inc",
        "Name":"Iphone",
        "Language":"English",
        "DisplayPriority":"999",
        "rowid":"books_9780596804299"
        }

Record2：Iphone Book

{
        "ListPrice":1184.526,
        "ShipsTo":1,
        "OID":"94694",
        "EAN":"9780321534101",
        "ISBN":"0321534107",
        "Author":"Kelby, Scott/ White, Terry",
        "product_type_fq":"Books",
        "ShipmentDurationDays":"21",
        "CurrencyValue":"24.9900",
        "ShipmentDurationText":"NORMALLY SHIPS IN 21 BUSINESS DAYS",
        "Availability":1,
        "COD":0,
        "PublicationDate":"2007-08-13 00:00:00.0",
        "Discount":"25",
        "SubCategory_fq":"Handheld Devices",
        "Binding":"Paperback",
        "BAMcategory_src":"Computers",
        "Category_fq":"Computers",
        "ShippingCharges":"0",
        "OIDType":8,
        "Pages":"219",
        "CallOrder":"0",
        "TrackInventory":"Ingram",
        "Author_fq":"Kelby, Scott/ White, Terry",
        "DisplayName":"The Iphone Book",
        "url":"/iphone-book-kelby-scott-white-terry/books/9780321534101.htm",
        "CurrencyType":"USD",
        "SubSubCategory":" Handheld Devices",
        "BAMcategory_fq":"Computers",
        "Mask":0,
        "Publisher":"Pearson P T R",
        "Name":"The Iphone Book",
        "Language":"English",        
        "DisplayPriority":"999",
        "rowid":"books_9780321534101"
        }

记录 3：iPhone

{
        "ListPrice":278.46,
        "ShipsTo":1,
        "OID":"694715",
        "EAN":"9781411423527",
        "ISBN":"1411423526",
        "Author":"Quamut (COR)",
        "product_type_fq":"Books",
        "ShipmentDurationDays":"21",
        "CurrencyValue":"5.9500",
        "ShipmentDurationText":"NORMALLY SHIPS IN 21 BUSINESS DAYS",
        "Availability":0,
        "COD":0,
        "PublicationDate":"2010-08-03 00:00:00.0",
        "Discount":"25",
        "SubCategory_fq":"Hardware",
        "Binding":"Paperback",
        "Category_fq":"Non Classifiable",
        "ShippingCharges":"0",
        "OIDType":8,
        "CallOrder":"0",        
        "TrackInventory":"BNT",
        "Author_fq":"Quamut (COR)",
        "DisplayName":"iPhone",
        "url":"/iphone-quamut-cor/books/9781411423527.htm",
        "CurrencyType":"USD",
        "SubSubCategory":"Handheld Devices",
        "Mask":0,
        "Publisher":"Sterling Pub Co Inc",
        "Name":"iPhone",
        "Language":"English",
        "DisplayPriority":"999",
        "rowid":"books_9781411423527"
        }

【问题讨论】：

  • 没有看到记录 1 和记录 3 之间有任何区别，所以我看不出结果会有什么不同。您能否确认分数属于记录 3 并提供更多关于查询解析器、字段类型和数据馈送的信息？
  • 您好，我正在使用 dismax 查询解析器，分数属于记录 3。我已经交叉检查，所有 3 条记录的分数都是我提到的。字段类型是 solrconfig.xml 中定义的默认“文本”字段类型。
  • 能否也提供数据馈送？
  • 嗨 Jayendra.. 我已经用提要编辑了我的问题.. 谢谢..

标签： solr scoring


【解决方案1】：

fieldnorm 考虑了字段长度，即术语的数量。
使用的字段类型是字段显示名称和名称的文本，其中包含停用词和单词分隔符过滤器。

记录 1 - Iphone
将生成一个令牌 - IPhone

记录 2 - The Iphone Book
将生成 2 个令牌 - Iphone, Book
将被停用词删除。

记录 3 - iPhone
还会生成 2 个令牌 - i,phone
由于 iPhone 有大小写变化，使用 splitOnCaseChange 的单词分隔符过滤器现在会将 iPhone 拆分为 2 个标记 i, Phone 并生成与 Record 2 相同的字段 norm

【讨论】：

  • 嗨 Jayendra，谢谢.. 有道理。但是对于另一个数据集，我得到了错误的相关性。 Record1：“真正的达芬奇密码”和Record2：“达芬奇密码”。当我搜索“达芬奇密码”时，两条记录的分数相同，因为 fieldNorm 是相同的。它使用相同的字段类型“文本”。使用 SolrAnalysis 我看到 Record1 更改为“real da vinci code”，record2 更改为“da vinci code”（索引时间）。那为什么两个分数一样呢。
  • 能否提供数据字段和调试分数？
  • 您好 Jayendra，数据字段是“文本”。我已将调试分数添加为答案的一部分。
【解决方案2】：

这是 user1021590 关于“达芬奇密码”搜索示例的后续问题/答案的答案。

所有文档获得相同分数的原因是由于 lengthNorm 的一个微妙的实现细节。 Lucence TFIDFSimilarity doc 声明了以下关于 norm(t, d) 的内容：

生成的规范值在存储之前被编码为单个字节。在搜索时，从索引目录中读取标准字节值并将其解码回浮点标准值。这种编码/解码虽然减少了索引大小，但也带来了精度损失——不能保证 decode(encode(x)) = x。例如，decode(encode(0.89)) = 0.75。

如果你深入研究代码，你会发现这个浮点到字节的编码是这样实现的：

public static byte floatToByte315(float f)
{
    int bits = Float.floatToRawIntBits(f);
    int smallfloat = bits >> (24 - 3);
    if (smallfloat <= ((63 - 15) << 3))
    {
        return (bits <= 0) ? (byte) 0 : (byte) 1;
    }
    if (smallfloat >= ((63 - 15) << 3) + 0x100)
    {
        return -1;
    }
    return (byte) (smallfloat - ((63 - 15) << 3));
}

将该字节解码为浮点数如下：

public static float byte315ToFloat(byte b)
{
    if (b == 0)
        return 0.0f;
    int bits = (b & 0xff) << (24 - 3);
    bits += (63 - 15) << 24;
    return Float.intBitsToFloat(bits);
}

lengthNorm 计算为1 / sqrt( number of terms in field )。然后使用floatToByte315 对其进行编码以进行存储。对于具有 3 个术语的字段，我们得到：

floatToByte315( 1/sqrt(3.0) ) = 120

对于一个有 4 个术语的字段，我们得到：

floatToByte315( 1/sqrt(4.0) ) = 120

所以它们都被解码为：

byte315ToFloat(120) = 0.5.

该文档还指出：

支持对规范值进行这种有损压缩的基本原理是，考虑到用户通过查询表达其真实信息需求的困难（和不准确），只有大的差异才重要。

更新：从 Solr 4.10 开始，此实现和相应的语句是 DefaultSimilarity 的一部分。

【讨论】：

  • 嘿，我面临同样的问题.. 对于 3 和 4 个术语的文档，fieldNorm 是相同的，因此会影响搜索结果。有什么解决办法吗？
  • 我将覆盖 lengthNorm 类方法并尝试特别处理 3 & 4 长度文档的情况..
猜你喜欢
  • 1970-01-01
  • 2011-04-04
  • 1970-01-01
  • 2014-05-13
  • 2016-08-24
  • 2018-04-06
  • 1970-01-01
  • 2011-03-07
  • 1970-01-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode