查询不会使用合适的索引答案

【问题标题】：query wouldn't use suitable index查询不会使用合适的索引
【发布时间】：2015-07-10 11:24:37
【问题描述】：

我有一个问题

db.Product.find({
    CategoryPath: /^399-305-352(-\d+)*$/,
    "Availability.Status": {
        $lt: 4
    },
    $or: [{
        _id: {
            $lt: 331000000
        }
    }, {
        _id: {
            $gt: 852000000,
            $lt: 853000000
        }
    }, {
        _id: {
            $gt: 972000000,
            $lt: 973000000
        }
    }]
}).sort({
    "Availability.Status": 1,
    Popularity: -1
});

解释我发现它使用索引Availability.Status_1_Popularity_-1:

{
    "cursor" : "BtreeCursor Availability.Status_1_Popularity_-1",
    "isMultiKey" : false,
    "n" : 913,
    "nscannedObjects" : 470239,
    "nscanned" : 470239,
    "nscannedObjectsAllPlans" : 1387264,
    "nscannedAllPlans" : 1387264,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 10838,
    "nChunkSkips" : 0,
    "millis" : 10117,
    "indexBounds" : {
        "Availability.Status" : [
            [
                -Infinity,
                4
            ]
        ],
        "Popularity" : [
            [
                {
                    "$maxElement" : 1
                },
                {
                    "$minElement" : 1
                }
            ]
        ]
    },
    "server" : "dal05mgo13.sl.dx:27017",
    "filterSet" : false
}

这对我来说很慢。我实际上还有另一个索引CategoryPath_1_Availability.Status_1，我认为这是一个更好的选择。但是当我强制 mongodb 将它与hint 一起使用时，我得到一个错误：

{
  "$err" : "Runner error: Overflow sort stage buffered data usage of 33581891 bytes exceeds internal limit of 33554432 bytes",
  "code" : 17144
}

现在我不明白的是，在find中指定的条件下，只选择了913的结果，即使没有索引，sort也不应该用掉32MB内存来对913条记录进行排序.谁能告诉我发生了什么？

我正在使用 MongoDB 2.6.10 x86_64

编辑：我的同事刚刚创建了一个新索引Availability.Status_1_Popularity_-1_CategoryPath_1，它现在正在从其他计划中获胜。我仍然不明白为什么。以下是详细解释信息：

{
    "cursor" : "BtreeCursor Availability.Status_1_Popularity_-1_CategoryPath_1",
    "isMultiKey" : true,
    "n" : 913,
    "nscannedObjects" : 1325,
    "nscanned" : 1930,
    "nscannedObjectsAllPlans" : 7729,
    "nscannedAllPlans" : 8334,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 64,
    "nChunkSkips" : 0,
    "millis" : 45,
    "indexBounds" : {
        "Availability.Status" : [
            [
                -Infinity,
                4
            ]
        ],
        "Popularity" : [
            [
                {
                    "$maxElement" : 1
                },
                {
                    "$minElement" : 1
                }
            ]
        ],
        "CategoryPath" : [
            [
                "399-305-352",
                "399-305-353"
            ],
            [
                /^399-305-352(-\d+)*$/,
                /^399-305-352(-\d+)*$/
            ]
        ]
    },
    "allPlans" : [
        {
            "cursor" : "BtreeCursor Availability.Status_1_Popularity_-1_CategoryPath_1",
            "isMultiKey" : true,
            "n" : 913,
            "nscannedObjects" : 1325,
            "nscanned" : 1930,
            "scanAndOrder" : false,
            "indexOnly" : false,
            "nChunkSkips" : 0,
            "indexBounds" : {
                "Availability.Status" : [
                    [
                        -Infinity,
                        4
                    ]
                ],
                "Popularity" : [
                    [
                        {
                            "$maxElement" : 1
                        },
                        {
                            "$minElement" : 1
                        }
                    ]
                ],
                "CategoryPath" : [
                    [
                        "399-305-352",
                        "399-305-353"
                    ],
                    [
                        /^399-305-352(-\d+)*$/,
                        /^399-305-352(-\d+)*$/
                    ]
                ]
            }
        },
        {
            "cursor" : "BtreeCursor Availability.Status_1_Popularity_-1",
            "isMultiKey" : false,
            "n" : 0,
            "nscannedObjects" : 306,
            "nscanned" : 306,
            "scanAndOrder" : false,
            "indexOnly" : false,
            "nChunkSkips" : 0,
            "indexBounds" : {
                "Availability.Status" : [
                    [
                        -Infinity,
                        4
                    ]
                ],
                "Popularity" : [
                    [
                        {
                            "$maxElement" : 1
                        },
                        {
                            "$minElement" : 1
                        }
                    ]
                ]
            }
        },
        {
            "clauses" : [
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 305,
                    "nscannedObjects" : 305,
                    "nscanned" : 305,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                852000000,
                                853000000
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                972000000,
                                973000000
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                -Infinity,
                                331000000
                            ]
                        ]
                    }
                }
            ],
            "cursor" : "QueryOptimizerCursor",
            "n" : 0,
            "nscannedObjects" : 305,
            "nscanned" : 305,
            "scanAndOrder" : true,
            "nChunkSkips" : 0
        },
        {
            "clauses" : [
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 7,
                    "nscannedObjects" : 7,
                    "nscanned" : 7,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                972000000,
                                973000000
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_1_Availability.Status_1_Popularity_-1",
                    "isMultiKey" : false,
                    "n" : 297,
                    "nscannedObjects" : 297,
                    "nscanned" : 297,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                852000000,
                                853000000
                            ]
                        ],
                        "Availability.Status" : [
                            [
                                {
                                    "$minElement" : 1
                                },
                                {
                                    "$maxElement" : 1
                                }
                            ]
                        ],
                        "Popularity" : [
                            [
                                {
                                    "$maxElement" : 1
                                },
                                {
                                    "$minElement" : 1
                                }
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                -Infinity,
                                331000000
                            ]
                        ]
                    }
                }
            ],
            "cursor" : "QueryOptimizerCursor",
            "n" : 0,
            "nscannedObjects" : 304,
            "nscanned" : 304,
            "scanAndOrder" : true,
            "nChunkSkips" : 0
        },
        {
            "clauses" : [
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 305,
                    "nscannedObjects" : 305,
                    "nscanned" : 305,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                852000000,
                                853000000
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_1_Availability.Status_1_Popularity_-1",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                972000000,
                                973000000
                            ]
                        ],
                        "Availability.Status" : [
                            [
                                {
                                    "$minElement" : 1
                                },
                                {
                                    "$maxElement" : 1
                                }
                            ]
                        ],
                        "Popularity" : [
                            [
                                {
                                    "$maxElement" : 1
                                },
                                {
                                    "$minElement" : 1
                                }
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                -Infinity,
                                331000000
                            ]
                        ]
                    }
                }
            ],
            "cursor" : "QueryOptimizerCursor",
            "n" : 0,
            "nscannedObjects" : 305,
            "nscanned" : 305,
            "scanAndOrder" : true,
            "nChunkSkips" : 0
        },
        {
            "clauses" : [
                {
                    "cursor" : "BtreeCursor _id_1_Availability.Status_1_Popularity_-1",
                    "isMultiKey" : false,
                    "n" : 305,
                    "nscannedObjects" : 305,
                    "nscanned" : 305,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                852000000,
                                853000000
                            ]
                        ],
                        "Availability.Status" : [
                            [
                                {
                                    "$minElement" : 1
                                },
                                {
                                    "$maxElement" : 1
                                }
                            ]
                        ],
                        "Popularity" : [
                            [
                                {
                                    "$maxElement" : 1
                                },
                                {
                                    "$minElement" : 1
                                }
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_1_Availability.Status_1_Popularity_-1",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                972000000,
                                973000000
                            ]
                        ],
                        "Availability.Status" : [
                            [
                                {
                                    "$minElement" : 1
                                },
                                {
                                    "$maxElement" : 1
                                }
                            ]
                        ],
                        "Popularity" : [
                            [
                                {
                                    "$maxElement" : 1
                                },
                                {
                                    "$minElement" : 1
                                }
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                -Infinity,
                                331000000
                            ]
                        ]
                    }
                }
            ],
            "cursor" : "QueryOptimizerCursor",
            "n" : 0,
            "nscannedObjects" : 305,
            "nscanned" : 305,
            "scanAndOrder" : true,
            "nChunkSkips" : 0
        },
        {
            "clauses" : [
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 305,
                    "nscannedObjects" : 305,
                    "nscanned" : 305,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                852000000,
                                853000000
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                972000000,
                                973000000
                            ]
                        ]
                    }
                },
                {
                    "cursor" : "BtreeCursor _id_1_Availability.Status_1_Popularity_-1",
                    "isMultiKey" : false,
                    "n" : 0,
                    "nscannedObjects" : 0,
                    "nscanned" : 0,
                    "scanAndOrder" : false,
                    "indexOnly" : false,
                    "nChunkSkips" : 0,
                    "indexBounds" : {
                        "_id" : [
                            [
                                -Infinity,
                                331000000
                            ]
                        ],
                        "Availability.Status" : [
                            [
                                {
                                    "$minElement" : 1
                                },
                                {
                                    "$maxElement" : 1
                                }
                            ]
                        ],
                        "Popularity" : [
                            [
                                {
                                    "$maxElement" : 1
                                },
                                {
                                    "$minElement" : 1
                                }
                            ]
                        ]
                    }
                }
            ],
            "cursor" : "QueryOptimizerCursor",
            "n" : 0,
            "nscannedObjects" : 305,
            "nscanned" : 305,
            "scanAndOrder" : true,
            "nChunkSkips" : 0
        }
    ],
    "server" : "dal05mgo12.sl.dx:27017",
    "filterSet" : false,
    "stats" : {
        "type" : "FETCH",
        "works" : 1931,
        "yields" : 64,
        "unyields" : 64,
        "invalidates" : 0,
        "advanced" : 913,
        "needTime" : 1017,
        "needFetch" : 0,
        "isEOF" : 1,
        "alreadyHasObj" : 0,
        "forcedFetches" : 0,
        "matchTested" : 913,
        "children" : [
            {
                "type" : "IXSCAN",
                "works" : 1931,
                "yields" : 64,
                "unyields" : 64,
                "invalidates" : 0,
                "advanced" : 1325,
                "needTime" : 605,
                "needFetch" : 0,
                "isEOF" : 1,
                "keyPattern" : "{ Availability.Status: 1.0, Popularity: -1.0, CategoryPath: 1.0 }",
                "isMultiKey" : 1,
                "boundsVerbose" : "field #0['Availability.Status']: [-inf.0, 4.0), field #1['Popularity']: [MaxKey, MinKey], field #2['CategoryPath']: [\"399-305-352\", \"399-305-353\"), [/^399-305-352(-\\d+)*$/, /^399-305-352(-\\d+)*$/]",
                "yieldMovedCursor" : 0,
                "dupsTested" : 1325,
                "dupsDropped" : 0,
                "seenInvalidated" : 0,
                "matchTested" : 0,
                "keysExamined" : 1930,
                "children" : [ ]
            }
        ]
    }
}

【问题讨论】：

向我们展示.hint("your index def").explain(1)，以便我们查看全部被拒绝的计划。
@BlakesSeven 那里的索引太多了，我必须在发布之前删除一些无意义的索引。请稍等。
@BlakesSeven 其实我不明白的是，mongodb在过滤之前是否对文档进行排序？这是Overflow 发生的唯一可能性。
除非您完全确定，否则请不要删除，因为您是提出问题的人，所以建议不要删除。但是“太多的指标”已经吓到我了，听起来有很多指标可能不应该存在。
这就是为什么至少“我”想看到完整的解释“详细”输出，因为看起来根本没有选择索引进行处理，或者至少有一个被拒绝的有力理由。

标签： mongodb mongodb-query

【解决方案1】：

这是因为索引中字段的顺序很重要。

CategoryPath_1_Availability.Status_1 转换为以下索引顺序：先按 CategoryPath 索引，然后按 Availability.Status。

但是您希望索引以Availability.Status 开头，mongo 检查是否有任何以Availability.Status 开头的索引（升序），而没有（在您的朋友创建新索引之前）。订单很重要:)

作为一般规则：假设您在集合中有以下字段：a、b 和 c。让我们在 a 和 b 上设置索引（a 升序，b 升序）。 只有当排序以 a（升序）、或 a（升序）和 b 开头时，您才能在排序中使用此索引（上升）。希望你明白:)

【讨论】：

这是一个问题，您可以看到只有 900 多个文档被选中。为什么 mongodb 不使用我的索引并在没有索引的情况下进行排序？这样会占用更少的资源。当我用索引提示时，内存溢出就来了。唯一的解释是mongodb是在过滤之前进行排序的。这是我不明白的部分。
那是因为您要求 mongo 对该查询使用错误的索引。
是的，我知道。我的意思是，mongo 可以使用我提供的索引，并在没有索引的情况下进行排序。我认为在内存中排序 900 条记录没什么大不了的。不知何故，mongo 决定在过滤之前对所有 500,000 条记录进行排序。这是我不明白的部分。
其实，mongo 是先应用排序，再应用过滤器，这就是它的工作原理