【问题标题】:Why is the performance of those 3 very similar neo4j cypher queries that drastically different?为什么这 3 个非常相似的 neo4j 密码查询的性能大不相同?
【发布时间】:2014-11-26 13:19:52
【问题描述】:

知道为什么第一个密码查询比第二个和第三个查询快得多吗?

第一个查询:

MATCH (source:Product { product_id:14603 }), 
      (destination:Product{product_id:286502}), 
      p = (source-[r]-()-[*0..3]-destination) 
RETURN p, length(p) as pathLength LIMIT 50

==> 14 秒

==> 50 rows
==> 
==> |             Operator | Rows | DbHits |                                      Identifiers |                                 Other |
==> +----------------------+------+--------+--------------------------------------------------+---------------------------------------+
==> |         ColumnFilter |   50 |      0 |                                                  |            keep columns p, pathLength |
==> |                Slice |   50 |      0 |                                                  |                          {  AUTOINT2} |
==> |              Extract |   50 |      0 |                                                  |                            pathLength |
==> |          ExtractPath |   50 |      0 |                                                p |                                       |
==> | SimplePatternMatcher |   50 |      0 | source,   UNNAMED90, destination, r,   UNNAMED89 |                                       |
==> |          SchemaIndex |   99 |    198 |                                   source, source |            {  AUTOINT0}; :Product(product_id) |
==> |     TraversalMatcher |   99 | 419669 |                                                  |   UNNAMED89,   UNNAMED90,   UNNAMED89 |
==> +----------------------+------+--------+--------------------------------------------------+---------------------------------------+
==> 
==> Total database accesses: 419867

第二次查询:

MATCH (source:Product { product_id:14603 }), 
      (destination:Product{product_id:286502}), 
      p = (source-[r]-()-[*0..2]-destination) 
RETURN p, length(p) as pathLength LIMIT 50

==> 140 秒

==> 13 rows
==>  
==> +----------------------+---------+---------+--------------------------------------------------+---------------------------------------+
==> |             Operator |    Rows |  DbHits |                                      Identifiers |                                 Other |
==> +----------------------+---------+---------+--------------------------------------------------+---------------------------------------+
==> |         ColumnFilter |      13 |       0 |                                                  |            keep columns p, pathLength |
==> |                Slice |      13 |       0 |                                                  |                          {  AUTOINT2} |
==> |              Extract |      13 |       0 |                                                  |                            pathLength |
==> |          ExtractPath |      13 |       0 |                                                p |                                       |
==> | SimplePatternMatcher |      13 |       0 | source,   UNNAMED90, destination, r,   UNNAMED89 |                                       |
==> |          SchemaIndex | 2266560 | 4533120 |                                   source, source |            {  AUTOINT0}; :Product(product_id) |
==> |     TraversalMatcher | 2266560 | 2266605 |                                                  |   UNNAMED89,   UNNAMED90,   UNNAMED89 |
==> +----------------------+---------+---------+--------------------------------------------------+---------------------------------------+
==> 
==> Total database accesses: 6799725

messages.log:

2014-11-26 03:41:18.282+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 139ms [total block time: 14.49s]
2014-11-26 03:41:20.086+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 101ms [total block time: 14.591s]
2014-11-26 03:41:29.022+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 144ms [total block time: 14.735s]
2014-11-26 03:41:43.230+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 151ms [total block time: 14.886s]
2014-11-26 03:41:45.160+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 127ms [total block time: 15.013s]
2014-11-26 03:41:48.830+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 135ms [total block time: 15.148s]
2014-11-26 03:42:01.956+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 134ms [total block time: 15.282s]
2014-11-26 03:42:05.731+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 171ms [total block time: 15.453s]
2014-11-26 03:42:07.555+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 122ms [total block time: 15.575s]
2014-11-26 03:42:13.271+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 144ms [total block time: 15.719s]
2014-11-26 03:42:36.465+0000 WARN  [o.n.k.EmbeddedGraphDatabase]: GC Monitor: Application threads blocked for an additional 115ms [total block time: 15.834s]

第三次查询:

MATCH (source:Product { product_id:14603 }), 
      (destination:Product{product_id:286502}), 
      p = (source-[*0..3]-destination)
RETURN p, length(p) as pathLength LIMIT 50

==> 无限运行时间(10 分钟用尽 CPU 后,我重新启动 neo4j 服务器)


服务器数据:

  • Neo4J 版本:community-2.1.1
  • 节点:650,000
  • 产品节点:550,000
  • 属性:8,000,000
  • 关系:6,000,000
  • 关系类型:9
  • 32GB 内存
  • 768GB 固态硬盘
  • 12 核 CPU

索引:

ON :Product(product_id) ONLINE(用于唯一性约束)
[...] 其他一些与这些查询无关的索引

neo4j.properties

  • neostore.nodestore.db.mapped_memory=50M
  • neostore.relationshipstore.db.mapped_memory=400M
  • neostore.propertystore.db.mapped_memory=400M
  • neostore.propertystore.db.strings.mapped_memory=400M
  • neostore.propertystore.db.arrays.mapped_memory=400M
  • node_cache_size=5G
  • relationship_cache_size=5G
  • 所有其他属性均为默认值

非常感谢!

【问题讨论】:

  • 您需要匹配每种类型的关系,还是可以在方括号[r:TAGGED][:TAGGED*0..2] 中放置关系类型约束? (我不明白为什么 1 比 2 快),您能否添加一些有关您的服务器/Neo4J 配置的详细信息。
  • 对于这个查询,所有的关系类型都是感兴趣的。服务器托管在 linode ==> RAM:32GB,CPU:12 核,HDD:768GB SSD 发布哪些新配置有意义?
  • conf/neo4j.properties 以及 messages.log
  • 您能否将PROFILE 输出添加到您的问题中格式化的所有 3 个查询?这将非常有帮助
  • 你可以尝试运行 Neo4j 2.1.5 吗?

标签: neo4j cypher database-performance


【解决方案1】:

如果您尝试将这些模式中的每一个作为单独的 MATCH 而不是逗号分隔,这有什么不同吗?例如

MATCH (source:Product { product_id:14603 })
MATCH  (destination:Product{product_id:286502})
MATCH p = (source-[r]-()-[*0..3]-destination) 
RETURN p, length(p) as pathLength LIMIT 50

此外,如果您在 /webadmin 上运行查询,那么您可以尝试在其前面加上“PROFILE”一词并将其输出粘贴到此处,我会看看它。

【讨论】:

  • 谢谢@Mark,遗憾的是,使用此查询完全耗尽了服务器。等待 5 分钟后,我不得不重新启动 neo4j。我使用原始查询 #1 运行配置文件:==> ColumnFilter | +切片 | +提取 | +提取路径 | +SimplePatternMatcher | +架构索引 | +TraversalMatcher ==> 数据库访问总数:419867 ColumnFilter、Slice、Extract、ExtractPath、SimplePatternMatcher 各有 50 行和 0 dbHits。 SchemaIndex 有 99 行和 198 个 DbHits。 ==> TraversalMatcher 有 99 行和 419669 个 DbHits :Product(product_id) 索引已被使用。
  • 您能否分享一个(可能是匿名的?)数据集版本,以便我可以使用它并弄清楚发生了什么?
  • 很遗憾,没有...您还可以使用任何其他信息吗?再次感谢
  • 有什么方法可以向最后一个模式添加方向?
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2014-06-20
  • 1970-01-01
  • 2011-10-25
  • 1970-01-01
  • 2011-04-19
  • 1970-01-01
  • 2013-10-31
相关资源
最近更新 更多