【问题标题】:Ignite service hangs when call cache remove in another cache's invoke processor, " Possible starvation in striped pool"?在另一个缓存的调用处理器中调用缓存删除时,Ignite 服务挂起,“条带池中可能出现饥饿”?
【发布时间】:2018-06-12 13:21:40
【问题描述】:

Ignite 日志出现饥饿警告并停止提供服务:

[12:55:22,080][WARNING][grid-timeout-worker-#71][G] >>> Possible starvation in striped pool.
    Thread name: sys-stripe-25-#26
    Deadlock: false
    Completed: 16272032
Thread [name="sys-stripe-25-#26", id=51, state=WAITING, blockCnt=79, waitCnt=15616666]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
    at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
    at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.remove0(GridDhtAtomicCache.java:716)
    at o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3084)
    at o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3065)
    at o.a.i.i.processors.cache.IgniteCacheProxyImpl.remove(IgniteCacheProxyImpl.java:1131)
    at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.remove(GatewayProtectedCacheProxy.java:998)
    at com.test.info.TestInfoBasicExecutor.handleCurrentLevel(TestInfoBasicExecutor.java:281)
    at com.test.info.TestInfoBasicExecutor$infoEntryProcessor.process(TestInfoBasicExecutor.java:514)
    at com.test.info.TestInfoBasicExecutor$infoEntryProcessor.process(TestInfoBasicExecutor.java:453)
    at o.a.i.i.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.runEntryProcessor(GridCacheMapEntry.java:5142)
    at o.a.i.i.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4550)
    at o.a.i.i.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4367)
    at o.a.i.i.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3051)
    at o.a.i.i.processors.cache.persistence.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2945)
    at o.a.i.i.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1717)
    at o.a.i.i.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1600)
    at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1199)
    at o.a.i.i.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1357)
    at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:345)
    at o.a.i.i.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1767)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2420)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1883)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1736)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1628)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3055)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:130)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)
    at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:261)
    at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
    at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
    at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
    at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
    at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
    at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
    at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
    at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
    at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
    at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
    at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
    at java.lang.Thread.run(Thread.java:745)

我使用invoke更新缓存A,在缓存A的etnryprocessor中, 我知道处理器已经被锁调用了,我只是为另一个缓存器更新这个条目, 我检查了缓存 A 的值,并根据该值更新缓存 B 条目,即在我的测试中放置或删除, put 没问题,但对于删除,删除原因服务似乎挂起:

    at com.test.info.TestInfoBasicExecutor.handleCurrentLevel(TestInfoBasicExecutor.java:281)
    at com.test.info.TestInfoBasicExecutor$infoEntryProcessor.process(TestInfoBasicExecutor.java:514)
    at com.test.info.TestInfoBasicExecutor$infoEntryProcessor.process(TestInfoBasicExecutor.java:453)

================================================ =======

更新 0702:

为了防止饥饿,我改变了我的代码:

在 Ignite Service A 的执行函数中:

cacheA.invoke(记录){ // 做一个记录过程

igniteQueue.put(processed_record);

}

在 Ignite Service B 的执行函数中:

saved_processed_record = igniteQueue.take();

==================

我已经尝试使用这种方式来防止饥饿,它运行顺利时 饥饿的旧代码(TPS 低),但是当我以高 TPS 运行时, “条纹池中可能的饥饿”又回来了,

看来我在 cache.invoke 中使用 igniteQueue 与 cache.invoke 中的先前缓存相比也不正确

当我想要对缓存中的每条记录进行处理时,然后根据处理后的记录来更新其他缓存,但似乎不可能?

【问题讨论】:

    标签: ignite


    【解决方案1】:

    您应该避免在入口处理器中执行缓存操作,即使这些操作属于其他缓存。原因是所有这些操作都将使用同一个线程池 - 这可能会导致饥饿。

    【讨论】:

    • 感谢您的回答,是否有任何建议如何对依赖于另一个缓存条目的缓存条目进行这种更新,我已尝试在我的帖子中将更新更改为缓存 B 以进行记录进入另一个 ignite 队列,并启动服务使用 take() 方法检查队列并以此为基础更新 ignite 缓存 B,它现在可以工作,但我想知道这种方法是否可以避免线程饥饿?
    • 你能澄清一下你所说的“依赖”是什么意思吗?您在寻找什么样的一致性/原子性保证?
    • 抱歉回复晚了,我已经更新了我的帖子以使我的问题更清楚,请查看原帖中的更新--0702更新,谢谢。
    • IgniteQueue在后台使用了Ignite缓存,所以Possible starvation in striped pool的原因是一样的。
    【解决方案2】:

    用于处理 Ignite 消息的条带池。看起来由于某种原因,该池中的所有线程都在等待某个操作(从日志中的缓存中删除)。这可能与网络问题或删除需要很长时间(例如您要删除所有数据)有关。

    能否附上线程转储和您的测试代码以供调查?

    【讨论】:

    • 我有 3 个客户端和 5 个客户端,客户端从客户端获取缓存信息,我不知道这个问题是否与太多客户端无法检索相同的缓存有关,对于回溯,服务器可以不会被转储它给出错误,所以我只是转储客户端的线程回溯,并将其粘贴在这里:pastebin.com/R2bpbimu
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2012-04-21
    • 1970-01-01
    • 2019-02-12
    • 1970-01-01
    • 1970-01-01
    • 2016-08-23
    • 1970-01-01
    相关资源
    最近更新 更多