【发布时间】：2018-01-16 05:16:42
【问题描述】：
目前面临一个问题，即在 uWSGI 下运行的烧瓶应用程序在一段时间后出现上述错误。
出现的异常信息是：
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})
这是 wsgi.py 的应用程序代码
import sys
import logging
sys.stdout = sys.stderr
from cqlengine.connection import (
cluster as cql_cluster, session as cql_session)
from url.settings import CLUSTER
log = logging.getLogger('uwsgi')
try:
from uwsgidecorators import postfork
except ImportError:
# We're not in a uWSGI context, no need to hook Cassandra session
# initialization to the postfork event.
pass
else:
@postfork
def cassandra_init():
""" Initialize a new Cassandra session in the context.
Ensures that a new session is returned for every new request.
"""
if cql_cluster is not None:
cql_cluster.shutdown()
if cql_session is not None:
cql_session.shutdown()
from url.settings import connect_cassandra
connect_cassandra()
log.info('Connection with cassandra completed')
from url.app import app as application
这是 connect_cassandra 方法的代码。
CLUSTER = ['XX.XX.XX.XX', 'XX.XX.XX.XX']
def connect_cassandra():
# next, setup the connection to your cassandra server(s)...
# see http://datastax.github.io/python-
driver/api/cassandra/cluster.html for options
# the list of hosts will be passed to create a Cluster() instance
from cassandra.cqlengine import connection
from cassandra.cqlengine.management import sync_table
from url.models import Links, LinksAnalytics
connection.setup(CLUSTER, "contentstudio", protocol_version=3)
sync_table(Links)
sync_table(LinksAnalytics)
重新启动应用程序可以正常工作 20-30 分钟后，它会停止给出此错误，有时它可以正常工作，这真的很令人沮丧。
此外，在收到此异常消息之间：
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: XX.XX.XX.XX dc1>: ConnectionException('Host has been marked down or removed',)})
使用 nodetool tpstats 更新 1
节点 1
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 1225692 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 853120 0 0
MutationStage 0 0 62573 0 0
MemtableReclaimMemory 0 0 1133 0 0
PendingRangeCalculator 0 0 2 0 0
GossipStage 0 0 4175516 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 0 0 0
RequestResponseStage 0 0 64064 0 0
Native-Transport-Requests 0 0 12887762 0 16587
ReadRepairStage 0 0 6887 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 34 0 0
MemtablePostFlush 0 0 1268 0 0
PerDiskMemtableFlushWriter_0 0 0 1123 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 1125 0 0
InternalResponseStage 0 0 45 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
节点 2
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 29325 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 407325 0 0
MutationStage 0 0 62573 0 0
MemtableReclaimMemory 0 0 1133 0 0
PendingRangeCalculator 0 0 4 0 0
GossipStage 0 0 4174442 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 0 0 0
RequestResponseStage 0 0 6845 0 0
Native-Transport-Requests 0 0 989812 0 0
ReadRepairStage 0 0 102 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 26 0 0
MemtablePostFlush 0 0 1268 0 0
PerDiskMemtableFlushWriter_0 0 0 1123 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 1125 0 0
InternalResponseStage 0 0 0 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
【问题讨论】：
-
如果您在应用程序发出这些消息时在节点上启动
nodetool tpstats，您会得到什么？你有掉过消息吗？ -
@ThomasArnaud 我已经添加了 nodetool tpstats，现在没有出现错误。我正在尝试重现此错误并将再次发布 nodetool tpstats。
-
如果你没有任何丢弃的消息，这意味着节点没有过载，问题出在客户端。也许负载对驱动程序来说太高了。我不是 python 专家，但似乎每个连接的默认最大运行请求为 100。Java 驱动程序建议对 LOCAL 主机使用 1024 最大请求。也许你应该尝试提升它：datastax.github.io/python-driver/api/cassandra/…
-
感谢您的帮助，我将尝试将最大请求数提高到 1024，看看问题是否仍然存在。
-
@Azhar 你找到解决这个问题的方法了吗？
标签： python python-3.x cassandra cassandra-3.0 nosql