如何在scrapy爬虫中使用用户名/密码进行身份验证？答案

【问题标题】：How to authenticate with username/password in scrapy crawler?如何在scrapy爬虫中使用用户名/密码进行身份验证？
【发布时间】：2016-08-08 14:49:48
【问题描述】：

我想在scrapy爬虫中使用用户名/密码对Pymongo客户端进行身份验证。使用这种方法但出现错误

   class MongoDBPipeline(object):
        def __init__(self):
            connection = pymongo.MongoClient(settings['MONGODB_HOST'], settings['MONGODB_PORT'])
            connection.the_database.authenticate(settings['MONGODB_USERNAME'],settings['MONGODB_PASSWORD'],source='$external', mechanism='PLAIN')
            db = connection[settings['MONGODB_DATABASE']]
            self.collection = db[settings['MONGODB_COLLECTION']]

错误：

 connection.the_database.authenticate(settings['MONGODB_USERNAME'],settings['MONGODB_PASSWORD'],source='$external', mechanism='PLAIN')
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/database.py", line 988, in authenticate
    connect=True)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 397, in _cache_credentials
    sock_info.authenticate(credentials)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/pool.py", line 287, in authenticate
    auth.authenticate(credentials, self)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/auth.py", line 407, in authenticate
    auth_func(credentials, sock_info)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/auth.py", line 329, in _authenticate_plain
    sock_info.command(source, cmd)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/pool.py", line 184, in command
    codec_options, check, allowable_errors)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/network.py", line 54, in command
    helpers._check_command_response(response_doc, msg, allowable_errors)
  File "/home/nikhil/.local/lib/python2.7/site-packages/pymongo/helpers.py", line 188, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: command SON([('saslStart', 1), ('mechanism', 'PLAIN'), ('payload', Binary('\x00nikhil\x00password', 0)), ('autoAuthorize', 1)]) on namespace $external.$cmd failed: no such cmd: saslStart

我想验证我的代码以将其写入 mongodb。

connection = pymongo.MongoClient("mongodb://localhost")

用户名和密码放在哪里？

【问题讨论】：

你看过here吗？您确定您在 PyMongo 中使用的身份验证方法与您设置数据库的方式相同吗？

标签： python mongodb scrapy pymongo

【解决方案1】：

scrapy-mongodb 会很容易使用。

但是，通常情况下，您需要从 mongodb 的 python 文档中引用 authentication examples。

-- 参考--

支持用户名和密码中的特殊字符

如果您的用户名或密码包含特殊字符（例如“/”、“”或“@”），您必须 %xx 转义它们以在 MongoDB URI 中使用。 PyMongo 使用 unquote_plus() 对它们进行解码。例如：

MongoClient('mongodb://user:' + password + '@127.0.0.1')

【讨论】：