【发布时间】:2020-05-04 05:03:01
【问题描述】:
我使用Python scrapy(1.8.0) 抓取一些数据并将其保存到mongodb 和pymongo。
当我使用Python2 和更早的pymongo 版本时,它可以工作。
但是当我尝试使用Python3 和pymongo 3.10.1 时,我发现插入文档不起作用。
我发现问题来自datetime,
像这样:
'releasedTime': [datetime.datetime(2020, 1, 17, 14, 30),datetime.datetime(2020, 1, 16, 18, 10),datetime.datetime(2020, 1, 17, 22, 0)]
日期数据来自如下:
stripSingleTime = singleTime.strip() # 14:30
myTime = datetime.datetime.strptime(stripSingleTime, '%H:%M').time() # 14:30:00
myDateTime = datetime.datetime.combine(datetime.date.today(), myTime) # 2020-01-17 14:30:00
# x is my for loop argument
myDateTimeArray[x].append(myDateTime) # [[datetime.datetime(2020, 1, 17, 14, 30)], []]
这里是关于insert_one:
import pymongo
from pymongo import MongoClient
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
import datetime
class MongoDBPipeline(object):
global theaters
theaters = []
def __init__(self):
connection = MongoClient(
settings['MONGODB_SERVER'],
settings['MONGODB_PORT'])
self.db = connection[settings['MONGODB_DB']]
self.collection = self.db[settings['MONGODB_COLLECTION']]
def open_spider(self, spider):
print ('Pipelines => open_spider =>', spider)
def process_item(self, item, spider):
global theaters
self.collection = self.db[type(item).__name__.replace('_Item','')]
if item['theater'] not in theaters:
theaters.append(item['theater'])
self.collection.remove({'theater': item['theater']})
# self.collection.insert_one(mydict) is working
mydict = { 'name': 'RUNOOB', 'alexa': '10001', 'url': 'https://www.runoob.com' }
test = {
'geometry': {'coordinates': [120.196866, 22.99322], 'type': 'Point'},
'phone': '06-2205151',
'theater': 'TodayTainan',
'movie': [{
'enName': 'The Lion King',
'goodMinePoint': 0.75,
'imdbScore': '8.5',
'photoHref': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/June2019/M07RYsvcBWpi3xJtjCQF-2714x3878.jpg',
'releasedTime': [datetime.datetime(2020, 1, 17, 14, 30)],
'rottenScore': '93%',
'videoId': ['MQuUkET0lQg', 'he2rj_8XwsE']
}]
}
self.collection.insert_one(test) # insert the dummy object test is not working
self.collection.create_index([("geometry", pymongo.GEOSPHERE)])
return item
为什么self.collection.insert_one(test) 不起作用?
我猜pymongo 和 mongodb 之间有问题。他们不处理我的datetime。
任何帮助将不胜感激。谢谢。
【问题讨论】:
-
定义“不工作”。它会出错吗?文件没有插入但没有错误?插入的文档包含意外数据?
-
如果我在
self.collection.insert_one(test)之后设置 print('test')。它不会被打印出来。如果我设置 try except 与self.collection.insert_one(test),代码将运行捕捉,但 print(err) 什么都没有。
标签: python-3.x mongodb scrapy pymongo