【发布时间】：2013-08-28 11:46:55
【问题描述】：
我遇到了许多错误，具体取决于插入/更新的内容。
这里是处理物品的代码：
def process_item(self, item, spider):
try:
if 'producer' in item:
self.cursor.execute("""INSERT INTO Producers (title, producer) VALUES (%s, %s)""", (item['title'], item['producer']))
elif 'actor' in item:
self.cursor.execute("""INSERT INTO Actors (title, actor) VALUES (%s, %s)""", (item['title'], item['actor']))
elif 'director' in item:
self.cursor.execute("""INSERT INTO Directors (title, director) VALUES (%s, %s)""", (item['title'], item['director']))
else:
self.cursor.execute("""UPDATE example_movie SET distributor=%S, rating=%s, genre=%s, budget=%s WHERE title=%s""", (item['distributor'], item['rating'], item['genre'], item['budget'], item['title']))
self.conn.commit()
except MySQLdb.Error, e:
print "Error %d: %s" % (e.args[0], e.args[1])
return item
以下是从爬虫返回的items 示例：
[{'budget': [u'N/A'], 'distributor': [u'Lorimar'], 'genre': [u'Action'], 'rating': [u'R'],'title': [u'Action Jackson']}, {'actor': u'Craig T. Nelson', 'title': [u'Action Jackson']}, {'actor': u'Sharon Stone', 'title': [u'Action Jackson']}, {'actor': u'Carl Weathers', 'title': [u'Action Jackson']}, {'producer': u'Joel Silver', 'title': [u'Action Jackson']}, {'director': u'Craig R. Baxley', 'title': [u'Action Jackson']}]
以下是返回的错误：
2013-08-25 23:04:57-0500 [ActorSpider] ERROR: Error processing {'budget': [u'N/A'],
'distributor': [u'Lorimar'],
'genre': [u'Action'],
'rating': [u'R'],
'title': [u'Action Jackson']}
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/scrapy/middleware.py", line 62, in _process_chain
return process_chain(self.methods[methodname], obj, *args)
File "/Library/Python/2.7/site-packages/scrapy/utils/defer.py", line 65, in process_chain
d.callback(input)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 361, in callback
self._startRunCallbacks(result)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 455, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 542, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/Users/fortylashes/Documents/Management_Work/BoxOfficeMojo/BoxOfficeMojo/pipelines.py", line 53, in process_item
self.cursor.execute("""UPDATE example_movie SET distributor=%S, rating=%s, genre=%s, budget=%s WHERE title=%s""", (item['distributor'], item['rating'], item['genre'], item['budget'], item['title']))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py", line 159, in execute
query = query % db.literal(args)
exceptions.ValueError: unsupported format character 'S' (0x53) at index 38
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), 'Craig T. Nelson')' at line 1
2013-08-25 23:04:57-0500 [ActorSpider] DEBUG: Scraped from <200 http://www.boxofficemojo.com/movies/?id=actionjackson.htm>
{'actor': u'Craig T. Nelson', 'title': [u'Action Jackson']}
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), 'Sharon Stone')' at line 1
2013-08-25 23:04:57-0500 [ActorSpider] DEBUG: Scraped from <200 http://www.boxofficemojo.com/movies/?id=actionjackson.htm>
{'actor': u'Sharon Stone', 'title': [u'Action Jackson']}
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), 'Carl Weathers')' at line 1
2013-08-25 23:04:57-0500 [ActorSpider] DEBUG: Scraped from <200 http://www.boxofficemojo.com/movies/?id=actionjackson.htm>
{'actor': u'Carl Weathers', 'title': [u'Action Jackson']}
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), 'Joel Silver')' at line 1
2013-08-25 23:04:57-0500 [ActorSpider] DEBUG: Scraped from <200 http://www.boxofficemojo.com/movies/?id=actionjackson.htm>
{'producer': u'Joel Silver', 'title': [u'Action Jackson']}
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), 'Craig R. Baxley')' at line 1
2013-08-25 23:04:57-0500 [ActorSpider] DEBUG: Scraped from <200 http://www.boxofficemojo.com/movies/?id=actionjackson.htm>
{'director': u'Craig R. Baxley', 'title': [u'Action Jackson']}
显然有很多问题。感谢您的阅读！非常感谢任何和所有建议或想法！
::::更新/更多信息::::
在总共 52 部测试集中，似乎有三部影片被插入到 Actors、Producers 和 Directors 表中。 注意：UPDATE 语句根本不起作用。
这些电影是：Abraham Lincoln: Vampire Hunter、Ace Ventura: Pet Detective 和 Ace Ventura: When Nature Calls
有趣的是，这些都是: 在title 中的所有电影- 我不确定这是什么意思，但如果有人有想法，请分享！
:::::插入已解决:::::
原来问题是由scraper 将单个项目放入列表中引起的。所以{'actor': [u'this one guy'] 而不是顶部{'actor': u'this one guy'}。
【问题讨论】：
标签： scrapy mysql-python pipeline