【问题标题】:scrapy pipeline error using mysql.connector使用mysql.connector的scrapy管道错误
【发布时间】:2015-12-31 22:12:40
【问题描述】:

我完全迷路了。这是我的管道。当我运行它时,我收到一个错误提示

      File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\Python27\bff\bff\pipelines.py", line 42, in process_item
    cursor.execute(add_Product)
  File "c:\python27\lib\site-packages\mysql\connector\cursor.py", line 492, in execute
    stmt = operation.encode(self._connection.python_charset)
AttributeError: 'tuple' object has no attribute 'encode'

从注释代码可以看出,我尝试了几种不同的方法。起初,我按照在示例中看到的那样进行操作,但是当我将 (item['StoreName']) 放入 VALUES 行而不是在上面将其定义为 Name = item 时,我得到了一个错误,即未定义 sadi item ['商店名称'] 我正在使用从 mqsql.org 网站安装的 mySQL.connector。在此先感谢

# -*- coding: utf-8 -*-
# Define your item pipelines here
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
from __future__ import print_function
from datetime import date, datetime, timedelta
import mysql.connector  
#from scrapy.extensions import DropItem
#from bff.items import ItemInfo

class mySQLPipeline(object):
def process_item(self, item, spider):

    Path = item['ProdPath']
    UPC = item['ProdUPC']
    Model = item['ProdModel']
    Desc = item['ProdDesc']
    Price = item['ProdPrice']
    Stock = item['InStock']
    #Ships = item['Ships']
    Name = item['StoreName']


    cnx = mysql.connector.connect(user='*****', password='*****',
                                  host='127.0.0.1',
                                  port='****',
                                  database='scrapyinfo')
    cursor = cnx.cursor()

    #add_Product = ("INSERT INTO walmart_products (ProdName)"
    #               "VALUES (%s), (Name);")

    add_Product = ("INSERT INTO walmart_products, (ProdName, ProdPath, ProdUPC, ProdModel, ProdDesc, ProdPrice, InStock, Ships, StoreName)"
                   "VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)", (Name, Path, UPC, Model, Desc, Price, Stock, Name))
                    #item['Ships'],



    #Add new product
    cursor.execute(add_Product)

    # Make sure data is committed to the database
    cnx.commit()

    cursor.close()
    cnx.close()
    return item

编辑。这是我的新代码

`

from __future__ import print_function
from datetime import date, datetime, timedelta
import mysql.connector  
#from scrapy.extensions import DropItem
#from bff.items import ItemInfo

class mySQLPipeline(object):
    def process_item(self, item, spider):

        Product = item['ProdName']      
        Path = item['ProdPath']
        UPC = item['ProdUPC']
        Model = item['ProdModel']
        Desc = item['ProdDesc']
        Price = item['ProdPrice']
        Stock = item['InStock']
        #Ships = item['Ships']
        Name = item['StoreName']


        cnx = mysql.connector.connect(user='****', password='****',
                                      host='127.0.0.1',
                                      port='****',
                                      database='****')
        cursor = cnx.cursor()
 #       add_Product = ("INSERT INTO walmart_products (ProdName, StoreName) VALUES (%s, %s,)", Product, Name,)
 #       add_Product = ("INSERT INTO walmart_products, (ProdName)"
 #                      "VALUES (%s)", (Name))
 #                      "VALUES (%(Name)s)")
        add_Product = ("INSERT INTO walmart_products "
                        "(ProdName, ProdPath, ProdUPC, ProdModel, ProdDesc, ProdPrice, InStock, StoreName) "
                        "VALUES (%s, %s, %s, %s, %s, %s, %s, %s)")
                        #item['Ships'],

        data_Product = (Product, Path, UPC, Model, Desc, Price, Stock, Name)                                        

        #Add new product
        cursor.execute(add_Product, data_Product)

        # Make sure data is committed to the database
        cnx.commit()

        cursor.close()
        cnx.close()
        return item

`

【问题讨论】:

  • 字段填充的是列表还是字符串?
  • 没有列表。这是我的 items.py # -*- coding: utf-8 -*- # Define here the models for your scraped items # # See documentation in: # http://doc.scrapy.org/en/latest/topics/items.html import scrapy class ItemInfo(scrapy.Item): ProdPath = scrapy.Field() ProdUPC = scrapy.Field() ProdName = scrapy.Field() ProdModel = scrapy.Field() ProdDesc = scrapy.Field() InStock = scrapy.Field() ProdPrice = scrapy.Field() StoreName = scrapy.Field() Ships = scrapy.Field() pass 的代码
  • 分享您如何填充项目,而不是项目声明。
  • 抱歉,我试图弄清楚如何让代码在回复中正确显示。这是我如何填充项目的示例。 ` def parse_item(self, response): sel = Selector (response) item = ItemInfo() item['ProdPath'] = sel.xpath('/html/head/meta[@property="og:url"]/@ content').extract() item['ProdUPC'] = sel.xpath('/html/head/meta[@property="og:upc"]/@content').extract()`
  • .extract() 给出一个列表。

标签: python mysql scrapy scrapy-pipeline


【解决方案1】:

万一有人被困在我原来的位置,我想不通。我很沮丧并停止使用 mysql.connector,而是使用了 MySQLdb。一旦我切换到那个,那么一切都很好。

【讨论】:

    猜你喜欢
    • 2012-05-13
    • 1970-01-01
    • 1970-01-01
    • 2013-07-28
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多