Python-xml 使用 beautifulsoup4 解析并将输出写入 mysql db - unicode 错误

Question

我正在尝试使用 beautifulsoup4 解析 xml 文件。 IDE：利利普斯 Python版本：2.7 xml编码：utf-8

示例 xml 文件：http://pastebin.com/RhjvyKDN

下面是我用来解析xml文件并将提取的信息写入本地mysql数据库的代码。

from bs4 import BeautifulSoup
import pymysql
import os, os.path

#strips apostrophes from the text and then just adds them at the beginning and end for the query
def apostro(text):

    text= text.replace("'","")
    text= text.replace(",","")
    text = "'"+text+"'"
    return text

#sets up the MYSQL connection
conn = pymysql.connect(host='127.0.0.1',  user='xxxx', passwd='xxxx', db='mysql', port= 3306 )
cur = conn.cursor()

#drop all of the previous values from the database
cur.execute("DELETE FROM db WHERE title is not null")
conn.commit()

#loop through all of the files
for root, _, files in os.walk("C:/usc/xml"):
    for f in files:
        #j is a counter for how many sections we have processed
        j=0
        #fullpath is the location of the file we're parsing
        fullpath = os.path.join(root, f)
        print(fullpath)

        #open file using BeautifulSoup
        soup = BeautifulSoup(open(""+fullpath+""), 'xml')
        sec = soup.find_all("section", {"style" : "-uslm-lc:I80"})
        t = soup.main.title
        t_num = t.num['value']

        #if not clauses are needed in case there is a blank, otherwise an error is thrown
        if not t.heading.text:
            t_head = ''
        else:
            t_head = t.heading.text.encode('ascii', 'ignore').encode("UTF-8")

        for element in sec:
            if not element.num['value']:
                section = ''
            else:
                section = element.num['value'].encode('ascii', 'ignore').encode("UTF-8")

            if not element.heading:
                s_head = ''
            else:
                s_head = element.heading.text.encode('ascii', 'ignore').encode("UTF-8")

            if not element.text:
                s_text = ''
            else:
                s_text = element.text.encode('ascii', 'ignore').encode("UTF-8")

            #inserttest is the sql command that 'cur' executes. counter is printed every time a section is written to let me know the program is still alive
            inserttest = "insert into deadlaws.usc_new (title, t_head, section, s_head, s_text) values (" + t_num + "," + apostro(t_head) + "," + apostro(section) + "," + apostro(s_head) + "," + apostro(s_text) +")"
            j=j+1
            cur.execute( inserttest)
            conn.commit()
            print(fullpath + " " +str(j))

conn.commit()
cur.close()
conn.close()

一切都很顺利，直到我注意到程序忽略了部分编号中的连字符'-'，这使得整个活动出错。

我知道我在编码语句中使用了'ignore'，但是连字符'-' 在ascii 中是合法字符，对吧？它不应该将字符写入数据库而不是忽略它吗？

我在 SO 和其他地方做了很多阅读。

我尝试在汤语句中包含from_encoding="utf-8"，在encode() 语句中包含'xmlrefreplace' 和其他方法，这导致了以下输出：它改写了这个a–（一些特殊的Unicode 字符）连字符'-' 到数据库中。

示例输出：

数据量很大，恐怕还有其他字符，例如-，被程序忽略了。如果它忽略来自t_head、s_head 和s_text 字段的特殊字符也没关系，因为它们是文本而不是节列。

我们将不胜感激任何解决此问题的帮助。

Answer 1

不要编码，MySQL 库完全能够将 Unicode 文本直接插入数据库。使用SQL parameters，而不是字符串插值，连接数据库时使用specify the character set：

conn = pymysql.connect(host='127.0.0.1', user='xxxx', passwd='xxxx',
                       db='mysql', port=3306,
                       charset='utf8')

不编码：

t_head = t.heading.text or ''

for element in sec:
    if not element.num['value']:
        section = ''
    else:
    section = element.num.get('value', '')
    s_head = element.heading.text or ''
    s_text = element.text or ''

    inserttest = "insert into deadlaws.usc_new (title, t_head, section, s_head, s_text) values (?, ?, ?, ?)"
    cur.execute(inserttest, (t_num, t_head, section, s_head, s_text))