Python抓取编码问题

Question

我正在尝试使用 beautifulsoup 抓取网站。我在很大程度上是成功的，但有两个问题

1. 从网站获取数据后，我将它们打印到屏幕上以及 将它们写入 CSV 文件。网站上有一个价格字段 实际金额中的卢比符号（价格样本结构 字段：₹10000）。当我将金额打印到控制台时，它打印得很好 没有问题。当我尝试将其写入 excel 工作表时，出现错误 “Unicodeencoeerror”编解码器“charmap”无法编码字符“\u20b9” 位置 28. 我正在将其他字段打印到控制台并显示问题 最多只有两个字段，一个带有货币符号，另一个带有 * 符号

2. 我有一个循环运行以从网页中获取特定页面的所有页面 搜索。搜索结果大约是 344 页，但循环在大约页面处停止 43，只有 HTML 错误 500 作为错误消息

   import bs4
   from urllib.request import urlopen as uReq
   
   from bs4 import BeautifulSoup as Soup
   filename = "data.csv"
   f = open(filename,"w")
   headers = "phone_name, phone_price, phone_rating,number_of_ratings, 
   memory, display, camera, battery, processor, Warrenty, security, OS\n"
   f.write(headers)
   
   
   for i in range(2):      # Number of pages minus one 
           my_url = 'https://www.flipkart.com/search?as=off&as-
           show=on&otracker=start&page=
           {}&q=cell+phones&viewType=list'.format(i+1)
           print(my_url)
   
           uClient=uReq(my_url)
   
           page_html=uClient.read()
   
           page_soup = Soup(page_html,"html.parser")
   
           containers=page_soup.findAll("a", {"class":"_1UoZlX"})
   
   
   
   
   for container in containers:          phone_name        =  
   container.find("div",{"class":"_3wU53n"}).text
   
      try:
      phone_price =  container.find("div",{"class":"_1vC4OE _2rQ-NK"}).text
   
      except:
      phone_price           =  'No Data'
   

非常感谢您提供的所有帮助！

Answer 1

为 Excel 编写 .CSV 文件时，应使用 utf-8-sig 编码正确支持任何 Unicode 字符。如果仅使用 utf8 并错误地显示字符，Excel 将假定 Windows 上的本地化 ANSI 编码。

#!python3
import csv
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as Soup

filename = "data.csv"
with open(filename,'w',newline='',encoding='utf-8-sig') as f:
    w = csv.writer(f)
    headers = 'phone_name phone_price phone_rating number_of_ratings memory display camera battery processor Warrenty security OS'
    w.writerow(headers.split())

    for i in range(2):      # Number of pages minus one 
            my_url = 'https://www.flipkart.com/search?as=off&as-show=on&otracker=start&page={}&q=cell+phones&viewType=list'.format(i+1)
            print(my_url)
            uClient=uReq(my_url)
            page_html=uClient.read()
            page_soup = Soup(page_html,"html.parser")
            containers=page_soup.findAll("a", {"class":"_1UoZlX"})

    for container in containers:
        phone_name = container.find("div",{"class":"_3wU53n"}).text

        try:
            phone_price = container.find("div",{"class":"_1vC4OE _2rQ-NK"}).text
        except:
            phone_price = 'No Data'

        w.writerow([phone_name,phone_price])

输出：

phone_name,phone_price,phone_rating,number_of_ratings,memory,display,camera,battery,processor,Warrenty,security,OS
"Asus Zenfone 3 Laser (Gold, 32 GB)","₹9,999"
"Intex Aqua Style III (Champagne/Champ, 16 GB)","₹3,999"
"iVooMi i1s (Platinum Gold, 32 GB)","₹7,499"
"Xolo ERA 3X (Posh Black, 16 GB)","₹6,999"
"iVooMi Me1 (Sunshine Gold, 8 GB)","₹3,599"
"Panasonic Eluga A4 (Mocha Gold, 32 GB)","₹9,790"
Samsung Metro 313 Dual Sim,"₹2,025"
"Samsung Galaxy J3 Pro (Gold, 16 GB)","₹6,990"
Samsung Guru Music 2,"₹1,625"
"Panasonic Eluga A4 (Marine Blue, 32 GB)","₹9,640"
"Asus Zenfone 4 Selfie (Black, 32 GB)","₹9,999"
Swipe Elite 3- 4G with VoLTE,"₹3,999"
"Asus Zenfone Max (Black, 16 GB)","₹7,486"
Swipe Elite 3- 4G with VoLTE,"₹3,999"
"Swipe Elite Power (Space Grey, 16 GB)","₹5,499"
"Celkon Diamond Mega (Grey, 16 GB)","₹5,499"
"Asus Zenfone Max (Black, 32 GB)","₹7,999"
"Swipe Elite Power (Champagne Gold, 16 GB)","₹5,499"
"Asus Zenfone 4 Selfie (Gold, 32 GB)","₹9,999"
"Karbonn Aura (Champagne, 8 GB)","₹3,199"
"Infinix Note 4 (Ice Blue, 32 GB)","₹8,999"
"Infinix Note 4 (Milan Black, 32 GB)","₹8,999"
"Moto G5s Plus (Blush Gold, 64 GB)","₹15,990"
"Moto G5s Plus (Lunar Grey, 64 GB)","₹15,940"

Excel：