Python：将先前抓取的数据保存在数据库中（已创建）

Question

我已经正确地用 Selenium 进行了刮擦，结果打印得很完美。我不知道如何将它们保存在数据库中，我已经使用 Sqlite 创建了表和字段。一般来说，我知道如何将数据保存在数据库中，但我不知道如何保存抓取数据。我不知道如何编写代码来保存抓取数据。

现在我只想将抓取的数据添加到数据库中并保存。数据库表名为 TableExample1。这些字段是：ID、Product_NameDB、Product_DescriptionDB、VendorDB、PriceDB。

我有两个问题

1. 保存在数据库中的文本如下：[
2. 只有一 (1) 行添加到数据库中。相反，应该添加数十或数百个（每个都像这样：Product_Name、Product_Description、Vendor、Price），具体取决于抓取的数据。

爬取打印数据的Python代码如下：

#Name of the scraped data 
Product_Name = (driver.find_element_by_class_name ("tablet-desktop-only"). Text)
Product_Description = (driver.find_element_by_class_name ("h-text-left"). Text)
Vendor = (driver.find_element_by_class_name ("in-vendor"). Text)
Price = (driver.find_element_by_class_name ("h-text-center"). Text)

for seq in Product_Name + Product_Description + Vendor + Price:
    print(seq.text)

更新 N.1 当前代码，遵循"Jeremy Kahan"对此问题的回答
在THIS 中（不同的是，我要求在控制台中打印所有抓取的结果，而不仅仅是 1）

这是当前最稳定且功能最强大的代码。抓取工作正常，但是您在另一个问题中建议的 Num_Groups 和 for i in range (Num_Groups) 只打印一个组，而不是全部。

我仍然有同样的 2 个问题

1. 保存在数据库中的文本如下： [

2. 只有一 (1) 行添加到数据库中。相反，应该添加数十个或数百个（每个都像这样：Product_Name、Product_Description、Vendor、Price），具体取决于抓取的数据。

    import sqlite3
    from datetime import datetime 
   
    #SCRAPING
    Product_Name=driver.find_elements_by_class_name("tablet-desktop-only")
    Product_Description=driver.find_elements_by_class_name("h-text-left")
    Vendor=driver.find_elements_by_class_name("in-match")
    Price=driver.find_elements_by_class_name("h-text-center")
   
   
    # How do I print the other data always with the same html name?
    # This is one row data. This is the code you wrote me in the  
    # other question. Print only one group. What to write to print 
    # all groups?  
    Num_Groups = min(len(Product_Name),len(Product_Description),len(Vendor), len(Price))
   
    for i in range(Num_Groups):    
        print(Product_Name[i].text)
        print(Product_Description[i].text)
        print(Vendor[i].text)
        print(Price[i].text)
   
   
    #INSERT DATA IN DATABASE
    con = sqlite3.connect('/home/mypc/Desktop/aaaaa/Database.db')
    cursor = con.cursor()
   
    ID=datetime.today().strftime('%Y%m%d%H%M%S')
    Values =   f"VALUES({ID},'{Product_Name}','{Product_Description}','{Vendor}','{Price}')"
    sqlite_insert_query = 'INSERT INTO TableExample1 (ID, Product_Name, Product_Description, Vendor, Price)' + Values
   
    count = cursor.execute(sqlite_insert_query)
    con.commit()
    print("Record inserted successfully ", cursor.rowcount)
    cursor.close()
   

2°上传（最终？） 问题：找到12个组，成功插入1条记录，一共添加了1条记录如何在数据库中插入12个谱表？

Num_Groups = min(len(Product_Name),len(Product_Description),len(Vendor), len(Price))

records_added = 0

 for i in range(Num_Groups):    
     print(Product_Name[i].text)
     print(Product_Description[i].text)
     print(Vendor[i].text)
     print(Price[i].text)


con = sqlite3.connect('/home/mypc/Scrivania/aaaa/Database.db')
cursor = con.cursor()

Values = f" VALUES ('{Product_Name[i].text}','{Product_Description[i].text}','{Vendor[i].text}','{Price[i].text}')"
sqlite_insert_query = 'INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price)' + Values

print("Qui, all'interno del ciclo, eseguiresti" + sqlite_insert_query)

count = cursor.execute(sqlite_insert_query)
con.commit()
print("Record inserted successfully ", cursor.rowcount)
records_added = records_added + 1
cursor.close()

print("")
print(f'Added a total of {records_added} records')
print(f"found {Num_Groups} groups")  # should be more than 1

Answer 1

好的。所以你告诉我你很擅长连接到数据库、获取游标、使用游标和连接执行 sql 以及提交更改和关闭游标。您需要的是要执行的 sql 字符串。

Values = f"VALUES('{Product_Name}','{Product_Description}','{Vendor}','{Price}')"
sqlite_insert_query = 'INSERT INTO TableExample1(Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB) ' + Values

那么你应该可以，假设cursor是光标设置：

count = cursor.execute(sqlite_insert_query)

我假设您进行了设置，因此 ID 是一个唯一键，如果您不指定它，则会生成它。

编辑：所以听起来数据库在 ID 上苦苦挣扎，并且没有制作一个以上的新 ID。假设您的 ID 只需要是连续的（并且只有您在使用它并且抓取时间超过一秒），您可以自己处理 ID，如下所示。在from datetime import datetime 之前

ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID},'{Product_Name}','{Product_Description}','{Vendor}','{Price}')"
sqlite_insert_query = 'INSERT INTO TableExample1(ID, Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB) ' + Values

此外，您提到在 Product_Name 中获取描述元素的文本，而不仅仅是 Product_Name（对于其他字段也是如此）。你需要：

Product_Name = driver.find_element_by_class_name("tablet-desktop-only").text

（或将(Product_Name.text) 放在我的 f 字符串中，但这似乎令人困惑）

第二次编辑： @Prophet 在https://stackoverflow.com/questions/68110293/problems-saving-scraped-data-to-database-python/68111109?noredirect=1#comment120399103_68111109 想说的内容如下。 当你说：

`Product_Name = driver.find_element_by_class_name("tablet-desktop-only").text`

Product_Name 是一个字符串，你的 for 循环在 Product_Name + Product_Description + Vendor + Price 正在循环一个连接的字符串，然后 seq 具有字符串的字符，一次取一个。然后seq.text 失败，正如您所经历的那样。这就是为什么我将那里的print 命令注释掉，并在稍后添加一个以打印Values 字符串。这是一种应该行之有效的方法。

如果你按照原来的方式留下东西

`Product_Name = driver.find_element_by_class_name("tablet-desktop-only")

Product_Name 是一个元素，当它合并到 Values 字符串中时，会转换为该元素的字符串表示形式，这就是为什么您会在数据库中看到所有不相关的文本的原因。我理解你离开它以便 for 循环可以工作，但是你应该这样做：

ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID},'{Product_Name.text}','{Product_Description.text}','{Vendor.text}','{Price.text}')"

或

Product_Name_Text = Product_Name.Text
Product_Description_Text = Product_Description.Text
Vendor_Text = Vendor.Text
Price_Text = Price.Text

然后

Values = f"VALUES({ID},'{Product_Name_Text}','{Product_Description_Text}','{Vendor_Text}','{Price_Text}')"

或

ID=datetime.today().strftime('%Y%m%d%H%M%S')
Values = f"VALUES({ID}"

for seq in Product_Name + Product_Description + Vendor + Price:
    print(seq.text)
    Values = Values + "," + "'" + seq.text + "'"
Values = Values + ")"

我肯定会推荐，至少在调试方面，打印Values 或sqlite_insert_query。如果您与我们分享这些结果，如果仍然无法正常工作，我们或许可以提供帮助。

在上述选项中，如果最终证明这不是问题，您可以省略我分配 ID。

我在您的代码中没有看到针对不同元素组的任何循环，因此我不确定您对插入多个新条目的期望。

第三次（决赛？）编辑： 你有2个问题。您在数据库中获得了额外的内容是因为您将每个元素的描述放入 SQL 的 Values 部分，而不是文本。我已经说明了如何通过将文本格式化为 Values 字符串来解决这个问题。 您遇到的第二个问题是，当您说 find_elements 时，您的定位器只能找到一件事（我无法调试您的定位器，因为我不知道该页面）。但是这里的代码应该告诉你匹配了多少东西。为了验证这个假设，我在下面编写了自己的版本，它可以（如果页面是永久性的）从亚马逊获取化妆品数据。细节并不重要，但代码应该说明必须发生的事情。另外，我相信我之前在不必要的情况下使用 ID 所做的事情，数据库会处理它。 由于我实际上不是数据库，因此我将其注释掉。您将取消注释。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

# launch and go to site, yours will vary
driver = webdriver.Firefox(executable_path='/usr/bin/geckodriver')
driver.maximize_window()
wait = WebDriverWait(driver, 15)
driver.get("https://www.amazon.com/b?node=18505451011&pd_rd_w=gVvMZ&pf_rd_p=b6363b44-58dd-4354-979f-1446a1c45f7a&pf_rd_r=5FYAS41AJR7GPQ5Q74J4&pd_rd_r=9bfd1639-2256-4c17-928c-99cf03e00d63&pd_rd_wg=UPUHV")
wait.until(EC.presence_of_element_located((By.LINK_TEXT, "See all results")))

# SCRAPING
# you will want to change back to your locators
Product_Name = driver.find_elements_by_class_name("apb-line-clamp-3")
# actually number of reviewers, picks up see all at the bottom, too, but it's ok because of min
Product_Description = driver.find_elements_by_class_name("a-color-link")
Vendor = driver.find_elements_by_class_name(
    "apb-browse-searchresults-product-byline")
Price = driver.find_elements_by_class_name("a-price")  # prices

# if the locators work well, these match multiple groups of products not just one
Num_Groups = min(len(Product_Name), len(
    Product_Description), len(Vendor), len(Price))
print(f"found {Num_Groups} groups")  # should be more than 1

# I have commented out the database code, because I am not actually using one
#con = sqlite3.connect('/home/mypc/Desktop/aaaaa/Database.db')
# connect to database outside the loop, not for each item
# I removed the code about generating ID's, which the database should handle
records_added = 0
for i in range(Num_Groups):  # should cause i to count from 0 up to and including Num_Groups-1
    print("")  # skip a line between stuff
    print(Product_Name[i].text)
    print(Product_Description[i].text)
    print(Vendor[i].text)
    print(Price[i].text)
    # note below I need to format the .text into the values string, not the text description of the element
    Values = f" VALUES ('{Product_Name[i].text}','{Product_Description[i].text}','{Vendor[i].text}','{Price[i].text}')"
    sqlite_insert_query = 'INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price)' + Values
    print("Here, inside the loop,  you would execute " + sqlite_insert_query)
    #cursor = con.cursor()
    #count = cursor.execute(sqlite_insert_query)
    # con.commit()
    #print("Record inserted successfully ", cursor.rowcount)
    records_added = records_added + 1
    # cursor.close()
print("")
print(f'Added a total of {records_added} records')

输出是：

found 12 groups

Crest 3D White Professional Effects Whitestrips 20 Treatments + Crest 3D White 1 Hour Express Whitestrips 2 Treatments - Teeth Whitening Kit
46,020
by Crest
$47
88
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Crest 3D White Professional Effects Whitestrips 20 Treatments + Crest 3D White 1 Hour Express Whitestrips 2 Treatments - Teeth Whitening Kit','46,020','by Crest','$47
88')

REVLON One-Step Hair Dryer And Volumizer Hot Air Brush, Black, Packaging May Vary
274,196
by REVLON
$41
99
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('REVLON One-Step Hair Dryer And Volumizer Hot Air Brush, Black, Packaging May Vary','274,196','by REVLON','$41
99')

Waterpik WP-660 Water Flosser Electric Dental Countertop Professional Oral Irrigator For Teeth, Aquarius, White
72,786
by Waterpik
$59.99
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Waterpik WP-660 Water Flosser Electric Dental Countertop Professional Oral Irrigator For Teeth, Aquarius, White','72,786','by Waterpik','$59.99')

Schick Hydro Silk Touch-Up Multipurpose Exfoliating Dermaplaning Tool, Eyebrow Razor, and Facial Razor with Precision Cover, 3 Count (Packaging May Vary)
113,269
by Schick Hydro Silk
$68
27
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Schick Hydro Silk Touch-Up Multipurpose Exfoliating Dermaplaning Tool, Eyebrow Razor, and Facial Razor with Precision Cover, 3 Count (Packaging May Vary)','113,269','by Schick Hydro Silk','$68
27')

Neutrogena Makeup Remover Cleansing Face Wipes, Daily Cleansing Facial Towelettes to Remove Waterproof Makeup and Mascara, Alcohol-Free, Value Twin Pack, 25 Count, 2 Pack
62,881
by Neutrogena
$4
99
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Neutrogena Makeup Remover Cleansing Face Wipes, Daily Cleansing Facial Towelettes to Remove Waterproof Makeup and Mascara, Alcohol-Free, Value Twin Pack, 25 Count, 2 Pack','62,881','by Neutrogena','$4
99')

Gillette Fusion Power Men's Razor Blades - 8 Refills
32,435
by Gillette
$6.99
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Gillette Fusion Power Men's Razor Blades - 8 Refills','32,435','by Gillette','$6.99')

Softsoap Moisturizing Liquid Hand Soap, Soothing Clean Aloe Vera - 7.5 Fluid Ounces (6 Pack)
47,238
by Softsoap
$8
12
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Softsoap Moisturizing Liquid Hand Soap, Soothing Clean Aloe Vera - 7.5 Fluid Ounces (6 Pack)','47,238','by Softsoap','$8
12')

Neutrogena Lightweight Body Oil for Dry Skin, Sheer Body Moisturizer in Light Sesame Formula, 16 fl. oz
13,011
by Neutrogena
$11.96
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Neutrogena Lightweight Body Oil for Dry Skin, Sheer Body Moisturizer in Light Sesame Formula, 16 fl. oz','13,011','by Neutrogena','$11.96')

Crest 3D White Whitestrips with Light, Teeth Whitening Strips Kit, 10 Treatments, 20 Individual Strips (Packaging May Vary)
9,172
by Crest
$23
91
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Crest 3D White Whitestrips with Light, Teeth Whitening Strips Kit, 10 Treatments, 20 Individual Strips (Packaging May Vary)','9,172','by Crest','$23
91')

Neutrogena Hydro Boost Hyaluronic Acid Hydrating Water Gel Daily Face Moisturizer for Dry Skin, Oil-Free, Non-Comedogenic Face Lotion, 1.7 fl. oz
51,417
by Neutrogena
$32.51
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Neutrogena Hydro Boost Hyaluronic Acid Hydrating Water Gel Daily Face Moisturizer for Dry Skin, Oil-Free, Non-Comedogenic Face Lotion, 1.7 fl. oz','51,417','by Neutrogena','$32.51')

CHI 44 Iron Guard Thermal Protection Spray 8 Fl Oz
17,660
by CHI
$5
33
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('CHI 44 Iron Guard Thermal Protection Spray 8 Fl Oz','17,660','by CHI','$5
33')

Crest 3D White Toothpaste Radiant Mint (3 Count of 4.1 oz Tubes), 12.3 oz (Packaging May Vary)
40,354
by Crest
$13.99
Here, inside the loop,  you would execute INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price) VALUES ('Crest 3D White Toothpaste Radiant Mint (3 Count of 4.1 oz Tubes), 12.3 oz (Packaging May Vary)','40,354','by Crest','$13.99')

added a total of 12 records

修订版 这是您的代码需要的样子。

Num_Groups = min(len(Product_Name),len(Product_Description),len(Vendor), len(Price))
con = sqlite3.connect('/home/mypc/Scrivania/aaaa/Database.db')
cursor = con.cursor()
records_added = 0

for i in range(Num_Groups):    
     print(Product_Name[i].text)
     print(Product_Description[i].text)
     print(Vendor[i].text)
     print(Price[i].text)
     Values = f" VALUES ('{Product_Name[i].text}','{Product_Description[i].text}','{Vendor[i].text}','{Price[i].text}')"
     sqlite_insert_query = 'INSERT INTO TableExample1 (Product_Name, Product_Description, Vendor, Price)' + Values
     print("Qui, all'interno del ciclo, eseguiresti" + sqlite_insert_query)
     count = cursor.execute(sqlite_insert_query)
     con.commit()
     print("Record inserted successfully ", cursor.rowcount)
     records_added = records_added + 1
cursor.close()

for: 下缩进的代码体针对 i 的每个值（在您提到的情况下，从 0 到 11）执行。现在，因为你的数据库插入在这个循环之外，它只执行一次。

真的是最后一次编辑： 我们将使用参数化查询让数据库引擎处理值中的撇号（以及我可能遗漏的任何其他内容）。根据我的阅读，无论如何这都是一种更安全的方法来帮助防止 SQL 注入攻击。

将Values= 行（保持相同的缩进）更改为

Values = (Product_Name[i].text, Product_Description[i].text, Vendor[i].text, Price[i].text)

将sqlite_insert_query = 行更改为（保持与现在相同的缩进）：

sqlite_insert_query = 'INSERT INTO TableExample1 (Product_NameDB, Product_DescriptionDB, VendorDB, PriceDB) VALUES (?, ?, ?, ?);'

将count=cursor.execute 行改为说

count = cursor.execute(sqlite_insert_query, Values)

这应该会更好，更安全。如果可行，您可以进行一些可选的清理工作。例如，您可以在循环之外设置 sqlite_insert_query，返回连接到数据库并初始化变量的位置。您也可以停止打印 sqlite_insert_query 并改为打印值（或者不打印任何内容，因为您之前有 4 行打印值）。