这个封闭的数据库怎么打开？

Question

这是我正在运行的代码：

import requests 
import records 
from bs4 import BeautifulSoup 
from urllib.parse import urljoin 
from sqlalchemy.exc import IntegrityError

db = records.Database('sqlite:///crawler_database.db')

db.query('''CREATE TABLE IF NOT EXISTS links (
            url text PRIMARY KEY,            
            created_at datetime,            
            visited_at datetime NULL)''') 
db.query('''CREATE TABLE IF NOT EXISTS numbers (url text, number integer,            
            PRIMARY KEY (url, number))''')

def store_link(url):    
    try:        
        db.query('''INSERT INTO links (url, created_at)                    
                    VALUES (:url, CURRENT_TIMESTAMP)''', url=url)

    except IntegrityError as ie:       
        # This link already exists, do nothing        
        pass

def store_number(url, number):    
    try:        
        db.query('''INSERT INTO numbers (url, number)                    
            VALUES (:url, :number)''', url=url, number=number)    
    except IntegrityError as ie:        
        # This number already exists, do nothing        
        pass

def mark_visited(url):    
    db.query('''UPDATE links SET visited_at=CURRENT_TIMESTAMP                
                WHERE url=:url''', url=url)

def get_random_unvisited_link():    
    link = db.query('''SELECT * FROM links                       
                       WHERE visited_at IS NULL                        
                       ORDER BY RANDOM() LIMIT 1''').first()    
    return None if link is None else link.url

def visit(url):    
    html = requests.get(url).text    
    html_soup = BeautifulSoup(html, 'html.parser')    
    new_links = []    
    for td in html_soup.find_all("td"):        
    store_number(url, int(td.text.strip()))    
    for link in html_soup.find_all("a"):        
        link_url = link.get('href')        
        if link_url is None:            
            continue        
        full_url = urljoin(url, link_url)        
        new_links.append(full_url)    
    return new_links

store_link('http://www.webscrapingfordatascience.com/crawler/') 
url_to_visit = get_random_unvisited_link() 
while url_to_visit is not None:    
    print('Now visiting:', url_to_visit)    
    new_links = visit(url_to_visit)    
    print(len(new_links), 'new link(s) found')    
    for link in new_links:        
        store_link(link)    
    mark_visited(url_to_visit)    
    url_to_visit = get_random_unvisited_link()

这是我的错误：

ProgrammingError: (sqlite3.ProgrammingError) 无法对已关闭的数据库进行操作。

回调错误将我指向这一行，url_to_visit = get_random_unvisited_link()。

我不明白它为什么关闭或发生了什么。谁愿意帮我解决这个问题？

这是完整的回溯：

ERROR:sqlalchemy.pool.impl.NullPool:Error closing cursor
Traceback (most recent call last):
  File "C:\Users\LENOVO\anaconda3\lib\site-packages\sqlalchemy\engine\result.py", line 1324, in fetchone
    row = self._fetchone_impl()
  File "C:\Users\LENOVO\anaconda3\lib\site-packages\sqlalchemy\engine\result.py", line 1204, in _fetchone_impl
    return self.cursor.fetchone()
sqlite3.ProgrammingError: Cannot operate on a closed database.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LENOVO\anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1339, in _safe_close_cursor
    cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.

Answer 1

在get_random_unvisited_link 返回的RecordCollection 上调用.first() 时会出现问题。我不确定，但我怀疑records 包中存在一个错误，由于cursor 被删除，行作为生成器RecordCollection 的返回出现故障。我相信这与GitHub page 上报告的问题相同。

详细说明，将您的get_random_unvisited_link 修改为：

def get_random_unvisited_link():                                                
    link = db.query('''SELECT * FROM links                                      
                       WHERE visited_at IS NULL                                 
                       ORDER BY RANDOM() LIMIT 1''')                            
    print(link)                                                                 
    for row in link:                                                            
        print(row)

输出：

<RecordCollection size=0 pending=True>
Error closing cursor
...Traceback from question...

这表明RecordCollection 仍有待处理的项目，但尝试访问这些项目会导致消息

错误关闭光标

接着是你看到的回溯。

有一个解决方法，使用fetchall 参数到query：

def get_random_unvisited_link():                                                
    rows = db.query('''SELECT * FROM links                                      
                       WHERE visited_at IS NULL                                 
                       ORDER BY RANDOM() LIMIT 1''', fetchall=True)             
    print(rows)                                                                 
    link = rows.first()                                                         
    print(link)                                                                 
    return None if link is None else link.url

打印输出：

<RecordCollection size=1 pending=False>
<Record {"url": "http://www.webscrapingfordatascience.com/crawler/", "created_at": "2020-04-14 06:57:45", "visited_at": null}>

现在，您可以看到我们已将查询结果加载到RecordCollection，因为它的size 是1，而pending 标志是False。