不能让脚本以常规方式完成任务

Question

我使用 selenium 编写了一个脚本，在其中实现了多处理，采用了 this answer 的想法。该脚本运行良好，我在控制台中看到了所有结果。但是，当执行完成时，我在 IDE 底部看不到任何此类迹象，这表明该过程已完成。

以下图片取自python的默认IDE和sublime text。

import threading
import concurrent.futures
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

threadLocal = threading.local()

def create_browser():
    driver = getattr(threadLocal, 'driver', None)
    if driver is None:
        options = webdriver.ChromeOptions()
        options.add_argument("--headless")
        driver = webdriver.Chrome(options=options)   
        setattr(threadLocal, 'driver', driver)
    return driver

def get_links(link):
    driver = create_browser()
    driver.get(link)
    for elem in WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".summary .question-hyperlink"))):
        yield elem.get_attribute("href")

def get_title(url):
    driver = create_browser()
    driver.get(url)
    title = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"h1[itemprop='name'] > a.question-hyperlink"))).text
    return title

if __name__ == '__main__':
    base = "https://stackoverflow.com{}"
    URL = "https://stackoverflow.com/questions/tagged/web-scraping?tab=newest&page=1&pagesize=50"
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_url = {executor.submit(get_title, link): link for link in get_links(URL)}
        for item in concurrent.futures.as_completed(future_to_url):
            print(item.result())

执行完成后如何终止进程？

Answer 1

我认为唯一可能存在的问题是，由于您试图高效地为每个线程创建单个 selenium 驱动程序，因此您忽略了在所有提交的作业完成时处理“退出”所有驱动程序并且那些驱动程序进程，尤其是在 IDE 中运行时，很可能不会终止。我会做出以下改变：

1. 添加类Driver，它将创建驱动程序实例并将其存储在线程本地存储中，但也有一个析构函数，当线程本地存储被删除时，它将quit驱动程序：

class Driver:
    def __init__(self):
        options = webdriver.ChromeOptions()
        options.add_argument("--headless")
        self.driver = webdriver.Chrome(options=options)

    def __del__(self):
        self.driver.quit() # clean up driver when we are cleaned up
        #print('The driver has been "quitted".')

1. create_browser 现在变为：

def create_browser():
    the_driver = getattr(threadLocal, 'the_driver', None)
    if the_driver is None:
        the_driver = Driver()
        setattr(threadLocal, 'the_driver', the_driver)
    return the_driver.driver

1. 最后，在获得所有Future 结果后，添加以下行以删除线程本地存储并强制调用Driver 实例的析构函数（希望如此）：

del threadLocal
import gc
gc.collect() # a little extra insurance

更新

我应该补充一点，我没有任何问题运行到完成（我在调用gc.colleect() 后打印“完成”）。但是，在我的 Windows 桌面上，我确实看到记录了以下消息：

[1024/092605.493:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092605.562:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092605.579:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092605.592:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092605.634:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
...
[1024/092617.865:INFO:CONSOLE(118)] "The deviceorientation events are blocked by feature policy. See https://github.com/WICG/feature-policy/blob/master/features.md#sensor-features", source: https://z.moatads.com/chaseusdcm562975626226/moatad.js (118)
[1024/092617.949:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.015:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.456:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.479:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.570:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.738:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.849:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)
[1024/092618.928:INFO:CONSOLE(0)] "Error with Feature-Policy header: Unrecognized feature: 'speaker'.", source:  (0)

这是我的输出：

ImportXML XPath issue using Google Sheets on a web scraping query
Scrapy meta or cb_kwargs not passing properly between multiple methods
How to seperate a list into table formate using python
How can I extract a table from wikipedia using Beautiful soup
Load a series of payload requests and perform pagination for each one of them
Pandas read_html not reading text properly
Getting text nested text in non-static webpage with httr in R [closed]
Scraping data with duplicate column headers [closed]
I keep getting [ TypeError: 'function' object is not iterable ] every time I try to iterate over the result of my function which returns an iterable [closed]
selnium and beutifulsoup scrapper very inconsistent
Web scraping the required content from a url link in R
Web-scrapping pop-up info generated by hovering over canvas element (Python/Selenium)
Daily leaderboard or price tracking data
Scrape PDF embedded in .php page
Beautiful Soup returning only the last URL of a txt file
Having trouble in scraping table data using beautiful soup
Authentication - Security Window - Rvest R
How can I read an iframe content inside another iframe using Puppeteer?
Xamarin.Forms: is there a way to update the style of web page displayed in a WebView with scraping?
Counter not working in for(i=0; ++i) loop node.js
Python: selenium can't read an specific table
Scraped json data want to output CSV file
Unable to scrape “shopee.com.my” top selling products page
How to click a menu item from mobile based website in selenium Python?
Does selenium in standalone mode has limitation for maximum number of sessions can be present at a time?
Error while capturing full website screen shot
How to retrieve SharePoint webpage code(html) or Scrape a sharepoint webpage?
API web data capture
Selenium select disappearing webelement
Python SQlite Query to select recently added data in the table
Webscraping with varying page numbers
How to extract contents between div tags with rvest and then bind rows
Why is the previous request aborting if I send a new request to the flask server? [closed]
Does anyone know how to click() on an href within data-bind using selenium? [closed]
Web Scraping on login sites with Python
How do I render image, title and link to template from views using one 'for loop'
Regex on List Comprehension Not Producing List But List of Lists Instead [duplicate]
How to get all tr id by using python selenium?
Scrapy - TypeError: can only concatenate str (not “list”) to str
I need to save scraped urls to a csv file in URI format. file won't write to csv
Scrapy keeps giving me the errot AttributeError: 'str' object has no attribute 'text'
How to scrape the different content with the same html attributes and values?
I can not scrape Google news with Beautiful soup. I am getting the error:TypeError: 'NoneType' object is not callable [closed]
selenium while loop error on load more button
Python- Selenium/BeautifulSoup PDF & Table scraper
Crawling all page with scrapy and FormRequest
Web Scrape COVID19 Data from Download Button in R
How do 3rd party app stores know when a new app is added to Google Play?
Scraping hidden leaderboard data from site
Cannot access a table shown in a Tableau Public Dashboard

更新 2

如果你在等待结果，你可以考虑使用超时：

if __name__ == '__main__':
    base = "https://stackoverflow.com{}"
    URL = "https://stackoverflow.com/questions/tagged/web-scraping?tab=newest&page=1&pagesize=50"
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_url = {executor.submit(get_title, link): link for link in get_links(URL)}
        for future in future_to_url:
            try:
                print(future.result(30))
            except concurrent.futures.TimeoutError as e:
                url = future_to_url[future]
                print('TimeoutError for URL', url)
    del threadLocal
    import gc
    gc.collect() # a little extra insurance
    print('done')

请注意，我不再使用as_completed，因为我希望能够指定超时值而不是无限期地等待结果。这里我指定了一个 30 秒的值，这应该足以让线程初始化驱动程序并获得第一个结果。如果您实际上是在等待结果，这应该打印一条 TimeoutError 消息并继续，

Answer 2

看起来进程终止正常，但如果您想确保进程已终止，只需 import sys 并在末尾包含 sys.exit。