为什么我的链接没有写在我的文件中答案

【问题标题】：why my links not writing in my file为什么我的链接没有写在我的文件中
【发布时间】：2016-10-04 03:42:31
【问题描述】：

import urllib
from bs4 import BeautifulSoup
import requests
import readability
import time
import http.client

seed_url = "https://en.wikipedia.org/wiki/Sustainable_energy"
root_url = "https://en.wikipedia.org"
max_limit=5
#file = open("file_crawled.txt", "w")

def get_urls(seed_url):
r = requests.get(seed_url)
soup = BeautifulSoup(r.content,"html.parser")
links = soup.findAll('a', href=True)
valid_links=[]
 for links in links:
   if 'wiki' in links['href'] and '.' not in links['href'] and ':' not in links['href'] and '#' not in links['href']:
     valid_links.append(root_url + links['href'])
 return valid_links


visited=[]
def crawl_dfs(seed_url, max_depth):
depth=1
file1 = open("file_crawled.txt", "w+")
visited.append(root_url)
   if depth<=max_depth:
      children=get_urls(seed_url)
      for child in children:
           if child not in visited:          
                file1.write(child)                                    
                time.sleep(1)
                visited.append(child)
                crawl_dfs(child,max_depth-1)
   file1.close()

crawl_dfs(seed_url,max_limit)

dfs爬取使用python 3.6 帮我写代码，请纠正我错的地方，我抓取的链接没有写入我的名为 file1 的文件。我不知道为什么我最终尝试了所有方法

【问题讨论】：

你到底尝试了什么？
在crawl_dfs()（打开文件）内部，你调用crawl_dfs()，它再次打开同一个文件（但你还没有关闭它）-很奇怪-也许在你调用@987654324之前打开它@第一次。
是的，你一直以'w+'模式打开文件，所以每次进入 crawl_dfs() 时它都会清除 .txt

标签： python python-3.x web web-crawler depth-first-search

【解决方案1】：

您只需打开和关闭文件一次 - 在第一个 crawl_dfs() 之前打开并在第一个 crawl_dfs() 之后关闭

测试：

import urllib
from bs4 import BeautifulSoup
import requests
#import readability
import time
import http.client

# --- functions ---

def get_urls(seed_url):
    r = requests.get(seed_url)
    soup = BeautifulSoup(r.content,"html.parser")
    links = soup.findAll('a', href=True)
    valid_links = []
    for links in links:
        if 'wiki' in links['href'] and '.' not in links['href'] and ':' not in links['href'] and '#' not in links['href']:
            valid_links.append(root_url + links['href'])
    return valid_links


def crawl_dfs(seed_url, max_depth, file_out):
    if max_depth >= 1:
       children = get_urls(seed_url)
       for child in children:
           if child not in visited:          
               file_out.write(child + "\n")                                    
               #time.sleep(1)
               visited.append(child)
               crawl_dfs(child, max_depth-1, file_out)

# --- main ---

seed_url = "https://en.wikipedia.org/wiki/Sustainable_energy"
root_url = "https://en.wikipedia.org"
max_limit = 1

visited=[root_url]

file1 = open("file_crawled.txt", "w+")

crawl_dfs(seed_url, max_limit, file1)

file1.close()

【讨论】：