用于抓取结果的脚本似乎不起作用？

Question

这是我发现的一个脚本，用于使用 grad cafe 网站抓取不同课程的研究生录取结果。但是，当我运行它以查找“政治科学”的结果时，它指出我有以下错误

Traceback (most recent call last):
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 57, in <module>
    data = get_data()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 52, in get_data
    pages = get_pages()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 45, in get_pages
    n = find_n_pages()
  File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 41, in find_n_pages
    reg = re.search('over\s([\d]*)\spages',html)
  File "C:\Users\lakna\AppData\Local\Programs\Python\Python36-32\lib\re.py", line 182, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object

我应该如何解决这个问题？以下是我使用的代码

def get_page(i=0, keyword="Political Science"):
    time.sleep(10)
    if i==0:
        #To change subjects, you want to change the keyword to say biostatistics,
        #test it by searching on the site to make sure you get what you want.
        url = "http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&o=&pp=250"
    else:
        url="http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&pp=250&o=&p="+str(i)
    response = urlopen(url)
    html = response.read().decode('utf-8')
    return
def find_n_pages():
    html = get_page()
    reg = re.search('over\s([\d]*)\spages',html)
    return int(reg.groups()[0])

def get_pages():
    n = find_n_pages()
    print ("Getting",n,"pages.")
    pages = [get_page(i) for i in range(1,n+1)]
    return pages

def get_data():
    data=[]
    pages = get_pages()
    for page in pages:
        data=get_data_from_page(page,data)
    return data

Answer 1

您的 get_page 函数没有返回 html，而是返回 None。

def get_page(i=0, keyword="Political Science"):
    ...
    html = response.read().decode('utf-8')
    return  # this is equivalent to return None (or not having this line at all)

应改为：

def get_page(i=0, keyword="Political Science"):
    ...
    return response.read().decode('utf-8')

因此错误：

In [11]: re.search("", None)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-1858f7517272> in <module>()
----> 1 re.search("", None)

/Users/andy/.miniconda3/lib/python3.6/re.py in search(pattern, string, flags)
    180     """Scan through string looking for a match to the pattern, returning
    181     a match object, or None if no match was found."""
--> 182     return _compile(pattern, flags).search(string)
    183
    184 def sub(pattern, repl, string, count=0, flags=0):

TypeError: expected string or bytes-like object