Python for cyber novel web crawler
Package in use
urllib, BeautifulSoup
urllib is a build-in package in Python and the most useful childpackage is .request.urlopen.
BeautifulSoup could be installed through Anaconda by yourself, and it could compel the .html webpage as an object.
Example
html = urlopen("http://www.shuhai.com/read/54351/1.html")
bsObj = BeautifulSoup(html)
chapter_content = bsObj.findAll("p")
for content in chapter_content:
print(content.get_text())
Extend
Use of the bsObj to check the construction of html body.
Use of .get_text() to return the text content in the object.
Use of .findAll()