【发布时间】：2014-01-01 15:47:27
【问题描述】：
我在打开带有 Python 请求包的页面时遇到了一些问题。该页面在浏览器中可以正常打开，但在尝试获取该站点时程序会挂起。
import bs4, requests
link = "http://s6.mediastreaming.it:8080/"
r = requests.get(link)
data=r.text
soup = bs4.BeautifulSoup(data)
soup.prettify()
print soup
任何帮助理解为什么这个网站挂起将不胜感激。该代码以http://google.com 作为链接运行良好。
编辑：添加信息以提供帮助。这是一个流媒体音乐的网站。我只想抓取页面中显示当前歌曲的部分。就这些。但也许该页面是音乐流媒体网站的事实导致了问题？我只想要源的文本。没有别的了。
编辑 2：尝试以下操作以查看我是否确实获得了流而不是我想要的页面。
import bs4, requests
link = "http://s6.mediastreaming.it:8080/"
r = requests.get(link,stream=True)
filename = "testfile.txt"
with open(filename,'wb') as fd:
for chunk in r.iter_content(100):
fd.write(chunk)
这是我得到的输出：
ICY 200 OK
icy-notice1:<BR>This stream requires <a href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2:SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name:Pig Radio - The Best Electronic & Indie Pop/Rock 24/7
icy-genre:Eclectic
icy-url:http://www.pigradio.com
content-type:audio/mpeg
icy-pub:1
icy-br:128
&Oç)goiYQŠ < 6Ê !‡À¡ö³ ‡/OFÌ)8…¨ÐU!ðiÁP¡¢ãÅ..........
当我在浏览器中打开页面并查看源代码时，我得到以下信息：
<HTML><HEAD><meta http-equiv="Content-Language" content="en-us"><meta http-equiv="Content-Type" content="text/html; charset=windows-1252"><meta http-equiv="Pragma" content="no-cache"><meta http-equiv="Expires" content="Mon, 01 Jan 1990 12:00:00 GMT"><title>SHOUTcast Administrator</title><style type="text/css"><!--a:link {color: blue; font-family:Arial, Helvetica; font-size:9pt;}a:visited {color: blue; font-family:Arial, Helvetica; font-size:9pt;}a:hover {color: red; font-family:Arial, Helvetica; font-size:9pt; }.default {color: White; font-family:Arial, Helvetica; font-size:9pt; font-weight: normal}.ST {color: White; font-family:Arial, Helvetica; font-size:8pt; font-weight: normal}.logoText {color: red; font-family: Arial Black, Helvetica, sans-serif; font-size: 25pt; font-weight: normal; letter-spacing : -2.5px;}.flagText {color: blue; font-family: webdings; font-size: 36pt; font-weight: normal; }.ltv {color: blue; font-family: Arial, Helvetica, sans-serif; font-size: 9pt; font-weight: normal;}.tnl {color: black; font-family: Arial, Helvetica, sans-serif; font-size: 10pt; font-weight: bold; text-decoration: none;}--></style></HEAD><BODY topmargin=0 leftmargin=0 marginheight=0 marginwidth=0 bgcolor=#000000 text=#EEEEEE link=#001155 vlink=#001155 alink=#FF0000><font class=default><table width=100% border=0 cellpadding=0 cellspacing=0><tr><td height=50><font class=flagText>U</font><font class=logoText> SHOUTcast D.N.A.S. Status</font></td></tr><tr><td height=14 align=right><font class=ltv><a id=ltv href="http://www.shoutcast.com/">SHOUTcast Server Version 1.9.8/Linux</a></font></td></tr><tr><td bgcolor=#DDDDDD height=20 align=center><table width=100% border=0 cellpadding=0 cellspacing=0><tr><td align=center><font class=tnl><a id=tnl href="index.html">Status</a></font></td><td align=center><font class=tnl> | </font></td><td align=center><font class=tnl><a id=tnl href="played.html">Song History</a></font></td><td align=center><font class=tnl> | </font></td><td align=center><font class=tnl><a id=tnl href="listen.pls">Listen</a></font></td><td align=center><font class=tnl> | </font></td><td align=center><font class=tnl><a id=tnl href="home.html">Stream URL</a></font></td><td align=center><font class=tnl> | </font></td><td align=center><font class=tnl><a id=tnl href="admin.cgi">Admin Login</a></font></td></tr></table></td></tr></table><br><table cellpadding=5 cellspacing=0 border=0 width=100%><tr><td bgcolor=#000025 colspan=2 align=center><font class=ST>Current Stream Information</font></td></tr></table><table cellpadding=2 cellspacing=0 border=0 align=center><tr><td width=100 nowrap><font class=default>Server Status: </font></td><td><font class=default><b>Server is currently up and public.</b></td></tr><tr><td width=100 nowrap><font class=default>Stream Status: </font></td><td><font class=default><b>Stream is up at 128 kbps with <B>45 of 300 listeners (45 unique)</b></b></td></tr><tr><td width=100 nowrap><font class=default>Listener Peak: </font></td><td><font class=default><b>200</b></td></tr><tr><td width=100 nowrap><font class=default>Average Listen Time: </font></td><td><font class=default><b>7h 13m 12s</b></td></tr><tr><td width=100 nowrap><font class=default>Stream Title: </font></td><td><font class=default><b>Pig Radio - The Best Electronic & Indie Pop/Rock 24/7</b></td></tr><tr><td width=100 nowrap><font class=default>Content Type: </font></td><td><font class=default><b>audio/mpeg</b></td></tr><tr><td width=100 nowrap><font class=default>Stream Genre: </font></td><td><font class=default><b>Eclectic</b></td></tr><tr><td width=100 nowrap><font class=default>Stream URL: </font></td><td><font class=default><b><a href="http://www.pigradio.com">http://www.pigradio.com</a></b></td></tr><tr><td width=100 nowrap><font class=default>Stream AIM: </font></td><td><font class=default><b><a href="aim:goim?screenname=N/A">N/A</a></b></td></tr><tr><td width=100 nowrap><font class=default>Stream IRC: </font></td><td><font class=default><b><a href="http://www.shoutcast.com/chat.phtml?dc=N%2FA">N/A</a></b></td></tr><tr><td width=100 nowrap><font class=default>**Current Song: </font></td><td><font class=default><b>Midlake - Young Bride (Cassettes Won't Listen Remix)**</b></td></tr></table><br><table cellpadding=0 cellspacing=0 border=0 width=100%> <tr><td bgcolor=#DDDDDD nowrap colspan=5 align=center><table cellspacing=0 cellpadding=0 border=0><tr><td><font class=ltv>Written by Stephen 'Tag Loomis, Tom Pepper and Justin Frankel</font></td></tr></table></td></tr><tr><td nowrap colspan=5 align=center><font class=ST><b><a href="http://www.shoutcast.com/disclaimer.phtml">Copyright Nullsoft Inc</a><a href="/llamacookie">.</a> 1998-2004</b></font></td></tr></table></font></body></html>
显然，这是我要为“当前歌曲”抓取的最后一点 HTML 我怎样才能得到这个 HTML？？
Edit3：解决了！我使用 Wireshark 来捕获浏览器发送的 GET，并将在那里看到的所有参数添加到我的 Python GET 的标头中。它看起来像这样：
import bs4, requests, urllib2
link = "http://173.255.137.244:8080/"
filename = "testfile.txt"
payload = {'Host':'173.255.137.244:8080','User-Agent':'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0','Accept-Encoding':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language':'en-US,en;q=0.5','Accept-Encoding':'gzip, deflate'}
r = requests.get(link,headers=payload)
data = r.text
soup = bs4.BeautifulSoup(data)
soup.prettify()
print soup
故事的寓意。 Wireshark 很酷。
【问题讨论】：
-
r = requests.get(link)
标签： python-2.7 beautifulsoup python-requests