JSoup 文档与页面源不同答案

【问题标题】：JSoup Document different to page sourceJSoup 文档与页面源不同
【发布时间】：2011-09-07 16:51:20
【问题描述】：

我在尝试使用 jSoup 解析网页时遇到问题。如果我在 chrome 中查看页面源代码，就会有您所期望的所有内容 - 整页。但是，当我使用 JSoup 连接到 url 时，Jsoup 文档仅包含少量页面的 html/xml。

知道这是为什么吗？

以下是差异的示例：

页面来源：

<title>Bath Rugby Official Website | First team</title> 
<base href="http://www.bathrugby.com/"></base> 
<meta name="description" content="The Official Website of Bath Rugby. The only place     for in-depth player, match and club news." /><meta property="og:title" content="Bath Rugby     Official Website | First team" /><meta property="og:url"     content="http://www.bathrugby.com/fixtures-and-results/first-team/" /> 
<meta property="og:type" content="sports_team" /> 
<!-- <meta property="og:image"     content="http://www.bathrugby.com/_assets/img/common/bath_rugby_logo.jpg" /> --> 
<meta property="og:site_name" content="Bath Rugby" /> 
<meta property="fb:admins" content="100001941260923" /> 


<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Latest News - RSS feed" href="/rss/rss-all-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Match News - RSS   feed" href="/rss/rss-match-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Team News - RSS feed" href="/rss/rss-team-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Club News - RSS feed" href="/rss/rss-club-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Academy News - RSS feed" href="/rss/rss-academy-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Ladies News - RSS feed" href="/rss/rss-ladies-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Community News - RSS feed" href="/rss/rss-community-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Programmes and Projects - Inclusion - RSS feed" href="/rss/rss-programmes-and-projects-inclusion.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Programmes and Projects - Education - RSS feed" href="/rss/rss-programmes-and-projects-education.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Programmes and Projects - Sports - RSS feed" href="/rss/rss-programmes-and-projects-sport.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Corporate News - RSS feed" href="/rss/rss-corporate-news.rss" /> 
<link rel="alternate" type="application/rss+xml" title="Bath Rugby - Redevelopment News - RSS feed" href="/rss/rss-redevelopment-news.rss" />

JSoup 文档：

<head> 
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
  <title>Bath Rugby - Fixtures and results</title> 
  <meta name="Author" content="Positive -&gt; http://www.positivestudio.co.uk" /> 
  <meta name="HandheldFriendly" content="True" /> 
  <meta name="viewport" content="width=device-width; initial-scale=1.0; minimum-scale=1.0; maximum-scale=1.0; user-scalable=0;" /> 
   <meta name="google-site-verification" content="7dRcSqMI5R3Rag9iNiHeFei6ycyBTd3dmhlZvF8QdhA" /> 
   <link rel="Shortcut Icon" type="image/ico" href="/_assets/favicon.ico" /> 
  <link rel="apple-touch-icon-precomposed" href="/_assets/img/icons/apple-touch-icon.png" /> 
   <link rel="stylesheet" type="text/css" media="all" href="/_assets/css/mobile.css" /> 
   <link rel="stylesheet" type="text/css" href="/_assets/css/retina.css" media="only screen and (-webkit-min-device-pixel-ratio: 2)" /> 
  <script type="text/javascript" src="/_assets/js/jquery-1.4.2.min.js"></script> 
   <script src="/_assets/js/css_browser_selector.js" type="text/javascript"></script> 
  <script type="text/javascript" src="/_assets/js/scripts.js"></script> 
  <script type="text/javascript">

    var _gaq = _gaq || [];
    _gaq.push(['_setAccount', 'UA-24788688-3']);
    _gaq.push(['_trackPageview']);

  (function() {

【问题讨论】：

标签： android parsing jsoup

【解决方案1】：

该网站似乎检测到您正在使用 Android 设备进行浏览，并提供了一个内容较少的适合移动设备的页面。比较您在 Chrome 和 Android 浏览器中查看 http://www.bathrugby.com 所获得的信息。

您可以通过执行connection.userAgent(params, "Mozilla/4.0") 之类的操作来影响通过 Jsoup 提供的内容。

【讨论】：

感谢您的回复。这很可能是原因，但我解决了如下。

【解决方案2】：

我通过检索页面的源 html 作为字符串来回答这个问题，然后将其传递给 jSoup。

HttpClient client = new DefaultHttpClient();
            HttpGet request = new HttpGet(url);
            HttpResponse response = client.execute(request);

            String html = "";
            InputStream in = response.getEntity().getContent();
            BufferedReader reader = new BufferedReader(new     InputStreamReader(in));
            StringBuilder str = new StringBuilder();
            String line = null;
            while((line = reader.readLine()) != null)
            {
                str.append(line);
            }
            in.close();
            html = str.toString();              

            doc = Jsoup.parse(html);

【讨论】：