如何使用 Simple-HTML-DOM 提取完整的子链接？答案

【问题标题】：How to extract complete sub links using Simple-HTML-DOM?如何使用 Simple-HTML-DOM 提取完整的子链接？
【发布时间】：2013-01-14 17:23:05
【问题描述】：

以下是我用来从页面中提取子链接的基本代码：

<?php
    include_once('simple_html_dom.php');
    function extract_links($target_url)
    {   
        $html = new simple_html_dom();
        $html->load_file($target_url);  
        $i=0;
        $crawl =array();
        foreach($html->find('a') as $link)
        {
            $crawl[$i] = $link->href;
            $i++;
        }
        var_dump($crawl);
    }
    extract_links('http://stackoverflow.com');
?>

输出如下：

array
  0 => string 'http://stackexchange.com' (length=24)
  1 => string '/users/login' (length=12)
  2 => string 'http://careers.stackoverflow.com' (length=32)
  3 => string 'http://chat.stackoverflow.com' (length=29)
  4 => string 'http://meta.stackoverflow.com' (length=29)
  5 => string '/about' (length=6)
  6 => string '/faq' (length=4)
  7 => string '/' (length=1)
  8 => string '/questions' (length=10)
  9 => string '/tags' (length=5)
  10 => string '/users' (length=6)
  11 => string '/badges' (length=7)
  12 => string '/unanswered' (length=11)
  13 => string '/questions/ask' (length=14)
  14 => string '?tab=interesting' (length=16)
  15 => string '?tab=featured' (length=13)
  16 => string '?tab=hot' (length=8)
  17 => string '?tab=week' (length=9)
  18 => string '?tab=month' (length=10)
  19 => string '/questions/14611052/basic-standalone-jpa-example-with-postgres-using-eclipse' (length=76)
  20 => string '/questions/tagged/eclipse' (length=25)
  21 => string '/questions/tagged/postgresql' (length=28)
  22 => string '/questions/tagged/jpa' (length=21)
  23 => string '/questions/14611052/basic-standalone-jpa-example-with-postgres-using-eclipse' (length=76)
  24 => string '/users/865448/tostao' (length=20)
  25 => string '/questions/14611172/unable-to-fully-print-a-page-containing-iframes-in-chrome' (length=77)
  26 => string '/questions/tagged/javascript' (length=28)
  27 => string '/questions/tagged/jquery' (length=24)
  28 => string '/questions/tagged/html' (length=22)
  29 => string '/questions/tagged/html5' (length=23)
  30 => string '/questions/tagged/google-chrome' (length=31)
  31 => string '/questions/14611172/unable-to-fully-print-a-page-containing-iframes-in-chrome' (length=77)
  32 => string '/users/962868/tejas' (length=19)
  33 => string '/questions/14609779/how-can-i-configure-bash-to-handle-crlf-shell-scripts' (length=73)
  34 => string '/questions/tagged/linux' (length=23)
  35 => string '/questions/tagged/windows' (length=25)
  36 => string '/questions/tagged/bash' (length=22)
  37 => string '/questions/tagged/line-endings' (length=30)
  38 => string '/questions/14609779/how-can-i-configure-bash-to-handle-crlf-shell-scripts/?lastactivity' (length=87)
  39 => string '/users/1899640/that-other-guy' (length=29)
  40 => string '/questions/14611169/using-one-socket-for-peer-to-peer-communication' (length=67)
  41 => string '/questions/tagged/sockets' (length=25)
  42 => string '/questions/tagged/p2p' (length=21)
  43 => string '/questions/14611169/using-one-socket-for-peer-to-peer-communication' (length=67)
  44 => string '/users/911651/xsnrg' (length=19)
  45 => string '/questions/14611166/possible-mistake-in-ios-dev-guide' (length=53)
  46 => string '/questions/tagged/iphone' (length=24)
  47 => string '/questions/tagged/ios' (length=21)
  48 => string '/questions/tagged/objective-c' (length=29)
  49 => string '/questions/14611166/possible-mistake-in-ios-dev-guide' (length=53)
  50 => string '/users/107715/matt-n' (length=20)
  51 => string '/questions/14611163/how-to-use-dispatcher-in-wpf-to-make-a-timer' (length=64)
  52 => string '/questions/tagged/wpf' (length=21)
  53 => string '/questions/tagged/timer' (length=23)
  54 => string '/questions/tagged/dispatcher' (length=28)
  55 => string '/questions/14611163/how-to-use-dispatcher-in-wpf-to-make-a-timer' (length=64)
  56 => string '/users/1741800/nashat' (length=21)
  57 => string '/questions/14610879/how-can-i-handle-an-access-violation-in-visual-studio-c' (length=75)
  58 => string '/questions/tagged/visual-c%2b%2b' (length=32)
  59 => string '/questions/tagged/exception-handling' (length=36)
  60 => string '/questions/tagged/access-violation' (length=34)
  61 => string '/questions/tagged/structured-exception' (length=38)
  62 => string '/questions/14610879/how-can-i-handle-an-access-violation-in-visual-studio-c/?lastactivity' (length=89)
  63 => string '/users/901812/big-endian' (length=24)
  64 => string '/questions/14611162/mvc-condintional-authorization' (length=50)
  65 => string '/questions/tagged/c%23' (length=22)
  66 => string '/questions/tagged/asp.net-mvc' (length=29)
  67 => string '/questions/tagged/asp.net-mvc-4' (length=31)
  68 => string '/questions/tagged/authorization' (length=31)
  69 => string '/questions/14611162/mvc-condintional-authorization' (length=50)
  70 => string '/users/644969/cadrell0' (length=22)
  71 => string '/questions/14611160/get-customer-role-nopcommerce' (length=49)
  72 => string '/questions/tagged/c%23' (length=22)
  73 => string '/questions/tagged/razor' (length=23)
  74 => string '/questions/tagged/nopcommerce' (length=29)
  75 => string '/questions/14611160/get-customer-role-nopcommerce' (length=49)
  76 => string '/users/1378841/mlg74' (length=20)
  77 => string '/questions/14611158/iframe-resizing-nested-in-gridview' (length=54)
  78 => string '/questions/tagged/resize' (length=24)
  79 => string '/questions/14611158/iframe-resizing-nested-in-gridview' (length=54)
  80 => string '/users/2026451/satish-patil' (length=27)
  81 => string '/questions/14611157/php-how-to-check-the-value-got-this-word-from-a-var' (length=71)
  82 => string '/questions/tagged/php' (length=21)
  83 => string '/questions/tagged/preg-match' (length=28)
  84 => string '/questions/tagged/strpos' (length=24)
  85 => string '/questions/14611157/php-how-to-check-the-value-got-this-word-from-a-var' (length=71)
  86 => string '/users/963414/samual99' (length=22)
  87 => string '/questions/14611155/how-to-get-the-coordinates-of-boundries-of-drawable-on-the-mapview' (length=86)
  88 => string '/questions/tagged/android' (length=25)
  89 => string '/questions/tagged/google-maps' (length=29)
  90 => string '/questions/14611155/how-to-get-the-coordinates-of-boundries-of-drawable-on-the-mapview' (length=86)
  91 => string '/users/1520564/blubar' (length=21)
  92 => string '/questions/14611153/why-css-is-empty-when-ssl-is-on-and-appcache-is-enabled-ipad-safari' (length=87)
  93 => string '/questions/tagged/css' (length=21)
  94 => string '/questions/tagged/ipad' (length=22)
  95 => string '/questions/tagged/ssl' (length=21)
  96 => string '/questions/tagged/mobile-safari' (length=31)
  97 => string '/questions/tagged/html5-appcache' (length=32)
  98 => string '/questions/14611153/why-css-is-empty-when-ssl-is-on-and-appcache-is-enabled-ipad-safari' (length=87)
  99 => string '/users/2026375/twoface' (length=22)
  100 => string '/questions/14611149/laravel-how-to-temporarily-store-eloquent-models-in-db-without-a-proper-schem' (length=97)
  101 => string '/questions/tagged/php' (length=21)
  102 => string '/questions/tagged/laravel' (length=25)
  103 => string '/questions/14611149/laravel-how-to-temporarily-store-eloquent-models-in-db-without-a-proper-schem' (length=97)
  104 => string '/users/291557/duality' (length=21)
  105 => string '/questions/13928812/xmlserializer-generateserializer-and-collections' (length=68)
  106 => string '/questions/tagged/c%23' (length=22)
  107 => string '/questions/tagged/xml-serialization' (length=35)
  108 => string '/questions/13928812/xmlserializer-generateserializer-and-collections/?lastactivity' (length=82)
  109 => string '/users/1200614/phil' (length=19)
  110 => string '/questions/14611145/keep-buttons-in-view-when-keyboard-opens-android' (length=68)
  111 => string '/questions/tagged/android' (length=25)
  112 => string '/questions/tagged/keyboard' (length=26)
  113 => string '/questions/tagged/resize' (length=24)
  114 => string '/questions/tagged/window' (length=24)
  115 => string '/questions/tagged/views' (length=23)
  116 => string '/questions/14611145/keep-buttons-in-view-when-keyboard-opens-android' (length=68)
  117 => string '/users/1137413/725623452362' (length=27)
  118 => string '/questions/14611144/ssdp-discovery-from-a-browser' (length=49)
  119 => string '/questions/tagged/silverlight' (length=29)
  120 => string '/questions/tagged/flash' (length=23)
  121 => string '/questions/14611144/ssdp-discovery-from-a-browser' (length=49)
  122 => string '/users/191882/legege' (length=20)
  123 => string '/questions/14611143/how-to-syncrhonize-on-site-in-memory-no-sql-datasources-with-central-database-in' (length=100)
  124 => string '/questions/tagged/architecture' (length=30)
  125 => string '/questions/tagged/nosql' (length=23)
  126 => string '/questions/tagged/java-ee-6' (length=27)
  127 => string '/questions/tagged/in-memory-database' (length=36)
  more elements...

现在考虑数组中的“/about”子链接。我希望它显示为“https://stackoverflow.com/about”。为什么只返回子链接的子部分，而在某些情况下返回完整的子链接？还有一些链接以“?”符号开头。如何清理这些链接？

编辑： 考虑“http://en.wikipedia.org/wiki/Web_crawler”。现在，如果我对其执行 extract_links，我会得到一个类似“http://en.wikipedia.org/wiki/Web_crawler/wiki/Web_search_engine”的子链接，它是无效的，并且大多数链接都是这种格式。正确的链接是“http://en.wikipedia.org/wiki/Web_search_engine”。我在另一个程序中使用这个函数，它将传递一个链接数组，所以我不能保持 if 条件 static。以下是我现在使用的代码片段：

foreach($html->find('a') as $link)
{   
    $href = $link->href;
    $fchr = substr($href, 0, 1);
    if ($fchr === '/')
    {
        $href = $target_url.$href;
    }
    else if ($fchr === '?')
    {
        $href = $target_url.'/'. $href;
    }
}

【问题讨论】：

小心 HTML 抓取 StackOverflow。见Etiquette of Screen-scraping Stack Overflow?。如果可以，请考虑使用API。
有several options available 用于将相对网址转换为绝对网址。选择其中之一。
@ithcy 感谢您的提醒。顺便说一句，我不是在刮它。只是为了展示一个例子。
@pguardiario 请检查问题的编辑部分。
检查什么？您应该为此使用库。您的代码将无法工作，甚至没有关闭。

标签： php hyperlink anchor simple-html-dom

【解决方案1】：

任何以“/”开头的链接都是从文档根目录开始的绝对路径。要获取完整的 URL，您需要预先添加找到该链接的主机名。对于相对链接，例如“?tab=etc”，您需要在前面添加找到链接的完整 URL。如果您想忽略查询字符串链接 ("?tab=etc")，请使用正则表达式。

【讨论】：

是的，正如你所说，我正在准备。但请检查问题的已编辑部分。如何处理这样的子链接？

【解决方案2】：

在 html 源代码中，此 a 元素的定义为 <a href="/about">about</a>，因此它返回现有的 href 值为 /about。

如果您想显示带有 http 和域的链接，试试这个（所有内部链接都可以以 / 或 ? 开头）；

foreach($html->find('a') as $link) {
    $href = $link->href;
    $fchr = substr($href, 0, 1);
    if ($fchr === '/') {
        $href = 'http://stackoverflow.com'. $href;
    } else if ($fchr === '?') {
        $href = 'http://stackoverflow.com/'. $href;
    }
    $crawl[] = $href;
}

PN：推送非关联数组时不需要$i，只需$array[] = .. 即可。

更新

假设您在使用“内部链接”时遇到问题（因此，这些链接可能不包含 http 或站点 url）。实际上我不知道如何检测内部链接了。我认为这抓住了所有这些：'foo', '/foo', '?foo', '#foo', '../foo', '/../foo'。但是假设这些似乎无效？ '../foo', '/../foo'.

// or more strong pattern
if (!preg_match('~^((ht|f)tp?s*)://.*~i', trim($href))) {
    $href = 'http://www.site.com/'. ltrim($href, '/');
}

【讨论】：

好的，这适用于单个链接。检查问题的已编辑部分。如果问题的编辑部分提到的特定链接，使用它会产生问题。
问题远比这个复杂。考虑所有：'foo'、'/foo'、'?foo'、'#foo'、'../foo'、'/../foo'

【解决方案3】：

@pguardiario 的评论

正如他在评论中所建议的那样，phpUri 是将相对 URL 转换为绝对 URL 的完美解决方案。你可以找到它here,

【讨论】：