【问题标题】:Link extraction from a google page in bash从 bash 中的谷歌页面提取链接
【发布时间】:2012-08-02 22:31:59
【问题描述】:

我正在制作一个脚本,它从 bash 中的谷歌页面获取所有链接。我使用w3m 实用程序和这个脚本获得了谷歌页面:

#!/bin/bash 
# performs a google search using a word in input

word=$1

touch .google

if [ -z $word ]
then
    echo "$word missing!"
    echo "Aborting..."
    exit
fi


a="www.google.com/search?q="  
search=$a$word

w3m -no-cookie $search > .google

sleep 1

接下来,我必须从此页面获取所有站点。我正在考虑采用所有以www. 开头并以/ 结尾的字符串

echo `grep -wo "www[^/]*" .google`> .temp

这样做的问题是我错过了很多不以www 开头的链接,同时当有一个不以/ 结尾的站点时,我冒着破坏一切的风险。

有什么更好的方法可以从此响应中获取网址?

【问题讨论】:

    标签: linux bash hyperlink w3m


    【解决方案1】:

    链接提取是一个难题。但是,lynx 程序有一个方便的-dump 选项,可以让您跳过大部分(或全部)html 解析。

    具体来说,请注意底部的References 部分。您可以从该行继续输出,并去掉前面的项目符号编号:

    $ lynx -dump 'http://www.seomoz.org/'
       #[1]RSS 2.0 [2]publisher
    
       [3]SEOmoz
         * [4]Log in
         * [5]Sign up
         * [6]Help
              + [7]Help Resources
              + [8]Support Forums
              + [9]Request a Feature
              + [10]Contact Us
    
         * [11]Features
         * [12]Pricing & Plans
         * [13]Community
              + [14]SEO Blog
              + [15]YOUmoz User Blog
              + [16]Top Users
              + [17]Events
              + [18]Recommended Companies
         * [19]Resources
              + [20]Learn SEO
              + [21]SEO Tools
              + [22]PRO Q&A Forum
              + [23]Mozscape API
         * [24]Blog
              + [25]SEO Blog
              + [26]YOUmoz User Blog
         * [27]About
              + [28]Our TAGFEE Mission
              + [29]Meet the Mozzers
              + [30]Contact Us
              + [31]Join Our Team
              + [32]Press & Awards
              + [33]Events
    
       Search SEOmoz
    
       ____________________ Search
    
    SEO & Social Monitoring
    
    Made Simple.
    
       SEOmoz PRO combines SEO management, social media monitoring, actionable
       recommendations, and so much more in one easy-to-use platform. Try it
       free for 30 days.
       [34]Try it for Free!
    
       [35]Take a tour of SEOmoz PRO
       or see [36]plans & pricing
    
         * Campaign Overview
         * Social Dashboard
         * Crawl Diagnostics
         * Dashboard
         * Google Analytics
         * Link Analysis
    
    Loved By...
    
         * Zillow
         * Disney
         * Overstock
         * Best Buy
         * Yelp
         * Sun Microsystems
    
    
       Roger Mozbot
    
    Be My Buddy...
    
         * [37]RSS
         * [38]Twitter
         * [39]Facebook
         * [40]Google+
    
    Effectively Manage Your SEO and Monitor Your Social Media
    
       [41]Link Analysis
    
       Analyze links and track key performance metrics in an efficient
       all-in-one dashboard.
       [42]Identify SEO Issues
    
       Identify critical SEO issues and get actionable recommendations.
       [43]Monitor Changes
    
       Automatically monitor changes to your rankings and take control of your
       organic traffic.
       Avinash Kaushik
    
         "SEOmoz tools provide best of class data. Their tools are a
         must-have for marketers looking to optimize their organic search
         results."
    
    Avinash Kaushik,
    
       Author, Web Analytics 1.0: An Hour A Day
       Patrick Altoft
    
         "SEOmoz has enabled us to scale our link-building process quickly
         without compromising on quality."
    
    Patrick Altoft,
    
       CEO, Branded3
    
    Latest from the SEOmoz Blog
         __________________________________________________________________
    
       [44]jennita
    
    [45]Winners of #MozCation 2012
    
       Posted by [46]jennita on 08/04/2012
       Whoa. Ever have one of those times where your expectations are
       completely blown out of the water? Well that's what happened during
       this year's nomination for a MozCation. Wait, wait, wait, before I get
       too far ahead of myself, I...
       [47]Read Full Entry
    
       13
    
       2
       [48]13 Comments
         __________________________________________________________________
    
    Latest from the Community YouMoz Blog
         __________________________________________________________________
    
       [49]larry.kim
    
    [50]Does SEO Even Work for Small Businesses?
    
       Posted by [51]larry.kim on 08/03/2012
       Clicks on paid search listings beat out organic listings by nearly a
       2:1 margin for keywords with high commercial intent in the US. Is SEO
       still a viable marketing tactic for the average small business owner?
       [52]Read Full Entry
    
       17
    
       3
       [53]28 Comments
         __________________________________________________________________
    
    Voted Best SEO Tool 2010!
    
       [54]Try it for Free!
    
    Looking for SEO consulting?
    
       SEOmoz doesn't provide consulting, but our friends at [55]Distilled
       still do. Rock on!
    
       Copyright ? 1996-2012 SEOmoz. All Rights Reserved.
    
    Product and Tools
    
         * [56]SEOmoz PRO
         * [57]Pricing and Plans
         * [58]Open Site Explorer
         * [59]SEO Toolbar
         * [60]Mozscape API
         * [61]More SEO Tools
    
    Company
    
         * [62]About
         * [63]SEO Blog
         * [64]YOUmoz Blog
         * [65]Affiliate Program
         * [66]Terms & Privacy Policy
         * [67]PRO Perks
    
    Popular Content
    
         * [68]Link Building
         * [69]Reputation Management
         * [70]Analytics
         * [71]Social Media
         * [72]Content & Blogging
         * [73]See All Categories
    
    Stay in Touch
    
         *
              + [74]RSS
              + [75]Twitter
              + [76]Facebook
              + [77]LinkedIn
         *
    
    
        SEOmoz
        119 Pine St. Suite 400
        Seattle, WA 98101
        206.632.3171
         * [78]Contact Us
         * [79]Sitemap
    
    References
    
       1. http://feeds.feedburner.com/seomoz
       2. https://plus.google.com/112544075040456048636
       3. http://www.seomoz.org/
       4. https://www.seomoz.org/users/login
       5. https://www.seomoz.org/users/register
       6. http://www.seomoz.org/
       7. http://www.seomoz.org/help
       8. http://www.seomoz.org/q
       9. http://seomoz.zendesk.com/forums/293194-seomoz-PRO-feature-requests
      10. http://www.seomoz.org/about/contact
      11. http://www.seomoz.org/features
      12. http://www.seomoz.org/plans
      13. http://www.seomoz.org/community
      14. http://www.seomoz.org/blog
      15. http://www.seomoz.org/ugc
      16. http://www.seomoz.org/users
      17. http://www.seomoz.org/about/events
      18. http://www.seomoz.org/article/recommended
      19. http://www.seomoz.org/resources
      20. http://www.seomoz.org/learn-seo
      21. http://www.seomoz.org/tools
      22. http://www.seomoz.org/q
      23. http://www.seomoz.org/api
      24. http://www.seomoz.org/blog
      25. http://www.seomoz.org/blog
      26. http://www.seomoz.org/ugc
      27. http://www.seomoz.org/about
      28. http://www.seomoz.org/about/mission
      29. http://www.seomoz.org/about/team
      30. http://www.seomoz.org/about/contact
      31. http://www.seomoz.org/about/jobs
      32. http://www.seomoz.org/about/press
      33. http://www.seomoz.org/about/seo-events
      34. http://www.seomoz.org/cart/freetrial?pg=home
      35. http://www.seomoz.org/features
      36. http://www.seomoz.org/plans
      37. http://feeds.feedburner.com/seomoz
      38. http://twitter.com/seomoz
      39. http://www.facebook.com/SEOmoz
      40. https://plus.google.com/112544075040456048636?prsrc=3
      41. http://www.seomoz.org/features
      42. http://www.seomoz.org/features
      43. http://www.seomoz.org/features
      44. http://www.seomoz.org/users/profile/81197
      45. http://www.seomoz.org/blog/winners-mozcation-2012
      46. http://www.seomoz.org/users/profile/81197
      47. http://www.seomoz.org/blog/winners-mozcation-2012
      48. http://www.seomoz.org/blog/winners-mozcation-2012#comments
      49. http://www.seomoz.org/users/profile/402613
      50. http://www.seomoz.org/ugc/does-seo-even-work-for-small-businesses
      51. http://www.seomoz.org/users/profile/402613
      52. http://www.seomoz.org/ugc/does-seo-even-work-for-small-businesses
      53. http://www.seomoz.org/ugc/does-seo-even-work-for-small-businesses#comments
      54. http://www.seomoz.org/cart/freetrial?pg=features
      55. http://www.seomoz.org/dp/distilled
      56. http://www.seomoz.org/features
      57. http://www.seomoz.org/plans
      58. http://www.opensiteexplorer.org/
      59. http://www.seomoz.org/seo-toolbar
      60. http://www.seomoz.org/api
      61. http://www.seomoz.org/tools
      62. http://www.seomoz.org/about
      63. http://www.seomoz.org/blog
      64. http://www.seomoz.org/ugc
      65. http://www.seomoz.org/dp/seomoz-pro-affiliate-program
      66. http://www.seomoz.org/terms-and-privacy
      67. http://www.seomoz.org/pro-perks
      68. http://www.seomoz.org/blog/category/4
      69. http://www.seomoz.org/blog/category/19
      70. http://www.seomoz.org/blog/category/8
      71. http://www.seomoz.org/blog/category/18
      72. http://www.seomoz.org/blog/category/1
      73. http://www.seomoz.org/blog
      74. http://feeds.feedburner.com/seomoz
      75. http://twitter.com/seomoz
      76. http://www.facebook.com/SEOmoz
      77. http://www.linkedin.com/groups?about=&gid=2976409&trk=anet_ug_grppro
      78. http://www.seomoz.org/about/contact
      79. http://www.seomoz.org/sitemap
    

    【讨论】:

      【解决方案2】:

      您可能希望使用 grep 查找 <a href=" 并将该值带到下一个引号符号。然后过滤掉所有 javascript 的东西。虽然这个解决方案可能也不是万无一失的。

      【讨论】:

      • 谢谢,但我得到的页面不是 html,它们看起来像普通的网页:O
      • 哦,你说的是 google 页面,我认为 google 会返回 HTML。那么这是一个不同的问题。我可以给你一个我多年来用来做 URL 匹配的正则表达式,虽然它也不是 100% 万无一失的
      • string expression = "(?:(?:ht|f)tps?://)*(?:[a-zA-Z0-9-]+\\.)+(?:com|net|org|info|mil|gov|edu|name|[a-z]{2}){1}(?::[0-9]+)?(?![a-z])(?:/[a-zA-Z0-9\\^\\.\\{\\}\\(\\)\\[\\]_\\?,'/\\\\+&%\\$:#=~-]*)*"; 你可能想用一些现代顶级域名来扩展它:)
      • 和成功率.. 嗯,不确定,我们用它来过滤聊天记录中的 URL,如果有的话,它会时不时地捕捉到误报,但到目前为止还没有注意到很多的垃圾。抱歉,您手头没有准确的统计数据。
      • 非常感谢,非常感谢。很抱歉这个菜鸟问题,但是我如何将它与我用 w3m 下载的文件一起使用?
      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2013-08-17
      • 1970-01-01
      • 2011-06-04
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多