【问题标题】:python - parsing an urlpython - 解析一个网址
【发布时间】:2017-03-25 21:41:33
【问题描述】:

我正在编写一个简单的脚本来检查网站是否存在于谷歌上,首先搜索确定的关键字。

现在,这是解析 url 并返回主机名的函数:

def parse_url(url):
    url = urlparse(url)
    hostname = url.netloc
    return hostname

并从以下选择的标签列表开始:

linkElems = soup.select('.r a') #in google first page the resulting urls have class r

这是我写的:

    for link in linkElems:
        l = link.get("href")[7:]
        url = parse_url(l)
        if "www.example.com" == url:
        #do stuff (ex store in a list, etc)

在最后一个,在第二行,我必须从第七个索引开始,因为所有 href 值都以'/url?q=' 开头。

我正在学习 python,所以我想知道是否有更好的方法来做到这一点,或者只是一种替代方法(可能使用正则表达式或替换方法或来自 urlparse 库)

【问题讨论】:

    标签: python urlparse


    【解决方案1】:

    您可以使用 python lxml 模块来做这也比 BeautifulSoup 快一个数量级。

    这可以这样做:

    import requests
    from lxml import html
    
    blah_url = "https://www.google.co.in/search?q=blah&oq=blah&aqs=chrome..69i57j0l5.1677j0j4&sourceid=chrome&ie=UTF-8"
    r = requests.get(blah_url).content
    root = html.fromstring(r)
    
    print(root.xpath('//h3[@class="r"]/a/@href')[0].replace('/url?q=', ''))
    print([url.replace('/url?q=', '') for url in root.xpath('//h3[@class="r"]/a/@href')])
    

    这将导致:

    http://www.urbandictionary.com/define.php%3Fterm%3Dblah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggTMAA&usg=AFQjCNFge5GFNmjpan7S_UCNjos1RP5vBA  
    ['http://www.urbandictionary.com/define.php%3Fterm%3Dblah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggTMAA&usg=AFQjCNFge5GFNmjpan7S_UCNjos1RP5vBA', 'http://www.dictionary.com/browse/blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggZMAE&usg=AFQjCNE1UVR3krIQHfEuIzHOeL0ZvB5TFQ', 'http://www.dictionary.com/browse/blah-blah-blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggeMAI&usg=AFQjCNFw8eiSqTzOm65PQGIFEoAz0yMUOA', 'https://en.wikipedia.org/wiki/Blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggjMAM&usg=AFQjCNFxEB8mEjEy6H3YFOaF4ZR1n3iusg', 'https://www.merriam-webster.com/dictionary/blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggpMAQ&usg=AFQjCNHYXX53LmMF-DOzo67S-XPzlg5eCQ', 'https://en.oxforddictionaries.com/definition/blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFgguMAU&usg=AFQjCNGlgcUx-BpZe0Hb-39XvmNua2n8UA', 'https://en.wiktionary.org/wiki/blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFggzMAY&usg=AFQjCNGc9VmmyQls_rOBOR_lMUnt1j3Flg', 'http://dictionary.cambridge.org/dictionary/english/blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFgg5MAc&usg=AFQjCNHJgZR1c6VY_WgFa6Rm-XNbdFJGmA', 'http://www.thesaurus.com/browse/blah&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQFgg-MAg&usg=AFQjCNEtnpmKxVJqUR7P1ss4VHnt34f4Kg', 'https://www.youtube.com/watch%3Fv%3D3taEuL4EHAg&sa=U&ved=0ahUKEwiyscHQ5_LSAhWFvI8KHctAC0IQtwIIRTAJ&usg=AFQjCNFnKlMFxHoYAIkl1MCrc_OXjgiClg']
    

    【讨论】:

      猜你喜欢
      • 2020-10-21
      • 2011-07-17
      • 2011-04-05
      • 1970-01-01
      • 1970-01-01
      • 2015-12-24
      • 2022-01-16
      • 2015-12-18
      相关资源
      最近更新 更多