从带有 403 错误 python 的网站中抓取链接

Question

我正在尝试从链接列表中抓取链接（所有链接都指向同一网站上的不同页面），但我一直运行 403 错误。这是我尝试抓取的链接示​​例

https://www.spectatornews.com/page/6/?s=band

https://www.spectatornews.com/page/7/?s=band

等等

这是我的代码：

getarticles = []

from bs4 import BeautifulSoup
import urllib.request

for i in listoflinks:
    resp = urllib.request.urlopen(i)
    soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'))

    for link in soup.find_all('a', href=True):

        getarticles.append(link['href'])

我一直在尝试使用来自HTTP error 403 in Python 3 Web Scraping 的一些答案，但我没有取得太大的成功。我不确定我是否正确地将它们应用于我的整个链接列表。我尝试通过使用标头来使用以下解决方案之一，但这会返回 HTTP 406 错误：不可接受

这是我试图修复的代码：

getarticles = []
from bs4 import BeautifulSoup

from bs4 import BeautifulSoup
import urllib.request

for i in listoflinks:
    req=urllib.request.Request(i, headers={'User-Agent': 'Mozilla/5.0'})
    resp = urllib.request.urlopen(req)
    soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'))

    for link in soup.find_all('a', href=True):

        getarticles.append(link['href'])

非常感谢任何帮助。我对此很陌生，所以尽可能多地解释和帮助是很好的。我只想从我的网站列表中收集链接！

谢谢

Answer 1

我要预先声明，我很少使用 urllib/3 库。但是，我确实尝试使用 scrapy 的 shell 终端命令以及使用没有用户代理的请求库并得到 200 响应。

我确实注意到您在声明“汤”时没有声明解析器的类型。

 soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'))

虽然我更喜欢使用scrapy的解析器，尽管它更重，但如果没记错你必须声明一个解析器类型，例如

soup = BeautifulSoup(resp, "lxml")

Bitto Benni-chan 说他设法让它响应 200 urllib.request，所以试试他的更改。这只是输入完整的用户代理名称。

我的建议是使用请求库。我认为这将是一个足够简单的改变。

from bs4 import BeautifulSoup
import requests

listoflinks = ['https://www.spectatornews.com/page/6/?s=band', 'https://www.spectatornews.com/page/7/?s=band']

getarticles = []

for i in listoflinks:
    resp = requests.get(i)
    soup = BeautifulSoup(resp.content, "lxml")

    for link in soup.find_all('a', href=True):

        getarticles.append(link['href'])

getarticles 列表输出如下：

'https://www.spectatornews.com/category/showcase/',
 'https://www.spectatornews.com/showcase/2003/02/06/minneapolis-band-trips-into-eau-claire/',
 'https://www.spectatornews.com/category/showcase/',
 'https://www.spectatornews.com/page/5/?s=band',
 'https://www.spectatornews.com/?s=band',
 'https://www.spectatornews.com/page/2/?s=band',
 'https://www.spectatornews.com/page/3/?s=band',
 'https://www.spectatornews.com/page/4/?s=band',
 'https://www.spectatornews.com/page/5/?s=band',
 'https://www.spectatornews.com/page/7/?s=band',
 'https://www.spectatornews.com/page/8/?s=band',
 'https://www.spectatornews.com/page/9/?s=band',
 'https://www.spectatornews.com/page/127/?s=band',
 'https://www.spectatornews.com/page/7/?s=band',
 'https://www.spectatornews.com',
 'https://www.spectatornews.com/feed/rss/',
 '#',
 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ',
 'https://www.snapchat.com/add/spectator news',
 'https://www.instagram.com/spectatornews/',
 'http://twitter.com/spectatornews',
 'http://facebook.com/spectatornews',
 '/',
 'https://snosites.com/why-sno/',
 'http://snosites.com',
 'https://www.spectatornews.com/wp-login.php',
 '#top',
 '/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/category/sports/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/category/multimedia-2/',
 'https://www.spectatornews.com/ads/banner-advertise-with-the-spectator/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/category/sports/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/category/multimedia-2/',
 '/',
 'https://www.spectatornews.com/about/',
 'https://www.spectatornews.com/about/editorial-policy/',
 'https://www.spectatornews.com/about/correction-policy/',
 'https://www.spectatornews.com/about/bylaws/',
 'https://www.spectatornews.com/advertise/',
 'https://www.spectatornews.com/contact/',
 'https://www.spectatornews.com/staff/',
 'https://www.spectatornews.com/submit-a-letter/',
 'https://www.spectatornews.com/submit-a-news-tip/',
 '/',
 'https://www.spectatornews.com',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/category/sports/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/category/multimedia-2/',
 '/',
 'https://www.spectatornews.com/feed/rss/',
 '#',
 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ',
 'https://www.snapchat.com/add/spectator news',
 'https://www.instagram.com/spectatornews/',
 'http://twitter.com/spectatornews',
 'http://facebook.com/spectatornews',
 'https://www.spectatornews.com/campus-news/2002/05/09/late-night-bus-service-idea-abandoned-due-to-expense/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/opinion/2002/03/21/yates-deserved-what-she-got-husband-also-to-blame/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/opinion/2001/11/29/air-force-concert-band-inspires-zorn-arena-audience/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/campus-news/2001/10/25/goth-style-bands-will-entertain-at-halloween-costume-concert/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/campus-news/2001/04/19/campus-group-will-host-hemp-event-with-bands-information/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/currents/2018/12/10/geekin-out/',
 'https://www.spectatornews.com/currents/2018/12/10/geekin-out/',
 'https://www.spectatornews.com/staff/?writer=Alanna%20Huggett',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/tag/geekcon/',
 'https://www.spectatornews.com/tag/tv10/',
 'https://www.spectatornews.com/tag/uwec/',
 'https://www.spectatornews.com/opinion/2018/12/07/keeping-up-with-the-kar-fashions-11/',
 'https://www.spectatornews.com/opinion/2018/12/07/keeping-up-with-the-kar-fashions-11/',
 'https://www.spectatornews.com/staff/?writer=Kar%20Wei%20Cheng',
 'https://www.spectatornews.com/category/column-2/',
 'https://www.spectatornews.com/category/multimedia-2/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/tag/accessories/',
 'https://www.spectatornews.com/tag/fashion/',
 'https://www.spectatornews.com/tag/multimedia/',
 'https://www.spectatornews.com/tag/winter/',
 'https://www.spectatornews.com/multimedia-2/2018/12/07/a-magical-night/',
 'https://www.spectatornews.com/multimedia-2/2018/12/07/a-magical-night/',
 'https://www.spectatornews.com/staff/?writer=Julia%20Van%20Allen',
 'https://www.spectatornews.com/category/multimedia-2/',
 'https://www.spectatornews.com/tag/dancing/',
 'https://www.spectatornews.com/tag/harry-potter/',
 'https://www.spectatornews.com/tag/smom/',
 'https://www.spectatornews.com/tag/student-ministry-of-magic/',
 'https://www.spectatornews.com/tag/uwec/',
 'https://www.spectatornews.com/tag/yule/',
 'https://www.spectatornews.com/tag/yule-ball/',
 'https://www.spectatornews.com/campus-news/2018/11/26/old-news-5/',
 'https://www.spectatornews.com/campus-news/2018/11/26/old-news-5/',
 'https://www.spectatornews.com/staff/?writer=Madeline%20Fuerstenberg',
 'https://www.spectatornews.com/category/column-2/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/tag/1950/',
 'https://www.spectatornews.com/tag/1975/',
 'https://www.spectatornews.com/tag/2000/',
 'https://www.spectatornews.com/tag/articles/',
 'https://www.spectatornews.com/tag/spectator/',
 'https://www.spectatornews.com/tag/throwback/',
 'https://www.spectatornews.com/currents/2018/11/21/boss-women-highlighting-businesswomen-in-eau-claire-6/',
 'https://www.spectatornews.com/currents/2018/11/21/boss-women-highlighting-businesswomen-in-eau-claire-6/',
 'https://www.spectatornews.com/staff/?writer=Taylor%20Reisdorf',
 'https://www.spectatornews.com/category/column-2/',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/tag/altoona/',
 'https://www.spectatornews.com/tag/boss-women/',
 'https://www.spectatornews.com/tag/business-women/',
 'https://www.spectatornews.com/tag/cherish-woodford/',
 'https://www.spectatornews.com/tag/crossfit/',
 'https://www.spectatornews.com/tag/crossfit-river-prairie/',
 'https://www.spectatornews.com/tag/eau-claire/',
 'https://www.spectatornews.com/tag/fitness/',
 'https://www.spectatornews.com/tag/gym/',
 'https://www.spectatornews.com/tag/local/',
 'https://www.spectatornews.com/tag/nicole-randall/',
 'https://www.spectatornews.com/tag/river-prairie/',
 'https://www.spectatornews.com/currents/2018/11/20/bad-art-good-music/',
 'https://www.spectatornews.com/currents/2018/11/20/bad-art-good-music/',
 'https://www.spectatornews.com/staff/?writer=Lea%20Kopke',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/tag/bad-art/',
 'https://www.spectatornews.com/tag/fmdown/',
 'https://www.spectatornews.com/tag/ghosts-of-the-sun/',
 'https://www.spectatornews.com/tag/music/',
 'https://www.spectatornews.com/tag/pablo-center/',
 'https://www.spectatornews.com/opinion/2018/11/14/the-tator-21/',
 'https://www.spectatornews.com/opinion/2018/11/14/the-tator-21/',
 'https://www.spectatornews.com/staff/?writer=Stephanie%20Janssen',
 'https://www.spectatornews.com/category/column-2/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/tag/satire/',
 'https://www.spectatornews.com/tag/sleepy/',
 'https://www.spectatornews.com/tag/tator/',
 'https://www.spectatornews.com/tag/uw-eau-claire/',
 'https://www.spectatornews.com/tag/uwec/',
 'https://www.spectatornews.com/page/6/?s=band',
 'https://www.spectatornews.com/?s=band',
 'https://www.spectatornews.com/page/2/?s=band',
 'https://www.spectatornews.com/page/3/?s=band',
 'https://www.spectatornews.com/page/4/?s=band',
 'https://www.spectatornews.com/page/5/?s=band',
 'https://www.spectatornews.com/page/6/?s=band',
 'https://www.spectatornews.com/page/8/?s=band',
 'https://www.spectatornews.com/page/9/?s=band',
 'https://www.spectatornews.com/page/10/?s=band',
 'https://www.spectatornews.com/page/127/?s=band',
 'https://www.spectatornews.com/page/8/?s=band',
 'https://www.spectatornews.com',
 'https://www.spectatornews.com/feed/rss/',
 '#',
 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ',
 'https://www.snapchat.com/add/spectator news',
 'https://www.instagram.com/spectatornews/',
 'http://twitter.com/spectatornews',
 'http://facebook.com/spectatornews',
 '/',
 'https://snosites.com/why-sno/',
 'http://snosites.com',
 'https://www.spectatornews.com/wp-login.php',
 '#top',
 '/',
 'https://www.spectatornews.com/category/campus-news/',
 'https://www.spectatornews.com/category/currents/',
 'https://www.spectatornews.com/category/sports/',
 'https://www.spectatornews.com/category/opinion/',
 'https://www.spectatornews.com/category/multimedia-2/']

Answer 2

403 FORBIDDEN

服务器理解请求但拒绝授权。

406 NOT ACCEPTABLE

目标资源没有当前的表示 被用户代理接受，根据主动 请求中收到的协商头字段，并且服务器是 不愿提供默认表示。

您的用户代理可能是问题所在。我能够通过更改它来获得输出

from bs4 import BeautifulSoup
import urllib.request
listoflinks=['https://www.spectatornews.com/page/6/?s=band','https://www.spectatornews.com/page/6/?s=band']
getarticles = []
for i in listoflinks:
    req = urllib.request.Request(
    i,
    headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
    )
    resp= urllib.request.urlopen(req)
    soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'),features="lxml")
    for link in soup.find_all('a', href=True):
        getarticles.append(link['href'])
print(getarticles)

输出

['https://www.spectatornews.com/ads/banner-advertise-with-the-spectator/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/category/currents/', 'https://www.spectatornews.com/category/sports/', 'https://www.spectatornews.com/category/opinion/', 'https://www.spectatornews.com/category/multimedia-2/', '/', 'https://www.spectatornews.com/about/', 'https://www.spectatornews.com/about/editorial-policy/', 'https://www.spectatornews.com/about/correction-policy/', 'https://www.spectatornews.com/about/bylaws/', 'https://www.spectatornews.com/advertise/', 'https://www.spectatornews.com/contact/', 'https://www.spectatornews.com/staff/', 'https://www.spectatornews.com/submit-a-letter/', 'https://www.spectatornews.com/submit-a-news-tip/', '/', 'https://www.spectatornews.com', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/category/currents/', 'https://www.spectatornews.com/category/sports/', 'https://www.spectatornews.com/category/opinion/', 'https://www.spectatornews.com/category/multimedia-2/', '/', 'https://www.spectatornews.com/feed/rss/', '#', 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ', 'https://www.snapchat.com/add/spectator news', 'https://www.instagram.com/spectatornews/', 'http://twitter.com/spectatornews', 'http://facebook.com/spectatornews', 'https://www.spectatornews.com/campus-news/2004/05/06/english-fest-draws-speakers-bands/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/campus-news/2004/05/03/burgers-on-the-grill-bands-on-the-scene/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/showcase/2004/04/29/hempfest-celebrates-its-10th-year-with-11-bands/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/04/29/pat-mcgee-band-rocks-mad-town/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/04/22/leinenkugels-battle-of-the-bands/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/04/08/on-the-music-scene-band-makes-mondays-better/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/03/18/on-the-music-scene-band-carries-on-duluozs-work/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2003/10/09/jamband-grooving-to-eau-claire/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2003/05/01/joepalooza-set-with-5-bands-one-drummer/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/campus-news/2003/05/01/hempfest-features-nine-bands/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/showcase/2003/02/17/houston-based-band-reaching-out-to-college-students-on-tour/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2003/02/06/minneapolis-band-trips-into-eau-claire/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/page/5/?s=band', 'https://www.spectatornews.com/?s=band', 'https://www.spectatornews.com/page/2/?s=band', 'https://www.spectatornews.com/page/3/?s=band', 'https://www.spectatornews.com/page/4/?s=band', 'https://www.spectatornews.com/page/5/?s=band', 'https://www.spectatornews.com/page/7/?s=band', 'https://www.spectatornews.com/page/8/?s=band', 'https://www.spectatornews.com/page/9/?s=band', 'https://www.spectatornews.com/page/127/?s=band', 'https://www.spectatornews.com/page/7/?s=band', 'https://www.spectatornews.com', 'https://www.spectatornews.com/feed/rss/', '#', 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ', 'https://www.snapchat.com/add/spectator news', 'https://www.instagram.com/spectatornews/', 'http://twitter.com/spectatornews', 'http://facebook.com/spectatornews', '/', 'https://snosites.com/why-sno/', 'http://snosites.com', 'https://www.spectatornews.com/wp-login.php', '#top', '/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/category/currents/', 'https://www.spectatornews.com/category/sports/', 'https://www.spectatornews.com/category/opinion/', 'https://www.spectatornews.com/category/multimedia-2/', 'https://www.spectatornews.com/ads/banner-advertise-with-the-spectator/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/category/currents/', 'https://www.spectatornews.com/category/sports/', 'https://www.spectatornews.com/category/opinion/', 'https://www.spectatornews.com/category/multimedia-2/', '/', 'https://www.spectatornews.com/about/', 'https://www.spectatornews.com/about/editorial-policy/', 'https://www.spectatornews.com/about/correction-policy/', 'https://www.spectatornews.com/about/bylaws/', 'https://www.spectatornews.com/advertise/', 'https://www.spectatornews.com/contact/', 'https://www.spectatornews.com/staff/', 'https://www.spectatornews.com/submit-a-letter/', 'https://www.spectatornews.com/submit-a-news-tip/', '/', 'https://www.spectatornews.com', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/category/currents/', 'https://www.spectatornews.com/category/sports/', 'https://www.spectatornews.com/category/opinion/', 'https://www.spectatornews.com/category/multimedia-2/', '/', 'https://www.spectatornews.com/feed/rss/', '#', 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ', 'https://www.snapchat.com/add/spectator news', 'https://www.instagram.com/spectatornews/', 'http://twitter.com/spectatornews', 'http://facebook.com/spectatornews', 'https://www.spectatornews.com/campus-news/2004/05/06/english-fest-draws-speakers-bands/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/campus-news/2004/05/03/burgers-on-the-grill-bands-on-the-scene/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/showcase/2004/04/29/hempfest-celebrates-its-10th-year-with-11-bands/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/04/29/pat-mcgee-band-rocks-mad-town/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/04/22/leinenkugels-battle-of-the-bands/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/04/08/on-the-music-scene-band-makes-mondays-better/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2004/03/18/on-the-music-scene-band-carries-on-duluozs-work/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2003/10/09/jamband-grooving-to-eau-claire/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2003/05/01/joepalooza-set-with-5-bands-one-drummer/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/campus-news/2003/05/01/hempfest-features-nine-bands/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/showcase/2003/02/17/houston-based-band-reaching-out-to-college-students-on-tour/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/showcase/2003/02/06/minneapolis-band-trips-into-eau-claire/', 'https://www.spectatornews.com/category/showcase/', 'https://www.spectatornews.com/page/5/?s=band', 'https://www.spectatornews.com/?s=band', 'https://www.spectatornews.com/page/2/?s=band', 'https://www.spectatornews.com/page/3/?s=band', 'https://www.spectatornews.com/page/4/?s=band', 'https://www.spectatornews.com/page/5/?s=band', 'https://www.spectatornews.com/page/7/?s=band', 'https://www.spectatornews.com/page/8/?s=band', 'https://www.spectatornews.com/page/9/?s=band', 'https://www.spectatornews.com/page/127/?s=band', 'https://www.spectatornews.com/page/7/?s=band', 'https://www.spectatornews.com', 'https://www.spectatornews.com/feed/rss/', '#', 'https://www.youtube.com/channel/UC1SM8q3lk_fQS1KuY77bDgQ', 'https://www.snapchat.com/add/spectator news', 'https://www.instagram.com/spectatornews/', 'http://twitter.com/spectatornews', 'http://facebook.com/spectatornews', '/', 'https://snosites.com/why-sno/', 'http://snosites.com', 'https://www.spectatornews.com/wp-login.php', '#top', '/', 'https://www.spectatornews.com/category/campus-news/', 'https://www.spectatornews.com/category/currents/', 'https://www.spectatornews.com/category/sports/', 'https://www.spectatornews.com/category/opinion/', 'https://www.spectatornews.com/category/multimedia-2/']

编辑以处理 404 错误：

您列表中的某些链接可能不可用。一种选择是使用 try-except 块来处理这些并处理剩余的链接

所以最终的代码是

from bs4 import BeautifulSoup
import urllib.request
listoflinks=['https://www.spectatornews.com/page/6/?s=band','https://www.spectatornews.com/page/6/?s=band','https://www.spectatornews.com/page/100099?s=band','http://sdfgsdjhgfjsgdhfgsj.com']
getarticles = []
for i in listoflinks:
    req = urllib.request.Request(
    i,
    headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
    )
    try:
        resp= urllib.request.urlopen(req)
    except urllib.error.HTTPError as e:
        if e.code == 404:
            print("Unavailable link",i," skipping---")
        else:
            raise e
    soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'),features="lxml")
    for link in soup.find_all('a', href=True):
        getarticles.append(link['href'])
print(getarticles)