【问题标题】:How to read the header content of a webpage in Django?如何在 Django 中读取网页的标题内容?
【发布时间】:2022-09-28 04:04:07
【问题描述】:

我在 Django 和 bs4 中创建了一个搜索引擎,它从 Ask.com 搜索引擎中抓取搜索结果。我想当 Django 从 Ask 获取搜索结果时,它会检查 X-Frame-Options 标头的值,以便根据条件的结果为我的 notAccept 布尔值提供一个值。

我从 Django 文档的this pagethis other page 中获得灵感,在测试了建议的答案后,我修改了我的代码,如下所示:

for result in result_listings:
                result_title = result.find(class_=\'PartialSearchResults-item-title\').text
                result_url = result.find(\'a\').get(\'href\')
                result_desc = result.find(class_=\'PartialSearchResults-item-abstract\').text

                res = requests.get(result_url)
              

                #for header in final_result[1]:
                response = res.headers[\'content-type\':\'X-Frame-Options\'] #the error is generated here
                if response in [\"DENY\", \"SAMEORIGIN\"]:
                    head = True
                    notAccept = bool(head)

但是当我测试时,我在终端中出现以下错误:

    Internal Server Error: /search
Traceback (most recent call last):
  File \"C:\\Python310\\lib\\site-packages\\django\\core\\handlers\\exception.py\", line 55, in inner
    response = get_response(request)
  File \"C:\\Python310\\lib\\site-packages\\django\\core\\handlers\\base.py\", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File \"C:\\Users\\user\\Documents\\AAprojects\\Whelpsgroups1\\searchEngine\\search\\views.py\", line 32, in search
    response = res.headers[\'content-type\':\'X-Frame-Options\']
  File \"C:\\Python310\\lib\\site-packages\\requests\\structures.py\", line 54, in __getitem__
    return self._store[key.lower()][1]
AttributeError: \'slice\' object has no attribute \'lower\'
[26/Sep/2022 22:57:24] \"GET /search?csrfmiddlewaretoken=1m8mRf9JWoHvzps2AemMyA7Wlb76PVzQ5UzuEtfH1p3PzwmZfqLlBHTkCvIDlot6&search=moto HTTP/1.1\" 500 93598

此错误与代码中指定的以下行有关。

response = res.headers[\'content-type\':\'X-Frame-Options\'] #the error is generated here

我像这样修改了这一行:

response = res.headers[\'X-Frame-Options\']

但现在我收到以下错误:

Traceback (most recent call last):
  File \"C:\\Python310\\lib\\site-packages\\django\\core\\handlers\\exception.py\", line 55, in inner
    response = get_response(request)
  File \"C:\\Python310\\lib\\site-packages\\django\\core\\handlers\\base.py\", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File \"C:\\Users\\user\\Documents\\AAprojects\\Whelpsgroups1\\searchEngine\\search\\views.py\", line 32, in search
    response = res.headers[\'X-Frame-Options\'] #the error is generated here
  File \"C:\\Python310\\lib\\site-packages\\requests\\structures.py\", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: \'x-frame-options\'

我在this page 上寻找解决方案,但找不到太多。

我不知道如何解决这个问题。我必须承认,我不太擅长处理标题。谢谢!

  • 我不明白你在哪里搜索这些值。你应该搜索res.headers
  • 在 Python 中,if 使用 or 代替 |and 代替 &not 代替 !
  • 始终将有问题的完整错误消息(从单词 \"Traceback\" 开始)(不在 cmets 中)作为文本(不是屏幕截图,不链接到外部门户)。完整的错误/回溯中还有其他有用的信息。
  • 如果您有不同数量的(),您可能会收到有关( 的消息

标签: python django http-headers


【解决方案1】:

这比 Django 更像是一个 Python 请求问题。根据我有限的知识,我不相信你能够只需查看链接即可获取页面的标题信息。您需要实际发送一个 GET 请求,因为它位于请求的标头中:

for l in links:
    response = requests.get(l)
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        head = True
        notAccept = bool(head)
    else:
        notAccept = bool(False)

我找不到任何事物关于CSP_FRAME_ANCESTORS tho..

希望你能找到一些有用的东西..至少现在你知道在这个话题上搜索python requests {x}


编辑

我将解释您添加的错误,无效索引:


# Final Result is an Array Filled with Tuples
#   OR just think of it as an Array filled with Arrays
# ---

# This would be the result on the first loop:

final_result = [
    (result_title0, result_url0, result_desc0), # index 0
    ]

# You used:
final_result[1] # => Undefined

# Correct Way:
# ---

# Grab first item in Array:
final_result[0] # => (result_title0, result_url0, result_desc0)

# Grab first item in Array + and then 2nd item in list:
final_result[0][1] # => result_url0

# Next Issue
# ---

# But you will run into this issue / always grabbing the first item
final_result = [
    (result_title0, result_url0, result_desc0), # index 0
    (result_title1, result_url1, result_desc1), # index 1
    ]


final_result[0][1] # => result_url0 **Wrong!**

# -1 should be used instead // (Last Item in list)
final_result[-1][1] # => result_url1 **Correct!**

# ^ Actual solution
# ---

但是因为您已经将 result_url 作为循环中的变量,您可能会在 GET 中使用它而不是尝试从该嵌套数组中获取它

for result in result_listings:
    result_title = result.find(class_='PartialSearchResults-item-title').text
    result_url = result.find('a').get('href')
    result_desc = result.find(class_='PartialSearchResults-item-abstract').text
    final_result.append((result_title, result_url, result_desc))

    # Ping URL found here: result.find('a').get('href')
    response = requests.get(result_url)

    # Check for header information in the response
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        # head = True
        notAccept = True
    else:
        notAccept = False

你不妨等到最后将该元组添加到最终结果列表中——甚至可以使用字典

等到结束

for result in result_listings:
    result_title = result.find(class_='PartialSearchResults-item-title').text
    result_url = result.find('a').get('href')
    result_desc = result.find(class_='PartialSearchResults-item-abstract').text

    # Ping URL found here: result.find('a').get('href')
    response = requests.get(result_url)

    # Check for header information in the response
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        # head = True
        notAccept = True
    else:
        notAccept = False

    # Add here! Last second.
    final_result.append((result_title, result_url, result_desc, notAccept))

等到结束+字典

for result in result_listings:
    result_title = result.find(class_='PartialSearchResults-item-title').text
    result_url = result.find('a').get('href')
    result_desc = result.find(class_='PartialSearchResults-item-abstract').text

    # Ping URL found here: result.find('a').get('href')
    response = requests.get(result_url)

    # Check for header information in the response
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        # head = True
        notAccept = True
    else:
        notAccept = False

    # Dict makes Code a little more readable last on when you use this data
    final_result.append({
        'title':result_title,
        'url': result_url,
        'desc': result_desc,
        'x-frame': notAccept
        })

【讨论】:

  • 我测试了你的解决方案,但我有一个问题。我的问题中有更多细节。谢谢 !
  • 我已经为您遇到的那个错误添加了一个编辑。我还解释了您会遇到的下一个错误 - 所以请确保您到达该代码块的底部!
  • 抱歉回复晚了,我有一些问题需要解决。感谢您的解决方案,它帮助我解决了循环问题,但我仍然遇到错误。我已在我编辑的问题中详细说明了该错误,并想澄清我的代码已更改。再次感谢您,祝您有美好的一天!
  • res.headers['content-type':'X-Frame-Options'] 中的 : 无效。你不能在字典中抓取两个这样的键,要么是response = res.headers['content-type'] 要么是response = res.headers['X-Frame-Options'] ..你可以这样做:content_type, x_frame_options = res.headers['content-type'], res.headers['X-Frame-Options'],但它必须是两个变量
猜你喜欢
  • 1970-01-01
  • 2010-12-23
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2011-10-03
  • 2015-06-01
  • 2013-05-01
  • 1970-01-01
相关资源
最近更新 更多