Foreach 循环使用 BeautifulSoup/Mechanize/Python 获取下一页链接

Question

我有一个观点：

def Processinitialscan(request):
    EnteredDomain = request.GET.get('domainNm')

    #get raw output
    getDomainLinksFromGoo = settings.GOOGLE_BASEURL_FOR_HARVEST+settings.GOO_RESULT_DOMAIN_QUERIED+EnteredDomain
    rawGatheredGooOutput = mechanizeBrowser.open(getDomainLinksFromGoo)

    beautifulSoupObj = BeautifulSoup(mechanizeBrowser.response().read()) #read the raw response
    getFirstPageLinks = beautifulSoupObj.find_all('cite') #get first page of urls

    pattern = re.compile('^.*start=')   #set regex to search on - find anything like: " <domain and path here>start= "
    getRemainingPageUrls = beautifulSoupObj.find_all('a',attrs={'class': 'fl', 'href': pattern})

    NumberOfUrlsFound = len(getRemainingPageUrls)

    MaxUrlsToGather = ((NumberOfUrlsFound*10)+settings.GOOGLE_RESULT_AMT_ACCOUNT_FOR_PAGE_1) # +10 because 10 represents the urls on the first page

    url_data = UrlData(NumberOfUrlsFound, pattern) 
    #return HttpResponse(MaxUrlsToGather)

    return render(request, 'VA/scan/process_scan.html', {
        'url_data':url_data,'EnteredDomain':EnteredDomain,'getDomainLinksFromGoo':getDomainLinksFromGoo,
        'getRemainingPageUrls' : getRemainingPageUrls, 'NumberOfUrlsFound':NumberOfUrlsFound,
        'getFirstPageLinks' : getFirstPageLinks, 'MaxUrlsToGather' : MaxUrlsToGather
    })

和一个模板：

{% block block_containercontent %}
    {% autoescape on %}
    <h1>{{ EnteredDomain }}</h1>
<strong>url used: </strong>{{ getDomainLinksFromGoo }}<br />
<hr>
<br>
<strong>first page of links</strong> {{ getFirstPageLinks }}
<hr>
<br><strong>number of "next" links</strong> {{ NumberOfUrlsFound }}
<hr>
<br>
<strong>remaining urls:</strong> {{ getRemainingPageUrls }}
    {% if url_data.num_of_urls > 1 %}
    {% for url in url_data.url_list %}
        {{ url }}
    {% endfor %}
{% endif %}
    {% endautoescape %}

{% endblock block_containercontent %}

此模板输出：

url used: https://www.google.com/search?q=site%3Aasite.com


first page of links [<cite>www.google.com/webmasters/</cite>, <cite>www.asite.com</cite>, <cite>www.asite.com/blog/</cite>, <cite>www.asite.com/blog/projects/</cite>, <cite>www.asite.com/blog/category/internet/</cite>, <cite>www.asite.com/blog/category/goals/</cite>, <cite>www.asite.com/blog/category/uncategorized/</cite>, <cite>www.asite.com/blog/why-i-left-facebook/2013/01/</cite>, <cite>www.asite.com/blog/category/startups-2/</cite>, <cite>www.asite.com/blog/category/goals/</cite>, <cite>www.asite.com/blog/category/internet/</cite>]


number of "next" links 2

我的问题：如何在模板内的循环中利用NumberOfUrlsFound 生成链接，例如：/search?q=site:entereddomain.com&start=10、/search?q=site:entereddomain.com&start=20，然后根据 NumberOfUrlsFound 的值使用 beautifulsoup 跟踪链接。所以如果 NumberOfUrlsFound = 2，则应该生成 urlssearch?q=site:asite.com&start=10,search?q=site:asite.com&start=20，此外：

（puesdo 代码..）：

if(NumberOfUrlsFound > 1)
    foreach(NumberOfUrlsFound)
        # generate url with start=n+10  
        ## asite.com?/search?start=10  
        ## Then ...
        ## asite.com?/search?start=20
        ## and so on ..
        # where n represents the previous number
        # this n number is determined by `NumberOfUrlsFound` which might have a value of 2 for example
        # this value of 2 represents a max value of start=20 value to generate urls on.

Answer 1

您可能可以创建一个数据对象来表示要在模板中显示的数据。

class UrlData(object):
    def __init__(self, num_of_urls, url_pattern):
        self.num_of_urls = num_of_urls
        self.url_pattern = url_pattern

    def url_list(self):
        # Returns a list of strings that represent the urls you want based on num_of_urls
        # e.g. asite.com/?search?start=10
        urls = []
        for i in xrange(self.num_of_urls):
            urls.append(self.url_pattern + 'start=' + str((i + 1) * 10))
        return urls

在你的views.py中

# Create a UrlData object from NumberOfUrlsFound and a url_pattern
# url_pattern being the asite.com/?search?start=
url_data = UrlData(NumberOfUrlsFound, getDomainLinksFromGoogle) 

return render(request, template, {'url_data': url_data, ...})

只需使用视图函数中的数据创建对象并将对象传递给模板。

在您的模板中，您可以执行以下操作：

# Mirroring your check
{% if url_data.num_of_urls > 1 %} 
    # We'll iterate through the url_list created from the function defined in UrlData
    {% for url in url_data.url_list %}
         {{ url }} # asite.com/?search...
    {% endfor %}
{% endif %}

在模板中，当你调用url_data.url_list时，它会运行UrlData中的函数