找到的元素列表中缺少 Webdriver HTML

Question

我正在检索一个元素列表，然后循环查找每个元素中的一个元素。前 15 个项目正常工作，但之后的每个项目（总共 217 个）我都遇到了问题。我发现这个尝试除外：

driver.get(url)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
sleep(5)

links = []
for card in driver.find_elements_by_class_name('product-card-grid__item-1-2'):
    try:
        link = card.find_element_by_tag_name('a').get_attribute("href")
        links.append(link)
    except:
        print(len(links))
        print(card.get_attribute('innerHTML'))

重复输出：

15
<div class="product-card-grid__inner"><div class="product-card"></div></div>
15
<div class="product-card-grid__inner"><div class="product-card"></div></div>
15
<div class="product-card-grid__inner"><div class="product-card"></div></div>
15
<div class="product-card-grid__inner"><div class="product-card"></div></div>

查看页面的 HTML，我检索到的每个 div 元素都具有相同的格式

<div>
    <div>
        <div>
            <div>
                <div>
                    <a>

但是在使用 webdriver 时，这在第 15 项之后会发生变化。我分配了检索到的元素并切片以查看第 15 项之后发生的更改。

cards = driver.find_elements_by_class_name('product-card-grid__item-1-2')
cards_slice = cards[0:20]
for card in cards_slice:
    print(card.get_attribute('innerHTML'))

输出：

...
<div class="product-card-grid__inner"><div class="product-card"><div id="product680985002"><div class="product-card__image-wrapper"><a href="https://www.gap.com/browse/product.do?pid=680985002&amp;cid=1173813&amp;pcid=1149858&amp;vid=1&amp;nav=hamnav%3AMen%3A%3AResponsibly-Made%20Shop&amp;grid=pds_13_217_1" target="_self" aria-hidden="false" class="css-0"><img alt="The Gen Good T-Shirt" src="https://www4.assets-gap.com/webcontent/0020/057/840/cn20057840.jpg" class="product-card__image" productid="680985002" productname="The Gen Good T-Shirt"><div style="box-sizing: border-box;"><div class="imgOverlayWrapper"><img class="imgInnerWrapper" src="/Asset_Archive/AllBrands/Iconography/gen_good_overlay_icon.svg" alt="overlay image"></div></div></a></div></div><a class="product-card__link" href="https://www.gap.com/browse/product.do?pid=680985002&amp;cid=1173813&amp;pcid=1149858&amp;vid=1&amp;nav=hamnav%3AMen%3A%3AResponsibly-Made%20Shop&amp;grid=pds_13_217_1"><div class="product-card__name">The Gen Good T-Shirt</div></a><div class="product-card__body"><div class="product-card-price"><div><span><span class="">$19.00</span></span></div></div><div class="product-card__marketing-flag">This Style Does Good. Excluded From Promotions.</div></div></div></div>
<div class="product-card-grid__inner"><div class="product-card"><div id="product680985012"><div class="product-card__image-wrapper"><a href="https://www.gap.com/browse/product.do?pid=680985012&amp;cid=1173813&amp;pcid=1149858&amp;vid=1&amp;nav=hamnav%3AMen%3A%3AResponsibly-Made%20Shop&amp;grid=pds_14_217_1" target="_self" aria-hidden="false" class="css-0"><img alt="The Gen Good T-Shirt" src="https://www3.assets-gap.com/webcontent/0020/046/627/cn20046627.jpg" class="product-card__image" productid="680985012" productname="The Gen Good T-Shirt"><div style="box-sizing: border-box;"><div class="imgOverlayWrapper"><img class="imgInnerWrapper" src="/Asset_Archive/AllBrands/Iconography/gen_good_overlay_icon.svg" alt="overlay image"></div></div></a></div></div><a class="product-card__link" href="https://www.gap.com/browse/product.do?pid=680985012&amp;cid=1173813&amp;pcid=1149858&amp;vid=1&amp;nav=hamnav%3AMen%3A%3AResponsibly-Made%20Shop&amp;grid=pds_14_217_1"><div class="product-card__name">The Gen Good T-Shirt</div></a><div class="product-card__body"><div class="product-card-price"><div><span><span class="">$19.00</span></span></div></div><div class="product-card__marketing-flag">This Style Does Good. Excluded From Promotions.</div></div></div></div>
<div class="product-card-grid__inner"><div class="product-card"></div></div>
<div class="product-card-grid__inner"><div class="product-card"></div></div>
<div class="product-card-grid__inner"><div class="product-card"></div></div>
...

更新：再次尝试完全相同的第一个代码后，我现在只能从前 8 个元素中获取链接？

我会理解，因为我滚动到底部，所以只有底部元素正在加载，但为什么它只是第一个？

Answer 1

我在循环外加了一个len(links)，让它结束，发现异常后居然发现了一些额外的元素。查看检索到的链接告诉我，链接是从顶部检索的，与我的想法相反，有些是从底部元素检索的——但仍然不是大多数中间的。

问题很可能只是滚动到底部不会加载所有中间元素/保持它们加载，因为它是一个长页面。但是第一个元素保持加载。我想我需要慢慢滚动到底部才能加载所有内容。

确认。我通过一次向下滚动一点来让它工作，它完全加载了所有元素。