如何使用 Selenium + Python 加载动态内容

Question

在 Quora 用户个人资料页面上，我想通过单击“更多”来显示每个用户答案的​​整个文本，但我无法获得“更多”元素列表，而是得到一个空列表，怎么办？

这是我的代码：

more_buttons = browser.find_elements_by_name('more')
for more_button in more_buttons:
        more_button.click()

以下是用户配置文件的示例： https://www.quora.com/profile/Ashley-Isabella-3

Answer 1

那是因为您使用了错误的 CSS 选择器。 尝试：

more_buttons = browser.find_elements_by_class_name('ui_qtext_more_link')

您可以通过在普通浏览器中右键单击您的元素并单击“检查”来找到正确的属性。

Answer 2

在 Quora 用户个人资料页面上单击文本为 (more) 的链接并检索用户答案，您可以使用以下解决方案：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  driver=webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
  driver.get("https://www.quora.com/profile/Ashley-Isabella-3")
  more_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.ui_qtext_more_link")))
  for more_button in more_buttons:
      more_button.click()
  my_text_elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.ui_qtext_expanded>span.ui_qtext_rendered_qtext>p>b")))
  for my_text_element in my_text_elements:
      print(my_text_element.text)
  

• 控制台输出：

  11 mistakes you make when trying to lose belly fat
  You could be doing the completely wrong exercise.
  1. You're not getting enough sleep
  2. You’re doing the wrong type of workout
  3. You’re consuming too much sugar
  4. You're not eating enough protein
  5. You’re feeling stressed or anxious
  6. You’re expecting a quick fix
  7. You’re not tracking your progress
  8. You’re crash-dieting
  9. You’re doing too much
  10. You haven't got the right exercise balance
  11. You’ve lost motivation
  
  9 Ways To Burn Fat Fast
  Implement these 9 fat-burning tips that use exercise and diet and watch the body fat melt like the butter you're no longer using.
  1. Stay Off The Scale
  2. Reduce Your Calories Gradually
  3. Vary Your Caloric Intake
  4. Train With Weights
  5. Do High-Intensity Intervals (HIIT)
  6. Eat More Fat
  7. Cut Carbs
  8. Increase Your Protein
  9. Eat 6 Smaller Meals Per Day, Not 2-3 Feasts
  Bonus:
  
  Researchers Explain 6 Reasons Why A Man Falls In Love
  1. HE’S CAPTIVATED BY THE WHOLE PACKAGE
  2. HE FEELS HE CAN MAKE HER HAPPY
  3. WHEN SHE’S OPEN TO LOVE
  4. THE WAY HE FEELS AROUND HER
  5. SHE SAYS “YES” TO LIFE
  6. HE FEELS PURPOSE AND PASSION
  Must Watch: Secret
  For You in Under 20 Minutes...
  These are the 7 weirdest things men find attractive in women
  Who knew anyone would ever find stubbornness attractive in a woman?
  1. Weakness
  2. Applying makeup
  3. Stubbornness
  4. Stretch marks
  5. Wearing glasses
  6. No make up
  7. Clumsiness
  Secret
  For You in Under 20 Minutes...
  5 surprising ways to get great skin
  There are so many bad ideas out there about how to achieve great skin. Here are five ways to get on the right track, regardless of 'the rules.'
  1. You don't need to wash your face with hot water to get it clean.
  2. Oil is good for your skin, not bad.
  3. You don't need to scrub to exfoliate.
  4. What matters most for healthy skin is not what you put on it, but what you eat.
  5. Chocolate doesn't cause acne, but bread and pasta might
  Make Your Skin Naturally Glow
  1. Skip the long, steamy showers and opt for shorter, cooler sprays.
  2. Check the dryness of your skin by scratching a small area on your arm or leg with your fingernail.
  3. Treat your neck and chest like an extension of your face.
  4. Run a humidifier every night in the winter to moisturize the air in your bedroom.
  
  12 easy tips for glowing skin
  1: Discover these glowing skin home remedies:
  2: Sweat it out for healthy skin
  3: Practice Yoga for Face Glow
  4: Know who you are!
  5: Follow a natural diet for glowing skin
  6: Indulge yourself…weekly
  7: Make Sudarshan Kriya your beauty mantra
  8: Meditate everyday to get Glowing Skin
  9: Silence is truly golden
  10: Save your mind at any cost
  11: 18 till I …why not?
  12: Flex those facial muscles
  15 Reasons Yoga Is Better Than The Gym
  1. Yoga benefits the mind, body and spirit.
  2. Yoga benefits your full body—externally and internally.
  3. Yoga teaches acceptance.
  4. Yoga is about focusing on yourself.
  5. Yoga will make you lean.
  6. Yoga is more efficient.
  7. You can do yoga anywhere.
  8. Yoga is kinder to the body.
  9. Yoga eases your aches and pains.
  10. Yoga helps you breathe easier.
  11. Yoga is calming.
  12. Yoga reduces stress.
  13. Everyone can practice yoga.
  14. Yoga improves concentration.
  15. Yogis rock.
  
  9 Ways to Tell If You’re Exercising Enough
  The sign: You never feel sore or tired the day after a workout
  The sign: You're not losing any weight
  The sign: You barely break a sweat
  The sign: The last rep feels hard, but you could probably do another
  The sign: You can chat with your friend throughout your workout
  The sign: You're not getting any more flexible in your yoga classes
  The sign: You run several days a week but aren't getting faster
  The sign: You're making strides in boot camp but are always stiff and un-flexible
  The sign: You take the same class regularly but aren't getting better at the exercises
  
  How To Build Muscle Mass After 50
  Exercise Plan
  Strength-Training Exercises
  Weight, Reps and Sets
  Aerobics Workouts
  
  Savannah Guthrie Says The Keto Diet Is Making Her Mind Sharp—And Her Body Sluggish
  She has zero desire to work out.
  
  How to Lose Weight in 10 Days at Home
  Flight of Stairs / Treadmill
  Lots of Water Fresh –
  Fruits and Vegetables –
  Low-Fat Foods Fish / Meat –
  Avoid Junk Food –
  Moderate Exercise -
  Drink lots of water -
  Day 1
  Day 2 and 3
  Day 4 and 5
  Day 6 and 7
  Day 8 and 9
  Day 10
  
  What are natural ways to make my face look younger?
  Smoothing Exfoliator
  Tightening Toner
  Plumping Moisturizing Facial Mask
  
  10 Ways to Lose 25 Pounds
  Follow these steps and weight loss won’t seem impossible.
  1. Do some detective work.
  2. Get moving.
  3. Plan ahead.
  4. Fill up on fruits and veggies.
  5. Don’t drink your calories.
  6. Don’t do anything drastic.
  7. Set smart calorie targets.
  8. Keep track.
  9. Do not make weight the only measure of success.
  10. Just practice; don’t try to be perfect.
  

Answer 3

实际上，您在关注链接（当您单击时），因此您可以收集链接，然后使用 .get 循环。使用return [...document.querySelectorAll ]语法

from selenium import webdriver

d = webdriver.Chrome()
d.get("https://www.quora.com/profile/Ashley-Isabella-3")
links = d.execute_script("return [...document.querySelectorAll('.ui_qtext_more_link')].map(link => link['href'])")
info = []
for link in links:
    d.get(link)
    pageInfo = d.execute_script("return [...document.querySelectorAll('.ui_qtext_para')].map(link => link.innerText)")
    info.append(pageInfo)
print(info)