我想刮取无限滚动实现的页面的所有数据。下面的python代码可以工作。

for i in range(100):
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(5)

这意味着每当我向下滚动到底部时,我都需要等待5秒,这通常足以让页面完成加载新生成的内容。但是,这可能并不省时。页面可能在5秒内完成新内容的加载。如何在每次向下滚动时检测页面是否完成了新内容的加载?如果我能检测到这一点,一旦我知道页面完成加载,我就可以再次向下滚动以查看更多内容。这样更节省时间。


当前回答

正如David Cullen的回答中提到的,我总是看到这样的建议:

element_present = EC.presence_of_element_located((By.ID, 'element_id'))
WebDriverWait(driver, timeout).until(element_present)

对于我来说,很难找到所有可以与By一起使用的定位器,所以我认为在这里提供列表会很有用。 根据Ryan Mitchell的Web Scraping with Python:

ID Used in the example; finds elements by their HTML id attribute CLASS_NAME Used to find elements by their HTML class attribute. Why is this function CLASS_NAME not simply CLASS? Using the form object.CLASS would create problems for Selenium's Java library, where .class is a reserved method. In order to keep the Selenium syntax consistent between different languages, CLASS_NAME was used instead. CSS_SELECTOR Finds elements by their class, id, or tag name, using the #idName, .className, tagName convention. LINK_TEXT Finds HTML tags by the text they contain. For example, a link that says "Next" can be selected using (By.LINK_TEXT, "Next"). PARTIAL_LINK_TEXT Similar to LINK_TEXT, but matches on a partial string. NAME Finds HTML tags by their name attribute. This is handy for HTML forms. TAG_NAME Finds HTML tags by their tag name. XPATH Uses an XPath expression ... to select matching elements.

其他回答

你试过driver.implicitly_wait吗?它就像驱动程序的一个设置,所以你只在会话中调用它一次,它基本上告诉驱动程序等待给定的时间,直到每个命令都可以执行。

driver = webdriver.Chrome()
driver.implicitly_wait(10)

因此,如果您设置等待时间为10秒,它将尽快执行命令,等待10秒后才放弃。我在类似的滚动场景中使用过这个,所以我不明白为什么它在您的情况下不起作用。希望这对你有帮助。

为了能够修复这个答案,我必须添加新的文本。确保在implicitly_wait中使用小写“w”。

Selenium无法检测页面是否完全加载,但javascript可以。我建议你试试这个。

from selenium.webdriver.support.ui import WebDriverWait
WebDriverWait(driver, 100).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

这将执行javascript代码,而不是使用python,因为javascript可以检测页面何时完全加载,它将显示“完成”。这个代码的意思是在100秒内,继续尝试这个文档。readyState直到complete显示。

如果您试图滚动并找到页面上的所有项目。您可以考虑使用以下方法。这是其他人在这里提到的一些方法的组合。它帮我完成了任务:

while True:
    try:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        driver.implicitly_wait(30)
        time.sleep(4)
        elem1 = WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "element-name")))
        len_elem_1 = len(elem1)
        print(f"A list Length {len_elem_1}")
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        driver.implicitly_wait(30)
        time.sleep(4)
        elem2 = WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "element-name")))
        len_elem_2 = len(elem2)
        print(f"B list Length {len_elem_2}")
        if len_elem_1 == len_elem_2:
            print(f"final length = {len_elem_1}")
            break
    except TimeoutException:
            print("Loading took too much time!")
nono = driver.current_url
driver.find_element(By.XPATH,"//button[@value='Send']").click()
  while driver.current_url == nono:
      pass
print("page loaded.")

你可以通过这个函数简单地做到这一点:

def page_is_loading(driver):
    while True:
        x = driver.execute_script("return document.readyState")
        if x == "complete":
            return True
        else:
            yield False

当你想在页面加载完成后做一些事情时,你可以使用:

Driver = webdriver.Firefox(options=Options, executable_path='geckodriver.exe')
Driver.get("https://www.google.com/")

while not page_is_loading(Driver):
    continue

Driver.execute_script("alert('page is loaded')")