我想刮取无限滚动实现的页面的所有数据。下面的python代码可以工作。
for i in range(100):
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
这意味着每当我向下滚动到底部时，我都需要等待5秒，这通常足以让页面完成加载新生成的内容。但是，这可能并不省时。页面可能在5秒内完成新内容的加载。如何在每次向下滚动时检测页面是否完成了新内容的加载?如果我能检测到这一点，一旦我知道页面完成加载，我就可以再次向下滚动以查看更多内容。这样更节省时间。
当前回答
这里我使用了一个相当简单的形式:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("url")
searchTxt=''
while not searchTxt:
try:
searchTxt=browser.find_element_by_name('NAME OF ELEMENT')
searchTxt.send_keys("USERNAME")
except:continue
其他回答
我挣扎了一点，让这个工作，因为它没有为我工作的预期。任何还在努力让它工作的人，可以检查一下。
我想等待一个元素出现在网页上，然后再继续我的操作。
我们可以使用WebDriverWait(driver, 10,1).until()，但catch是until()期望一个函数，它可以执行一段时间的超时提供(在我们的情况下是10)每1秒。所以保持它如下对我有用。
element_found = wait_for_element.until(lambda x: x.find_element_by_class_name("MY_ELEMENT_CLASS_NAME").is_displayed())
下面是until()在幕后所做的事情
def until(self, method, message=''):
"""Calls the method provided with the driver as an argument until the \
return value is not False."""
screen = None
stacktrace = None
end_time = time.time() + self._timeout
while True:
try:
value = method(self._driver)
if value:
return value
except self._ignored_exceptions as exc:
screen = getattr(exc, 'screen', None)
stacktrace = getattr(exc, 'stacktrace', None)
time.sleep(self._poll)
if time.time() > end_time:
break
raise TimeoutException(message, screen, stacktrace)
正如David Cullen的回答中提到的，我总是看到这样的建议:
element_present = EC.presence_of_element_located((By.ID, 'element_id'))
WebDriverWait(driver, timeout).until(element_present)
对于我来说，很难找到所有可以与By一起使用的定位器，所以我认为在这里提供列表会很有用。 根据Ryan Mitchell的Web Scraping with Python:
ID Used in the example; finds elements by their HTML id attribute CLASS_NAME Used to find elements by their HTML class attribute. Why is this function CLASS_NAME not simply CLASS? Using the form object.CLASS would create problems for Selenium's Java library, where .class is a reserved method. In order to keep the Selenium syntax consistent between different languages, CLASS_NAME was used instead. CSS_SELECTOR Finds elements by their class, id, or tag name, using the #idName, .className, tagName convention. LINK_TEXT Finds HTML tags by the text they contain. For example, a link that says "Next" can be selected using (By.LINK_TEXT, "Next"). PARTIAL_LINK_TEXT Similar to LINK_TEXT, but matches on a partial string. NAME Finds HTML tags by their name attribute. This is handy for HTML forms. TAG_NAME Finds HTML tags by their tag name. XPATH Uses an XPath expression ... to select matching elements.
另外，您可以检查DOM是否没有更多的修改，而不是向下滚动100次(在页面底部是AJAX惰性加载的情况下)
def scrollDown(driver, value):
driver.execute_script("window.scrollBy(0,"+str(value)+")")
# Scroll down the page
def scrollDownAllTheWay(driver):
old_page = driver.page_source
while True:
logging.debug("Scrolling loop")
for i in range(2):
scrollDown(driver, 500)
time.sleep(2)
new_page = driver.page_source
if new_page != old_page:
old_page = new_page
else:
break
return True
试图将find_element_by_id传递给构造函数的presence_of_element_locate(如已接受的答案所示)会引发NoSuchElementException异常。我不得不在fragles的评论中使用语法:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get('url')
timeout = 5
try:
element_present = EC.presence_of_element_located((By.ID, 'element_id'))
WebDriverWait(driver, timeout).until(element_present)
except TimeoutException:
print "Timed out waiting for page to load"
这与文档中的示例相匹配。这里是By文档的链接。
回答得很好。等待XPATH的快速示例。
# wait for sizes to load - 2s timeout
try:
WebDriverWait(driver, 2).until(expected_conditions.presence_of_element_located(
(By.XPATH, "//div[@id='stockSizes']//a")))
except TimeoutException:
pass
