Selenium是抓取JS和Ajax内容的最佳工具。

查看这篇文章，了解如何使用Python从web中提取数据

$ pip install selenium

然后下载Chrome webdriver。

from selenium import webdriver

browser = webdriver.Chrome()

browser.get("https://www.python.org/")

nav = browser.find_element_by_id("mainnav")

print(nav.text)

容易,对吧?

也许硒可以做到。

from selenium import webdriver
import time

driver = webdriver.Firefox()
driver.get(url)
time.sleep(5)
htmlSource = driver.page_source

听起来好像你真正要找的数据可以通过主页面上的一些javascript调用的辅助URL访问。

虽然您可以尝试在服务器上运行javascript来处理这个问题，但一种更简单的方法可能是使用Firefox加载页面，并使用Charles或Firebug之类的工具来准确识别辅助URL。然后，您可以直接查询该URL以获得您感兴趣的数据。

把BeautifulSoup和Selenium混合在一起对我来说效果很好。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup as bs

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
    try:
        element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))) #waits 10 seconds until element is located. Can have other wait conditions  such as visibility_of_element_located or text_to_be_present_in_element

        html = driver.page_source
        soup = bs(html, "lxml")
        dynamic_text = soup.find_all("p", {"class":"class_name"}) #or other attributes, optional
    else:
        print("Couldnt locate element")

附注:你可以在这里找到更多的等待条件

简单快捷的解决方案:

我也遇到过同样的问题。我想刮一些数据是用JavaScript构建的。如果我只用BeautifulSoup从这个网站抓取文本，那么我就以文本中的标签结束。 我想渲染这个标签，并将从中抓取信息。 另外，我不想使用像Scrapy和selenium这样的笨重框架。

我发现请求模块的get方法接受url，它实际上呈现脚本标签。

例子:

import requests
custom_User_agent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
url = "https://www.abc.xyz/your/url"
response = requests.get(url, headers={"User-Agent": custom_User_agent})
html_text = response.text

这将呈现加载站点和呈现标签。

希望这将有助于作为快速和简单的解决方案，渲染网站加载脚本标签。

You'll want to use urllib, requests, beautifulSoup and selenium web driver in your script for different parts of the page, (to name a few). Sometimes you'll get what you need with just one of these modules. Sometimes you'll need two, three, or all of these modules. Sometimes you'll need to switch off the js on your browser. Sometimes you'll need header info in your script. No websites can be scraped the same way and no website can be scraped in the same way forever without having to modify your crawler, usually after a few months. But they can all be scraped! Where there's a will there's a way for sure. If you need scraped data continuously into the future just scrape everything you need and store it in .dat files with pickle. Just keep searching how to try what with these modules and copying and pasting your errors into the Google.