你也可以使用webdriver执行javascript。

from selenium import webdriver

driver = webdriver.Firefox()
driver.get(url)
driver.execute_script('document.title')

或者将值存储在变量中

result = driver.execute_script('var text = document.title ; return text')

这似乎是一个很好的解决方案，从一个伟大的博客文章

import sys  
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 

#Take this class for granted.Just use result of rendering.
class Render(QWebPage):  
  def __init__(self, url):  
    self.app = QApplication(sys.argv)  
    QWebPage.__init__(self)  
    self.loadFinished.connect(self._loadFinished)  
    self.mainFrame().load(QUrl(url))  
    self.app.exec_()  

  def _loadFinished(self, result):  
    self.frame = self.mainFrame()  
    self.app.quit()  

url = 'http://pycoders.com/archive/'  
r = Render(url)  
result = r.frame.toHtml()
# This step is important.Converting QString to Ascii for lxml to process

# The following returns an lxml element tree
archive_links = html.fromstring(str(result.toAscii()))
print archive_links

# The following returns an array containing the URLs
raw_links = archive_links.xpath('//div[@class="campaign"]/a/@href')
print raw_links

也许硒可以做到。

from selenium import webdriver
import time

driver = webdriver.Firefox()
driver.get(url)
time.sleep(5)
htmlSource = driver.page_source

如前所述，Selenium是呈现JavaScript结果的好选择:

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True
browser = Firefox(executable_path="/usr/local/bin/geckodriver", options=options)

url = "https://www.example.com"
browser.get(url)

gazpacho是一个非常容易解析渲染html的库:

from gazpacho import Soup

soup = Soup(browser.page_source)
soup.find("a").attrs['href']

我们没有得到正确的结果，因为任何javascript生成的内容都需要在DOM上呈现。当我们获取一个HTML页面时，我们获取初始的，未经javascript修改的DOM。

因此，我们需要在抓取页面之前呈现javascript内容。

由于selenium已经在本线程中多次提到(有时也提到了它的速度有多慢)，我将列出其他两种可能的解决方案。

解决方案1:这是一个关于如何使用Scrapy抓取javascript生成内容的非常好的教程，我们将遵循这一点。

我们需要:

Docker installed in our machine. This is a plus over other solutions until this point, as it utilizes an OS-independent platform. Install Splash following the instruction listed for our corresponding OS.Quoting from splash documentation: Splash is a javascript rendering service. It’s a lightweight web browser with an HTTP API, implemented in Python 3 using Twisted and QT5. Essentially we are going to use Splash to render Javascript generated content. Run the splash server: sudo docker run -p 8050:8050 scrapinghub/splash. Install the scrapy-splash plugin: pip install scrapy-splash Assuming that we already have a Scrapy project created (if not, let's make one), we will follow the guide and update the settings.py: Then go to your scrapy project’s settings.py and set these middlewares: DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware': 725, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810, } The URL of the Splash server(if you’re using Win or OSX this should be the URL of the docker machine: How to get a Docker container's IP address from the host?): SPLASH_URL = 'http://localhost:8050' And finally you need to set these values too: DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter' HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' Finally, we can use a SplashRequest: In a normal spider you have Request objects which you can use to open URLs. If the page you want to open contains JS generated data you have to use SplashRequest(or SplashFormRequest) to render the page. Here’s a simple example: class MySpider(scrapy.Spider): name = "jsscraper" start_urls = ["http://quotes.toscrape.com/js/"] def start_requests(self): for url in self.start_urls: yield SplashRequest( url=url, callback=self.parse, endpoint='render.html' ) def parse(self, response): for q in response.css("div.quote"): quote = QuoteItem() quote["author"] = q.css(".author::text").extract_first() quote["quote"] = q.css(".text::text").extract_first() yield quote SplashRequest renders the URL as html and returns the response which you can use in the callback(parse) method.

解决方案2:我们暂且称之为实验性的(2018年5月)…… 此解决方案仅适用于Python版本3.6(目前)。

你知道请求模块吗(谁不知道呢)? 现在它有了一个网络爬行的小兄弟:requests-HTML:

这个库旨在使解析HTML(例如抓取网页)尽可能简单和直观。

安装请求-html: pipenv 对页面的url进行请求: 导入HTMLSession 会话= HTMLSession() R = session.get(a_page_url) 渲染响应以获得Javascript生成的比特: r.html.render ()

最后，该模块似乎提供了抓取功能。 或者，我们也可以尝试使用我们刚刚渲染的r.html对象来使用BeautifulSoup。

Pyppeteer

你可以考虑Pyppeteer，它是Chrome/Chromium驱动程序前端的Python移植版本。

下面是一个简单的例子，展示了如何使用pyppeterer动态地访问被注入到页面中的数据:

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch({"headless": True})
    [page] = await browser.pages()

    # normally, you go to a live site...
    #await page.goto("http://www.example.com")
    # but for this example, just set the HTML directly:
    await page.setContent("""
    <body>
    <script>
    // inject content dynamically with JS, not part of the static HTML!
    document.body.innerHTML = `<p>hello world</p>`; 
    </script>
    </body>
    """)
    print(await page.content()) # shows that the `<p>` was inserted

    # evaluate a JS expression in browser context and scrape the data
    expr = "document.querySelector('p').textContent"
    print(await page.evaluate(expr, force_expr=True)) # => hello world

    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

请参阅pyppeterer的参考文档。