You'll want to use urllib, requests, beautifulSoup and selenium web driver in your script for different parts of the page, (to name a few). Sometimes you'll get what you need with just one of these modules. Sometimes you'll need two, three, or all of these modules. Sometimes you'll need to switch off the js on your browser. Sometimes you'll need header info in your script. No websites can be scraped the same way and no website can be scraped in the same way forever without having to modify your crawler, usually after a few months. But they can all be scraped! Where there's a will there's a way for sure. If you need scraped data continuously into the future just scrape everything you need and store it in .dat files with pickle. Just keep searching how to try what with these modules and copying and pasting your errors into the Google.

简单快捷的解决方案:

我也遇到过同样的问题。我想刮一些数据是用JavaScript构建的。如果我只用BeautifulSoup从这个网站抓取文本，那么我就以文本中的标签结束。 我想渲染这个标签，并将从中抓取信息。 另外，我不想使用像Scrapy和selenium这样的笨重框架。

我发现请求模块的get方法接受url，它实际上呈现脚本标签。

例子:

import requests
custom_User_agent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"
url = "https://www.abc.xyz/your/url"
response = requests.get(url, headers={"User-Agent": custom_User_agent})
html_text = response.text

这将呈现加载站点和呈现标签。

希望这将有助于作为快速和简单的解决方案，渲染网站加载脚本标签。

使用PyQt5

from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl
from PyQt5.QtWebEngineWidgets import QWebEnginePage
import sys
import bs4 as bs
import urllib.request


class Client(QWebEnginePage):
    def __init__(self,url):
        global app
        self.app = QApplication(sys.argv)
        QWebEnginePage.__init__(self)
        self.html = ""
        self.loadFinished.connect(self.on_load_finished)
        self.load(QUrl(url))
        self.app.exec_()

    def on_load_finished(self):
        self.html = self.toHtml(self.Callable)
        print("Load Finished")

    def Callable(self,data):
        self.html = data
        self.app.quit()

# url = ""
# client_response = Client(url)
# print(client_response.html)

如果你以前曾经使用过python的Requests模块，我最近发现开发人员创建了一个名为Requests- html的新模块，现在它也有呈现JavaScript的能力。

你也可以访问https://html.python-requests.org/来了解更多关于这个模块的信息，或者如果你只对呈现JavaScript感兴趣，那么你可以访问https://html.python-requests.org/?#javascript-support来直接学习如何使用该模块使用Python来呈现JavaScript。

从本质上讲，一旦你正确安装了Requests-HTML模块，下面的例子，在上面的链接中显示，展示了你如何使用这个模块来抓取一个网站，并呈现网站中包含的JavaScript:

from requests_html import HTMLSession
session = HTMLSession()

r = session.get('http://python-requests.org/')

r.html.render()

r.html.search('Python 2 will retire in only {months} months!')['months']

'<time>25</time>' #This is the result.

我最近从YouTube上的一个视频中了解到这一点。点击这里!观看YouTube上演示该模块如何工作的视频。

Selenium是抓取JS和Ajax内容的最佳工具。

查看这篇文章，了解如何使用Python从web中提取数据

$ pip install selenium

然后下载Chrome webdriver。

from selenium import webdriver

browser = webdriver.Chrome()

browser.get("https://www.python.org/")

nav = browser.find_element_by_id("mainnav")

print(nav.text)

容易,对吧?

把BeautifulSoup和Selenium混合在一起对我来说效果很好。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup as bs

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
    try:
        element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))) #waits 10 seconds until element is located. Can have other wait conditions  such as visibility_of_element_located or text_to_be_present_in_element

        html = driver.page_source
        soup = bs(html, "lxml")
        dynamic_text = soup.find_all("p", {"class":"class_name"}) #or other attributes, optional
    else:
        print("Couldnt locate element")

附注:你可以在这里找到更多的等待条件