我试图开发一个简单的网页刮板。我想提取没有HTML代码的文本。它适用于普通HTML，但不适用于JavaScript代码添加文本的某些页面。
例如，如果一些JavaScript代码添加了一些文本，我不能看到它，因为当我调用:
response = urllib2.urlopen(request)
我得到了原始文本而没有添加的文本(因为JavaScript是在客户端执行的)。
所以，我正在寻找一些解决这个问题的想法。
当前回答
使用PyQt5
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl
from PyQt5.QtWebEngineWidgets import QWebEnginePage
import sys
import bs4 as bs
import urllib.request
class Client(QWebEnginePage):
def __init__(self,url):
global app
self.app = QApplication(sys.argv)
QWebEnginePage.__init__(self)
self.html = ""
self.loadFinished.connect(self.on_load_finished)
self.load(QUrl(url))
self.app.exec_()
def on_load_finished(self):
self.html = self.toHtml(self.Callable)
print("Load Finished")
def Callable(self,data):
self.html = data
self.app.quit()
# url = ""
# client_response = Client(url)
# print(client_response.html)
其他回答
Pyppeteer
你可以考虑Pyppeteer，它是Chrome/Chromium驱动程序前端的Python移植版本。
下面是一个简单的例子，展示了如何使用pyppeterer动态地访问被注入到页面中的数据:
import asyncio
from pyppeteer import launch
async def main():
browser = await launch({"headless": True})
[page] = await browser.pages()
# normally, you go to a live site...
#await page.goto("http://www.example.com")
# but for this example, just set the HTML directly:
await page.setContent("""
<body>
<script>
// inject content dynamically with JS, not part of the static HTML!
document.body.innerHTML = `<p>hello world</p>`;
</script>
</body>
""")
print(await page.content()) # shows that the `<p>` was inserted
# evaluate a JS expression in browser context and scrape the data
expr = "document.querySelector('p').textContent"
print(await page.evaluate(expr, force_expr=True)) # => hello world
await browser.close()
asyncio.get_event_loop().run_until_complete(main())
请参阅pyppeterer的参考文档。
我最近使用requests_html库来解决这个问题。
他们的扩展文档在readthedocs。IO非常好(跳过pypi.org上的带注释的版本)。如果您的用例是基本的，那么您可能会取得一些成功。
from requests_html import HTMLSession
session = HTMLSession()
response = session.request(method="get",url="www.google.com/")
response.html.render()
如果你在使用response.html.render()呈现你需要的数据时遇到麻烦，你可以将一些javascript传递给呈现函数来呈现你需要的特定js对象。这是从他们的文档中复制的，但这可能正是你需要的:
如果指定了script，它将在 运行时。例子:
script = """
() => {
return {
width: document.documentElement.clientWidth,
height: document.documentElement.clientHeight,
deviceScaleFactor: window.devicePixelRatio,
}
}
"""
返回执行脚本的返回值，如果有的话:
>>> response.html.render(script=script)
{'width': 800, 'height': 600, 'deviceScaleFactor': 1}
In my case, the data I wanted were the arrays that populated a javascript plot but the data wasn't getting rendered as text anywhere in the html. Sometimes its not clear at all what the object names are of the data you want if the data is populated dynamically. If you can't track down the js objects directly from view source or inspect, you can type in "window" followed by ENTER in the debugger console in the browser (Chrome) to pull up a full list of objects rendered by the browser. If you make a few educated guesses about where the data is stored, you might have some luck finding it there. My graph data was under window.view.data in the console, so in the "script" variable passed to the .render() method quoted above, I used:
return {
data: window.view.data
}
Playwright-Python
还有一种选择是剧作家- Python，它是微软剧作家(本身是受木偶大师影响的浏览器自动化库)到Python的移植。
下面是选择一个元素并抓取它的文本的最小示例:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("http://whatsmyuseragent.org/")
ua = page.query_selector(".user-agent");
print(ua.text_content())
browser.close()
我个人更喜欢在单独的容器中使用scrapy和selenium和dockerizing。通过这种方式，你既可以轻松安装，也可以抓取几乎所有包含某种形式javascript的现代网站。这里有一个例子:
使用scrapy startproject创建你的scraper并编写你的蜘蛛，骨架可以像这样简单:
import scrapy
class MySpider(scrapy.Spider):
name = 'my_spider'
start_urls = ['https://somewhere.com']
def start_requests(self):
yield scrapy.Request(url=self.start_urls[0])
def parse(self, response):
# do stuff with results, scrape items etc.
# now were just checking everything worked
print(response.body)
真正的魔力发生在middleware .py中。重写下载中间件中的两个方法__init__和process_request，方法如下:
# import some additional modules that we need
import os
from copy import deepcopy
from time import sleep
from scrapy import signals
from scrapy.http import HtmlResponse
from selenium import webdriver
class SampleProjectDownloaderMiddleware(object):
def __init__(self):
SELENIUM_LOCATION = os.environ.get('SELENIUM_LOCATION', 'NOT_HERE')
SELENIUM_URL = f'http://{SELENIUM_LOCATION}:4444/wd/hub'
chrome_options = webdriver.ChromeOptions()
# chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
self.driver = webdriver.Remote(command_executor=SELENIUM_URL,
desired_capabilities=chrome_options.to_capabilities())
def process_request(self, request, spider):
self.driver.get(request.url)
# sleep a bit so the page has time to load
# or monitor items on page to continue as soon as page ready
sleep(4)
# if you need to manipulate the page content like clicking and scrolling, you do it here
# self.driver.find_element_by_css_selector('.my-class').click()
# you only need the now properly and completely rendered html from your page to get results
body = deepcopy(self.driver.page_source)
# copy the current url in case of redirects
url = deepcopy(self.driver.current_url)
return HtmlResponse(url, body=body, encoding='utf-8', request=request)
不要忘记在settings.py文件中取消下一行的注释来启用这个中间件:
DOWNLOADER_MIDDLEWARES = {
'sample_project.middlewares.SampleProjectDownloaderMiddleware': 543,}
接下来是dockerization。从一个轻量级映像创建Dockerfile(我在这里使用python Alpine)，复制你的项目目录到它，安装要求:
# Use an official Python runtime as a parent image
FROM python:3.6-alpine
# install some packages necessary to scrapy and then curl because it's handy for debugging
RUN apk --update add linux-headers libffi-dev openssl-dev build-base libxslt-dev libxml2-dev curl python-dev
WORKDIR /my_scraper
ADD requirements.txt /my_scraper/
RUN pip install -r requirements.txt
ADD . /scrapers
最后在docker-compose.yaml中把所有这些都整合在一起:
version: '2'
services:
selenium:
image: selenium/standalone-chrome
ports:
- "4444:4444"
shm_size: 1G
my_scraper:
build: .
depends_on:
- "selenium"
environment:
- SELENIUM_LOCATION=samplecrawler_selenium_1
volumes:
- .:/my_scraper
# use this command to keep the container running
command: tail -f /dev/null
运行docker-compose up -d。如果你是第一次这样做，它将需要一段时间来获取最新的硒/独立铬和构建你的刮刀图像以及。
完成后，您可以检查容器是否使用docker ps运行，还可以检查selenium容器的名称是否与传递给scraper容器的环境变量的名称相匹配(在这里，它是SELENIUM_LOCATION=samplecrawler_selenium_1)。
使用docker exec -ti YOUR_CONTAINER_NAME sh进入你的刮板容器，我的命令是docker exec -ti samplecrawler_my_scraper_1 sh, cd到正确的目录下，并用scrapy爬行my_spider运行你的刮板。
所有内容都在我的github页面上，你可以从这里获取
You'll want to use urllib, requests, beautifulSoup and selenium web driver in your script for different parts of the page, (to name a few). Sometimes you'll get what you need with just one of these modules. Sometimes you'll need two, three, or all of these modules. Sometimes you'll need to switch off the js on your browser. Sometimes you'll need header info in your script. No websites can be scraped the same way and no website can be scraped in the same way forever without having to modify your crawler, usually after a few months. But they can all be scraped! Where there's a will there's a way for sure. If you need scraped data continuously into the future just scrape everything you need and store it in .dat files with pickle. Just keep searching how to try what with these modules and copying and pasting your errors into the Google.
推荐文章
- 证书验证失败:无法获得本地颁发者证书
- 当使用pip3安装包时，“Python中的ssl模块不可用”
- Babel 6改变了它导出默认值的方式
- 无法切换Python与pyenv
- Python if not == vs if !=
- 如何配置历史记录?
- ES6模板文字可以在运行时被替换(或重用)吗?
- [Vue警告]:找不到元素
- 如何从scikit-learn决策树中提取决策规则?
- 为什么在Mac OS X v10.9 (Mavericks)的终端中apt-get功能不起作用?
- 可以在setInterval()内部调用clearInterval()吗?
- AngularJS控制器的生命周期是什么?
- 无法读取未定义的属性“msie”- jQuery工具
- 将旋转的xtick标签与各自的xtick对齐
- 我的蛋蛋怎么不见了?
