我想使用Python从HTML文件中提取文本。我想从本质上得到相同的输出,如果我从浏览器复制文本,并将其粘贴到记事本。
我想要一些更健壮的东西,而不是使用正则表达式,正则表达式可能会在格式不佳的HTML上失败。我见过很多人推荐Beautiful Soup,但我在使用它时遇到了一些问题。首先,它会抓取不需要的文本,比如JavaScript源代码。此外,它也不解释HTML实体。例如,我会期望'在HTML源代码中转换为文本中的撇号,就像我将浏览器内容粘贴到记事本一样。
更新html2text看起来很有希望。它正确地处理HTML实体,而忽略JavaScript。然而,它并不完全生成纯文本;它产生的降价,然后必须转换成纯文本。它没有示例或文档,但代码看起来很干净。
相关问题:
在python中过滤HTML标签并解析实体
在Python中将XML/HTML实体转换为Unicode字符串
我发现的最好的一段代码提取文本没有javascript或不想要的东西:
from urllib.request import urlopen
from bs4 import BeautifulSoup
url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")
# kill all script and style elements
for script in soup(["script", "style"]):
script.extract() # rip it out
# get text
text = soup.get_text()
# break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
# drop blank lines
text = '\n'.join(chunk for chunk in chunks if chunk)
print(text)
你只需要安装BeautifulSoup:
pip install beautifulsoup4
@PeYoTIL使用BeautifulSoup并删除样式和脚本内容的回答对我来说并不管用。我尝试用分解代替提取,但它仍然没有工作。所以我创建了自己的格式文本使用<p>标签和替换<a>标签与href链接。也处理文本内的链接。可在此主旨与测试文档嵌入。
from bs4 import BeautifulSoup, NavigableString
def html_to_text(html):
"Creates a formatted text email message as a string from a rendered html template (page)"
soup = BeautifulSoup(html, 'html.parser')
# Ignore anything in head
body, text = soup.body, []
for element in body.descendants:
# We use type and not isinstance since comments, cdata, etc are subclasses that we don't want
if type(element) == NavigableString:
# We use the assumption that other tags can't be inside a script or style
if element.parent.name in ('script', 'style'):
continue
# remove any multiple and leading/trailing whitespace
string = ' '.join(element.string.split())
if string:
if element.parent.name == 'a':
a_tag = element.parent
# replace link text with the link
string = a_tag['href']
# concatenate with any non-empty immediately previous string
if ( type(a_tag.previous_sibling) == NavigableString and
a_tag.previous_sibling.string.strip() ):
text[-1] = text[-1] + ' ' + string
continue
elif element.previous_sibling and element.previous_sibling.name == 'a':
text[-1] = text[-1] + ' ' + string
continue
elif element.parent.name == 'p':
# Add extra paragraph formatting newline
string = '\n' + string
text += [string]
doc = '\n'.join(text)
return doc
另一个在Python 2.7.9+中使用BeautifulSoup4的例子
包括:
import urllib2
from bs4 import BeautifulSoup
代码:
def read_website_to_text(url):
page = urllib2.urlopen(url)
soup = BeautifulSoup(page, 'html.parser')
for script in soup(["script", "style"]):
script.extract()
text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = '\n'.join(chunk for chunk in chunks if chunk)
return str(text.encode('utf-8'))
解释道:
将url数据读入为html(使用BeautifulSoup),删除所有脚本和样式元素,并使用.get_text()仅获取文本。分割成行,删除每个标题的开头和结尾空格,然后将多个标题分割成一行,each chunks = (phrase.strip() for line in line for phrase in line。(" "))。然后使用text = '\n'。加入,删除空行,最后返回为批准的utf-8。
注:
一些系统这是运行在https://连接失败,因为SSL问题,你可以关闭验证来解决这个问题。修复示例:http://blog.pengyifan.com/how-to-fix-python-ssl-certificate_verify_failed/
Python < 2.7.9在运行时可能会遇到一些问题
text.encode('utf-8')可能会留下奇怪的编码,可能只需要返回str(text)即可。