似乎应该有一种比以下更简单的方法:
import string
s = "string. With. Punctuation?" # Sample string
out = s.translate(string.maketrans("",""), string.punctuation)
有?
似乎应该有一种比以下更简单的方法:
import string
s = "string. With. Punctuation?" # Sample string
out = s.translate(string.maketrans("",""), string.punctuation)
有?
当前回答
对于严肃的自然语言处理(NLP),您应该让像SpaCy这样的库通过标记化处理标点符号,然后您可以根据需要手动调整。
例如,您希望如何处理单词中的连字符?例外情况,如缩写?开始和结束引号?URL?在NLP中,将“let’s”这样的收缩分隔为“let”和“s”以进行进一步处理通常很有用。
其他回答
这里有一个使用RegEx的简单方法
import re
punct = re.compile(r'(\w+)')
sentence = 'This ! is : a # sample $ sentence.' # Text with punctuation
tokenized = [m.group() for m in punct.finditer(sentence)]
sentence = ' '.join(tokenized)
print(sentence)
'This is a sample sentence'
正则表达式很简单,如果你知道的话。
import re
s = "string. With. Punctuation?"
s = re.sub(r'[^\w\s]','',s)
显然,我无法对所选答案进行编辑,所以这里有一个适用于Python3的更新。在进行非平凡转换时,转换方法仍然是最有效的选择。
上面的@Brian为最初的繁重工作做出了贡献。感谢@ddejohn对原始测试的改进建议。
#!/usr/bin/env python3
"""Determination of most efficient way to remove punctuation in Python 3.
Results in Python 3.8.10 on my system using the default arguments:
set : 51.897
regex : 17.901
translate : 2.059
replace : 13.209
"""
import argparse
import re
import string
import timeit
parser = argparse.ArgumentParser()
parser.add_argument("--filename", "-f", default=argparse.__file__)
parser.add_argument("--iterations", "-i", type=int, default=10000)
opts = parser.parse_args()
with open(opts.filename) as fp:
s = fp.read()
exclude = set(string.punctuation)
table = str.maketrans("", "", string.punctuation)
regex = re.compile(f"[{re.escape(string.punctuation)}]")
def test_set(s):
return "".join(ch for ch in s if ch not in exclude)
def test_regex(s): # From Vinko's solution, with fix.
return regex.sub("", s)
def test_translate(s):
return s.translate(table)
def test_replace(s): # From S.Lott's solution
for c in string.punctuation:
s = s.replace(c, "")
return s
opts = dict(globals=globals(), number=opts.iterations)
solutions = "set", "regex", "translate", "replace"
for solution in solutions:
elapsed = timeit.timeit(f"test_{solution}(s)", **opts)
print(f"{solution:<10}: {elapsed:6.3f}")
import re
s = "string. With. Punctuation?" # Sample string
out = re.sub(r'[^a-zA-Z0-9\s]', '', s)
在处理Unicode字符串时,我建议使用PyPi正则表达式模块,因为它同时支持Unicode属性类(如\p{X}/\p{X})和POSIX字符类(如[:name:])。
只需在终端中键入pipinstallregex(或pip3installregex)并按回车键即可安装软件包。
如果您需要删除任何类型的标点符号(即除字母、数字和空格之外的任何其他符号),您可以使用
regex.sub(r'[\p{P}\p{S}]', '', text) # to remove one by one
regex.sub(r'[\p{P}\p{S}]+', '', text) # to remove all consecutive punctuation/symbols with one go
regex.sub(r'[[:punct:]]+', '', text) # Same with a POSIX character class
在线观看Python演示:
import regex
text = 'भारत India <><>^$.,,! 002'
new_text = regex.sub(r'[\p{P}\p{S}\s]+', ' ', text).lower().strip()
# OR
# new_text = regex.sub(r'[[:punct:]\s]+', ' ', text).lower().strip()
print(new_text)
# => भारत india 002
在这里,我向字符类添加了空白模式