使用Python从文本文件中删除停止词

print('====THIS IS HOW TO REMOVE STOP WORS====')

with open('one.txt','r')as myFile:

    str1=myFile.read()

    stop_words ="not", "is", "it", "By","between","This","By","A","when","And","up","Then","was","by","It","If","can","an","he","This","or","And","a","i","it","am","at","on","in","of","to","is","so","too","my","the","and","but","are","very","here","even","from","them","then","than","this","that","though","be","But","these"

    myList=[]

    myList.extend(str1.split(" "))

    for i in myList:

        if i not in stop_words:

            print ("____________")

            print(i,end='\n')

我在寻找一个非常简单的解决方案。这是我得到的：

import re 

s = "string. With. Punctuation?" 
s = re.sub(r'[\W\s]', ' ', s)

print(s)
'string  With  Punctuation '

这可能不是最好的解决方案，但我就是这样做的。

import string
f = lambda x: ''.join([i for i in x if i not in string.punctuation])

从效率的角度来看，你不会击败

s.translate(None, string.punctuation)

对于更高版本的Python，请使用以下代码：

s.translate(str.maketrans('', '', string.punctuation))

它使用查找表在C语言中执行原始字符串操作——除了编写自己的C代码之外，没有什么能比这更好的了。

如果速度不令人担忧，另一个选择是：

exclude = set(string.punctuation)
s = ''.join(ch for ch in s if ch not in exclude)

这比用每个字符替换s.replace更快，但不会像正则表达式或字符串转换等非纯python方法那样执行得好，正如您从下面的计时中看到的那样。对于这种类型的问题，在尽可能低的水平上解决是有回报的。

计时代码：

import re, string, timeit

s = "string. With. Punctuation"
exclude = set(string.punctuation)
table = string.maketrans("","")
regex = re.compile('[%s]' % re.escape(string.punctuation))

def test_set(s):
    return ''.join(ch for ch in s if ch not in exclude)

def test_re(s):  # From Vinko's solution, with fix.
    return regex.sub('', s)

def test_trans(s):
    return s.translate(table, string.punctuation)

def test_repl(s):  # From S.Lott's solution
    for c in string.punctuation:
        s=s.replace(c,"")
    return s

print "sets      :",timeit.Timer('f(s)', 'from __main__ import s,test_set as f').timeit(1000000)
print "regex     :",timeit.Timer('f(s)', 'from __main__ import s,test_re as f').timeit(1000000)
print "translate :",timeit.Timer('f(s)', 'from __main__ import s,test_trans as f').timeit(1000000)
print "replace   :",timeit.Timer('f(s)', 'from __main__ import s,test_repl as f').timeit(1000000)

结果如下：

sets      : 19.8566138744
regex     : 6.86155414581
translate : 2.12455511093
replace   : 28.4436721802

字符串标点符号仅为ASCII！一种更正确（但也慢得多）的方法是使用unicodedata模块：

# -*- coding: utf-8 -*-
from unicodedata import category
s = u'String — with -  «punctation »...'
s = ''.join(ch for ch in s if category(ch)[0] != 'P')
print 'stripped', s

您还可以概括和剥离其他类型的字符：

''.join(ch for ch in s if category(ch)[0] not in 'SP')

它还将删除~*+§$等字符，这些字符可能是“标点符号”，也可能不是“标点符号。

这个问题没有太多细节，所以我采取的方法是用最简单的解释来解决这个问题：去掉标点符号。

请注意，所提供的解决方案不考虑收缩词（例如，you are）或连字符词（例如肛门保留）。。。这是关于它们是否应该被视为标点符号的争论。。。也不能解释非英语字符集或类似的东西。。。因为问题中没有提到这些细节。有人认为空格是标点符号，这在技术上是正确的。。。但对我来说，这在当前问题的背景下毫无意义。

# using lambda
''.join(filter(lambda c: c not in string.punctuation, s))

# using list comprehension
''.join('' if c in string.punctuation else c for c in s)