在Python中对正则表达式使用compile有什么好处吗?
h = re.compile('hello')
h.match('hello world')
vs
re.match('hello', 'hello world')
在Python中对正则表达式使用compile有什么好处吗?
h = re.compile('hello')
h.match('hello world')
vs
re.match('hello', 'hello world')
当前回答
这个答案可能姗姗来迟,但却是一个有趣的发现。如果你打算多次使用regex,使用compile真的可以节省你的时间(这在文档中也有提到)。下面你可以看到,当直接调用match方法时,使用编译后的正则表达式是最快的。将一个编译好的正则表达式传递给re.match会使它更慢,而将re.match与patter字符串传递在中间的某个地方。
>>> ipr = r'\D+((([0-2][0-5]?[0-5]?)\.){3}([0-2][0-5]?[0-5]?))\D+'
>>> average(*timeit.repeat("re.match(ipr, 'abcd100.10.255.255 ')", globals={'ipr': ipr, 're': re}))
1.5077415757028423
>>> ipr = re.compile(ipr)
>>> average(*timeit.repeat("re.match(ipr, 'abcd100.10.255.255 ')", globals={'ipr': ipr, 're': re}))
1.8324008992184038
>>> average(*timeit.repeat("ipr.match('abcd100.10.255.255 ')", globals={'ipr': ipr, 're': re}))
0.9187896518778871
其他回答
下面是一个使用re.compile的示例,在请求时速度超过50倍。
这一点与我在上面的评论中所说的是一样的,即当您的使用从编译缓存中获益不多时,使用re.compile可能是一个显著的优势。这种情况至少发生在一个特定的情况下(我在实践中遇到过),即当以下所有情况都成立时:
您有很多regex模式(不仅仅是re._MAXCACHE,它目前的默认值是512),以及 你经常使用这些正则表达式,而且 相同模式的连续使用之间被多个re._MAXCACHE其他正则表达式分隔,因此每个正则表达式在连续使用之间从缓存中刷新。
import re
import time
def setup(N=1000):
# Patterns 'a.*a', 'a.*b', ..., 'z.*z'
patterns = [chr(i) + '.*' + chr(j)
for i in range(ord('a'), ord('z') + 1)
for j in range(ord('a'), ord('z') + 1)]
# If this assertion below fails, just add more (distinct) patterns.
# assert(re._MAXCACHE < len(patterns))
# N strings. Increase N for larger effect.
strings = ['abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz'] * N
return (patterns, strings)
def without_compile():
print('Without re.compile:')
patterns, strings = setup()
print('searching')
count = 0
for s in strings:
for pat in patterns:
count += bool(re.search(pat, s))
return count
def without_compile_cache_friendly():
print('Without re.compile, cache-friendly order:')
patterns, strings = setup()
print('searching')
count = 0
for pat in patterns:
for s in strings:
count += bool(re.search(pat, s))
return count
def with_compile():
print('With re.compile:')
patterns, strings = setup()
print('compiling')
compiled = [re.compile(pattern) for pattern in patterns]
print('searching')
count = 0
for s in strings:
for regex in compiled:
count += bool(regex.search(s))
return count
start = time.time()
print(with_compile())
d1 = time.time() - start
print(f'-- That took {d1:.2f} seconds.\n')
start = time.time()
print(without_compile_cache_friendly())
d2 = time.time() - start
print(f'-- That took {d2:.2f} seconds.\n')
start = time.time()
print(without_compile())
d3 = time.time() - start
print(f'-- That took {d3:.2f} seconds.\n')
print(f'Ratio: {d3/d1:.2f}')
我在笔记本电脑上获得的示例输出(Python 3.7.7):
With re.compile:
compiling
searching
676000
-- That took 0.33 seconds.
Without re.compile, cache-friendly order:
searching
676000
-- That took 0.67 seconds.
Without re.compile:
searching
676000
-- That took 23.54 seconds.
Ratio: 70.89
I didn't bother with timeit as the difference is so stark, but I get qualitatively similar numbers each time. Note that even without re.compile, using the same regex multiple times and moving on to the next one wasn't so bad (only about 2 times as slow as with re.compile), but in the other order (looping through many regexes), it is significantly worse, as expected. Also, increasing the cache size works too: simply setting re._MAXCACHE = len(patterns) in setup() above (of course I don't recommend doing such things in production as names with underscores are conventionally “private”) drops the ~23 seconds back down to ~0.7 seconds, which also matches our understanding.
大多数情况下,是否使用re.compile没有什么区别。在内部,所有函数都是按照编译步骤实现的:
def match(pattern, string, flags=0):
return _compile(pattern, flags).match(string)
def fullmatch(pattern, string, flags=0):
return _compile(pattern, flags).fullmatch(string)
def search(pattern, string, flags=0):
return _compile(pattern, flags).search(string)
def sub(pattern, repl, string, count=0, flags=0):
return _compile(pattern, flags).sub(repl, string, count)
def subn(pattern, repl, string, count=0, flags=0):
return _compile(pattern, flags).subn(repl, string, count)
def split(pattern, string, maxsplit=0, flags=0):
return _compile(pattern, flags).split(string, maxsplit)
def findall(pattern, string, flags=0):
return _compile(pattern, flags).findall(string)
def finditer(pattern, string, flags=0):
return _compile(pattern, flags).finditer(string)
此外,re.compile()绕过了额外的间接和缓存逻辑:
_cache = {}
_pattern_type = type(sre_compile.compile("", 0))
_MAXCACHE = 512
def _compile(pattern, flags):
# internal: compile pattern
try:
p, loc = _cache[type(pattern), pattern, flags]
if loc is None or loc == _locale.setlocale(_locale.LC_CTYPE):
return p
except KeyError:
pass
if isinstance(pattern, _pattern_type):
if flags:
raise ValueError(
"cannot process flags argument with a compiled pattern")
return pattern
if not sre_compile.isstring(pattern):
raise TypeError("first argument must be string or compiled pattern")
p = sre_compile.compile(pattern, flags)
if not (flags & DEBUG):
if len(_cache) >= _MAXCACHE:
_cache.clear()
if p.flags & LOCALE:
if not _locale:
return p
loc = _locale.setlocale(_locale.LC_CTYPE)
else:
loc = None
_cache[type(pattern), pattern, flags] = p, loc
return p
除了使用re.compile带来的小速度好处外,人们还喜欢命名潜在复杂的模式规范并将其与应用的业务逻辑分离所带来的可读性:
#### Patterns ############################################################
number_pattern = re.compile(r'\d+(\.\d*)?') # Integer or decimal number
assign_pattern = re.compile(r':=') # Assignment operator
identifier_pattern = re.compile(r'[A-Za-z]+') # Identifiers
whitespace_pattern = re.compile(r'[\t ]+') # Spaces and tabs
#### Applications ########################################################
if whitespace_pattern.match(s): business_logic_rule_1()
if assign_pattern.match(s): business_logic_rule_2()
注意,另一位受访者错误地认为pyc文件直接存储已编译的模式;然而,在现实中,每次PYC加载时,它们都会被重新构建:
>>> from dis import dis
>>> with open('tmp.pyc', 'rb') as f:
f.read(8)
dis(marshal.load(f))
1 0 LOAD_CONST 0 (-1)
3 LOAD_CONST 1 (None)
6 IMPORT_NAME 0 (re)
9 STORE_NAME 0 (re)
3 12 LOAD_NAME 0 (re)
15 LOAD_ATTR 1 (compile)
18 LOAD_CONST 2 ('[aeiou]{2,5}')
21 CALL_FUNCTION 1
24 STORE_NAME 2 (lc_vowels)
27 LOAD_CONST 1 (None)
30 RETURN_VALUE
上面的分解来自于一个包含tmp.py的PYC文件:
import re
lc_vowels = re.compile(r'[aeiou]{2,5}')
在无意中看到这里的讨论之前,我运行了这个测试。然而,在运行它之后,我想我至少会发布我的结果。
我剽窃了Jeff Friedl的“精通正则表达式”中的例子。这是在一台运行OSX 10.6 (2Ghz英特尔酷睿2双核,4GB内存)的macbook上。Python版本为2.6.1。
运行1 -使用re.compile
import re
import time
import fpformat
Regex1 = re.compile('^(a|b|c|d|e|f|g)+$')
Regex2 = re.compile('^[a-g]+$')
TimesToDo = 1000
TestString = ""
for i in range(1000):
TestString += "abababdedfg"
StartTime = time.time()
for i in range(TimesToDo):
Regex1.search(TestString)
Seconds = time.time() - StartTime
print "Alternation takes " + fpformat.fix(Seconds,3) + " seconds"
StartTime = time.time()
for i in range(TimesToDo):
Regex2.search(TestString)
Seconds = time.time() - StartTime
print "Character Class takes " + fpformat.fix(Seconds,3) + " seconds"
Alternation takes 2.299 seconds
Character Class takes 0.107 seconds
运行2 -不使用re.compile
import re
import time
import fpformat
TimesToDo = 1000
TestString = ""
for i in range(1000):
TestString += "abababdedfg"
StartTime = time.time()
for i in range(TimesToDo):
re.search('^(a|b|c|d|e|f|g)+$',TestString)
Seconds = time.time() - StartTime
print "Alternation takes " + fpformat.fix(Seconds,3) + " seconds"
StartTime = time.time()
for i in range(TimesToDo):
re.search('^[a-g]+$',TestString)
Seconds = time.time() - StartTime
print "Character Class takes " + fpformat.fix(Seconds,3) + " seconds"
Alternation takes 2.508 seconds
Character Class takes 0.109 seconds
我想说的是,预编译在概念上和“字面上”(如在“文学编程”中)都是有利的。看看这段代码片段:
from re import compile as _Re
class TYPO:
def text_has_foobar( self, text ):
return self._text_has_foobar_re_search( text ) is not None
_text_has_foobar_re_search = _Re( r"""(?i)foobar""" ).search
TYPO = TYPO()
在你的应用程序中,你可以这样写:
from TYPO import TYPO
print( TYPO.text_has_foobar( 'FOObar ) )
this is about as simple in terms of functionality as it can get. because this is example is so short, i conflated the way to get _text_has_foobar_re_search all in one line. the disadvantage of this code is that it occupies a little memory for whatever the lifetime of the TYPO library object is; the advantage is that when doing a foobar search, you'll get away with two function calls and two class dictionary lookups. how many regexes are cached by re and the overhead of that cache are irrelevant here.
将其与更常见的风格进行比较,如下所示:
import re
class Typo:
def text_has_foobar( self, text ):
return re.compile( r"""(?i)foobar""" ).search( text ) is not None
在应用中:
typo = Typo()
print( typo.text_has_foobar( 'FOObar ) )
我很乐意承认我的风格在python中是非常不寻常的,甚至可能是有争议的。然而,在更接近python的使用方式的示例中,为了进行一次匹配,我们必须实例化一个对象,进行三次实例字典查找,并执行三次函数调用;此外,当使用超过100个正则表达式时,我们可能会遇到重新缓存的麻烦。此外,正则表达式被隐藏在方法体中,这在大多数情况下并不是一个好主意。
可以说,每一个措施的子集——有针对性的,别名的import语句;别名方法(如适用);减少函数调用和对象字典查找——可以帮助减少计算和概念的复杂性。
我真的很尊重上面所有的答案。在我看来 是的!当然,使用re.compile而不是一次又一次地编译正则表达式是值得的。
使用re.compile可以使代码更加动态,因为您可以调用已经编译好的正则表达式,而不是一次又一次地编译。这对你有好处:
处理器的努力 时间复杂度。 使正则表达式通用。(可以在findall, search, match中使用) 并使您的程序看起来很酷。
例子:
example_string = "The room number of her room is 26A7B."
find_alpha_numeric_string = re.compile(r"\b\w+\b")
在Findall中使用
find_alpha_numeric_string.findall(example_string)
在搜索中使用
find_alpha_numeric_string.search(example_string)
类似地,您可以将它用于:Match和Substitute