我试图理解多处理相对于线程的优势。我知道多处理绕过了全局解释器锁,但是还有什么其他的优势,线程不能做同样的事情吗?
当前回答
关键的优势是隔离。进程崩溃不会导致其他进程崩溃,而线程崩溃可能会对其他线程造成严重破坏。
其他回答
多处理
Multiprocessing通过增加cpu来提高计算能力。 多个进程同时执行。 创建流程既耗时又耗费资源。 多处理可以是对称的也可以是非对称的。
Python中的多处理库使用独立的内存空间,多个CPU核心,绕过CPython中的GIL限制,子进程是可杀死的(例如程序中的函数调用),并且更容易使用。 该模块的一些注意事项是内存占用较大,IPC稍微复杂一些,开销更大。
多线程
多线程创建单个进程的多个线程,以提高计算能力。 一个进程的多个线程并发执行。 线程的创建在时间和资源上都是经济的。
多线程库是轻量级的,共享内存,负责响应式UI,并用于I/O绑定应用程序。 该模块不可杀死,并受GIL约束。 多个线程生活在同一个进程中的同一个空间中,每个线程将执行特定的任务,有自己的代码,自己的堆栈内存,指令指针,并共享堆内存。 如果一个线程有内存泄漏,它会损害其他线程和父进程。
使用Python的多线程和多处理示例
Python 3有启动并行任务的功能。这使我们的工作更容易。
它有线程池和进程池。
下面让我们来了解一下:
ThreadPoolExecutor例子
import concurrent.futures
import urllib.request
URLS = ['http://www.foxnews.com/',
'http://www.cnn.com/',
'http://europe.wsj.com/',
'http://www.bbc.co.uk/',
'http://some-made-up-domain.com/']
# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
with urllib.request.urlopen(url, timeout=timeout) as conn:
return conn.read()
# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
# Start the load operations and mark each future with its URL
future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
data = future.result()
except Exception as exc:
print('%r generated an exception: %s' % (url, exc))
else:
print('%r page is %d bytes' % (url, len(data)))
ProcessPoolExecutor
import concurrent.futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
def main():
with concurrent.futures.ProcessPoolExecutor() as executor:
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
print('%d is prime: %s' % (number, prime))
if __name__ == '__main__':
main()
另一件没有提到的事情是,它取决于你使用的是什么操作系统。在Windows中,进程是昂贵的,所以线程在Windows中会更好,但在unix中,进程比它们的Windows变体更快,所以在unix中使用进程要安全得多,而且生成速度快。
Threading's job is to enable applications to be responsive. Suppose you have a database connection and you need to respond to user input. Without threading, if the database connection is busy the application will not be able to respond to the user. By splitting off the database connection into a separate thread you can make the application more responsive. Also because both threads are in the same process, they can access the same data structures - good performance, plus a flexible software design.
注意,由于GIL,应用程序实际上并没有同时做两件事,但我们所做的是将数据库上的资源锁放在一个单独的线程中,这样CPU时间就可以在它和用户交互之间切换。CPU时间在线程之间分配。
Multiprocessing is for times when you really do want more than one thing to be done at any given time. Suppose your application needs to connect to 6 databases and perform a complex matrix transformation on each dataset. Putting each job in a separate thread might help a little because when one connection is idle another one could get some CPU time, but the processing would not be done in parallel because the GIL means that you're only ever using the resources of one CPU. By putting each job in a Multiprocessing process, each can run on it's own CPU and run at full efficiency.
Other answers have focused more on the multithreading vs multiprocessing aspect, but in python Global Interpreter Lock (GIL) has to be taken into account. When more number (say k) of threads are created, generally they will not increase the performance by k times, as it will still be running as a single threaded application. GIL is a global lock which locks everything out and allows only single thread execution utilizing only a single core. The performance does increase in places where C extensions like numpy, Network, I/O are being used, where a lot of background work is done and GIL is released. So when threading is used, there is only a single operating system level thread while python creates pseudo-threads which are completely managed by threading itself but are essentially running as a single process. Preemption takes place between these pseudo threads. If the CPU runs at maximum capacity, you may want to switch to multiprocessing. Now in case of self-contained instances of execution, you can instead opt for pool. But in case of overlapping data, where you may want processes communicating you should use multiprocessing.Process.
Python文档引用
这个答案的规范版本现在是双重问题:线程模块和多处理模块之间有什么区别?
我已经突出显示了Python文档中关于进程vs线程和GIL的关键引用:什么是CPython中的全局解释器锁(GIL) ?
进程与线程实验
为了更具体地展示差异,我做了一些基准测试。
在基准测试中,我对8超线程CPU上不同数量的线程进行了CPU和IO限制。每个线程提供的功总是相同的,因此线程越多,提供的总功就越多。
结果如下:
图数据。
结论:
对于CPU约束的工作,多处理总是更快,可能是由于GIL IO绑定工作。两者的速度完全相同 线程只能扩展到大约4倍,而不是预期的8倍,因为我使用的是8超线程机器。 与此相比,C POSIX cpu绑定的工作达到了预期的8倍加速:'real', 'user'和'sys'在time(1)的输出中是什么意思? 我不知道这是什么原因,肯定有其他Python的低效率因素在起作用。
测试代码:
#!/usr/bin/env python3
import multiprocessing
import threading
import time
import sys
def cpu_func(result, niters):
'''
A useless CPU bound function.
'''
for i in range(niters):
result = (result * result * i + 2 * result * i * i + 3) % 10000000
return result
class CpuThread(threading.Thread):
def __init__(self, niters):
super().__init__()
self.niters = niters
self.result = 1
def run(self):
self.result = cpu_func(self.result, self.niters)
class CpuProcess(multiprocessing.Process):
def __init__(self, niters):
super().__init__()
self.niters = niters
self.result = 1
def run(self):
self.result = cpu_func(self.result, self.niters)
class IoThread(threading.Thread):
def __init__(self, sleep):
super().__init__()
self.sleep = sleep
self.result = self.sleep
def run(self):
time.sleep(self.sleep)
class IoProcess(multiprocessing.Process):
def __init__(self, sleep):
super().__init__()
self.sleep = sleep
self.result = self.sleep
def run(self):
time.sleep(self.sleep)
if __name__ == '__main__':
cpu_n_iters = int(sys.argv[1])
sleep = 1
cpu_count = multiprocessing.cpu_count()
input_params = [
(CpuThread, cpu_n_iters),
(CpuProcess, cpu_n_iters),
(IoThread, sleep),
(IoProcess, sleep),
]
header = ['nthreads']
for thread_class, _ in input_params:
header.append(thread_class.__name__)
print(' '.join(header))
for nthreads in range(1, 2 * cpu_count):
results = [nthreads]
for thread_class, work_size in input_params:
start_time = time.time()
threads = []
for i in range(nthreads):
thread = thread_class(work_size)
threads.append(thread)
thread.start()
for i, thread in enumerate(threads):
thread.join()
results.append(time.time() - start_time)
print(' '.join('{:.6e}'.format(result) for result in results))
相同目录上的GitHub上游+绘图代码。
在Ubuntu 18.10, Python 3.6.7,联想ThinkPad P51笔记本电脑上测试,CPU:英特尔酷睿i7-7820HQ CPU(4核/ 8线程),RAM: 2倍三星M471A2K43BB1-CRC(2倍16GiB), SSD:三星MZVLB512HAJQ-000L7 (3000 MB/s)。
可视化给定时间哪些线程正在运行
这篇文章https://rohanvarma.me/GIL/告诉我,你可以运行一个回调每当线程调度与目标=参数的线程。线程和multiprocessing.Process。
这允许我们准确地查看每次运行的线程。当这完成后,我们会看到(我制作了这张特殊的图表):
+--------------------------------------+
+ Active threads / processes +
+-----------+--------------------------------------+
|Thread 1 |******** ************ |
| 2 | ***** *************|
+-----------+--------------------------------------+
|Process 1 |*** ************** ****** **** |
| 2 |** **** ****** ** ********* **********|
+-----------+--------------------------------------+
+ Time --> +
+--------------------------------------+
这将表明:
线程由GIL完全序列化 进程可以并行运行
推荐文章
- 将Pandas或Numpy Nan替换为None以用于MysqlDB
- 使用pandas对同一列进行多个聚合
- 使用Python解析HTML
- django MultiValueDictKeyError错误,我如何处理它
- 如何在for循环期间修改列表条目?
- 我如何在Django中创建一个鼻涕虫?
- 没有名为'django.core.urlresolvers'的模块
- 蟒蛇导出环境文件
- Django - makemigrations -未检测到任何更改
- SQLAlchemy:引擎、连接和会话差异
- 在Python Pandas中删除多个列中的所有重复行
- 更改pandas DataFrame中的特定列名
- 将Pandas多索引转换为列
- 熊猫在每组中获得最高的n个记录
- 熊猫数据帧得到每组的第一行