如果您希望获得尽可能好的性能，您可能会考虑使用异步I/O而不是线程。与成千上万个操作系统线程相关的开销是不小的，Python解释器内的上下文切换甚至增加了更多的开销。线程当然可以完成工作，但我怀疑异步路由将提供更好的整体性能。

具体来说，我建议使用Twisted库中的异步web客户端(http://www.twistedmatrix.com)。它有一个公认的陡峭的学习曲线，但一旦你很好地掌握了Twisted的异步编程风格，它就很容易使用。

Twisted的异步web客户端API的HowTo可以在以下地址找到:

http://twistedmatrix.com/documents/current/web/howto/client.html

最简单的方法是使用Python的内置线程库。它们不是“真正的”/内核线程。它们有问题(比如序列化)，但足够好了。你需要一个队列和线程池。这里有一个选项，但是编写自己的选项很简单。您无法并行处理所有100,000个调用，但可以同时发出100个(或左右)调用。

自从2010年这篇文章发布以来，事情发生了很大的变化，我还没有尝试过所有其他的答案，但我尝试了一些，我发现使用python3.6对我来说这是最好的。

在AWS上运行时，我每秒可以获取大约150个独特的域名。

import concurrent.futures
import requests
import time

out = []
CONNECTIONS = 100
TIMEOUT = 5

tlds = open('../data/sample_1k.txt').read().splitlines()
urls = ['http://{}'.format(x) for x in tlds[1:]]

def load_url(url, timeout):
    ans = requests.head(url, timeout=timeout)
    return ans.status_code

with concurrent.futures.ThreadPoolExecutor(max_workers=CONNECTIONS) as executor:
    future_to_url = (executor.submit(load_url, url, TIMEOUT) for url in urls)
    time1 = time.time()
    for future in concurrent.futures.as_completed(future_to_url):
        try:
            data = future.result()
        except Exception as exc:
            data = str(type(exc))
        finally:
            out.append(data)

            print(str(len(out)),end="\r")

    time2 = time.time()

print(f'Took {time2-time1:.2f} s')

线程绝对不是这里的答案。它们将提供进程和内核瓶颈，以及吞吐量限制，如果总体目标是“最快的方式”，这些限制是不可接受的。

稍微扭曲一点，它的异步HTTP客户端会给你更好的结果。

解决这个问题的一个好方法是首先编写获得一个结果所需的代码，然后合并线程代码来并行化应用程序。

In a perfect world this would simply mean simultaneously starting 100,000 threads which output their results into a dictionary or list for later processing, but in practice you are limited in how many parallel HTTP requests you can issue in this fashion. Locally, you have limits in how many sockets you can open concurrently, how many threads of execution your Python interpreter will allow. Remotely, you may be limited in the number of simultaneous connections if all the requests are against one server, or many. These limitations will probably necessitate that you write the script in such a way as to only poll a small fraction of the URLs at any one time (100, as another poster mentioned, is probably a decent thread pool size, although you may find that you can successfully deploy many more).

您可以遵循以下设计模式来解决上述问题:

Start a thread which launches new request threads until the number of currently running threads (you can track them via threading.active_count() or by pushing the thread objects into a data structure) is >= your maximum number of simultaneous requests (say 100), then sleeps for a short timeout. This thread should terminate when there is are no more URLs to process. Thus, the thread will keep waking up, launching new threads, and sleeping until your are finished. Have the request threads store their results in some data structure for later retrieval and output. If the structure you are storing the results in is a list or dict in CPython, you can safely append or insert unique items from your threads without locks, but if you write to a file or require in more complex cross-thread data interaction you should use a mutual exclusion lock to protect this state from corruption.

我建议您使用threading模块。您可以使用它来启动和跟踪正在运行的线程。Python的线程支持是完全的，但是对问题的描述表明它完全满足了您的需求。

最后，如果您希望看到用Python编写的并行网络应用程序的相当简单的应用程序，请查看ssh.py。它是一个小型库，使用Python线程并行处理许多SSH连接。该设计非常接近您的需求，您可能会发现它是一个很好的资源。

考虑使用风车，虽然风车可能不能做那么多线程。

您可以在5台机器上使用手卷Python脚本，每台机器使用端口40000-60000连接出站，打开100,000个端口连接。

另外，使用一个线程良好的QA应用程序(如OpenSTA)做一个示例测试可能会有所帮助，以了解每个服务器可以处理多少。

另外，试着在LWP::ConnCache类中使用简单的Perl。这样您可能会获得更好的性能(更多的连接)。