Urlretrieve和请求。得到的都很简单，然而现实却不。 我已经为两个站点提取了数据，包括文本和图像，以上两个可能解决了大部分任务。但如果想要更通用的解决方案，我建议使用urlopen。由于它包含在Python 3标准库中，您的代码可以在任何运行Python 3的机器上运行，而无需预先安装site-package

import urllib.request
url_request = urllib.request.Request(url, headers=headers)
url_connect = urllib.request.urlopen(url_request)

#remember to open file in bytes mode
with open(filename, 'wb') as f:
    while True:
        buffer = url_connect.read(buffer_size)
        if not buffer: break

        #an integer value of size of written data
        data_wrote = f.write(buffer)

#you could probably use with-open-as manner
url_connect.close()

当使用Python通过HTTP下载文件时，这个答案提供了HTTP 403禁止的解决方案。我只尝试了请求和urllib模块，其他模块可能会提供更好的东西，但这是我用来解决大多数问题的一个。

2012年，使用python请求库

>>> import requests
>>> 
>>> url = "http://download.thinkbroadband.com/10MB.zip"
>>> r = requests.get(url)
>>> print len(r.content)
10485760

您可以运行pip install请求来获取它。

请求比替代方法有很多优点，因为API简单得多。如果必须进行身份验证，则尤其如此。Urllib和urllib2在这种情况下非常不直观和痛苦。

2015-12-30

人们对进度条表示钦佩。这当然很酷。现在有几种现成的解决方案，包括tqdm:

from tqdm import tqdm
import requests

url = "http://download.thinkbroadband.com/10MB.zip"
response = requests.get(url, stream=True)

with open("10MB", "wb") as handle:
    for data in tqdm(response.iter_content()):
        handle.write(data)

这实际上是@kvance在30个月前描述的实现。

以下是python中下载文件最常用的调用:

urllib。Urlretrieve ('url_to_file'， file_name) urllib2.urlopen(“url_to_file”) requests.get (url) wget。下载(“url”,file_name)

注意:urlopen和urlretrieve在下载大文件(大小为> 500 MB)时表现相对较差。请求。Get将文件存储在内存中，直到下载完成。

使用urllib.request.urlopen ():

import urllib.request
with urllib.request.urlopen('http://www.example.com/') as f:
    html = f.read().decode('utf-8')

这是使用库的最基本的方法，没有任何错误处理。您还可以执行更复杂的操作，例如更改头文件。

在Python 2中，该方法在urllib2中:

import urllib2
response = urllib2.urlopen('http://www.example.com/')
html = response.read()

你可以使用python请求

import os
import requests


outfile = os.path.join(SAVE_DIR, file_name)
response = requests.get(URL, stream=True)
with open(outfile,'wb') as output:
  output.write(response.content)

你可以使用shutil

import os
import requests
import shutil
 
outfile = os.path.join(SAVE_DIR, file_name)
response = requests.get(url, stream = True)
with open(outfile, 'wb') as f:
  shutil.copyfileobj(response.content, f)

如果你从受限的url下载，不要忘记在标题中包含访问令牌

Python 2/3的PabloG代码的改进版本:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import ( division, absolute_import, print_function, unicode_literals )

import sys, os, tempfile, logging

if sys.version_info >= (3,):
    import urllib.request as urllib2
    import urllib.parse as urlparse
else:
    import urllib2
    import urlparse

def download_file(url, dest=None):
    """ 
    Download and save a file specified by url to dest directory,
    """
    u = urllib2.urlopen(url)

    scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
    filename = os.path.basename(path)
    if not filename:
        filename = 'downloaded.file'
    if dest:
        filename = os.path.join(dest, filename)

    with open(filename, 'wb') as f:
        meta = u.info()
        meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all
        meta_length = meta_func("Content-Length")
        file_size = None
        if meta_length:
            file_size = int(meta_length[0])
        print("Downloading: {0} Bytes: {1}".format(url, file_size))

        file_size_dl = 0
        block_sz = 8192
        while True:
            buffer = u.read(block_sz)
            if not buffer:
                break

            file_size_dl += len(buffer)
            f.write(buffer)

            status = "{0:16}".format(file_size_dl)
            if file_size:
                status += "   [{0:6.2f}%]".format(file_size_dl * 100 / file_size)
            status += chr(13)
            print(status, end="")
        print()

    return filename

if __name__ == "__main__":  # Only run if this file is called directly
    print("Testing with 10MB download")
    url = "http://download.thinkbroadband.com/10MB.zip"
    filename = download_file(url)
    print(filename)