我试图获得应用商店>业务的内容:

import requests
from lxml import html

page = requests.get("https://itunes.apple.com/in/genre/ios-business/id6000?mt=8")
tree = html.fromstring(page.text)

flist = []
plist = []
for i in range(0, 100):
    app = tree.xpath("//div[@class='column first']/ul/li/a/@href")
    ap = app[0]
    page1 = requests.get(ap)

当我尝试(0,2)的范围,它工作,但当我把范围在100,它显示这个错误:

Traceback (most recent call last):
  File "/home/preetham/Desktop/eg.py", line 17, in <module>
    page1 = requests.get(ap)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max retries exceeded with url: /in/app/adobe-reader/id469337564?mt=8 (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

当前回答

这里发生的是itunes服务器拒绝你的连接(你在短时间内从同一个ip地址发送了太多的请求)

url: /in/app/adobe-reader/id469337564?太= 8

错误跟踪是误导性的,它应该是类似于“无法建立连接,因为目标机器主动拒绝它”。

关于python有一个问题。请求库在Github,看看这里

为了克服这个问题(与其说是一个问题,不如说是误导调试跟踪),你应该像这样捕捉连接相关的异常:

try:
    page1 = requests.get(ap)
except requests.exceptions.ConnectionError:
    r.status_code = "Connection refused"

另一种克服这个问题的方法是,如果你使用足够的时间间隔来发送请求到服务器,这可以通过python中的sleep(timeinsec)函数来实现(不要忘记导入sleep)

from time import sleep

所有的请求都是很棒的python库,希望能解决你的问题。

其他回答

只需使用请求功能:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


session = requests.Session()
retry = Retry(connect=3, backoff_factor=0.5)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

session.get(url)

这将获取URL并重试3次,以防出现requests.exceptions.ConnectionError。Backoff_factor将有助于在尝试之间应用延迟,以避免在定期请求配额的情况下再次失败。

看一下urllib3.util.retry。重试,它有许多选项来简化重试。

就这么做,

将下面的代码粘贴到page = requests.get(url)的位置:

import time

page = ''
while page == '':
    try:
        page = requests.get(url)
        break
    except:
        print("Connection refused by the server..")
        print("Let me sleep for 5 seconds")
        print("ZZzzzz...")
        time.sleep(5)
        print("Was a nice sleep, now let me continue...")
        continue

不客气:)

实现异常处理总是好的。它不仅有助于避免脚本意外退出,还有助于记录错误和信息通知。当使用Python请求时,我更喜欢捕获这样的异常:

    try:
        res = requests.get(adress,timeout=30)
    except requests.ConnectionError as e:
        print("OOPS!! Connection Error. Make sure you are connected to Internet. Technical Details given below.\n")
        print(str(e))            
        renewIPadress()
        continue
    except requests.Timeout as e:
        print("OOPS!! Timeout Error")
        print(str(e))
        renewIPadress()
        continue
    except requests.RequestException as e:
        print("OOPS!! General Error")
        print(str(e))
        renewIPadress()
        continue
    except KeyboardInterrupt:
        print("Someone closed the program")

这里的renewIPadress()是一个用户定义函数,它可以在IP地址被阻塞时更改IP地址。你可以不用这个函数。

加上我自己的经验:

r = requests.get(download_url)

当我试图下载url中指定的文件时。

错误在于

HTTPSConnectionPool(host, port=443): Max retries exceeded with url (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

我通过在函数中添加verify = False来纠正它,如下所示:

r = requests.get(download_url + filename)
open(filename, 'wb').write(r.content)

我正在用Gauge编写一个测试,我也遇到了这个错误,这是因为我试图在没有激活VPN的情况下请求内部URL。