我一直在用Chromedriver测试Selenium,我注意到一些页面可以检测到你正在使用Selenium,即使根本没有自动化。甚至当我手动使用Chrome通过Selenium和Xephyr浏览时,我经常会看到一个页面说检测到可疑活动。我已经检查了我的用户代理和浏览器指纹,它们都与正常的Chrome浏览器完全相同。

当我在普通的Chrome浏览器中浏览这些网站时,一切都很好,但当我使用Selenium时,我被检测到。

理论上,chromedriver和Chrome在任何web服务器上看起来应该是完全一样的,但不知何故它们可以检测到它。

如果你想要一些测试代码,试试这个:

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=1, size=(1600, 902))
display.start()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('--profile-directory=Default')
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-plugins-discovery");
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.delete_all_cookies()
driver.set_window_size(800,800)
driver.set_window_position(0,0)
print 'arguments done'
driver.get('http://stubhub.com')

如果你在stubhub周围浏览,你会在一两个请求内被重定向和“阻止”。我一直在研究这个问题,但我不知道他们是如何判断用户正在使用Selenium的。

他们是怎么做到的?

我在Firefox中安装了Selenium IDE插件,当我在普通的Firefox浏览器中只使用附加插件访问stubhub.com时,我被禁止了。

当我使用Fiddler查看来回发送的HTTP请求时,我注意到“假浏览器”的请求经常在响应头中有“无缓存”。

是否有一种方法可以从JavaScript检测我是否在Selenium Webdriver页面中?建议当你在使用网络驱动程序时没有办法检测。但这些证据表明情况并非如此。

该网站将指纹上传到他们的服务器上,但我检查了一下,Selenium的指纹与使用Chrome时的指纹是相同的。

这是他们发送到服务器上的指纹载荷之一:

{"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en-
US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins":
{"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionMo
dule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes":
{"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave-
flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContent
DecryptionModuleapplication/x-ppapi-widevine-
cdm","4":"NativeClientExecutableapplication/x-
nacl","5":"PortableNativeClientExecutableapplication/x-
pnacl","6":"PortableDocumentFormatapplication/x-google-chrome-
pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts":
{"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"Trebu
chetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationM
ono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}

它在Selenium和Chrome中是相同的。

vpn只用于一次使用,但在加载第一个页面后就会被检测到。显然,正在运行一些JavaScript代码来检测Selenium。


当前回答

我所要做的就是

my_options = webdriver.ChromeOptions()
my_options.add_argument( '--disable-blink-features=AutomationControlled' )

Some more information to this: This relates to website skyscanner.com. In the past I have been able to scrape it. Yes, it did detect the browser automation and it gave me a captcha to press and hold a button. I used to be able to complete the captcha manually, then search flights and then scrape. But this time around after completing the captcha I get the same captcha again and again, just can't seem to escape from it. I tried some of the most popular suggestions to avoid automation being detected, but they didn't work. Then I found this article which did work, and by process of elimination I found out it only took the option above to get around their browser automation detection. Now I don't even get the captcha and everything else seems to be working normally.

我目前正在运行的版本:

操作系统:Windows 7 64位 win32上的Python 3.8.0 (tags/v3.8.0:fa919fd, 2019-10-14) (MSC v.1916 64位(AMD64)) 浏览器:Chrome版本100.0.4896.60(官方 构建)(64位) 4.1.3硒 ChromeDriver 100.0.4896.60 chromedriver_win32.zip 930ff33ae8babeaa74e0dd1ce1dae7ff

其他回答

我也有同样的问题,用下面的配置解决了这个问题(用c#)

options.AddArguments("start-maximized");
options.AddArguments("--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36");

options.AddExcludedArgument("enable-automation"); // For hiding chrome being controlled by automation..
options.AddAdditionalCapability("useAutomationExtension", false);

// Import cookies
options.AddArguments("user-data-dir=" + userDataDir);

options.AddArguments("profile-directory=" + profileDir);

我还发现一些网站使用了一个检查用户代理的平台。如果该值包含:"HeadlessChrome",则使用无头模式时行为可能会很奇怪。

解决方法是重写用户代理的值,例如在Java中:

chromeOptions.addArguments("--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36");

部分接口导航器{只读属性布尔webdriver;};

Navigator接口的webdriver IDL属性必须返回webdriver-active标志的值,初始值为false。

此属性允许网站确定用户代理受WebDriver控制,并可用于帮助减轻拒绝服务攻击。

直接摘自2017年W3C WebDriver编辑草案。这在很大程度上意味着,至少Selenium驱动程序的未来迭代将是可识别的,以防止误用。最终,如果没有源代码,就很难判断到底是什么原因导致chrome驱动程序被检测到。

它适用于一些网站,从导航器中删除属性webdriver

from selenium import webdriver
driver = webdriver.Chrome()
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source":
        "const newProto = navigator.__proto__;"
        "delete newProto.webdriver;"
        "navigator.__proto__ = newProto;"
    })

一些网站发现了这一点:

function d() {
try {
    if (window.document.$cdc_asdjflasutopfhvcZLmcfl_.cache_)
        return !0
} catch (e) {}

try {
    //if (window.document.documentElement.getAttribute(decodeURIComponent("%77%65%62%64%72%69%76%65%72")))
    if (window.document.documentElement.getAttribute("webdriver"))
        return !0
} catch (e) {}

try {
    //if (decodeURIComponent("%5F%53%65%6C%65%6E%69%75%6D%5F%49%44%45%5F%52%65%63%6F%72%64%65%72") in window)
    if ("_Selenium_IDE_Recorder" in window)
        return !0
} catch (e) {}

try {
    //if (decodeURIComponent("%5F%5F%77%65%62%64%72%69%76%65%72%5F%73%63%72%69%70%74%5F%66%6E") in document)
    if ("__webdriver_script_fn" in document)
        return !0
} catch (e) {}