我还发现一些网站使用了一个检查用户代理的平台。如果该值包含:"HeadlessChrome"，则使用无头模式时行为可能会很奇怪。

解决方法是重写用户代理的值，例如在Java中:

chromeOptions.addArguments("--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36");

正如我们已经在问题和发布的答案中发现的那样，这里有一个名为“蒸馏网络”的反网络抓取和机器人检测服务。根据该公司CEO的采访:

尽管他们可以创造新的机器人，但我们找到了识别的方法 硒是他们使用的一个工具，所以我们阻止硒不 不管他们在机器人上迭代了多少次。我们现在正在做 使用Python和许多不同的技术。一旦我们发现了规律 从一种机器人中脱颖而出，然后我们对其进行逆向工程 他们使用的技术并将其识别为恶意的。

要了解他们究竟是如何检测硒的，还需要时间和更多的挑战，但目前我们可以肯定地说:

it's not related to the actions you take with Selenium. Once you navigate to the site, you get immediately detected and banned. I've tried to add artificial random delays between actions, take a pause after the page is loaded - nothing helped it's not about browser fingerprint either. I tried it in multiple browsers with clean profiles and not, incognito modes, but nothing helped since, according to the hint in the interview, this was "reverse engineering", I suspect this is done with some JavaScript code being executed in the browser revealing that this is a browser automated via Selenium WebDriver

我决定把它作为一个答案，因为很明显:

当你用chromedriver使用selenium时，网站能检测到吗?

Yes.

Also, I haven't experimented with older Selenium and older browser versions. In theory, there could be something implemented/added to Selenium at a certain point that Distil Networks bot detector currently relies on. Then, if this is the case, we might detect (yeah, let's detect the detector) at what point/version a relevant change was made, look into changelog and changesets and, may be, this could give us more information on where to look and what is it they use to detect a webdriver-powered browser. It's just a theory that needs to be tested.

在wellsfargo.com上实现的例子:

try {
 if (window.document.documentElement.getAttribute("webdriver")) return !+[]
} catch (IDLMrxxel) {}
try {
 if ("_Selenium_IDE_Recorder" in window) return !+""
} catch (KknKsUayS) {}
try {
 if ("__webdriver_script_fn" in document) return !+""

基本上，Selenium检测的工作方式是测试与Selenium一起运行时出现的预定义JavaScript变量。机器人检测脚本通常在任何变量(在窗口对象上)中查找包含单词“selenium”/“webdriver”的任何内容，也记录名为$cdc_和$wdc_的变量。当然，所有这些都取决于您使用的是哪种浏览器。所有不同的浏览器都公开不同的内容。

对我来说，我使用Chrome，所以，我所要做的就是确保$cdc_不再存在作为文档变量，和voilà(下载chromedriver源代码，修改chromedriver和重新编译$cdc_在不同的名称。)

这是我在chromedriver中修改的函数:

文件call_function.js:

function getPageCache(opt_doc) {
  var doc = opt_doc || document;
  //var key = '$cdc_asdjflasutopfhvcZLmcfl_';
  var key = 'randomblabla_';
  if (!(key in doc))
    doc[key] = new Cache();
  return doc[key];
}

(注意注释。我所做的就是把$cdc_变成randomblabla_。)

下面是演示机器人网络可能使用的一些技术的伪代码:

runBotDetection = function () {
    var documentDetectionKeys = [
        "__webdriver_evaluate",
        "__selenium_evaluate",
        "__webdriver_script_function",
        "__webdriver_script_func",
        "__webdriver_script_fn",
        "__fxdriver_evaluate",
        "__driver_unwrapped",
        "__webdriver_unwrapped",
        "__driver_evaluate",
        "__selenium_unwrapped",
        "__fxdriver_unwrapped",
    ];

    var windowDetectionKeys = [
        "_phantom",
        "__nightmare",
        "_selenium",
        "callPhantom",
        "callSelenium",
        "_Selenium_IDE_Recorder",
    ];

    for (const windowDetectionKey in windowDetectionKeys) {
        const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
        if (window[windowDetectionKeyValue]) {
            return true;
        }
    };
    for (const documentDetectionKey in documentDetectionKeys) {
        const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
        if (window['document'][documentDetectionKeyValue]) {
            return true;
        }
    };

    for (const documentKey in window['document']) {
        if (documentKey.match(/\$[a-z]dc_/) && window['document'][documentKey]['cache_']) {
            return true;
        }
    }

    if (window['external'] && window['external'].toString() && (window['external'].toString()['indexOf']('Sequentum') != -1)) return true;

    if (window['document']['documentElement']['getAttribute']('selenium')) return true;
    if (window['document']['documentElement']['getAttribute']('webdriver')) return true;
    if (window['document']['documentElement']['getAttribute']('driver')) return true;

    return false;
};

根据用户szx的说法，也可以简单地在十六进制编辑器中打开chromedriver.exe，只需手动进行替换，而无需实际进行任何编译。

它适用于一些网站，从导航器中删除属性webdriver

from selenium import webdriver
driver = webdriver.Chrome()
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source":
        "const newProto = navigator.__proto__;"
        "delete newProto.webdriver;"
        "navigator.__proto__ = newProto;"
    })

我发现改变JavaScript的“key”变量是这样的:

//Fools the website into believing a human is navigating it
((JavascriptExecutor)driver).executeScript("window.key = \"blahblah\";");

当使用Selenium WebDriver和谷歌Chrome时，可以在一些网站上使用，因为许多网站都会检查这个变量，以避免被Selenium刮擦。