我有一个相当大的音乐网站，有一个很大的艺术家数据库。我一直注意到其他音乐网站在窃取我们网站的数据(我在这里和那里输入假艺人的名字，然后进行谷歌搜索)。
如何防止屏幕刮擦?这可能吗?
当前回答
快速解决这个问题的方法是设置一个陷阱。
Make a page that if it's opened a certain amount of times or even opened at all, will collect certain information like the IP and whatnot (you can also consider irregularities or patterns but this page shouldn't have to be opened at all). Make a link to this in your page that is hidden with CSS display:none; or left:-9999px; positon:absolute; try to place it in places that are less unlikely to be ignored like where your content falls under and not your footer as sometimes bots can choose to forget about certain parts of a page. In your robots.txt file set a whole bunch of disallow rules to pages you don't want friendly bots (LOL, like they have happy faces!) to gather information on and set this page as one of them. Now, If a friendly bot comes through it should ignore that page. Right but that still isn't good enough. Make a couple more of these pages or somehow re-route a page to accept differnt names. and then place more disallow rules to these trap pages in your robots.txt file alongside pages you want ignored. Collect the IP of these bots or anyone that enters into these pages, don't ban them but make a function to display noodled text in your content like random numbers, copyright notices, specific text strings, display scary pictures, basically anything to hinder your good content. You can also set links that point to a page which will take forever to load ie. in php you can use the sleep() function. This will fight the crawler back if it has some sort of detection to bypass pages that take way too long to load as some well written bots are set to process X amount of links at a time. If you have made specific text strings/sentences why not go to your favorite search engine and search for them, it might show you where your content is ending up.
无论如何，如果你从战术和创造性的角度思考，这可能是一个很好的起点。最好的办法就是学习机器人是如何工作的。
我还会考虑打乱一些ID或页面元素上的属性显示方式:
<a class="someclass" href="../xyz/abc" rel="nofollow" title="sometitle">
每次都会改变它的形式，因为一些机器人可能会在你的页面或目标元素中寻找特定的模式。
<a title="sometitle" href="../xyz/abc" rel="nofollow" class="someclass">
id="p-12802" > id="p-00392"
其他回答
屏幕抓取器通过处理HTML来工作。如果他们决定要获取你的数据，从技术上讲你也无能为力，因为人的眼球可以处理任何东西。法律上已经指出，你可能有一些追索权，这是我的建议。
但是，您可以通过使用非基于html的表示逻辑来隐藏数据的关键部分
为每个艺术家/专辑等生成一个Flash文件。 为每个艺术家内容生成一个图像。也许只是一个艺术家名字的图像，等等就足够了。通过将文本呈现到服务器上的JPEG/PNG文件并链接到该图像来实现这一点。
请记住，这可能会影响你的搜索排名。
当然，这是可能的。为了100%的成功，让你的网站离线。
在现实中，你可以做一些事情，让抓取变得更加困难。谷歌进行浏览器检查，以确保您不是一个抓取搜索结果的机器人(尽管这和大多数其他事情一样，可以被欺骗)。
你可以做一些事情，比如在第一次连接到你的网站和随后的点击之间需要几秒钟。我不确定理想的时间是什么，也不知道具体怎么做，但这是另一个想法。
我相信还有其他一些人有更多的经验，但我希望这些想法至少有一定的帮助。
快速解决这个问题的方法是设置一个陷阱。
Make a page that if it's opened a certain amount of times or even opened at all, will collect certain information like the IP and whatnot (you can also consider irregularities or patterns but this page shouldn't have to be opened at all). Make a link to this in your page that is hidden with CSS display:none; or left:-9999px; positon:absolute; try to place it in places that are less unlikely to be ignored like where your content falls under and not your footer as sometimes bots can choose to forget about certain parts of a page. In your robots.txt file set a whole bunch of disallow rules to pages you don't want friendly bots (LOL, like they have happy faces!) to gather information on and set this page as one of them. Now, If a friendly bot comes through it should ignore that page. Right but that still isn't good enough. Make a couple more of these pages or somehow re-route a page to accept differnt names. and then place more disallow rules to these trap pages in your robots.txt file alongside pages you want ignored. Collect the IP of these bots or anyone that enters into these pages, don't ban them but make a function to display noodled text in your content like random numbers, copyright notices, specific text strings, display scary pictures, basically anything to hinder your good content. You can also set links that point to a page which will take forever to load ie. in php you can use the sleep() function. This will fight the crawler back if it has some sort of detection to bypass pages that take way too long to load as some well written bots are set to process X amount of links at a time. If you have made specific text strings/sentences why not go to your favorite search engine and search for them, it might show you where your content is ending up.
无论如何，如果你从战术和创造性的角度思考，这可能是一个很好的起点。最好的办法就是学习机器人是如何工作的。
我还会考虑打乱一些ID或页面元素上的属性显示方式:
<a class="someclass" href="../xyz/abc" rel="nofollow" title="sometitle">
每次都会改变它的形式，因为一些机器人可能会在你的页面或目标元素中寻找特定的模式。
<a title="sometitle" href="../xyz/abc" rel="nofollow" class="someclass">
id="p-12802" > id="p-00392"
方法一(小型场地适用): 提供加密/编码的数据。我使用python (urllib, requests, beautifulSoup等)对web进行了Scape，发现许多网站提供加密/编码的数据，这些数据在任何编程语言中都无法解密，因为加密方法不存在。
我在一个PHP网站上通过加密和最小化输出实现了这一点(警告:这对大型网站来说不是一个好主意)，响应总是混乱的内容。
最小化PHP输出的例子(如何最小化PHP页面html输出?):
<?php
function sanitize_output($buffer) {
$search = array(
'/\>[^\S ]+/s', // strip whitespaces after tags, except space
'/[^\S ]+\</s', // strip whitespaces before tags, except space
'/(\s)+/s' // shorten multiple whitespace sequences
);
$replace = array('>', '<', '\\1');
$buffer = preg_replace($search, $replace, $buffer);
return $buffer;
}
ob_start("sanitize_output");
?>
方法二: 如果你不能阻止他们，那就给他们提供虚假/无用的数据作为回应。
方法三: 阻止常见的抓取用户代理，你会在主要/大型网站上看到这一点，因为你的user - agent不可能用“python3.4”来抓取它们。
方法四: 确保所有的用户标题都是有效的，我有时提供尽可能多的标题，使我的刮板看起来像一个真实的用户，其中一些甚至不是真实或有效的像en-FU:)。 下面是我通常提供的一些头文件的列表。
headers = {
"Requested-URI": "/example",
"Request-Method": "GET",
"Remote-IP-Address": "656.787.909.121",
"Remote-IP-Port": "69696",
"Protocol-version": "HTTP/1.1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip,deflate",
"Accept-Language": "en-FU,en;q=0.8",
"Cache-Control": "max-age=0",
"Connection": "keep-alive",
"Dnt": "1",
"Host": "http://example.com",
"Referer": "http://example.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"
}
一种方法是将内容作为XML属性、URL编码的字符串、使用HTML编码的JSON预格式化的文本或数据uri提供，然后在客户机上将其转换为HTML。以下是一些这样做的网站:
Skechers: XML <document filename="" height="" width="" title="SKECHERS" linkType="" linkUrl="" imageMap="" href="http://www.bobsfromskechers.com" alt="BOBS from Skechers" title="BOBS from Skechers" /> Chrome Web Store: JSON <script type="text/javascript" src="https://apis.google.com/js/plusone.js">{"lang": "en", "parsetags": "explicit"}</script> Bing News: data URL <script type="text/javascript"> //<