MySQL数据库在什么时候开始失去性能?
物理数据库大小重要吗? 记录的数量重要吗? 性能下降是线性的还是指数级的?
我有一个我相信是一个大的数据库,大约有1500万条记录,占用了近2GB。基于这些数字,我是否有任何动机清理数据,或者我是否可以允许它继续扩展几年?
MySQL数据库在什么时候开始失去性能?
物理数据库大小重要吗? 记录的数量重要吗? 性能下降是线性的还是指数级的?
我有一个我相信是一个大的数据库,大约有1500万条记录,占用了近2GB。基于这些数字,我是否有任何动机清理数据,或者我是否可以允许它继续扩展几年?
当前回答
I'm currently managing a MySQL database on Amazon's cloud infrastructure that has grown to 160 GB. Query performance is fine. What has become a nightmare is backups, restores, adding slaves, or anything else that deals with the whole dataset, or even DDL on large tables. Getting a clean import of a dump file has become problematic. In order to make the process stable enough to automate, various choices needed to be made to prioritize stability over performance. If we ever had to recover from a disaster using a SQL backup, we'd be down for days.
Horizontally scaling SQL is also pretty painful, and in most cases leads to using it in ways you probably did not intend when you chose to put your data in SQL in the first place. Shards, read slaves, multi-master, et al, they are all really shitty solutions that add complexity to everything you ever do with the DB, and not one of them solves the problem; only mitigates it in some ways. I would strongly suggest looking at moving some of your data out of MySQL (or really any SQL) when you start approaching a dataset of a size where these types of things become an issue.
更新:几年后,我们的数据集已经增长到大约800 GiB。此外,我们还有一个200+ GiB的表和其他一些50-100 GiB的表。我之前说的都成立。它的性能仍然很好,但运行完整数据集操作的问题变得更糟了。
其他回答
物理数据库大小无关紧要。记录的数量并不重要。
In my experience the biggest problem that you are going to run in to is not size, but the number of queries you can handle at a time. Most likely you are going to have to move to a master/slave configuration so that the read queries can run against the slaves and the write queries run against the master. However if you are not ready for this yet, you can always tweak your indexes for the queries you are running to speed up the response times. Also there is a lot of tweaking you can do to the network stack and kernel in Linux that will help.
我的内存达到了10GB,只有中等数量的连接,它处理请求还不错。
我将首先关注您的索引,然后让服务器管理员查看您的操作系统,如果所有这些都没有帮助,那么可能是时候实现主/从配置了。
总的来说,这是一个非常微妙的问题,无论如何都不是微不足道的。我建议你阅读mysqlperformanceblog.com和高性能MySQL。我真的认为这个问题没有普遍的答案。
我正在做一个项目,它有一个MySQL数据库,几乎有1TB的数据。最重要的可伸缩性因素是RAM。如果您的表的索引适合内存,并且您的查询得到了高度优化,那么您可以使用普通机器处理合理数量的请求。
记录的数量确实很重要,这取决于表的外观。有很多varchar字段和只有几个int或long类型是有区别的。
数据库的物理大小也很重要:例如,考虑备份。根据你的引擎,你的物理db文件会增长,但不会缩小,例如innodb。因此,删除大量的行,并不有助于缩小您的物理文件。
这个问题有很多,在很多情况下,细节决定成败。
I'm currently managing a MySQL database on Amazon's cloud infrastructure that has grown to 160 GB. Query performance is fine. What has become a nightmare is backups, restores, adding slaves, or anything else that deals with the whole dataset, or even DDL on large tables. Getting a clean import of a dump file has become problematic. In order to make the process stable enough to automate, various choices needed to be made to prioritize stability over performance. If we ever had to recover from a disaster using a SQL backup, we'd be down for days.
Horizontally scaling SQL is also pretty painful, and in most cases leads to using it in ways you probably did not intend when you chose to put your data in SQL in the first place. Shards, read slaves, multi-master, et al, they are all really shitty solutions that add complexity to everything you ever do with the DB, and not one of them solves the problem; only mitigates it in some ways. I would strongly suggest looking at moving some of your data out of MySQL (or really any SQL) when you start approaching a dataset of a size where these types of things become an issue.
更新:几年后,我们的数据集已经增长到大约800 GiB。此外,我们还有一个200+ GiB的表和其他一些50-100 GiB的表。我之前说的都成立。它的性能仍然很好,但运行完整数据集操作的问题变得更糟了。
2GB和约15M条记录是一个非常小的数据库-我在奔腾III上运行过更大的数据库(!),一切仍然运行得非常快。如果你的慢,那是数据库/应用程序设计的问题,而不是mysql的问题。
如果数据库设计不当,性能可能会在几千行中下降。
如果你有合适的索引,使用合适的引擎(不要使用MyISAM,因为需要多个dml),使用分区,根据使用情况分配正确的内存,当然还有良好的服务器配置,MySQL可以处理tb级的数据!
总有办法提高数据库性能。