在一个目录中保存多少文件有关系吗?如果是,一个目录中有多少文件是太多的,拥有太多文件的影响是什么?(这是在Linux服务器上。)

Background: I have a photo album website, and every image uploaded is renamed to an 8-hex-digit id (say, a58f375c.jpg). This is to avoid filename conflicts (if lots of "IMG0001.JPG" files are uploaded, for example). The original filename and any useful metadata is stored in a database. Right now, I have somewhere around 1500 files in the images directory. This makes listing the files in the directory (through FTP or SSH client) take a few seconds. But I can't see that it has any effect other than that. In particular, there doesn't seem to be any impact on how quickly an image file is served to the user.

我想过通过创建16个子目录来减少图像的数量:0-9和a-f。然后我将根据文件名的第一个十六进制数字将图像移动到子目录中。但是除了偶尔通过FTP/SSH列出目录之外,我不确定这样做是否有任何理由。


当前回答

我也有同样的问题。试图在ext4的Ubuntu服务器中存储数百万个文件。结束运行我自己的基准测试。发现平面目录执行得更好,同时使用起来更简单:

写了一篇文章。

其他回答

如果实现目录分区方案所涉及的时间是最少的,我赞成它。当您第一次调试涉及通过控制台操作10,000个文件目录的问题时,您将能够理解。

例如,F-Spot将照片文件存储为YYYY\MM\DD\filename。ext,这意味着在手动操作我的~20000张照片集合时,我必须处理的最大目录大约有800个文件。这也使文件更容易从第三方应用程序中浏览。永远不要以为只有你的软件会访问你的软件文件。

没有一个数字是“太多”的,只要它不超过操作系统的限制。然而,不管哪种操作系统,一个目录中的文件越多,访问任何单个文件所需的时间就越长,而且在大多数操作系统上,性能是非线性的,因此从10,000个文件中找到一个文件所需的时间是在1,000个文件中找到一个文件所需的时间的10倍以上。

与目录中有大量文件相关的次要问题包括通配符展开失败。为了降低风险,您可以考虑根据上传日期或其他有用的元数据对目录进行排序。

上面的大多数答案都没有说明,对于最初的问题,没有“一刀切”的答案。

In today's environment we have a large conglomerate of different hardware and software -- some is 32 bit, some is 64 bit, some is cutting edge and some is tried and true - reliable and never changing. Added to that is a variety of older and newer hardware, older and newer OSes, different vendors (Windows, Unixes, Apple, etc.) and a myriad of utilities and servers that go along. As hardware has improved and software is converted to 64 bit compatibility, there has necessarily been considerable delay in getting all the pieces of this very large and complex world to play nicely with the rapid pace of changes.

恕我直言,没有一种方法可以解决问题。解决办法是研究各种可能性,然后通过反复试验找到最适合你特定需求的方法。每个用户必须确定什么适合他们的系统,而不是使用千篇一律的方法。

I for example have a media server with a few very large files. The result is only about 400 files filling a 3 TB drive. Only 1% of the inodes are used but 95% of the total space is used. Someone else, with a lot of smaller files may run out of inodes before they come near to filling the space. (On ext4 filesystems as a rule of thumb, 1 inode is used for each file/directory.) While theoretically the total number of files that may be contained within a directory is nearly infinite, practicality determines that the overall usage determine realistic units, not just filesystem capabilities.

我希望以上所有不同的答案都能促进思考和解决问题,而不是成为进步的不可逾越的障碍。

请记住,在Linux上,如果目录中有太多文件,shell可能无法展开通配符。我在Linux上托管的相册有这个问题。它将所有调整大小的图像存储在一个目录中。虽然文件系统可以处理许多文件,但shell不能。例子:

-shell-3.00$ ls A*
-shell: /bin/ls: Argument list too long

or

-shell-3.00$ chmod 644 *jpg
-shell: /bin/chmod: Argument list too long

我也遇到过类似的问题。我试图访问一个有一万多个文件的目录。构建文件列表和在任何文件上运行任何类型的命令花费的时间太长了。

我想了一个小php脚本来为自己做这件事,并试图找出一种方法来防止它在浏览器中超时。

下面是我编写的php脚本来解决这个问题。

列出FTP目录中文件太多的文件

如何帮助别人