Aws s3 ls s3://bucket-name/folder-prefix-if-any——recursive | wc -l .使用实例

在s3cmd中，简单地运行以下命令(在Ubuntu系统上):

s3cmd ls -r s3://mybucket | wc -l

您可以使用s3的AWS cloudwatch指标来查看每个桶的确切计数。

现在S3 API有一个简单的解决方案(在AWS cli中可用):

aws s3api list-objects --bucket BUCKETNAME --output json --query "[length(Contents[])]"

或针对特定文件夹:

aws s3api list-objects --bucket BUCKETNAME --prefix "folder/subfolder/" --output json --query "[length(Contents[])]"

以下是如何使用java客户端。

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.11.519</version>
</dependency>

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectListing;

public class AmazonS3Service {

    private static final String S3_ACCESS_KEY_ID = "ACCESS_KEY";
    private static final String S3_SECRET_KEY = "SECRET_KEY";
    private static final String S3_ENDPOINT = "S3_URL";

    private AmazonS3 amazonS3;

    public AmazonS3Service() {
        ClientConfiguration clientConfiguration = new ClientConfiguration();
        clientConfiguration.setProtocol(Protocol.HTTPS);
        clientConfiguration.setSignerOverride("S3SignerType");
        BasicAWSCredentials credentials = new BasicAWSCredentials(S3_ACCESS_KEY_ID, S3_SECRET_KEY);
        AWSStaticCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(credentials);
        AmazonS3ClientBuilder.EndpointConfiguration endpointConfiguration = new AmazonS3ClientBuilder.EndpointConfiguration(S3_ENDPOINT, null);
        amazonS3 = AmazonS3ClientBuilder.standard().withCredentials(credentialsProvider).withClientConfiguration(clientConfiguration)
                .withPathStyleAccessEnabled(true).withEndpointConfiguration(endpointConfiguration).build();
    }

    public int countObjects(String bucketName) {
        int count = 0;
        ObjectListing objectListing = amazonS3.listObjects(bucketName);
        int currentBatchCount = objectListing.getObjectSummaries().size();
        while (currentBatchCount != 0) {
            count += currentBatchCount;
            objectListing = amazonS3.listNextBatchOfObjects(objectListing);
            currentBatchCount = objectListing.getObjectSummaries().size();
        }
        return count;
    }
}

在AWS CLI的命令行中，使用ls加——summary。它将为您提供所有项目的列表以及特定bucket中的文档总数。我还没有尝试过包含子桶的桶:

aws s3 ls "s3://MyBucket" --summarize

这可能会花费一些时间(列出我16+K的文档大约需要4分钟)，但这比一次计算1K要快。