首先，创建一个s3客户端对象:

s3_client = boto3.client('s3')

接下来，创建一个变量来保存bucket名称和文件夹。注意文件夹名后面的斜杠“/”:

bucket_name = 'my-bucket'
folder = 'some-folder/'

接下来，调用s3_client。List_objects_v2获取文件夹内容对象的元数据:

response = s3_client.list_objects_v2(
  Bucket=bucket_name,
  Prefix=folder
)

最后，使用对象的元数据，您可以通过调用s3_client来获取S3对象。get_object功能:

for object_metadata in response['Contents']:
    object_key = object_metadata['Key']
    response = s3_client.get_object(
        Bucket=bucket_name,
        Key=object_key
    )
    object_body = response['Body'].read()
    print(object_body)

如你所见，字符串格式的对象内容可以通过调用response['Body'].read()来获得。

如果你想传递ACCESS和SECRET密钥(你不应该这样做，因为这是不安全的):

from boto3.session import Session

ACCESS_KEY='your_access_key'
SECRET_KEY='your_secret_key'

session = Session(aws_access_key_id=ACCESS_KEY,
                  aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('your_bucket')

for s3_file in your_bucket.objects.all():
    print(s3_file.key)

为了处理大型键列表(即当目录列表大于1000项时)，我使用以下代码将多个列表中的键值(即文件名)累积起来(感谢上面的阿梅里奥的第一行)。代码是针对python3的:

    from boto3  import client
    bucket_name = "my_bucket"
    prefix      = "my_key/sub_key/lots_o_files"

    s3_conn   = client('s3')  # type: BaseClient  ## again assumes boto.cfg setup, assume AWS S3
    s3_result =  s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")

    if 'Contents' not in s3_result:
        #print(s3_result)
        return []

    file_list = []
    for key in s3_result['Contents']:
        file_list.append(key['Key'])
    print(f"List count = {len(file_list)}")

    while s3_result['IsTruncated']:
        continuation_key = s3_result['NextContinuationToken']
        s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
        for key in s3_result['Contents']:
            file_list.append(key['Key'])
        print(f"List count = {len(file_list)}")
    return file_list

我花了一整晚的时间在这个问题上因为我只想知道子文件夹下的文件数但它也返回了一个额外的文件在子文件夹本身的内容中，

在研究之后，我发现这就是s3的工作方式 一个场景，我卸载数据从红移在以下目录

s3://bucket_name/subfolder/<10 number of files>

当我用

paginator.paginate(Bucket=price_signal_bucket_name,Prefix=new_files_folder_path+"/")

它只会返回10个文件，但当我在s3桶上创建文件夹时，它也会返回子文件夹

结论

如果整个文件夹都上传到s3，那么列出only将返回前缀下的文件 但是如果文件夹是在s3桶本身创建的，那么使用boto3客户端列出它也将返回子文件夹和文件

使用cloudpathlib

cloudpathlib提供了一个方便的包装器，这样您就可以使用简单的pathlib API与AWS S3(以及Azure blob存储、GCS等)进行交互。你可以用pip install "cloudpathlib[s3]"来安装。

像pathlib一样，你可以使用glob或iterdir来列出目录的内容。

下面是一个带有公共AWS S3桶的示例，您可以复制并过去运行该桶。

from cloudpathlib import CloudPath

s3_path = CloudPath("s3://ladi/Images/FEMA_CAP/2020/70349")

# list items with glob
list(
    s3_path.glob("*")
)[:3]
#> [ S3Path('s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0001_5a63d42e-27c6-448a-84f1-bfc632125b8e.jpg'),
#>   S3Path('s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0002_a89f1b79-786f-4dac-9dcc-609fb1a977b1.jpg'),
#>   S3Path('s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0003_02c30af6-911e-4e01-8c24-7644da2b8672.jpg')]

# list items with iterdir
list(
    s3_path.iterdir()
)[:3]
#> [ S3Path('s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0001_5a63d42e-27c6-448a-84f1-bfc632125b8e.jpg'),
#>   S3Path('s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0002_a89f1b79-786f-4dac-9dcc-609fb1a977b1.jpg'),
#>   S3Path('s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0003_02c30af6-911e-4e01-8c24-7644da2b8672.jpg')]

创建于2021-05-21 20:38:47 PDT由reprexlite v0.4.2创建

ObjectSummary:

有两个标识符附加到ObjectSummary:

bucket_name 关键

boto3 S3: ObjectSummary

有关AWS S3文档中的对象键的更多信息:

Object Keys: When you create an object, you specify the key name, which uniquely identifies the object in the bucket. For example, in the Amazon S3 console (see AWS Management Console), when you highlight a bucket, a list of objects in your bucket appears. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is at most 1024 bytes long. The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. Suppose that your bucket (admin-created) has four objects with the following object keys: Development/Projects1.xls Finance/statement1.pdf Private/taxdocument.pdf s3-dg.pdf Reference: AWS S3: Object Keys

下面是一些示例代码，演示如何获取桶名和对象键。

例子:

import boto3
from pprint import pprint

def main():

    def enumerate_s3():
        s3 = boto3.resource('s3')
        for bucket in s3.buckets.all():
             print("Name: {}".format(bucket.name))
             print("Creation Date: {}".format(bucket.creation_date))
             for object in bucket.objects.all():
                 print("Object: {}".format(object))
                 print("Object bucket_name: {}".format(object.bucket_name))
                 print("Object key: {}".format(object.key))

    enumerate_s3()


if __name__ == '__main__':
    main()