File.readlines()接受一个可选的size参数，它近似于在返回的行中读取的行数。

bigfile = open('bigfilename','r')
tmp_lines = bigfile.readlines(BUF_SIZE)
while tmp_lines:
    process([line for line in tmp_lines])
    tmp_lines = bigfile.readlines(BUF_SIZE)

如果您的计算机、操作系统和python是64位的，那么您可以使用mmap模块将文件的内容映射到内存中，并使用索引和切片访问它。下面是文档中的一个例子:

import mmap
with open("hello.txt", "r+") as f:
    # memory-map the file, size 0 means whole file
    map = mmap.mmap(f.fileno(), 0)
    # read content via standard file methods
    print map.readline()  # prints "Hello Python!"
    # read content via slice notation
    print map[:5]  # prints "Hello"
    # update content using slice notation;
    # note that new content must have same size
    map[6:] = " world!\n"
    # ... and read again using standard file methods
    map.seek(0)
    print map.readline()  # prints "Hello  world!"
    # close the map
    map.close()

如果你的计算机、操作系统或python是32位的，那么映射大文件会占用你的大部分地址空间，并耗尽程序的内存。

更新: 也可以使用file_object。如果你想让数据块以完整的行形式给出结果我的意思是没有未完成的行会出现在结果中。

例如:-

def read_in_chunks(file_object, chunk_size=1024):
    """Lazy function (generator) to read a file piece by piece.
    Default chunk size: 1k."""
    while True:
        data = file_object.readlines(chunk_size)
        if not data:
            break
        yield data

——在给出的答案上加上——

当我在chunk中读取文件时，让我们假设一个名为split.txt的文本文件，我在chunk中读取时面临的问题是，我有一个用例，我正在逐行处理数据，只是因为我在chunk中读取文本文件，它(文件块)有时以部分行结束，最终破坏了我的代码(因为它期望完整的行被处理)

阅读之后,我知道我能克服这个问题通过保持一块跟踪的最后一点我做的是如果块/ n,这意味着块包含一个完整的线,否则我通常存储部分最后一行和保持它在一个变量中,以便我可以利用这一点,将它与下一个未完成的线在接下来的一部分与我成功地克服这个问题。

示例代码:-

# in this function i am reading the file in chunks
def read_in_chunks(file_object, chunk_size=1024):
    """Lazy function (generator) to read a file piece by piece.
    Default chunk size: 1k."""
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data

# file where i am writing my final output
write_file=open('split.txt','w')

# variable i am using to store the last partial line from the chunk
placeholder= ''
file_count=1

try:
    with open('/Users/rahulkumarmandal/Desktop/combined.txt') as f:
        for piece in read_in_chunks(f):
            #print('---->>>',piece,'<<<--')
            line_by_line = piece.split('\n')

            for one_line in line_by_line:
                # if placeholder exist before that means last chunk have a partial line that we need to concatenate with the current one 
                if placeholder:
                    # print('----->',placeholder)
                    # concatinating the previous partial line with the current one
                    one_line=placeholder+one_line
                    # then setting the placeholder empty so that next time if there's a partial line in the chunk we can place it in the variable to be concatenated further
                    placeholder=''
                
                # futher logic that revolves around my specific use case
                segregated_data= one_line.split('~')
                #print(len(segregated_data),type(segregated_data), one_line)
                if len(segregated_data) < 18:
                    placeholder=one_line
                    continue
                else:
                    placeholder=''
                #print('--------',segregated_data)
                if segregated_data[2]=='2020' and segregated_data[3]=='2021':
                    #write this
                    data=str("~".join(segregated_data))
                    #print('data',data)
                    #f.write(data)
                    write_file.write(data)
                    write_file.write('\n')
                    print(write_file.tell())
                elif segregated_data[2]=='2021' and segregated_data[3]=='2022':
                    #write this
                    data=str("-".join(segregated_data))
                    write_file.write(data)
                    write_file.write('\n')
                    print(write_file.tell())
except Exception as e:
    print('error is', e)                

由于我的低声誉，我不允许评论，但SilentGhosts解决方案应该更容易与file.readlines([sizehint])

Python文件方法

编辑:SilentGhost是对的，但这应该比:

s = "" 
for i in xrange(100): 
   s += file.next()

f = ... # file-like object, i.e. supporting read(size) function and 
        # returning empty string '' when there is nothing to read

def chunked(file, chunk_size):
    return iter(lambda: file.read(chunk_size), '')

for data in chunked(f, 65536):
    # process the data

更新:该方法最好在https://stackoverflow.com/a/4566523/38592中解释

File.readlines()接受一个可选的size参数，它近似于在返回的行中读取的行数。

bigfile = open('bigfilename','r')
tmp_lines = bigfile.readlines(BUF_SIZE)
while tmp_lines:
    process([line for line in tmp_lines])
    tmp_lines = bigfile.readlines(BUF_SIZE)