我想从目录中读取几个CSV文件到熊猫，并将它们连接到一个大的DataFrame。不过我还没弄明白。以下是我目前所掌握的:
import glob
import pandas as pd
# Get data file names
path = r'C:\DRO\DCL_rawdata_files'
filenames = glob.glob(path + "/*.csv")
dfs = []
for filename in filenames:
dfs.append(pd.read_csv(filename))
# Concatenate all data into one DataFrame
big_frame = pd.concat(dfs, ignore_index=True)
我想我在for循环中需要一些帮助?
当前回答
darindaCoder的答案的替代方案:
path = r'C:\DRO\DCL_rawdata_files' # use your path
all_files = glob.glob(os.path.join(path, "*.csv")) # advisable to use os.path.join as this makes concatenation OS independent
df_from_each_file = (pd.read_csv(f) for f in all_files)
concatenated_df = pd.concat(df_from_each_file, ignore_index=True)
# doesn't create a list, nor does it append to one
其他回答
Dask库可以从多个文件中读取数据帧:
>>> import dask.dataframe as dd
>>> df = dd.read_csv('data*.csv')
(来源:https://examples.dask.org/dataframes/01-data-access.html # Read-CSV-files)
Dask数据框架实现了Pandas数据框架API的一个子集。如果所有的数据都适合内存，你可以调用df.compute()将数据帧转换为Pandas数据帧。
可选择使用pathlib库(通常优先于os.path)。
该方法避免了重复使用pandas concat()/ apping()。
从熊猫文档中可以看到: 值得注意的是，concat()(因此append())会生成数据的完整副本，并且不断重用此函数会产生显著的性能影响。如果需要对多个数据集使用操作，请使用列表推导式。
import pandas as pd
from pathlib import Path
dir = Path("../relevant_directory")
df = (pd.read_csv(f) for f in dir.glob("*.csv"))
df = pd.concat(df)
import pandas as pd
import glob
path = r'C:\DRO\DCL_rawdata_files' # use your path
file_path_list = glob.glob(path + "/*.csv")
file_iter = iter(file_path_list)
list_df_csv = []
list_df_csv.append(pd.read_csv(next(file_iter)))
for file in file_iter:
lsit_df_csv.append(pd.read_csv(file, header=0))
df = pd.concat(lsit_df_csv, ignore_index=True)
如果多个CSV文件被压缩，您可以使用zipfile读取所有文件并按以下方式连接:
import zipfile
import pandas as pd
ziptrain = zipfile.ZipFile('yourpath/yourfile.zip')
train = []
train = [ pd.read_csv(ziptrain.open(f)) for f in ziptrain.namelist() ]
df = pd.concat(train)
你也可以这样做:
import pandas as pd
import os
new_df = pd.DataFrame()
for r, d, f in os.walk(csv_folder_path):
for file in f:
complete_file_path = csv_folder_path+file
read_file = pd.read_csv(complete_file_path)
new_df = new_df.append(read_file, ignore_index=True)
new_df.shape
