我在日常工作中使用SAS，它的核心支持非常棒。然而，SAS作为一款软件，由于许多其他原因而很糟糕。

有一天，我希望用python和pandas来取代我对SAS的使用，但我目前缺乏用于大型数据集的核心工作流。我不是在谈论需要分布式网络的“大数据”，而是那些大到无法装入内存，但又小到可以装入硬盘的文件。

我的第一个想法是使用HDFStore将大型数据集保存在磁盘上，只将我需要的数据块放入数据框架中进行分析。其他人提到MongoDB是一种更容易使用的替代方案。我的问题是:

完成以下任务的最佳实践工作流程是什么:

将平面文件加载到永久的磁盘数据库结构中 查询该数据库以检索数据以输入pandas数据结构 在操作熊猫的碎片后更新数据库

现实世界的例子将非常受欢迎，尤其是那些在“大数据”上使用熊猫的人。

编辑—我希望这样工作的一个例子:

迭代地导入一个大型平面文件，并将其存储在一个永久的磁盘数据库结构中。这些文件通常太大，无法装入内存。 为了使用Pandas，我希望读取这些数据的子集(通常一次只有几列)，这些子集可以放入内存中。 我将通过对所选列执行各种操作来创建新列。 然后，我必须将这些新列追加到数据库结构中。

我正在努力寻找执行这些步骤的最佳实践方法。阅读关于熊猫和pytables的链接，似乎添加一个新列可能是一个问题。

编辑——具体回答杰夫的问题:

I am building consumer credit risk models. The kinds of data include phone, SSN and address characteristics; property values; derogatory information like criminal records, bankruptcies, etc... The datasets I use every day have nearly 1,000 to 2,000 fields on average of mixed data types: continuous, nominal and ordinal variables of both numeric and character data. I rarely append rows, but I do perform many operations that create new columns. Typical operations involve combining several columns using conditional logic into a new, compound column. For example, if var1 > 2 then newvar = 'A' elif var2 = 4 then newvar = 'B'. The result of these operations is a new column for every record in my dataset. Finally, I would like to append these new columns into the on-disk data structure. I would repeat step 2, exploring the data with crosstabs and descriptive statistics trying to find interesting, intuitive relationships to model. A typical project file is usually about 1GB. Files are organized into such a manner where a row consists of a record of consumer data. Each row has the same number of columns for every record. This will always be the case. It's pretty rare that I would subset by rows when creating a new column. However, it's pretty common for me to subset on rows when creating reports or generating descriptive statistics. For example, I might want to create a simple frequency for a specific line of business, say Retail credit cards. To do this, I would select only those records where the line of business = retail in addition to whichever columns I want to report on. When creating new columns, however, I would pull all rows of data and only the columns I need for the operations. The modeling process requires that I analyze every column, look for interesting relationships with some outcome variable, and create new compound columns that describe those relationships. The columns that I explore are usually done in small sets. For example, I will focus on a set of say 20 columns just dealing with property values and observe how they relate to defaulting on a loan. Once those are explored and new columns are created, I then move on to another group of columns, say college education, and repeat the process. What I'm doing is creating candidate variables that explain the relationship between my data and some outcome. At the very end of this process, I apply some learning techniques that create an equation out of those compound columns.

我很少向数据集中添加行。我几乎总是会创建新的列(统计/机器学习术语中的变量或特征)。