人们使用什么技巧来管理交互式R会话的可用内存?我使用下面的函数[基于Petr Pikal和David Hinds在2004年发布的r-help列表]来列出(和/或排序)最大的对象,并偶尔rm()其中一些对象。但到目前为止最有效的解决办法是……在64位Linux下运行,有充足的内存。

大家还有什么想分享的妙招吗?请每人寄一份。

# improved list of objects
.ls.objects <- function (pos = 1, pattern, order.by,
                        decreasing=FALSE, head=FALSE, n=5) {
    napply <- function(names, fn) sapply(names, function(x)
                                         fn(get(x, pos = pos)))
    names <- ls(pos = pos, pattern = pattern)
    obj.class <- napply(names, function(x) as.character(class(x))[1])
    obj.mode <- napply(names, mode)
    obj.type <- ifelse(is.na(obj.class), obj.mode, obj.class)
    obj.size <- napply(names, object.size)
    obj.dim <- t(napply(names, function(x)
                        as.numeric(dim(x))[1:2]))
    vec <- is.na(obj.dim)[, 1] & (obj.type != "function")
    obj.dim[vec, 1] <- napply(names, length)[vec]
    out <- data.frame(obj.type, obj.size, obj.dim)
    names(out) <- c("Type", "Size", "Rows", "Columns")
    if (!missing(order.by))
        out <- out[order(out[[order.by]], decreasing=decreasing), ]
    if (head)
        out <- head(out, n)
    out
}
# shorthand
lsos <- function(..., n=10) {
    .ls.objects(..., order.by="Size", decreasing=TRUE, head=TRUE, n=n)
}

当前回答

Unfortunately I did not have time to test it extensively but here is a memory tip that I have not seen before. For me the required memory was reduced with more than 50%. When you read stuff into R with for example read.csv they require a certain amount of memory. After this you can save them with save("Destinationfile",list=ls()) The next time you open R you can use load("Destinationfile") Now the memory usage might have decreased. It would be nice if anyone could confirm whether this produces similar results with a different dataset.

其他回答

Unfortunately I did not have time to test it extensively but here is a memory tip that I have not seen before. For me the required memory was reduced with more than 50%. When you read stuff into R with for example read.csv they require a certain amount of memory. After this you can save them with save("Destinationfile",list=ls()) The next time you open R you can use load("Destinationfile") Now the memory usage might have decreased. It would be nice if anyone could confirm whether this produces similar results with a different dataset.

Tip for dealing with objects requiring heavy intermediate calculation: When using objects that require a lot of heavy calculation and intermediate steps to create, I often find it useful to write a chunk of code with the function to create the object, and then a separate chunk of code that gives me the option either to generate and save the object as an rmd file, or load it externally from an rmd file I have already previously saved. This is especially easy to do in R Markdown using the following code-chunk structure.

```{r Create OBJECT}

COMPLICATED.FUNCTION <- function(...) { Do heavy calculations needing lots of memory;
                                        Output OBJECT; }

```
```{r Generate or load OBJECT}

LOAD <- TRUE
SAVE <- TRUE
#NOTE: Set LOAD to TRUE if you want to load saved file
#NOTE: Set LOAD to FALSE if you want to generate the object from scratch
#NOTE: Set SAVE to TRUE if you want to save the object externally

if(LOAD) { 
  OBJECT <- readRDS(file = 'MySavedObject.rds') 
} else {
  OBJECT <- COMPLICATED.FUNCTION(x, y, z)
  if (SAVE) { saveRDS(file = 'MySavedObject.rds', object = OBJECT) } }

```

With this code structure, all I need to do is to change LOAD depending on whether I want to generate the object, or load it directly from an existing saved file. (Of course, I have to generate it and save it the first time, but after this I have the option of loading it.) Setting LOAD <- TRUE bypasses use of my complicated function and avoids all of the heavy computation therein. This method still requires enough memory to store the object of interest, but it saves you from having to calculate it each time you run your code. For objects that require a lot of heavy calculation of intermediate steps (e.g., for calculations involving loops over large arrays) this can save a substantial amount of time and computation.

当我在一个有很多中间步骤的大型项目中工作时,我会尽量减少对象的数量。而不是创建许多唯一的对象

Dataframe -> step1 -> step2 -> step3 -> result

raster->多pliedrast -> meanRastF -> sqrtRast -> resultRast

我使用临时对象,我称之为temp。

Dataframe -> temp -> temp -> temp -> result

这样就少了一些中间文件,多了一些概览。

raster  <- raster('file.tif')
temp <- raster * 10
temp <- mean(temp)
resultRast <- sqrt(temp)

为了节省更多内存,我可以在不再需要时简单地删除temp。

rm(temp)

如果我需要几个中间文件,我使用temp1, temp2, temp3。

对于测试,我使用test, test2,…

除了以上回答中给出的更通用的内存管理技术外,我总是尽可能地减小对象的大小。例如,我处理非常大但非常稀疏的矩阵,换句话说,大多数值为零的矩阵。使用“矩阵”包(大写很重要),我能够将我的平均对象大小从~2GB减小到~200MB,简单如下:

my.matrix <- Matrix(my.matrix)

Matrix包包含的数据格式可以像常规矩阵一样使用(不需要更改其他代码),但能够更有效地存储稀疏数据,无论是加载到内存中还是保存到磁盘中。

此外,我收到的原始文件是“长”格式的,其中每个数据点都有变量x, y, z, I。将数据转换为只有变量I的x * y * z维度数组更有效。

了解你的数据并使用一些常识。

gData包中的llfunction也可以显示每个对象的内存使用情况。

gdata::ll(unit='MB')