今天,我运行了文件系统索引的脚本来刷新RAID文件索引,4h后它崩溃了,出现以下错误:

[md5:]  241613/241627 97.5%  
[md5:]  241614/241627 97.5%  
[md5:]  241625/241627 98.1%
Creating missing list... (79570 files missing)
Creating new files list... (241627 new files)

<--- Last few GCs --->

11629672 ms: Mark-sweep 1174.6 (1426.5) -> 1172.4 (1418.3) MB, 659.9 / 0 ms [allocation failure] [GC in old space requested].
11630371 ms: Mark-sweep 1172.4 (1418.3) -> 1172.4 (1411.3) MB, 698.9 / 0 ms [allocation failure] [GC in old space requested].
11631105 ms: Mark-sweep 1172.4 (1411.3) -> 1172.4 (1389.3) MB, 733.5 / 0 ms [last resort gc].
11631778 ms: Mark-sweep 1172.4 (1389.3) -> 1172.4 (1368.3) MB, 673.6 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3d1d329c9e59 <JS Object>
1: SparseJoinWithSeparatorJS(aka SparseJoinWithSeparatorJS) [native array.js:~84] [pc=0x3629ef689ad0] (this=0x3d1d32904189 <undefined>,w=0x2b690ce91071 <JS Array[241627]>,L=241627,M=0x3d1d329b4a11 <JS Function ConvertToString (SharedFunctionInfo 0x3d1d3294ef79)>,N=0x7c953bf4d49 <String[4]\: ,\n  >)
2: Join(aka Join) [native array.js:143] [pc=0x3629ef616696] (this=0x3d1d32904189 <undefin...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/bin/node]
 2: 0xe2c5fc [/usr/bin/node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/bin/node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/bin/node]
 6: v8::internal::Runtime_SparseJoinWithSeparator(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
 7: 0x3629ef50961b

服务器配置16gb RAM和24gb SSD交换盘。我非常怀疑我的脚本内存超过了36gb。至少不应该是这样

脚本创建文件索引存储为对象数组与文件元数据(修改日期,权限等,没有大数据)

以下是完整的脚本代码: http://pastebin.com/mjaD76c3

我已经经历了奇怪的节点问题在过去与这个脚本迫使我eg。分割索引到多个文件作为节点是故障时,工作在这样的大文件字符串。对于庞大的数据集,有什么方法可以改善nodejs的内存管理吗?


当前回答

你可以通过以下方法修复Node.js中的“堆出内存”错误。

Increase the amount of memory allocated to the Node.js process by using the --max-old-space-size flag when starting the application. For example, you can increase the limit to 4GB by running node --max-old-space-size=4096 index.js. Use a memory leak detection tool, such as the Node.js heap dump module, to identify and fix memory leaks in your application. You can also use the node inspector and use chrome://inspect to check memory usage. Optimize your code to reduce the amount of memory needed. This might involve reducing the size of data structures, reusing objects instead of creating new ones, or using more efficient algorithms. Use a garbage collector (GC) algorithm to manage memory automatically. Node.js uses the V8 engine's garbage collector by default, but you can also use other GC algorithms such as the Garbage Collection in Node.js Use a containerization technology like Docker which limits the amount of memory available to the container. Use a process manager like pm2 which allows to automatically restart the node application if it goes out of memory.

其他回答

最近,在我的一个项目中遇到了同样的问题。尝试了一些事情,任何人都可以尝试调试以确定根本原因:

正如每个人建议的那样,通过添加以下命令来增加节点的内存限制: { "脚本":{ "server":"node——max-old-space-size={size-value} server/index.js" } }

这里size-value我已经为我的应用程序定义了1536(因为我的kubernetes pod内存是2 GB的限制,请求1.5 GB)

因此,始终根据前端基础设施/架构限制(略小于限制)定义size-value。

在上面的命令中,使用——max-old-space-size命令在node之后,而不是文件名server/index.js之后。

If you have ngnix config file then check following things: worker_connections: 16384 (for heavy frontend applications) [nginx default is 512 connections per worker, which is too low for modern applications] use: epoll (efficient method) [nginx supports a variety of connection processing methods] http: add following things to free your worker from getting busy in handling some unwanted task. (client_body_timeout , reset_timeout_connection , client_header_timeout,keepalive_timeout ,send_timeout). Remove all logging/tracking tools like APM , Kafka , UTM tracking, Prerender (SEO) etc middlewares or turn off. Now code level debugging: In your main server file , remove unwanted console.log which is just printing a message. Now check for every server route i.e app.get() , app.post() ... below scenarios:

Data => if(Data) res.send(Data) //你真的需要等待数据或API返回一些我必须等待的响应吗??,如果不是这样修改:

data => res.send(data) // this will not block your thread, apply everywhere where it's needed

else part: if there is no error coming then simply return res.send({}) , NO console.log here. error part: some people define as error or err which creates confusion and mistakes. like this: `error => { next(err) } // here err is undefined` `err => {next(error) } // here error is undefined` `app.get(API , (re,res) =>{ error => next(error) // here next is not defined })` remove winston , elastic-epm-node other unused libraries using npx depcheck command. In the axios service file , check the methods and logging properly or not like : if(successCB) console.log("success") successCB(response.data) // here it's wrong statement, because on success you are just logging and then `successCB` sending outside the if block which return in failure case also. Save yourself from using stringify , parse etc on accessive large dataset. (which i can see in your above shown logs too.

最后但并非最不重要的是,每当应用程序崩溃或pod重新启动时,都要检查日志。在日志中特别查找这部分:安全上下文 这将告诉你为什么,在哪里,谁是背后的崩溃的罪魁祸首。

检查是否没有在64位计算机上安装32位版本的节点。如果你在64位或32位机器上运行node,那么nodejs文件夹应该分别位于Program Files和Program Files (x86)中。

为了防止人们在使用产生大量日志记录的nodejs应用程序时遇到这个问题,一位同事通过将标准输出管道到一个文件来解决这个问题。

使用这个选项——优化大小。它将专注于使用更少的内存。

我遇到这个问题时,试图调试与VSCode,所以只是想添加这是如何添加参数到您的调试设置。

你可以将它添加到launch.json中配置的runtimeArgs属性中。

参见下面的示例。

{
"version": "0.2.0",
"configurations": [{
        "type": "node",
        "request": "launch",
        "name": "Launch Program",
        "program": "${workspaceRoot}\\server.js"
    },
    {
        "type": "node",
        "request": "launch",
        "name": "Launch Training Script",
        "program": "${workspaceRoot}\\training-script.js",
        "runtimeArgs": [
            "--max-old-space-size=4096"
        ]
    }
]}