今天，我运行了文件系统索引的脚本来刷新RAID文件索引，4h后它崩溃了，出现以下错误:
[md5:] 241613/241627 97.5%
[md5:] 241614/241627 97.5%
[md5:] 241625/241627 98.1%
Creating missing list... (79570 files missing)
Creating new files list... (241627 new files)
<--- Last few GCs --->
11629672 ms: Mark-sweep 1174.6 (1426.5) -> 1172.4 (1418.3) MB, 659.9 / 0 ms [allocation failure] [GC in old space requested].
11630371 ms: Mark-sweep 1172.4 (1418.3) -> 1172.4 (1411.3) MB, 698.9 / 0 ms [allocation failure] [GC in old space requested].
11631105 ms: Mark-sweep 1172.4 (1411.3) -> 1172.4 (1389.3) MB, 733.5 / 0 ms [last resort gc].
11631778 ms: Mark-sweep 1172.4 (1389.3) -> 1172.4 (1368.3) MB, 673.6 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x3d1d329c9e59 <JS Object>
1: SparseJoinWithSeparatorJS(aka SparseJoinWithSeparatorJS) [native array.js:~84] [pc=0x3629ef689ad0] (this=0x3d1d32904189 <undefined>,w=0x2b690ce91071 <JS Array[241627]>,L=241627,M=0x3d1d329b4a11 <JS Function ConvertToString (SharedFunctionInfo 0x3d1d3294ef79)>,N=0x7c953bf4d49 <String[4]\: ,\n >)
2: Join(aka Join) [native array.js:143] [pc=0x3629ef616696] (this=0x3d1d32904189 <undefin...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/bin/node]
2: 0xe2c5fc [/usr/bin/node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/bin/node]
6: v8::internal::Runtime_SparseJoinWithSeparator(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
7: 0x3629ef50961b
服务器配置16gb RAM和24gb SSD交换盘。我非常怀疑我的脚本内存超过了36gb。至少不应该是这样
脚本创建文件索引存储为对象数组与文件元数据(修改日期，权限等，没有大数据)
以下是完整的脚本代码:
http://pastebin.com/mjaD76c3
我已经经历了奇怪的节点问题在过去与这个脚本迫使我eg。分割索引到多个文件作为节点是故障时，工作在这样的大文件字符串。对于庞大的数据集，有什么方法可以改善nodejs的内存管理吗?
最近，在我的一个项目中遇到了同样的问题。尝试了一些事情，任何人都可以尝试调试以确定根本原因:
正如每个人建议的那样，通过添加以下命令来增加节点的内存限制:
｛
"脚本":{
"server":"node——max-old-space-size={size-value} server/index.js"
｝
｝
这里size-value我已经为我的应用程序定义了1536(因为我的kubernetes pod内存是2 GB的限制，请求1.5 GB)
因此，始终根据前端基础设施/架构限制(略小于限制)定义size-value。
在上面的命令中，使用——max-old-space-size命令在node之后，而不是文件名server/index.js之后。
If you have ngnix config file then check following things:
worker_connections: 16384 (for heavy frontend applications)
[nginx default is 512 connections per worker, which is too low for modern applications]
use: epoll (efficient method) [nginx supports a variety of connection processing methods]
http: add following things to free your worker from getting busy in handling some unwanted task. (client_body_timeout , reset_timeout_connection , client_header_timeout,keepalive_timeout ,send_timeout).
Remove all logging/tracking tools like APM , Kafka , UTM tracking, Prerender (SEO) etc middlewares or turn off.
Now code level debugging: In your main server file , remove unwanted console.log which is just printing a message.
Now check for every server route i.e app.get() , app.post() ... below scenarios:
Data => if(Data) res.send(Data) //你真的需要等待数据或API返回一些我必须等待的响应吗??，如果不是这样修改:
data => res.send(data) // this will not block your thread, apply everywhere where it's needed
else part: if there is no error coming then simply return res.send({}) , NO console.log here.
error part: some people define as error or err which creates confusion and mistakes. like this:
`error => { next(err) } // here err is undefined`
`err => {next(error) } // here error is undefined`
`app.get(API , (re,res) =>{
error => next(error) // here next is not defined
})`
remove winston , elastic-epm-node other unused libraries using npx depcheck command.
In the axios service file , check the methods and logging properly or not like :
if(successCB) console.log("success") successCB(response.data) // here it's wrong statement, because on success you are just logging and then `successCB` sending outside the if block which return in failure case also.
Save yourself from using stringify , parse etc on accessive large dataset. (which i can see in your above shown logs too.
最后但并非最不重要的是，每当应用程序崩溃或pod重新启动时，都要检查日志。在日志中特别查找这部分:安全上下文
这将告诉你为什么，在哪里，谁是背后的崩溃的罪魁祸首。
