The business process stops or restarts abnormally. You can judge whether OOM has occurred according to the /var/log/messages log. If so, what process occupies a lot of memory space to trigger OOM Killer
After running nmap for an hour, it will be terminated. Checking the log shows that the system memory is insufficient, the oom mechanism is triggered, the nmap process is killed, but the historical memory usage is not continuously reached the peak value.
Filtering the log found that there are multiple Out of memory log information in the messages log, triggering the Kill process action
Nov 2211:09:03 VM_0_14_centos kernel:[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Nov 2211:09:03 VM_0_14_centos kernel:[328]032826240945600 systemd-journal
Nov 2211:09:03 VM_0_14_centos kernel:[346]034629149762500 lvmetad
Nov 2211:09:03 VM_0_14_centos kernel:[349]034911121186240-1000 systemd-udevd
Nov 2211:09:03 VM_0_14_centos kernel:[469]046913877113270-1000 auditd
Nov 2211:09:03 VM_0_14_centos kernel:[505]05056594751800 systemd-logind
Nov 2211:09:03 VM_0_14_centos kernel:[506]8150615044143340-900 dbus-daemon
Nov 2211:09:03 VM_0_14_centos kernel:[511]99951113513119096100 polkitd
Nov 2211:09:03 VM_0_14_centos kernel:[512]9985122144371000 lsmd
Nov 2211:09:03 VM_0_14_centos kernel:[516]0516109634800 acpid
Nov 2211:09:03 VM_0_14_centos kernel:[518]38518123301802900 ntpd
Nov 2211:09:03 VM_0_14_centos kernel:[729]0729315701551900 crond
Nov 2211:09:03 VM_0_14_centos kernel:[730]07306476521800 atd
Nov 2211:09:03 VM_0_14_centos kernel:[803]0803268454985000 dhclient
Nov 2211:09:03 VM_0_14_centos kernel:[881]088114397527579600 tuned
Nov 2211:09:03 VM_0_14_centos kernel:[883]0883802294877900 rsyslogd
Nov 2211:09:03 VM_0_14_centos kernel:[967]096727522331200 agetty
Nov 2211:09:03 VM_0_14_centos kernel:[968]096827522331000 agetty
Nov 2211:09:03 VM_0_14_centos kernel:[1324]0132428199256600-1000 sshd
Nov 2211:09:03 VM_0_14_centos kernel:[1345]01345243651151700 sgagent
Nov 2211:09:03 VM_0_14_centos kernel:[1393]013933881916263000 barad_agent
Nov 2211:09:03 VM_0_14_centos kernel:[1402]014023954718303100 barad_agent
Nov 2211:09:03 VM_0_14_centos kernel:[1403]0140316945928635100 barad_agent
Nov 2211:09:03 VM_0_14_centos kernel:[1480]0148025249982100 YDLive
Nov 2211:09:03 VM_0_14_centos kernel:[3343]03343320071981900 screen
Nov 2211:09:03 VM_0_14_centos kernel:[3344]03344290232721400 bash
Nov 2211:09:03 VM_0_14_centos kernel:[13683]01368314284017033600 YDService
Nov 2211:09:03 VM_0_14_centos kernel:[19272]019272678212422656132900 nmap
Nov 2211:09:03 VM_0_14_centos kernel: Out of memory: Kill process 19272(nmap) score 873 or sacrifice child
Nov 2211:09:03 VM_0_14_centos kernel: Killed process 19272(nmap) total-vm:2712848kB, anon-rss:1690624kB, file-rss:0kB, shmem-rss:0kB
Here you can confirm from the anon-rss value how much physical memory the process actually occupied before being killed:
total-vm indicates how much memory space it needs to occupy if a program completely resides in memory
How much memory is actually occupied by the anon-rss process
Note: If there are multiple processes that take up high memory in the system, you can directly view the actual memory size of the killed process may differ from the actual memory size of the system. Then you can calculate the total memory of the process that takes up high memory in the current system. Compare and confirm which processes consume a lot of system memory:
Calculation of actual memory usage:
The RSS (physical memory page) size is 4kB. You can view the RSS value printed in the messages log (the number of physical memory pages occupied by the process)
For example, here we see that the nmap process occupies the highest, the actual physical memory page occupied is 422656, multiplied by 4KB equals 1690624KB, divided by 1024 equals 1651MB