It is found that the home space of a server is full, and useless files need to be cleared, but after deleting the files, it is found that the available space has not changed
os：centos6.0
phenomenon:
**Found the current disk space usage: **
[ root@ticketb ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 981M 203M 729M 22% /
none 16G 0 16G 0%/dev/shm
/dev/sda9 2.9G 37M 2.7G 2%/tmp
/dev/sda7 4.9G 1.9G 2.7G 42%/usr
/dev/sda8 2.9G 145M 2.6G 6%/var/dev/mapper/vghome-lvhome
20 G 19G 11M 100%/home
/dev/mapper/vgoradata-lvoradata
144 G 48G 90G 35%/u01/oradata
/dev/mapper/vgbackup-lvbackup
193 G 7.8G 175G 5%/u01/backup
Use the following command to find useless files, and then delete
[ root@ticketb ~]# find /home/oracle/admin/dbticb/udump/ -name "dbticb_*.trc" -mtime +50 | xargs rm -rf
Then after checking the disk space usage, there is no change in /home space
[ root@ticketb ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 981M 203M 729M 22% /
none 16G 0 16G 0%/dev/shm
/dev/sda9 2.9G 37M 2.7G 2%/tmp
/dev/sda7 4.9G 1.9G 2.7G 42%/usr
/dev/sda8 2.9G 145M 2.6G 6%/var/dev/mapper/vghome-lvhome
20 G 19G 11M 100%/home
/dev/mapper/vgoradata-lvoradata
144 G 48G 90G 35%/u01/oradata
/dev/mapper/vgbackup-lvbackup
193 G 7.8G 175G 5%/u01/backup
This is depressing, I have deleted the file, why the space has not been released, the rm command should be deleted directly, check what is the space occupied under /home
[ root@ticketb ~]# du -h --max-depth=1/home
16 K /home/lost+found
2.6 G /home/oracle
2.6 G /home
But the display space here has been released, so under google,
Reasons for not releasing disk space:
In Linux or Unix systems, deleting files through rm or file manager will unlink (unlink) from the directory structure of the file system. However, if the file is
Open (there is a process in use), then the process will still be able to read the file, and the disk space will always be occupied. And what I deleted is the oracle alarm log file
The file should be in use when deleted
Solution
First get a list of files that have been deleted but still occupied by the application, as shown below:
[ root@ticketb ~]# lsof |grep deleted
oracle 12639 oracle 5w REG 253,0648215907/home/oracle/admin/dbticb/udump/dbticb_ora_12637.trc(deleted)
oracle 12639 oracle 6w REG 253,016749822091215748/home/oracle/admin/dbticb/bdump/alert_dbticb.log(deleted)
oracle 12639 oracle 7u REG 253,0036282/home/oracle/oracle/product/10.2.0/db_1/dbs/lkinstdbticb(deleted)
oracle 12639 oracle 8w REG 253,016749822091215748/home/oracle/admin/dbticb/bdump/alert_dbticb.log(deleted)
oracle 12641 oracle 5w REG 253,0648215907/home/oracle/admin/dbticb/udump/dbticb_ora_12637.trc(deleted)
oracle 12641 oracle 6w REG 253,016749822091215748/home/oracle/admin/dbticb/bdump/alert_dbticb.log(deleted)
。
。
oracle 23492 oracle 6w REG 253,016749822091215748/home/oracle/admin/dbticb/bdump/alert_dbticb.log(deleted)
oracle 23492 oracle 7u REG 253,0036282/home/oracle/oracle/product/10.2.0/db_1/dbs/lkinstdbticb(deleted)
oracle 23492 oracle 8w REG 253,016749822091215748/home/oracle/admin/dbticb/bdump/alert_dbticb.log(deleted)
oracle 23494 oracle 10u REG 253,0036307/home/oracle/oracle/product/10.2.0/db_1/dbs/lkinstrmandb(deleted)
From the output, you can see that /home/oracle/admin/dbticb/bdump/alert_dbticb.log is still used and the space is not released
How to release the process?
One method is to kill the corresponding process, or stop the application that uses this file, and let os automatically reclaim the disk space
This file is used by many processes in my environment, it is a bit troublesome to stop the process, and there is a great risk
When linux opens a file, the Linux kernel will create a pid for each process in the /proc/ "/proc/nnnn/fd/ directory (nnnn is pid)"
The named directory is used to store information about the process, and its subdirectory fd stores the fd (fd: file descriptor) of all files opened by the process.
The kill process can force the system to reclaim the files allocated for use by truncating the files in the proc file system.
This is an advanced technique, used only when the administrator is sure that it will not affect the running process. Application to this
Support is not good. When a file in use is truncated, it may cause unpredictable problems
So I still use stop application to solve
Restart the oracle database and find that the space corresponding to /home/oracle/admin/dbticb/bdump/alert_dbticb.log is released
After checking the disk space usage, I found that the space has been reclaimed
[ root@ticketb ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 981M 203M 729M 22% /
none 16G 0 16G 0%/dev/shm
/dev/sda9 2.9G 37M 2.7G 2%/tmp
/dev/sda7 4.9G 1.9G 2.7G 42%/usr
/dev/sda8 2.9G 145M 2.6G 6%/var/dev/mapper/vghome-lvhome
20 G 2.6G 16G 15%/home
/dev/mapper/vgoradata-lvoradata
144 G 48G 90G 35%/u01/oradata
/dev/mapper/vgbackup-lvbackup
193 G 7.8G 175G 5%/u01/backup
ok, the problem is solved, and then do the finishing work
Learn the lsof command
The full name of lsof is list opened files, which is to list the files that have been opened in the system. We all know that in the Linux environment, everything is a file.
Devices are files, directories are files, and even sockets are files. Therefore, the lsof command is very helpful for daily linux management.
lsof is one of the most commonly used commands in Linux, and the usual output format is:
Quote
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
Commonly include the following fields: more visible manual.
1、 COMMAND
The command name displayed by default with a length of 9 characters. You can use the +c parameter to specify the width of the display. If the parameter followed by +c is zero, the full name of the command will be displayed
2、 PID: ID number of the process
3、 PPID
The IP number of the parent process is not displayed by default. It can be opened when the -R parameter is used.
4、 PGID
The ID number of the process group is not displayed by default. It can be opened when the -g parameter is used.
5、 USER
The execution UID of the command or the name of the user logged in in the system. The default display is the user name, when the -l parameter is used, the UID can be displayed.
6、 FD
Is the File Descriptor number of the file, or the following content:
(It is difficult to translate the corresponding meaning here, keep English)
Quote
cwd current working directory;
Lnn library references(AIX);
jld jail directory(FreeBSD);
ltx shared library text(code and data);
Mxx hex memory-mapped type number xx.
m86 DOS Merge mapped file;
mem memory-mapped file;
mmap memory-mapped device;
pd parent directory;
rtd root directory;
tr kernel trace file(OpenBSD);
txt program text(code and data);
v86 VP/ix mapped file;
The File Descriptor number display modes of the file are:
Quote
r for read access;
w for write access;
u for read and write access;
N for a Solaris NFS lock of unknown type;
r for read lock on part of the file;
R for a read lock on the entire file;
w for a write lock on part of the file;
W for a write lock on the entire file;
u for a read and write lock of any length;
U for a lock of unknown type;
x for an SCO OpenServer Xenix lock on part of the file;
X for an SCO OpenServer Xenix lock on the entire file;
space if there is no lock.
7、 TYPE
Quote
IPv4 IPv4 package;
IPv6 uses IPv6 format packets, even if the address is IPv4, it will be displayed as IPv6 and mapped to IPv6 addresses;
DIR directory
LINK link file
See more notes in the manual for details.
8、 DEVICE
Device number represented by character special and block special
9、 SIZE
The size of the file, if it cannot be expressed by size, it will be left blank. Use the -s parameter to control.
10、 NODE
The node code of the local file, or the protocol, such as TCP, etc.
11、 NAME
The full path of the mount point and the file (the link will be resolved to the actual path), or the address, port, status, etc. of the connected parties
**Common examples: **
[ root@svr-db-test ~]# lsof /home/oracle/10.2.0/db_1/bin/tnslsnr
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
tnslsnr 3520 oracle txt REG 253,543106211408866/home/oracle/10.2.0/db_1/bin/tnslsnr
[ root@svr-db-test ~]# lsof -i :22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 3101 root 3u IPv6 8670 TCP *:ssh(LISTEN)
sshd 4545 root 3u IPv6 4237972 TCP 203.aibo.com:ssh->win-avbmq9e8ka7.gdgg.local:nsjtp-ctrl(ESTABLISHED)
[ root@svr-db-test ~]# lsof -c init
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
init 1 root cwd DIR 253,040962/
init 1 root rtd DIR 253,040962/
init 1 root txt REG 253,043496524446/sbin/init
init 1 root mem REG 253,0130448917826/lib64/ld-2.5.so
init 1 root mem REG 253,01678480917827/lib64/libc-2.5.so
init 1 root mem REG 253,023520917686/lib64/libdl-2.5.so
init 1 root mem REG 253,0247528917844/lib64/libsepol.so.1
init 1 root mem REG 253,095480917845/lib64/libselinux.so.1
init 1 root 10u FIFO 0,162311/dev/initctl
[ root@svr-db-test ~]# lsof -p 1
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
init 1 root cwd DIR 253,040962/
init 1 root rtd DIR 253,040962/
init 1 root txt REG 253,043496524446/sbin/init
init 1 root mem REG 253,0130448917826/lib64/ld-2.5.so
init 1 root mem REG 253,01678480917827/lib64/libc-2.5.so
init 1 root mem REG 253,023520917686/lib64/libdl-2.5.so
init 1 root mem REG 253,0247528917844/lib64/libsepol.so.1
init 1 root mem REG 253,095480917845/lib64/libselinux.so.1
init 1 root 10u FIFO 0,162311/dev/initctl
[ root@svr-db-test ~]# lsof -g 3520
COMMAND PID PGID USER FD TYPE DEVICE SIZE NODE NAME
tnslsnr 35203520 oracle cwd DIR 253,5409611059201/home/oracle
tnslsnr 35203520 oracle rtd DIR 253,040962/
tnslsnr 35203520 oracle txt REG 253,543106211408866/home/oracle/10.2.0/db_1/bin/tnslsnr
tnslsnr 35203520 oracle mem REG 253,0130448917826/lib64/ld-2.5.so
tnslsnr 35203520 oracle mem REG 253,01678480917827/lib64/libc-2.5.so
tnslsnr 35203520 oracle mem REG 253,023520917686/lib64/libdl-2.5.so
tnslsnr 35203520 oracle mem REG 253,0615136917834/lib64/libm-2.5.so
tnslsnr 35203520 oracle mem REG 253,0141208917829/lib64/libpthread-2.5.so
tnslsnr 35203520 oracle mem REG 253,0109824917839/lib64/libnsl-2.5.so
tnslsnr 35203520 oracle mem REG 253,52070662211405436/home/oracle/10.2.0/db_1/lib/libclntsh.so.10.1
tnslsnr 35203520 oracle mem REG 253,5380309711410641/home/oracle/10.2.0/db_1/lib/libnnz10.so
tnslsnr 35203520 oracle mem REG 253,58349311407251/home/oracle/10.2.0/db_1/lib/libons.so
tnslsnr 35203520 oracle mem REG 253,053880917532/lib64/libnss_files-2.5.so
tnslsnr 35203520 oracle mem REG 253,5854511407615/home/oracle/10.2.0/db_1/lib/libskgxn2.so
tnslsnr 35203520 oracle mem REG 253,551370511410332/home/oracle/10.2.0/db_1/lib/libocrutl10.so
tnslsnr 35203520 oracle mem REG 253,563616111410330/home/oracle/10.2.0/db_1/lib/libocr10.so
tnslsnr 35203520 oracle mem REG 253,565782511410331/home/oracle/10.2.0/db_1/lib/libocrb10.so
tnslsnr 35203520 oracle mem REG 253,5174576911410365/home/oracle/10.2.0/db_1/lib/libhasgen10.so
tnslsnr 35203520 oracle mem REG 253,56198511410366/home/oracle/10.2.0/db_1/lib/libclsra10.so
tnslsnr 35203520 oracle 0u CHR 1,32553/dev/null
tnslsnr 35203520 oracle 1u CHR 1,32553/dev/null
tnslsnr 35203520 oracle 2u CHR 1,32553/dev/null
tnslsnr 35203520 oracle 3w REG 253,531885301211633459/home/oracle/10.2.0/db_1/network/log/listener.log
tnslsnr 35203520 oracle 4r FIFO 0,615661 pipe
tnslsnr 35203520 oracle 5r REG 253,51177611410579/home/oracle/10.2.0/db_1/network/mesg/nlus.msb
tnslsnr 35203520 oracle 6r REG 253,54659211407160/home/oracle/10.2.0/db_1/network/mesg/tnsus.msb
tnslsnr 35203520 oracle 7w FIFO 0,615662 pipe
tnslsnr 35203520 oracle 8u IPv4 15665 TCP 203.aibo.com:ncube-lm(LISTEN)
tnslsnr 35203520 oracle 9u unix 0xffff81021b7d698015666/var/tmp/.oracle/s#3520.1
tnslsnr 35203520 oracle 10u unix 0xffff81021b7d66c015668/var/tmp/.oracle/s#3520.2
[ root@svr-db-test ~]# lsof +d /home/oracle
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
tnslsnr 3520 oracle cwd DIR 253,5409611059201/home/oracle
[ root@svr-db-test ~]# lsof +D /home/oracle
Syntax: lsof -i[46] [protocol][@hostname|hostaddr][:service|port]
46 - - > IPv4 or IPv6
protocol --> TCP or UDP
hostname --> Internet host name
hostaddr --> IPv4 location
service --> service name in /etc/service (can be more than one)
port --> port number (more than one is possible)
example:
[ root@svr-db-test ~]# lsof -i [email protected]:1521-n
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
oracle 15633 oracle 16u IPv4 4069605 TCP 192.168.2.203:31580->192.168.2.245:ncube-lm(ESTABLISHED)
or
[ root@svr-db-test ~]# lsof -i [email protected]:1521
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
oracle 15633 oracle 16u IPv4 4069605 TCP 203.aibo.com:31580->192.168.2.245:ncube-lm(ESTABLISHED)
**lsof -n does not convert IP to hostname, the default is not to add -n parameter **
[ root@svr-db-test ~]# lsof -u oracle
or
[ root@svr-db-test ~]# lsof -u 0
[ root@svr-db-test ~]# lsof -tc sshd
3101
4545
attention:
Process debugging commands: truss, strace and ltrace
The process cannot be started, the software running speed suddenly slows down, the "SegmentFault" of the program, etc. are all headaches for every Unix system user, and these problems can be solved by using the three common debugging tools of truss, strace and ltrace Quickly diagnose the "intractable diseases" of the software.
Recommended Posts