Core dump debugging in ubuntu

When executing the program in the Linux environment, sometimes a segment fault ('segment fault') will appear, and core dumped will be displayed at the same time, like the following:

[1]15428 segmentation fault(core dumped)./a.out

Below is the definition and description of the segfault I found online:

A segmentation fault(often shortened to segfault) is a particular error condition that can occur during the operation of computer software. In short, a segmentation fault occurs when a program attempts to access a memory location that it is not allowed to access, or attempts to access a memory location in a way that is not allowed(e.g., attempts to write to a read-only location, or to overwrite part of the operating system). Systems based on processors like the Motorola 68000 tend to refer to these events as Address or Bus errors.

Segmentation is one approach to memory management and protection in the operating system. It has been superseded by paging for most purposes, but much of the terminology of segmentation is still used,"segmentation fault" being an example. Some operating systems still have segmentation at some logical level although paging is used as the main memory management policy.

On Unix-like operating systems, a process that accesses invalid memory receives the SIGSEGV signal. On Microsoft Windows, a process that accesses invalid memory receives the STATUS_ACCESS_VIOLATION exception.

The simple understanding is that a segfault will occur if the memory that should not be accessed is accessed.
The core dump is a way to write information such as the call stack at the time of an error into a file to facilitate subsequent debugging. Under Ubuntu, some settings are required to debug core dump correctly. The following is a detailed description.

ulimit settings###

ulimit is a tool to limit the system resources occupied by the shell startup process. For detailed instructions, please see here. Here we need to set ulimit, because under Ubuntu, the default core file size is 0. You can execute ulimit -a to view all the option settings:

$ ulimit -a
- t: cpu time(seconds)              unlimited
- f: file size(blocks)              unlimited
- d: data seg size(kbytes)          unlimited
- s: stack size(kbytes)8192-c: core file size(blocks)0-m: resident setsize(kbytes)      unlimited
- u: processes                       2063144-n: file descriptors                1024-l: locked-in-memory size(kbytes)64-v: address space(kbytes)          unlimited
- x: file locks                      unlimited
- i: pending signals                 2063144-q: bytes in POSIX msg queues       819200-e: max nice                        0-r: max rt priority                 0-N 15:                              unlimited

You can see that the value of -c: core file size (blocks) in the above result is 0, so when a core dump occurs in a segmentation fault, the core file will not be generated by default.
How to modify the size of the core file? The value of ulimit can be set in the form of ulimit -kv, where -k is the first column in the above result, and v is the set value, and the maximum can be set to unlimited, so We can set it up like this:

ulimit -c unlimited

Use ulimit -a again to check and find that the modification of -c has taken effect:

$ ulimit -a
- t: cpu time(seconds)              unlimited
- f: file size(blocks)              unlimited
- d: data seg size(kbytes)          unlimited
- s: stack size(kbytes)8192-c: core file size(blocks)         unlimited
- m: resident setsize(kbytes)      unlimited
- u: processes                       2063144-n: file descriptors                1024-l: locked-in-memory size(kbytes)64-v: address space(kbytes)          unlimited
- x: file locks                      unlimited
- i: pending signals                 2063144-q: bytes in POSIX msg queues       819200-e: max nice                        0-r: max rt priority                 0-N 15:                              unlimited
Note: When opening a new Shell, the ulimit options are restored to the default options, and the value needs to be reset

Set core_pattern

After the ulimit setting above, we also need to set the core_pattern, that is, what operations are performed on the core file after sending the core dump. This can be obtained by viewing the /proc/sys/kernel/core_pattern file, which is on Ubuntu 16.04 , The content of the above file is as follows:

$ cat /proc/sys/kernel/core_pattern
| /usr/share/apport/apport %p %s %c %P

The l means to execute the following command, and the latter apport is a tool for Ubuntu bug feedback. Therefore, under Ubuntu, the default core dump segfault handling mechanism is to treat it as a bug for bug inspection. If it is a bug, report it.
Under this setting, we cannot use gdb to debug the errors of our program.
So here we have to modify the content of core_pattern and change it to core. But there is no way to modify the core_pattern file, because it is not a physical file itself, so here is a little trick to achieve this function: Pause the apport service:

sudo service apport stop

Then check the contents of core_pattern:

$ cat /proc/sys/kernel/core_pattern
core

You can see the operation here and change it to core to generate a core file.

gcc/g++ set debug mode###

In addition to the above two settings, you also need to enable the debug mode by adding the -g parameter when compiling the code, so that debugging information will be added to the generated executable file:

g++-g xxx.cpp
gcc -g xxx.c

Use gdb to debug the program###

After completing the above settings, you can use gdb to debug. When a segmentation error occurs in the program and the core file is generated, start debugging by executing the following command:

gdb ./a.out core

Where ./a.out is the path to the executable file, and core is the file generated by core dump.
Then execute the bt command in the gdb debugging environment to locate the location of the error, and then use the search engine to find a solution based on the error message. The following is one of my debugging site information:

$ gdb ./build/tools/caffe.bin core
GNU gdb(Ubuntu 7.11.1-0ubuntu1~16.5)7.11.1Copyright(C)2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty"for details.
This GDB was configured as"x86_64-linux-gnu".
Type "show configuration"for configuration details.
For bug reporting instructions, please see:<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from./build/tools/caffe.bin...(no debugging symbols found)...done.[New LWP38035][Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./build/tools/caffe.bin'.
Program terminated with signal SIGSEGV, Segmentation fault.
#00 x00007f99b6ff5f66in google::protobuf::Arena::AllocateAligned(std::type_info const*, unsigned long)()from/usr/local/lib/libopencv_dnn.so.3.3(gdb) bt
#00 x00007f99b6ff5f66in google::protobuf::Arena::AllocateAligned(std::type_info const*, unsigned long)()from/usr/local/lib/libopencv_dnn.so.3.3
#10 x00007f99b6ff5f89in google::protobuf::Arena::AddListNode(void*,void(*)(void*))()from/usr/local/lib/libopencv_dnn.so.3.3
#20 x00007f99b70473aein google::protobuf::FileDescriptorProto::New(google::protobuf::Arena*)const[clone .localalias.409]()from/usr/local/lib/libopencv_dnn.so.3.3
#30 x00007f99b6147038in google::protobuf::MessageLite::ParseFromArray(voidconst*, int)()from/usr/lib/x86_64-linux-gnu/libprotobuf.so.9
#40 x00007f99b61901b6in google::protobuf::EncodedDescriptorDatabase::Add(voidconst*, int)()from/usr/lib/x86_64-linux-gnu/libprotobuf.so.9
#50 x00007f99b61519b8in google::protobuf::DescriptorPool::InternalAddGeneratedFile(voidconst*, int)()from/usr/lib/x86_64-linux-gnu/libprotobuf.so.9
#60 x00007f99b618048cin google::protobuf::protobuf_AddDesc_google_2fprotobuf_2fdescriptor_2eproto()()from/usr/lib/x86_64-linux-gnu/libprotobuf.so.9
#70 x00007f99b7e346baincall_init(l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffe8743a4d8, env=env@entry=0x7ffe8743a4e8) at dl-init.c:72
#80 x00007f99b7e347cbincall_init(env=0x7ffe8743a4e8, argv=0x7ffe8743a4d8, argc=1, l=<optimized out>) at dl-init.c:30
#9_ dl_init(main_map=0x7f99b804b168, argc=1, argv=0x7ffe8743a4d8, env=0x7ffe8743a4e8) at dl-init.c:120
#100 x00007f99b7e24c6ain_dl_start_user()from/lib64/ld-linux-x86-64.so.2
#110 x0000000000000001in??()
#120 x00007ffe8743c1cbin??()
#130 x0000000000000000in??()(gdb)

reference###

  1. https://www.ibm.com/developerworks/cn/linux/l-gccrtl/index.html
  2. https://www.ibm.com/developerworks/cn/linux/l-cn-ulimit/

Recommended Posts

Core dump debugging in ubuntu
Install mysql-pytho in Ubuntu
Nagios3 in ubuntu serve
Use supervisor in ubuntu
Install python in Ubuntu
Install JDK in Ubuntu19.10
Use of Anaconda in Ubuntu
How to install Helm in Ubuntu
Happy installation of Jenkins in ubuntu
How to use hanlp in ubuntu
Use disk quota in ubuntu multi-user
How to install mysql in Ubuntu 14.04
Install and configure Docker in Ubuntu
Installation instructions for gitlab in Ubuntu
How to install mysql in Ubuntu 14.04
Linux basics (root privileges in Ubuntu)
Steps to upgrade Laravel 5.4 to 5.6 in Ubuntu 16.04