Detailed explanation of common tools for Python process control

In our work, we use ddagent ver. 5 As a collection tool, collect and report the performance indicators of the hosting server, and customize ddagent to a certain extent. After many functional iterations, it was found that a batch of online hosting servers with a long running time took up too much memory. Analyzing the occupancy of each node in the process tree on the problem machine, it can be seen that the memory occupancy of the ddagent collection process is still high.

We will learn the most basic process control tools, such as

if conditional judgment for loop while loop

As a monitoring component that guarantees the stable operation of the business system, there is a memory leak, which is naturally very serious, so we started our "troubleshooting journey".

if-elif-else judgment

analysis

There are many tools to analyze and export the memory status of Python programs. Here, we use pyrasite, which can attach to a running Python program, generate a memory snapshot, and check which object types currently occupy how much memory, and sort them from largest to smallest.

The command is very simple: pyrasite-memory-viewer <PID ，同时会生成一份快照文件： /tmp/pyrasite-<PID -objects.json。

Due to the inability to provide real production data, all the data mentioned below are from the sampling of the problem version after 12 hours of running in the test environment.

In the Cui view provided by pyrasite, we can clearly see that dictionary-type object instances occupy the most memory, reaching 3.4mb, with 6621 instances:

while loop

While loop is also a common loop method. This kind of loop usually ends in a loop body or conditional way. It cannot go on indefinitely.

Regarding the leakage, we have the following facts and guesses:

One or more locations continue to create empty dictionary objects and cannot reclaim them, resulting in memory leaks and memory leaks increase over time. In the index collection business, the leak is likely to be caused during each collection process, and it is triggered repeatedly after the interval. We can't see the currently dependent ddagent version has unclosed related issues, which may be in our customization process Errors introduced in

[[["... /embedded/lib/python2.7/threading.py",774,"__bootstrap","self.__bootstrap_inner()"],[".../embedded/lib/python2.7/threading.py",801,"__bootstrap_inner","self.run()"],[".../modules/monitor/bot/schedule.py",51,"run","task.run()"],[".../modules/monitor/bot/task.py",50,"run","super(RepeatTask, self).run()"],[".../modules/monitor/bot/task.py",18,"run","self.check()"],[".../modules/monitor/checks/collector.py",223,"wrapper","_check.run()"],[".../modules/monitor/checks/__init__.py",630,"run","self._roll_up_instance_metadata()"],[".../modules/monitor/checks/__init__.py",498,"_roll_up_instance_metadata","dict((k, v) for (k, v) in self._instance_metadata))"],[".../modules/monitor/tracer.py",33,"__init__","self.trace_info = traceback.extract_stack()"]],[[".../embedded/lib/python2.7/threading.py",774,"__bootstrap","self.__bootstrap_inner()"],[".../embedded/lib/python2.7/threading.py",801,"__bootstrap_inner","self.run()"],[".../modules/monitor/bot/schedule.py",51,"run","task.run()"],[".../modules/monitor/bot/task.py",50,"run","super(RepeatTask, self).run()"],[".../modules/monitor/bot/task.py",18,"run","self.check()"],[".../modules/monitor/checks/collector.py",223,"wrapper","_check.run()"],[".../modules/monitor/checks/__init__.py",630,"run","self._roll_up_instance_metadata()"],[".../modules/monitor/checks/__init__.py",498,"_roll_up_instance_metadata","dict((k, v) for (k, v) in self._instance_metadata))"],[".../modules/monitor/tracer.py",33,"__init__","self.trace_info = traceback.extract_stack()"]],[[".../embedded/lib/python2.7/threading.py",774,"__bootstrap","self.__bootstrap_inner()"],[".../embedded/lib/python2.7/threading.py",801,"__bootstrap_inner","self.run()"],[".../modules/monitor/bot/schedule.py",51,"run","task.run()"],[".../modules/monitor/bot/task.py",50,"run","super(RepeatTask, self).run()"],[".../modules/monitor/bot/task.py",18,"run","self.check()"],[".../modules/monitor/checks/collector.py",223,"wrapper","_check.run()"],[".../modules/monitor/checks/__init__.py",630,"run","self._roll_up_instance_metadata()"],[".../modules/monitor/checks/__init__.py",498,"_roll_up_instance_metadata","dict((k, v) for (k, v) in self._instance_metadata))"],[".../modules/monitor/tracer.py",33,"__init__","self.trace_info = traceback.extract_stack()"]],...

Replay

We don't mention the old clichés such as "doing a good job in design review and specification review" and "strengthening quality inspection work in the experimental phase", which is also worthy of our reflection.

It is almost impossible to completely prevent and control memory leaks, and a secure programming language like rust cannot guarantee that the program will not leak memory.

Many behaviors that cause memory insecurity, such as array access out of bounds, access to released memory, etc., can be avoided by formulating stricter programming models (such as ownership + life cycle rules proposed by rust) and even data competition issues.

However, behaviors that trigger memory leaks, such as race conditions, require developers to combine development components with business rules. Imagine a data queue that needs to be refreshed manually. As a result, we forgot to call it when pushing data. This memory leak cannot be identified by any conventional inspection rules.

The keyword function is to more vividly explain the location and specific usage of the incoming parameters. If a function has four or five parameters, and too many parameters are passed in at one time, it is hard to dazzle people. If key=value is used to pass in.

to sum up

Through the study in this section, we have learned to control the code flow and execute different codes under different conditions through if else. There are two different loop methods for for/while and how to define a function, including the return value and parameter passing method of the function, the position parameter passing and the key=value parameter passing when passing parameters to the function.

The above is the whole content of this article, I hope it will be helpful to everyone's study.