Detailed implementation of Python plug-in mechanism

The plug-in mechanism is a method of injecting code/function reverse dependency into the main program. The compiled language implements the plug-in by dynamically loading the dynamic library. For scripting languages like Python, the plug-in mechanism is simpler.

mechanism

Python's import method can dynamically load Python files, that is, use the file name of a certain py script as the parameter of import, and load the py script program module when the program is running. The corresponding import keyword is to statically load the dependent py module.

description

__ The import__() function is used to dynamically load classes and functions.

If a module changes frequently, you can use import() to dynamically load it.

grammar

__ import__ syntax:

__ import__(name[, globals[, locals[, fromlist[, level]]]])

Parameter Description:

name — module name

If the py script that needs to be dynamically loaded is stored in any directory, you need to add the script search path first:

sys.path.append(modulePath)

Application example

# Increase search path
sys.path.append(modulePath)
# Load script
module =__import__(moduleName) 
# Save the script object, otherwise it will be destroyed
self.modules[moduleName]= module
# Call the method initialization in the plugin
module.InitModule(self)

to sum up

Use the plug-in mechanism to achieve high cohesion and low coupling programs.

In practice, the tasks we deal with are completed by several executable programs. The executable programs can be C++, .Net, Java, or even other script programs. At this time, we use Python as the glue to define the main task flow. The framework uses a plug-in mechanism to dynamically inject tasks that need to be performed.

In addition, when different exe needs to be used in different situations, we only need to use json to define the required exe combination, and the main program can meet the transformed business needs without any changes.

Supplementary knowledge: Kusto uses python plugin

The whole process is that kusto data is automatically converted into pandas DataFrame when entering the python script,

The output of the python script is automatically converted to a kusto table, where the column names and variables remain unchanged.

Python script followed by Kusto's output

Pay attention to the following points

typeof is the output parameter of the python script
The data type in typeof is exactly the same as the pandas DataFrame column output by the python script, including variable names and variable types. If they are inconsistent, an error will be reported.
In typeof, * indicates the data type of multiplexed input, for example (*, age:int) indicates that the input has multiple age attributes on the basis of output
The input of the python script is the kusto table converted to DataFrame, and its variable name in the python script is df (it will be automatically matched). At the same time, we need to name the output DataFrame as result, and the program will automatically output
Python can accept external parameters, through the form of kargs["topK"], kargs is the system default variable to pass parameters, and kusto at the end of the python script passes the form of pack("topK", 10) to the python script Parameters

6 . The python script can be written directly in the kusto code or accessed in the form of a link

The python in kusto runs the enterprise version of anaconda, and individuals cannot easily install the packages they want, so if you want to use certain packages, it is best to write their functions in the most basic packages. The sandbox of kusto running mirrors supports numpy, pd, and tensorflow, keras, torch hdbscan, xgboost and other popular packages.

The above detailed explanation of the implementation of the Python plug-in mechanism is all the content shared by the editor. I hope to give you a reference.