Python black magic metaclass

It takes about 5 minutes to read this article.

Python has a lot of black magic, in order not to divide your heart, today only talk about metaclass. There are two extreme views on this feature of metaclass:

• This feature is too powerful, it is the omnipotent Aladdin's magic lamp, you must find opportunities to use it to show your Python strength. • This feature is too dangerous and will confuse people to abuse it. Once opened, it will release demons, making the code difficult to maintain.

Today we will take a look at whether metaclass is Aladdin’s magic lamp or Pandora’s box.

What is metaclass

Many books will be translated into metaclass, which is only understood literally, meta is indeed a meta, the source, the translation is fine. However, when understanding, you should understand meta as transcending data that describes data. In fact, the meta of metaclass originated from the Greek word meta, which contains two meanings:

• "Beyond", such as the technical term metadata, means beyond data that describes data. • "Change", such as the technical word metamorphosis, means the form of change.

Therefore, it can be understood that metaclass is a superclass describing a class, and at the same time, the form of a subclass can be changed. You may ask, is this similar to the definition of metadata? What is the use of this feature in programming?

Very useful. In the absence of a metaclass, the subclass inherits the parent class, and the parent class cannot perform operations on the subclass, but with the metaclass, you can operate on the subclass, just like a decorator, which can be dynamically customized and modified Class, metaclass can be dynamically customized or modified to inherit its subclasses.

**What problems can metaclass solve? **

You already know that a metaclass can be customized and modified like a decorator to inherit its subclasses. Here is what practical problems it can solve. For example, in a large-scale intelligent voice assistant project, we have 10,000 voice dialogue scenarios, and each scenario is developed by a different team. As a core team member of the intelligent voice assistant, it is impossible for you to understand the implementation details of each sub-scenario.

When experimenting with different scenes in dynamic configuration, it is often to experiment with the configuration of scenes A and B today, and the configuration of experiments B and C tomorrow. The configuration file alone has the order of tens of thousands of lines, and the workload is not small. Using this dynamic configuration concept, I can let the engine dynamically load the required Python classes according to my text configuration file.

If you are not very clear, then you should know YAML. It is a well-known Python tool that can easily serialize and deserialize data. YAMLObject can let any of its subclasses support serialization and deserialization (serialization & deserialization). deserialization). You should be clear about serialization and deserialization:

•Serialization: When the program is running, all variables or objects are stored in memory. Once the program is called, the memory occupied by these variables or objects will be recycled. In order to achieve persistent storage of variables and objects to disk or transmission over the network, we need to convert variables or objects into a binary stream. The process of converting it into a binary stream is serialization. • Deserialization: Deserialization means that the program cannot be read from the disk when the program is running. The serialized object or variable needs to be transferred from the disk to the memory, and the binary stream will be converted into the original data. format. We call this process deserialization.

Now you have 10,000 YAML configuration files in different formats. Originally, you need to write 10,000 classes to load these configuration files. With a metaclass, you only need to implement a metaclass superclass, and then implement a subclass to inherit this metaclass. Different classes can be automatically pulled according to different configuration files, which greatly improves efficiency.

To understand metaclass through an example

Please manually code in ipython and see what is output at each step, so that you can thoroughly understand the steps of class creation and instantiation.

In[15]:classMymeta(type):...:     def __init__(self, name, bases, dic):...:super().__init__(name, bases, dic)...:print('===>Mymeta.__init__')...:print(self.__name__)...:print(dic)...:print(self.yaml_tag)...:...:     def __new__(cls,*args,**kwargs):...:print('===>Mymeta.__new__')...:print(cls.__name__)...:return type.__new__(cls,*args,**kwargs)...:...:     def __call__(cls,*args,**kwargs):...:print('===>Mymeta.__call__')...:         obj = cls.__new__(cls)...:         cls.__init__(cls,*args,**kwargs)...:return obj
   ...: 
In[16]: 
In[16]: 
In[16]:classFoo(metaclass=Mymeta):...:     yaml_tag ='!Foo'...:...:     def __init__(self, name):...:print('Foo.__init__')...:         self.name = name
 ...:...:  def __new__(cls,*args,**kwargs):...:print('Foo.__new__')...:return object.__new__(cls)...:===>Mymeta.__new__
Mymeta
===> Mymeta.__init__
Foo
{'__ module__':'__main__','__qualname__':'Foo','yaml_tag':'!Foo','__init__':<function Foo.__init__ at 0x0000000007EF3828>,'__new__':<function Foo.__new__ at 0x0000000007EF3558>}!Foo

In[17]: foo =Foo('foo')===>Mymeta.__call__
Foo.__new__
Foo.__init__

In[18]:

From the above running results, we can find that when defining the class Foo() definition, MyMeta’s __new__ and __init__ methods will be called in turn to build the Foo class, and then when foo = Foo() is called to create an instance of the class, it will Call the __call__ method of MyMeta to call the __new__ and __init__ methods of the Foo class.

After running the above example, you will understand a lot. Under normal circumstances, we cannot operate the properties of the subclass in the parent class, but the metaclass can. Another way of understanding: metaclasses, decorators, and class decorators can all be classified as metaprogramming.

**How does the Python underlying language design level implement metaclass? **

To understand the underlying principles of metaclass, you need a deep understanding of the Python type model. Below, it will be explained in three points.

**First, all Python user-defined classes are instances of type. **

It may surprise you. In fact, the class itself is just an instance of a class called type. In the Python type world, the type type is the God of creation. This can be verified in the code:

In [2]: #Python 3 is similar to Python 2
 ...: classMyClass:...:   pass
 ...:...: instance =MyClass()...:in[3]:type(instance)...:
Out[2]: __main__.MyClass
In [4]:type(MyClass)...:
Out[4]: type
In [5]:

As you can see, instance is an instance of MyClass, and MyClass is just an instance of "God" type.

Second, the user-defined class is nothing but the __call__ operator overload of the type class

When we define a class at the end of the statement, what really happens is that Python calls the __call__ operator of type. Simply put, when you define a class, write the following:

classMyClass:
 data =1

What Python actually executes is the following code:

class=type(classname, superclasses, attributedict)

The type(classname, superclasses, attributedict) on the right side of the equal sign here is the __call__ operator overload of type, which will further call:

type.__new__(typeclass, classname, superclasses, attributedict)
type.__init__(class, classname, superclasses, attributedict)

Of course, all of this can be verified by code, such as

In [5]:classMyClass:...:     data =1...:...: instance =MyClass()...:

In [6]: MyClass, instance
   ...:
Out[6]:(__main__.MyClass,<__main__.MyClass at 0x4ef5188>)

In [7]: instance.data
   ...:
Out[7]:1

In [8]: MyClass =type('MyClass',(),{'data':1})...: instance =MyClass()...:

In [9]: MyClass, instance
   ...:
Out[9]:(__main__.MyClass,<__main__.MyClass at 0x4f40748>)

In [10]: instance.data
    ...:
Out[10]:1

In [11]:

It can be seen that the normal definition of MyClass is exactly the same as the result of manually calling the type operator.

Third, metaclass is a subclass of type. By replacing the __call__ operator overload mechanism of type, the normal class "beyond the deformation"

In fact, after understanding the above points, we will understand that it is Python's class creation mechanism that gives metaclass the opportunity to show its talents.

Once you set the metaclass of a type MyClass to MyMeta, MyClass will no longer be created by the native type, but will call the __call__ operator overload of MyMeta.

class=type(classname, superclasses, attributedict) 
# Becomes
class=MyMeta(classname, superclasses, attributedict)

Risks of using metaclass

However, all things have advantages and disadvantages, especially the existence of metaclass. As you can see, the metaclass will "distort" the normal Python type model. Therefore, if it is used carelessly, the risk to the entire code base is immeasurable.

In other words, metaclass is only used by a small group of Python developers when developing framework-level Python libraries. In the application layer, metaclass is often not a good choice.

to sum up

This article helps you understand the role of metaclass from the process of Python class creation.

Metaclass is black magic. If used properly, it means heaven, otherwise it means hell.

(Finish)