I won't talk too much nonsense, let's just look at the code!
import os
import time
import mmap
filename ='test.txt'
# If it does not exist, create it.
if not os.path.exists(filename):open(filename,'w')print(os.path.isdir(filename))if os.path.isfile(filename):print(time.ctime(os.path.getctime(filename)))
fd = os.open(filename, os.O_RDWR)
m = mmap.mmap(fd,50, access=mmap.ACCESS_WRITE) #1024-byte file.
m.seek(2)
buf1 =bytes(b'Zhang')
m[2:len(buf1)+2]= buf1
buf2 = b'Phil'
m.seek(20) #Position the writing position.
m.write(buf2) #Write byte data.
m.close()
fd = os.open(filename, os.O_RDWR)
m = mmap.mmap(fd,50, access=mmap.ACCESS_READ)
m.seek(20)
buf3 = m.read(len(buf2))
m.close()print(list(buf3))
The contents of the test.txt file after the operation:
Zhang Phil
Supplementary knowledge: Python does memory mapping to binary files, and random access to content efficiently and gracefully
Reading and writing binary files are still using the open function?
Are the various combinations of seek(), read() and write() tired?
Using mmap module to realize the memory mapping of files, let us read and write binary files as efficient and elegant as manipulating arrays.
First, give a practical function to demonstrate how to open a file and perform memory mapping operations on it.
def memory_map(filename, access=mmap.ACCESS_WRITE):
size = os.path.getsize(filename)
fd = os.open(filename, os.O_RDWR)return mmap.mmap(fd, size, access=access)
To use this function, you need to prepare an already created file and fill it with some data.
size =1000000withopen('data','wb')as f:
f.seek(size-1)
f.write(b'\x00')
Then we can use the memory_map() function to memory map the contents of the file, and the mmap object returned by it allows us to read and write binary files like an array!
m =memory_map('data')len(m)1000000
m[:10]
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
m[0]0
m[0:11]=b'Hello World'#Change a slice
m.close()withopen('data','rb')as f:#Verify that the changes have taken effect
print(f.read(11))
b'Hello World'
Since the mmap object returned by mmap() can also be used as a context manager, in this case, the underlying file will be automatically closed.
withmemory_map('data')as m:print(len(m))print(m[0:11])1000000
b'Hello World'
By default, the file opened by the memory_map() function can be read or written. Any changes to the data will be copied back to the original file.
If you need read-only access, you can provide the mmap.ACCESS_READ value for the access parameter.
m = memory_map(filename, mmap.ACCESS_READ)
If you only want to modify the data locally and do not want to write these modifications back to the original file, you can use the mmap.ACCESS_COPY parameter.
m = memory_map(filename, mmap.ACCESS_COPY)
to sum up:
After mapping the file to memory through mmap, we can efficiently and elegantly perform random access to the content of the file.
Instead of opening the file and accessing it by combining various seek(), read() and write() calls, it is better to simply map the file to memory and then access the data through the slice operation.
It should be emphasized that memory mapping of a file does not cause the entire file to be read into memory. In other words, the file will not be copied to some kind of memory buffer or array. In contrast, the operating system only reserves a piece of virtual memory for the contents of the file.
When accessing different areas of the file, these areas of the file will be read and mapped to the memory area as needed. However, the parts of the file that have never been accessed are simply left on the disk. All this is done behind the scenes in a transparent way.
If multiple Python interpreters have memory mapped the same file, the mmap object obtained can be used to exchange data between the interpreters. In other words, all interpreters can read/write data at the same time, and changes made to the data in one interpreter will automatically be reflected in other interpreters.
Although some additional steps are needed to deal with synchronization issues, sometimes this method can be used as an alternative to transferring data through pipes or sockets.
The above reading and writing method of Python memory mapped file is all the content shared by the editor, I hope to give you a reference.
Recommended Posts