Python implements panoramic image stitching
Image panorama stitching includes three parts: feature point extraction and matching, image registration, and image fusion.
1、 SIFT-based feature point extraction and matching
Use Sift to extract the local features of the image, find the extreme point in the scale space, and extract its position, scale, and direction information.
Specific steps:
1 ). Generate Gaussian Difference Pyramid (DOG Pyramid), build scale space
2 ). Spatial extreme point detection (preliminary exploration of key points)
3). Accurate positioning of stable key points
4 ). Stable key point direction information distribution
5). Key point description
6). Feature point matching
2、 Image registration
Image registration is a technology to determine the overlap area and overlap position between images to be stitched, and it is the core of the entire image stitching. This section adopts the method of image registration based on feature points, that is, the transformation matrix between image sequences is constructed by matching point pairs to complete the mosaic of panoramic images.
The solution of transformation matrix H is the core of image registration, and the algorithm flow of its solution is as follows.
1 ) Detect feature points in each image.
2 ) Calculate the matching between feature points.
3 ) Calculate the initial value of the transformation matrix between images.
4 ) Iteratively refine the H transformation matrix.
5 ) Boot matching. Use the estimated H to define the search area near the epipolar line to further determine the correspondence of feature points.
6 ) Repeat iterations 4) and 5) until the number of corresponding points is stable.
Let the transformation between image sequences be projection transformation
The 8 degrees of freedom parameters hi=(i=0,1,...,7) of the H matrix can be calculated using 4 sets of best matches, and use this as the initial value.
In order to improve the accuracy of image registration, this section uses the RANSAC algorithm to solve and refine the image transformation matrix to achieve a better image mosaic effect. The idea of the RANSAC algorithm is simple and clever: first, two points are randomly selected, these two points determine a straight line, and the points within a certain range of the straight line are called the support of this straight line. This random selection is repeated several times, and then, the straight line with the largest support set is confirmed as the fitting of the sample point set. The points within the error distance of the fitting are considered as interior points, and they constitute a consistent set, otherwise, they are exterior points. According to the algorithm description, it can be quickly judged that if there are only a small number of outer points, then the line determined by the randomly selected initial point set containing the outer points will not receive much support. It is worth noting that an excessively large proportion of outer points will lead to The RANSAC algorithm failed. In the example of straight line fitting, at least two points are required to determine a straight line from a point set; while for perspective transformation, such a minimum set requires 4 points.
3、 Image fusion
Because of the difference between the camera and the light intensity, it will cause uneven brightness within an image and between the images. The spliced image will alternate between light and dark, which causes great inconvenience to observation. Brightness and color balance processing, the usual processing method is to correct the unevenness of the illumination inside an image through the camera's illumination model, and then establish the relationship between the adjacent two images through the relationship between the overlapping areas of the two adjacent images Histogram mapping table, through the mapping table to do the overall mapping transformation of the two images, and finally achieve the overall brightness and color consistency.
Implementation:
import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt
if __name__ =='__main__':
top, bot, left, right =100,100,0,500
img1 = cv.imread('1.jpg')
img2 = cv.imread('2.jpg')
srcImg = cv.copyMakeBorder(img1, top, bot, left, right, cv.BORDER_CONSTANT, value=(0,0,0))
testImg = cv.copyMakeBorder(img2, top, bot, left, right, cv.BORDER_CONSTANT, value=(0,0,0))
img1gray = cv.cvtColor(srcImg, cv.COLOR_BGR2GRAY)
img2gray = cv.cvtColor(testImg, cv.COLOR_BGR2GRAY)
sift = cv.xfeatures2d_SIFT().create()
# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1gray, None)
kp2, des2 = sift.detectAndCompute(img2gray, None)
# FLANN parameters
FLANN_INDEX_KDTREE =1
index_params =dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
search_params =dict(checks=50)
flann = cv.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1, des2, k=2)
# Need to draw only good matches, so create a mask
matchesMask =[[0,0]for i inrange(len(matches))]
good =[]
pts1 =[]
pts2 =[]
# ratio test as per Lowe's paper
for i,(m, n)inenumerate(matches):if m.distance <0.7*n.distance:
good.append(m)
pts2.append(kp2[m.trainIdx].pt)
pts1.append(kp1[m.queryIdx].pt)
matchesMask[i]=[1,0]
draw_params =dict(matchColor=(0,255,0),
singlePointColor=(255,0,0),
matchesMask=matchesMask,
flags=0)
img3 = cv.drawMatchesKnn(img1gray, kp1, img2gray, kp2, matches, None,**draw_params)
plt.imshow(img3,), plt.show()
rows, cols = srcImg.shape[:2]
MIN_MATCH_COUNT =10iflen(good) MIN_MATCH_COUNT:
src_pts = np.float32([kp1[m.queryIdx].pt for m in good]).reshape(-1,1,2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in good]).reshape(-1,1,2)
M, mask = cv.findHomography(src_pts, dst_pts, cv.RANSAC,5.0)
warpImg = cv.warpPerspective(testImg, np.array(M),(testImg.shape[1], testImg.shape[0]), flags=cv.WARP_INVERSE_MAP)for col inrange(0, cols):if srcImg[:, col].any() and warpImg[:, col].any():
left = col
breakfor col inrange(cols-1,0,-1):if srcImg[:, col].any() and warpImg[:, col].any():
right = col
break
res = np.zeros([rows, cols,3], np.uint8)for row inrange(0, rows):for col inrange(0, cols):if not srcImg[row, col].any():
res[row, col]= warpImg[row, col]
elif not warpImg[row, col].any():
res[row, col]= srcImg[row, col]else:
srcImgLen =float(abs(col - left))
testImgLen =float(abs(col - right))
alpha = srcImgLen /(srcImgLen + testImgLen)
res[row, col]= np.clip(srcImg[row, col]*(1-alpha)+ warpImg[row, col]* alpha,0,255)
# opencv is bgr, matplotlib is rgb
res = cv.cvtColor(res, cv.COLOR_BGR2RGB)
# show the result
plt.figure()
plt.imshow(res)
plt.show()else:print("Not enough matches are found - {}/{}".format(len(good), MIN_MATCH_COUNT))
matchesMask = None
**Experimental results: **
1、 Indoor scene:
Original image 1
Original image 2
After splicing:
2、 Outdoor scene:
scene 1:
Original image 1
Original image 2
After splicing:
Scene 2:
Original image 1
Original image 2
After splicing:
Scene 3:
Original image 1
Original image 2
After splicing:
to sum up:
In this paper, the two images are stitched in panoramic view for indoor and outdoor situations, and it is found that the stitching effect is better in indoor situations. In the case of outdoor scene 1, the two images have a combination of near and far. After the two images are spliced, the close-in image is enlarged and tilted to a certain extent; in scene 2, the two images are both distant views, and the splicing effect is still Not bad, but there are stitching seams in the middle and upper part of the stitched image; scene 3 is the stitching of images at different brightness levels. It can be found that the stitched image has obvious light and dark gaps, and the stitching seams are obviously not good. Spliced together, there are many places that do not overlap.
This experiment originally used the opencv-contrib 3.4.5 version, but it could not be used due to sift's patent restrictions, and then the opencv-contriv3.4.2 code can be run without problems. Method: first uninstall the current version of opencv and install:
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple opencv-contrib-python==3.4.2.16
This article has been included in the topic "Python Image Processing Operations", welcome to click to learn more exciting content.
The above is the whole content of this article, I hope it will be helpful to everyone's study.
Recommended Posts