Python，OpenCV——对齐和覆盖多个图像，一个接一个

Question

我的项目是对齐航拍照片以制作马赛克地图。我的计划是从两张照片开始，将第二张与第一张对齐，然后从两张对齐的图像中创建一个“初始马赛克”。完成后，我将第三张照片与初始马赛克对齐，然后将第四张照片与结果对齐，依此类推，从而逐步构建地图。

我有两种技术可以做到这一点，但更准确的一种是使用calcOpticalFlowPyrLK()，它只适用于双图像阶段，因为两个输入图像的大小必须相同。因此我尝试了一个新的解决方案，但它不太准确，而且每一步引入的错误都会堆积起来，最终产生一个荒谬的结果。

我的问题有两个方面，但如果您知道其中一个问题的答案，则不必同时回答两个问题，除非您愿意。首先，有没有办法使用类似于calcOpticalFlowPyrLK() 但有两个不同大小的图像（这包括任何潜在的解决方法）？其次，有没有办法修改检测器/描述符解决方案以使其更准确？

这是仅适用于两个图像的准确版本：

# load images
base = cv2.imread("images/1.jpg")
curr = cv2.imread("images/2.jpg")

# convert to grayscale
base_gray = cv2.cvtColor(base, cv2.COLOR_BGR2GRAY)

# find the coordinates of good features to track  in base
base_features = cv2.goodFeaturesToTrack(base_gray, 3000, .01, 10)

# find corresponding features in current photo
curr_features = np.array([])
curr_features, pyr_stati, _ = cv2.calcOpticalFlowPyrLK(base, curr, base_features, curr_features, flags=1)

# only add features for which a match was found to the pruned arrays
base_features_pruned = []
curr_features_pruned = []
for index, status in enumerate(pyr_stati):
    if status == 1:
        base_features_pruned.append(base_features[index])
        curr_features_pruned.append(curr_features[index])

# convert lists to numpy arrays so they can be passed to opencv function
bf_final = np.asarray(base_features_pruned)
cf_final = np.asarray(curr_features_pruned)

# find perspective transformation using the arrays of corresponding points
transformation, hom_stati = cv2.findHomography(cf_final, bf_final, method=cv2.RANSAC, ransacReprojThreshold=1)

# transform the images and overlay them to see if they align properly
# not what I do in the actual program, just for use in the example code
# so that you can see how they align, if you decide to run it
height, width = curr.shape[:2]
mod_photo = cv2.warpPerspective(curr, transformation, (width, height))
new_image = cv2.addWeighted(mod_photo, .5, base, .5, 1)

这是适用于多个图像的不准确的一个（直到错误变得太大）：

# load images
base = cv2.imread("images/1.jpg")
curr = cv2.imread("images/2.jpg")


# convert to grayscale
base_gray = cv2.cvtColor(self.base, cv2.COLOR_BGR2GRAY)

# DIFFERENCES START
curr_gray = cv2.cvtColor(self.curr_photo, cv2.COLOR_BGR2GRAY)

# create detector, get keypoints and descriptors
detector = cv2.ORB_create()
base_keys, base_desc = detector.detectAndCompute(base_gray, None)
curr_keys, curr_desc = detector.detectAndCompute(curr_gray, None)

matcher = cv2.DescriptorMatcher_create("BruteForce-Hamming")

max_dist = 0.0
min_dist = 100.0

for match in matches:
     dist = match.distance
     min_dist = dist if dist < min_dist else min_dist
     max_dist = dist if dist > max_dist else max_dist

good_matches = [match for match in matches if match.distance <= 3 * min_dist ]

base_matches = []
curr_matches = []
for match in good_matches:
    base_matches.append(base_keys[match.queryIdx].pt)
    curr_matches.append(curr_keys[match.trainIdx].pt)

bf_final = np.asarray(base_matches)
cf_final = np.asarray(curr_matches)

# SAME AS BEFORE

# find perspective transformation using the arrays of corresponding points
transformation, hom_stati = cv2.findHomography(cf_final, bf_final, method=cv2.RANSAC, ransacReprojThreshold=1)

# transform the images and overlay them to see if they align properly
# not what I do in the actual program, just for use in the example code
# so that you can see how they align, if you decide to run it
height, width = curr.shape[:2]
mod_photo = cv2.warpPerspective(curr, transformation, (width, height))
new_image = cv2.addWeighted(mod_photo, .5, base, .5, 1)

最后，这里有一些我正在使用的图片：

Answer 1

单应性组成，所以如果你有img1和img2之间的单应性，以及img2和img3之间的单应性，那么这两个单应性的组合给出了img1和img3之间的单应性。

您的尺寸当然不正确，因为您试图将img3 与包含img1 和img2 的拼接图像匹配。但你不需要这样做。在您获得每对连续图像之间的所有单应性之前，不要缝合它们。然后您可以通过以下两种方式之一进行；从后面工作或从前面工作。我将用于例如h31 是指将img3 扭曲成img1 坐标的单应性。

从前面（伪代码）：

warp img2 into coordinates of img1 with h21
warp img3 into coordinates of img1 with h31 = h32 @ h21
warp img4 into coordinates of img1 with h41 = h43 @ h31
...
stitch/blend images together

这里@ 是矩阵乘法运算符，它将实现我们的单应性组合（注意，最安全的做法是除以单应性中的最后一项，以确保它们都缩放相同）。

从后面（伪代码）：

...
warp prev stitched img into coordinates of img3 with h43
stitch warped stitched img with img3
warp prev stitched img into coordinates of img2 with h32
stitch warped stitched img with img2
warp prev stitched img into coordinates of img1 with h21
stitch warped stitched img with img1

这个想法是要么从前面开始，将所有内容变形到第一个图像坐标框架中，或者从后面开始，变形到前一个图像并缝合，然后将缝合的图像变形到前一个图像，然后重复.我认为第一种方法可能更容易。在任何一种情况下，您都必须担心单应性估计中错误的传播，因为它们会在多个组合单应性上累积。

这是将多个图像与单应性混合在一起的幼稚方法。更复杂的方法是使用捆绑调整，它考虑了所有图像的特征点。然后为了获得良好的混合，步骤是增益补偿以消除相机增益调整和渐晕，然后是多波段混合以防止模糊。请参阅 Brown 和 Lowe here 的开创性论文以及精彩的示例和免费演示软件 here。