Post


where I express ideas
Image Stitching

Image Stitching In Computer Vision

Ever wondered how image stitching is done for panorama, here's an algorithm using RANSAC.

1 Pairwise stitching

In this part of the assignment, the task is to stitch together two images. The general setup is described as the following:

  1. Preprocessing images
    • The two input images were converted to double and turned to gray scale
  2. Detecting interesting spots
    • Harris corner detector was used and applied to both images
  3. Extracting features from detected blobs
    • SIFT feature was extracted from each detected blob
  4. Putative matching
    • A similarity matrix between 2 sets of features was build using pairwise squared Euclidean distance
    • Nearest Neighbor algorithm was applied on the similarity matrix to match features in one image to those of the other
  5. Model selection
    • RANSAC was implemented
      • s = minimum points was chosen to be 4
      • t = inlier threshold was chosen to be 30
      • N = number of iteration was chosen to be 8000
      • d = consensus size was chosen to be 0.1 of number of features (I expect the inlier ratio to be 0.1)
      • 4 pairs of feature matching was selected in each iteration of the algorithm
      • A homogeneous least square equation was solved using SVD
      • using the found transformation, the sum of square error was calculated
      • inliers were selected by accepting all features with SSE < t
      • if the number of inliers is larger than d, the homogeneous equation for homography is recalculated by fitting all points, and the number of inliers is updated
      • Thae final output is the homography transformation with the largest number of inliers
  6. Image stitching
    • images are stitched together by applying the best transformation on the first image and superimpose the transformed image on the second image (the overlapping region used the second image for color output, and took average for the grayscale output)

Outputs:
Number of homography inliers is 371
Average residual is 327016.491630

/static/blog_static/image_stitching/inlier_matches.jpg

/static/blog_static/image_stitching/uttower_bw.jpg

/static/blog_static/image_stitching/uttower_color.jpg

2 Multiple image stitching

The second part of the assignment is to stitch multiple images together. My solution is outlined as the following:

  1. Image preprocessing
  2. images were read in and stored in a data structure called imCell
  3. Image stitching
  4. while imCell has more than 1 element for each element in imCell except the first one compute the number of inliers for the stitch between this element and imCell{1} using code from part1 (the order of stitching is important because I used nearest neighbor to match the features from the first image to the second image which means the transformation is unlikely to be one to one. I used the current element as the first argument of stitch because it doesn’t have overlapping features so that most of its distinctive points are preserved. I cannot use imCell{1} because it contains too much irrelevant details after several iteration inside the while loop)

    check that the max number of inliers is greater than 50 (arbitrarily chosen so as to assure that the found transformation is reliable) and proceed to the next step. Otherwise, go back to the previous for loop if less than 10 times going back tried. If more than 10 times tried, restart from step 2 by replacing imCell with the original imCell

    update imCell{1} with the color image output from the stitch(imCell{chosen}, imCell{1}) where ‘chosen’ is the id of the element that gives the most inliers

    delete the imCell(chosen)

Output:
The order of stitching for Pier is:
1.JPG stitched with 2.JPG
the combined JPG was stitched with 3.JPG

/static/blog_static/image_stitching/pier_stitched.jpg

Graph 2.1 Pier stitched image

The order of stitching for Hill is: 1.JPG stitched with 2.JPG the combined JPG was stitched with 3.JPG

/static/blog_static/image_stitching/hill_stitched.jpg

Graph 2.2 Hill stitched image

The order of stitching for Ledge is: 1.JPG stitched with 2.JPG the combined JPG was stitched with 3.JPG

/static/blog_static/image_stitching/ledge_stitched.jpg

Graph 2.3 Ledge stitched image

Discussion: There are some artifact in stitching multiple images.

  1. Stitched image has inconsistent colors
    Since images comes from different sources, when stitching them together, the overall color is inconsistent as well. This could be addressed by using one image as a baseline and reconstruct the other image using its gradient.
  2. Stitched image has obvious separating lines between images
    In performing transformation, some of the pixels on the boundary get assigned no colors. It could be an artifact of matlab.
  3. On rare occasions the algorithm didn’t succeed
    There are several reasons that could induce this problem
    • no iteration reached the desired inlier ratio (addressed by lowering inlier ratio and increasing number of iterations)
    • too little inliers (addressed in my implementation by either reentering the for loop or restarting the algorithm)
    • matlab artifact in applying transformation (in rare cases, matlab imtransform function rescales the transformed image, can be addressed by setting ‘XYScale’ to 1)

Date: 2015-03-08T15:59-0400

Author: Jiaxuan Wang