Image registration is the process of transforming different sets of data into one coordinate system. Data may be multiple photographs, data from different sensors, times, depths, or viewpoints.[1] It is used in computer vision, medical imaging,[2] military automatic target recognition, and compiling and analyzing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from these different measurements.
Algorithm classification
editIntensity-based vs feature-based
editImage registration or image alignment algorithms can be classified into intensity-based and feature-based.[3] One of the images is referred to as the moving or source and the others are referred to as the target, fixed or sensed images. Image registration involves spatially transforming the source/moving image(s) to align with the target image. The reference frame in the target image is stationary, while the other datasets are transformed to match to the target.[3] Intensity-based methods compare intensity patterns in images via correlation metrics, while feature-based methods find correspondence between image features such as points, lines, and contours.[3] Intensity-based methods register entire images or sub-images. If sub-images are registered, centers of corresponding sub images are treated as corresponding feature points. Feature-based methods establish a correspondence between a number of especially distinct points in images. Knowing the correspondence between a number of points in images, a geometrical transformation is then determined to map the target image to the reference images, thereby establishing point-by-point correspondence between the reference and target images.[3] Methods combining intensity-based and feature-based information have also been developed.[4]
Transformation models
editImage registration algorithms can also be classified according to the transformation models they use to relate the target image space to the reference image space. The first broad category of transformation models includes linear transformations, which include rotation, scaling, translation, and other affine transforms.[5] Linear transformations are global in nature, thus, they cannot model local geometric differences between images.[3]
The second category of transformations allow 'elastic' or 'nonrigid' transformations. These transformations are capable of locally warping the target image to align with the reference image. Nonrigid transformations include radial basis functions (thin-plate or surface splines, multiquadrics, and compactly-supported transformations[3]), physical continuum models (viscous fluids), and large deformation models (diffeomorphisms).
Transformations are commonly described by a parametrization, where the model dictates the number of parameters. For instance, the translation of a full image can be described by a single parameter, a translation vector. These models are called parametric models. Non-parametric models on the other hand, do not follow any parameterization, allowing each image element to be displaced arbitrarily.[6]
There are a number of programs that implement both estimation and application of a warp-field. It is a part of the SPM and AIR programs.
Transformations of coordinates via the law of function composition rather than addition
editAlternatively, many advanced methods for spatial normalization are building on structure preserving transformations homeomorphisms and diffeomorphisms since they carry smooth submanifolds smoothly during transformation. Diffeomorphisms are generated in the modern field of Computational Anatomy based on flows since diffeomorphisms are not additive although they form a group, but a group under the law of function composition. For this reason, flows which generalize the ideas of additive groups allow for generating large deformations that preserve topology, providing 1-1 and onto transformations. Computational methods for generating such transformation are often called LDDMM[7][8][9][10] which provide flows of diffeomorphisms as the main computational tool for connecting coordinate systems corresponding to the geodesic flows of Computational Anatomy.
There are a number of programs which generate diffeomorphic transformations of coordinates via diffeomorphic mapping including MRI Studio[11] and MRI Cloud.org[12]
Spatial vs frequency domain methods
editSpatial methods operate in the image domain, matching intensity patterns or features in images. Some of the feature matching algorithms are outgrowths of traditional techniques for performing manual image registration, in which an operator chooses corresponding control points (CP) in images. When the number of control points exceeds the minimum required to define the appropriate transformation model, iterative algorithms like RANSAC can be used to robustly estimate the parameters of a particular transformation type (e.g. affine) for registration of the images.
Frequency-domain methods find the transformation parameters for registration of the images while working in the transform domain. Such methods work for simple transformations, such as translation, rotation, and scaling. Applying the phase correlation method to a pair of images produces a third image which contains a single peak. The location of this peak corresponds to the relative translation between the images. Unlike many spatial-domain algorithms, the phase correlation method is resilient to noise, occlusions, and other defects typical of medical or satellite images. Additionally, the phase correlation uses the fast Fourier transform to compute the cross-correlation between the two images, generally resulting in large performance gains. The method can be extended to determine rotation and scaling differences between two images by first converting the images to log-polar coordinates.[13][14] Due to properties of the Fourier transform, the rotation and scaling parameters can be determined in a manner invariant to translation.
Single- vs multi-modality methods
editAnother classification can be made between single-modality and multi-modality methods. Single-modality methods tend to register images in the same modality acquired by the same scanner/sensor type, while multi-modality registration methods tended to register images acquired by different scanner/sensor types.
Multi-modality registration methods are often used in medical imaging as images of a subject are frequently obtained from different scanners. Examples include registration of brain CT/MRI images or whole body PET/CT images for tumor localization, registration of contrast-enhanced CT images against non-contrast-enhanced CT images[15] for segmentation of specific parts of the anatomy, and registration of ultrasound and CT images for prostate localization in radiotherapy.
Automatic vs interactive methods
editRegistration methods may be classified based on the level of automation they provide. Manual, interactive, semi-automatic, and automatic methods have been developed. Manual methods provide tools to align the images manually. Interactive methods reduce user bias by performing certain key operations automatically while still relying on the user to guide the registration. Semi-automatic methods perform more of the registration steps automatically but depend on the user to verify the correctness of a registration. Automatic methods do not allow any user interaction and perform all registration steps automatically.
Similarity measures for image registration
editImage similarities are broadly used in medical imaging. An image similarity measure quantifies the degree of similarity between intensity patterns in two images.[3] The choice of an image similarity measure depends on the modality of the images to be registered. Common examples of image similarity measures include cross-correlation, mutual information, sum of squared intensity differences, and ratio image uniformity. Mutual information and normalized mutual information are the most popular image similarity measures for registration of multimodality images. Cross-correlation, sum of squared intensity differences and ratio image uniformity are commonly used for registration of images in the same modality.
Many new features have been derived for cost functions based on matching methods via large deformations have emerged in the field Computational Anatomy including Measure matching which are pointsets or landmarks without correspondence, Curve matching and Surface matching via mathematical currents and varifolds.
Uncertainty
editThere is a level of uncertainty associated with registering images that have any spatio-temporal differences. A confident registration with a measure of uncertainty is critical for many change detection applications such as medical diagnostics.
In remote sensing applications where a digital image pixel may represent several kilometers of spatial distance (such as NASA's LANDSAT imagery), an uncertain image registration can mean that a solution could be several kilometers from ground truth. Several notable papers have attempted to quantify uncertainty in image registration in order to compare results.[16][17] However, many approaches to quantifying uncertainty or estimating deformations are computationally intensive or are only applicable to limited sets of spatial transformations.
Applications
edit
Image registration has applications in remote sensing (cartography updating), and computer vision. Due to the vast range of applications to which image registration can be applied, it is impossible to develop a general method that is optimized for all uses.
Medical image registration (for data of the same patient taken at different points in time such as change detection or tumor monitoring) often additionally involves elastic (also known as nonrigid) registration to cope with deformation of the subject (due to breathing, anatomical changes, and so forth).[18][19][20] Nonrigid registration of medical images can also be used to register a patient's data to an anatomical atlas, such as the Talairach atlas for neuroimaging.
In astrophotography, image alignment and stacking are often used to increase the signal to noise ratio for faint objects. Without stacking it may be used to produce a timelapse of events such as a planet's rotation of a transit across the Sun. Using control points (automatically or manually entered), the computer performs transformations on one image to make major features align with a second or multiple images. This technique may also be used for images of different sizes, to allow images taken through different telescopes or lenses to be combined.
In cryo-TEM, instability causes specimen drift and many fast acquisitions with accurate image registration is required to preserve high resolution and obtain high signal to noise images. For low SNR data, the best image registration is achieved by cross-correlating all permutations of images in an image stack.[21]
Image registration is an essential part of panoramic image creation. There are many different techniques that can be implemented in real time and run on embedded devices like cameras and camera-phones.
See also
edit
References
edit- ^ Lisa Gottesfeld Brown, A survey of image registration techniques (abstract), ACM Computing Surveys archive, volume 24, issue 4, December 1992), pages 325 - 376
- ^ biological imaging and brain mapping
- ^ a b c d e f g A. Ardeshir Goshtasby: 2-D and 3-D Image Registration for Medical, Remote Sensing, and Industrial Applications, Wiley Press, 2005.
- ^ Papademetris, Xenophon; Jackowski, Andrea P.; Schultz, Robert T.; Staib, Lawrence H.; Duncan, James S. (2004). "Integrated Intensity and Point-Feature Nonrigid Registration". Medical Image Computing and Computer-Assisted Intervention – MICCAI 2004. Lecture Notes in Computer Science. Vol. 3216. pp. 763–770. doi:10.1007/978-3-540-30135-6_93. ISBN 978-3-540-22976-6. ISSN 0302-9743.
- ^ http://www.comp.nus.edu.sg/~cs4243/lecture/register.pdf [bare URL PDF]
- ^ Sotiras, A.; Davatzikos, C.; Paragios, N. (July 2013). "Deformable Medical Image Registration: A Survey". IEEE Transactions on Medical Imaging. 32 (7): 1153–1190. doi:10.1109/TMI.2013.2265603. PMC 3745275. PMID 23739795.
- ^ Toga, Arthur W. (1998-11-17). Brain Warping. Academic Press. ISBN 9780080525549.
- ^ "Landmark matching on brain surfaces via large deformation diffeomorphisms on the sphere — University of Utah". utah.pure.elsevier.com. Archived from the original on 2018-06-29. Retrieved 2016-03-21.
- ^ Beg, M. Faisal; Miller, Michael I.; Trouvé, Alain; Younes, Laurent (2005). "Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms". International Journal of Computer Vision. 61 (2): 139–157. doi:10.1023/B:VISI.0000043755.93987.aa. S2CID 17772076. Retrieved 2016-03-21.
- ^ Joshi, S. C.; Miller, M. I. (2000-01-01). "Landmark matching via large deformation diffeomorphisms". IEEE Transactions on Image Processing. 9 (8): 1357–1370. Bibcode:2000ITIP....9.1357J. doi:10.1109/83.855431. ISSN 1057-7149. PMID 18262973.
- ^ "MRI Studio".
- ^ "MRICloud Brain Mapping".
- ^ B. Srinivasa Reddy; B. N. Chatterji (Aug 1996). "An FFT-Based Technique for Translation, Rotation and Scale-Invariant Image Registration". IEEE Transactions on Image Processing. 5 (8): 1266–1271. Bibcode:1996ITIP....5.1266R. doi:10.1109/83.506761. PMID 18285214. S2CID 6562358.
- ^ Zokai, S., Wolberg, G., "Image Registration Using Log-Polar Mappings for Recovery of Large-Scale Similarity and Projective Transformations". IEEE Transactions on Image Processing, vol. 14, No. 10, October, 2005.
- ^ Ristea, Nicolae-Catalin; Miron, Andreea-Iuliana; Savencu, Olivian; Georgescu, Mariana-Iuliana; Verga, Nicolae; Khan, Fahad Shahbaz; Ionescu, Radu Tudor (2023). "Cy Tran: A cycle-consistent transformer with multi-level consistency for non-contrast to contrast CT translation". Neurocomputing. 538: 126211. arXiv:2110.06400. doi:10.1016/j.neucom.2023.03.072. S2CID 257952429.
- ^ Simonson, K., Drescher, S., Tanner, F., A Statistics Based Approach to Binary Image Registration with Uncertainty Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 1, January 2007
- ^ Domokos, C., Kato, Z., Francos, J., Parametric estimation of affine deformations of binary images. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, 2008
- ^ Zhao, Shengyu; Lau, Tingfung; Luo, Ji; Chang, Eric I-Chao; Xu, Yan (2020). "Unsupervised 3D End-to-End Medical Image Registration With Volume Tweening Network". IEEE Journal of Biomedical and Health Informatics. 24 (5): 1394–1404. arXiv:1902.05020. doi:10.1109/JBHI.2019.2951024. ISSN 2168-2208. PMID 31689224. S2CID 61153704.
- ^ Chen, Junyu; He, Yufan; Frey, Eric C.; Li, Ye; Du, Yong (2021-04-13). "ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration". arXiv:2104.06468 [eess.IV].
- ^ Burduja, Mihail; Ionescu, Radu Tudor (2021). "Unsupervised Medical Image Alignment with Curriculum Learning". 2021 IEEE International Conference on Image Processing (ICIP). pp. 3787–3791. arXiv:2102.10438. doi:10.1109/ICIP42928.2021.9506067. ISBN 978-1-6654-4115-5. S2CID 231986287.
- ^ Savitsky; El Baggari; Clement; Hovden; Kourkoutis (2018). "Image registration of low signal-to-noise cryo-STEM data". Ultramicroscopy. 191: 56–65. arXiv:1710.09281. doi:10.1016/j.ultramic.2018.04.008. PMID 29843097. S2CID 26983019.
External links
edit
- Richard Szeliski, Image Alignment and Stitching: A Tutorial. Foundations and Trends in Computer Graphics and Computer Vision, 2:1-104, 2006.
- B. Fischer, J. Modersitzki: Ill-posed medicine – an introduction to image registration. Inverse Problems, 24:1–19, 2008
- Barbara Zitová, Jan Flusser: Image registration methods: a survey. Image Vision Comput. 21(11): 977-1000 (2003).
- C. Je and H.-M. Park. Optimized Hierarchical Block Matching for Fast and Accurate Image Registration. Signal Processing: Image Communication, Volume 28, Issue 7, pp. 779–791, August, 2013.
- Registering Multimodal MRI Images using Matlab.
- elastix Archived 2012-04-19 at the Wayback Machine: a toolbox for rigid and nonrigid registration of images.
- niftyreg: a toolbox for doing near real-time robust rigid, affine (using block matching) and non-rigid image registration (using a refactored version of the free form deformation algorithm).
- Image Registration techniques using MATLAB
- Image Compare application automatically compares a pair of images and highlights their differences. This application runs in desktop and mobile phone browsers without requiring installation.