Structure from motion
Structure from motion (SfM) is a photogrammetric range imaging technique for estimating three-dimensional structures from two-dimensional image sequences that may be coupled with local motion signals. It is studied in the fields of computer vision and visual perception. In biological vision, SfM refers to the phenomenon by which humans (and other living creatures) can recover 3D structure from the projected 2D (retinal) motion field of a moving object or scene.
Humans perceive a lot of information about the three-dimensional structure in their environment by moving around it. When the observer moves and the objects around the observer move, information is obtained from images sensed over time.
Finding structure from motion presents a similar problem to finding structure from stereo vision. In both instances, the correspondence between images and the reconstruction of 3D object needs to be found.
To find correspondence between images, features such as corner points (edges with gradients in multiple directions) are tracked from one image to the next. One of the most widely used feature detectors is the scale-invariant feature transform (SIFT). It uses the maxima from a difference-of-Gaussians (DOG) pyramid as features. The first step in SIFT is finding a dominant gradient direction. To make it rotation-invariant, the descriptor is rotated to fit this orientation. Another common feature detector is the SURF (speeded-up robust features). In SURF, the DOG is replaced with a Hessian matrix-based blob detector. Also, instead of evaluating the gradient histograms, SURF computes for the sums of gradient components and the sums of their absolute values. Another type of feature recently made practical for structure from motion are general curves (e.g., locally an edge with gradients in one direction), part of a technology known as "Pointless" SfM, useful when point features are insufficient, common in man-made environments.
Sometimes some of the matched features are incorrectly matched. This is why the matches should also be filtered. RANSAC (random sample consensus) is the algorithm that is usually used to remove the outlier correspondences. In the paper of Fischler and Bolles, RANSAC is used to solve the location determination problem (LDP), where the objective is to determine the points in space that project onto an image into a set of landmarks with known locations.
The feature trajectories over time are then used to reconstruct their 3D positions and the camera's motion. An alternative is given by so-called direct approaches, where geometric information (3D structure and camera motion) is directly estimated from the images, without intermediate abstraction to features or corners.
There are several approaches to structure from motion. In incremental SFM, camera poses are solved for and added one by one to the collection. In global SFM , the poses of all cameras are solved for at the same time. A somewhat intermediate approach is out-of-core SFM, where several partial reconstructions are computed that are then integrated into a global solution.
Structure from Motion photogrammetry with multi-view stereo provides hyperscale landform models using images acquired from a range of digital cameras and optionally a network of ground control points. The technique is not limited in temporal frequency and can provide point cloud data comparable in density and accuracy to those generated by terrestrial and airborne laser scanning at a fraction of the cost.. Structure from motion is also useful in remote or rugged environments where terrestrial laser scanning is limited by equipment portability and airborne laser scanning is limited by terrain roughness causing loss of data and image foreshortening. The technique has been applied in many settings such as rivers, badlands, sandy coastlines, fault zones, landslides, and coral reef settings. SfM has been also successfully applied for the characterization of rock masses through the determination of some properties as the orientation, persistence, etc. of discontinuities. A full range of digital cameras can be utilized, including digital SLR's, compact digital cameras and even smart phones. Generally though, higher accuracy data will be achieved with more expensive cameras, which include lenses of higher optical quality. The technique therefore offers exciting opportunities to characterize surface topography in unprecedented detail and, with multi-temporal data, to detect elevation, position and volumetric changes that are symptomatic of earth surface processes. Structure from Motion can be placed in the context of other digital surveying methods.
Cultural heritage is present everywhere. Its structural control, documentation and conservation is one of humanity's main duties (UNESCO). Under this point of view, SfM is used in order to properly estimate situations as well as planning and maintenance efforts and costs, control and restoration. Because serious constraints often exist connected to the accessibility of the site and impossibility to install invasive surveying pillars that did not permit the use of traditional surveying routines (like total stations), SfM provides a non-invasive approach for the structure, without the direct interaction between the structure and any operator. The use is accurate as only qualitative considerations are needed. It is fast enough to respond to the monument’s immediate management needs. The first operational phase is an accurate preparation of the photogrammetric surveying where is established the relation between best distance from the object, focal length, the ground sampling distance (GSD) and the sensor’s resolution. With this information the programmed photographic acquisitions must be made using vertical overlapping of at least 60% (figure 02).
- Jonathan L. Carrivick, Mark W. Smith, Duncan J. Quincey (2016). Structure from Motion in the Geosciences. Wiley-Blackwell. 208 pages. ISBN 978-1-118-89584-9
- Richard Hartley & Andrew Zisserman (2003). Multiple View Geometry in Computer Vision. Cambridge University Press. ISBN 978-0-521-54051-3.
- Olivier Faugeras and Quang-Tuan Luong and Theodore Papadopoulo (2001). The Geometry of Multiple Images. MIT Press. ISBN 978-0-262-06220-6.
- Yi Ma; S. Shankar Sastry; Jana Kosecka; Stefano Soatto (November 2003). An Invitation to 3-D Vision: From Images to Geometric Models. Interdisciplinary Applied Mathematics Series, #26. Springer-Verlag New York, LLC. ISBN 978-0-387-00893-6.
- S. Ullman (1979). "The interpretation of structure from motion". Proceedings of the Royal Society of London. 203 (1153): 405–426. doi:10.1098/rspb.1979.0006.
- Linda G. Shapiro; George C. Stockman (2001). Computer Vision. Prentice Hall. ISBN 978-0-13-030796-5.
- D. G. Lowe (2004). "Distinctive image features from scale-invariant keypoints". International Journal of Computer Vision. 60 (2): 91–110. CiteSeerX 10.1.1.73.2924. doi:10.1023/b:visi.0000029664.99615.94.
- H. Bay; T. Tuytelaars & L. Van Gool (2006). "Surf: Speeded up robust features". 9th European Conference on Computer Vision.
- K. Häming & G. Peters (2010). "The structure-from-motion reconstruction pipeline – a survey with focus on short image sequences". Kybernetika. 46 (5): 926–937.
- Nurutdinova, Andrew; Fitzgibbon, Andrew (2015). "Towards Pointless Structure from Motion: 3D reconstruction and camera parameters from general 3D curves" (PDF). Proceedings of the IEEE International Conference on Computer Vision (ICCV 2015): 2363--2371. doi:10.1109/ICCV.2015.272.
- Fabbri, Ricardo; Giblin, Peter; Kimia, Benjamin (2012). "Camera Pose Estimation Using First-Order Curve Differential Geometry" (PDF). Lecture Notes in Computer Science (ECCV 2012). 7575: 231–244. doi:10.1007/978-3-642-33765-9_17.
- Apple, ARKIT team (2018). "Understanding ARKit Tracking and Detection". WWDC.
- B. D. Lucas & T. Kanade. "An iterative image registration technique with an application to stereo vision". Ijcai81.
- M. A. Fischler & R. C. Bolles (1981). "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography". Commun. ACM. 24 (6): 381–395. doi:10.1145/358669.358692.
- F. Dellaert; S. Seitz; C. Thorpe & S. Thrun (2000). "Structure from Motion without Correspondence" (PDF). IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
- Engel, Jakob; Schöps, Thomas; Cremers, Daniel (2014). "LSD-SLAM: Large-Scale Direct Monocular SLAM". European Conference on Computer Vision (ECCV) 2014 (PDF).
- J.L. Schönberger & J.M. Frahm (2016). "Structure-from-Motion Revisited" (PDF). IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
- C. Tomasi & T. Kanade (1992). "Shape and motion from image streams under orthography: a factorization method". International Journal of Computer Vision. 9 (2): 137–154. CiteSeerX 10.1.1.131.9807. doi:10.1007/BF00129684.
- V.M. Govindu (2001). "Combining two-view constraints for motion estimation". IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
- Westoby, M. J.; Brasington, J.; Glasser, N. F.; Hambrey, M. J.; Reynolds, J. M. (2012-12-15). "'Structure-from-Motion' photogrammetry: A low-cost, effective tool for geoscience applications". Geomorphology. 179: 300–314. doi:10.1016/j.geomorph.2012.08.021.
- James, M. R.; Robson, S. (2012-09-01). "Straightforward reconstruction of 3D surfaces and topography with a camera: Accuracy and geoscience application". Journal of Geophysical Research: Earth Surface. 117 (F3): F03017. doi:10.1029/2011jf002289. ISSN 2156-2202.
- Fonstad, Mark A.; Dietrich, James T.; Courville, Brittany C.; Jensen, Jennifer L.; Carbonneau, Patrice E. (2013-03-30). "Topographic structure from motion: a new development in photogrammetric measurement" (PDF). Earth Surface Processes and Landforms. 38 (4): 421–430. doi:10.1002/esp.3366. ISSN 1096-9837.
- Javernick, L.; Brasington, J.; Caruso, B. (2014). "Modeling the topography of shallow braided rivers using Structure-from-Motion photogrammetry". Geomorphology. 213: 166–182. doi:10.1016/j.geomorph.2014.01.006.
- Smith, Mark William; Vericat, Damià (2015-09-30). "From experimental plots to experimental landscapes: topography, erosion and deposition in sub-humid badlands from Structure-from-Motion photogrammetry" (PDF). Earth Surface Processes and Landforms. 40 (12): 1656–1671. doi:10.1002/esp.3747. ISSN 1096-9837.
- Goldstein, Evan B; Oliver, Amber R; deVries, Elsemarie; Moore, Laura J; Jass, Theo (2015-10-22). "Ground control point requirements for structure-from-motion derived topography in low-slope coastal environments". PeerJ PrePrints. doi:10.7287/peerj.preprints.1444v1. ISSN 2167-9843.
- Mancini, Francesco; Dubbini, Marco; Gattelli, Mario; Stecchi, Francesco; Fabbri, Stefano; Gabbianelli, Giovanni (2013-12-09). "Using Unmanned Aerial Vehicles (UAV) for High-Resolution Reconstruction of Topography: The Structure from Motion Approach on Coastal Environments". Remote Sensing. 5 (12): 6880–6898. doi:10.3390/rs5126880.
- Johnson, Kendra; Nissen, Edwin; Saripalli, Srikanth; Arrowsmith, J. Ramón; McGarey, Patrick; Scharer, Katherine; Williams, Patrick; Blisniuk, Kimberly (2014-10-01). "Rapid mapping of ultrafine fault zone topography with structure from motion". Geosphere. 10 (5): 969–986. doi:10.1130/GES01017.1.
- Del Soldato, M.; Riquelme, A.; Bianchini, S.; Tomàs, R.; Di Martire, D.; De Vita, P.; Moretti, S.; Calcaterra, D. (2018-06-06). "Multisource data integration to investigate one century of evolution for the Agnone landslide (Molise, southern Italy)". Landslides. 15 (11): 2113–2128. doi:10.1007/s10346-018-1015-z. ISSN 1612-510X.
- Bryson, Mitch; Duce, Stephanie; Harris, Dan; Webster, Jody M.; Thompson, Alisha; Vila-Concejo, Ana; Williams, Stefan B. (2016). "Geomorphic changes of a coral shingle cay measured using Kite Aerial Photography". Geomorphology. 270: 1–8. doi:10.1016/j.geomorph.2016.06.018.
- Riquelme, A.; Cano, M.; Tomás, R.; Abellán, A. (2017-01-01). "Identification of Rock Slope Discontinuity Sets from Laser Scanner and Photogrammetric Point Clouds: A Comparative Analysis". Procedia Engineering. 191: 838–845. doi:10.1016/j.proeng.2017.05.251. ISSN 1877-7058.
- Jordá Bordehore, Luis; Riquelme, Adrian; Cano, Miguel; Tomás, Roberto (2017-09-01). "Comparing manual and remote sensing field discontinuity collection used in kinematic stability assessment of failed rock slopes". International Journal of Rock Mechanics and Mining Sciences. 97: 24–32. doi:10.1016/j.ijrmms.2017.06.004. ISSN 1365-1609.
- Guidi. G.; Beraldin, J.A.; Atzeni, C. High accuracy 3D modelling of cultural heritage: The digitizing of Donatello. IEEE Trans. Image Process. 2004, 13, 370–380
- Kraus, K., 2007. Photogrammetry: Geometry from Image and Laser Scans. Walter de Gruyter, 459 pp. ISBN 978-3-11-019007-6
Structure from motion software toolboxes
Open source solutions
- Trying all the free Photogrammetry-Tools
- Bundler – Structure from Motion for Unordered Photo Collections by Noah Snavely
- openMVG An Open Multiple View Geometry library + Structure from Motion demonstrators
- openMVS Multi-View Reconstruction Software
- Libmv – A Structure from Motion library
- Theia: A Fast and scalable structure-from-motion library released under the BSD license
- COLMAP: General-purpose Structure-from-Motion pipeline with a graphical and command-line interface, licensed under the GPL.
- MicMac, a SFM open-source code released by the Institut national de l'information géographique et forestière
- LSD-SLAM: Large-Scale Direct Monocular SLAM in real-time, by Jakob Engel
- MVE – The Multi-View Environment by Simon Fuhrmann, TU Darmstadt.
- ceres-solver for general non-linear least squares. Has features for bundle adjustment. Previously used by Google internally for google maps. Released to the public in 2012.
- SBA for generic bundle adjustment by Manolis Lourakis.
- SFMToolkit a complete photogrammetry solution based on open-source software
- AliceVision an open-source (MPLv2) Photogrammetric Computer Vision Framework (SfM+MVS)
- Structure from Motion toolbox for Matlab by Vincent Rabaud
- Matlab Functions for Multiple View Geometry by Andrew Zissermann
- Structure and Motion Toolkit by Phil Torr
- Matlab Code for Non-Rigid Structure from Motion by Lorenzo Torresani
- Python Photogrammetry Toolbox GUI – an open-source SFM GUI (Easy SfM and dense point cloud estimation launcher) by Pierre Moulon and Arc-Team
- OpenSfM, a Structure from Motion library written in Python on top of OpenCV, used by Mapillary, Simplified BSD License.
- Catena Python Abstract Workflow Framework with SfM components.
Closed source software
- Agisoft Photoscan, a photogrammetry solution by Agisoft integrating SfM and ground control. Supports editing/classifying, batch operations, Python scripting, and import/export other formats.
- RealityCapture, a photogrammetry solution by Capturing Reality that can process images and/or laser-scans automatically. It can also export SFM to bundle.out and other formats.
- Smart3DCapture, a complete photogrammetry solution by Acute3D.
- 3DF Samantha – Command line structure from Motion pipeline for Windows, by 3Dflow srl. Free for non-commercial purposes.
- 3DF Zephyr, a photogrammetry solution by 3Dflow srl, based on 3DF Samantha.
- Automatic Camera Tracking System (ACTS), a structure-from-motion with dense depth recovery system for Microsoft Windows, by Vision Group of State Key Lab of CAD&CG, Zhejiang University.
- Large-Scale Automatic Camera Tracking System (LS-ACTS), a large-scale structure-from-motion system for Microsoft Windows, by Vision Group of State Key Lab of CAD&CG, Zhejiang University.
- VisualSFM: A Visual Structure from Motion System, by Changchang Wu
- MountainsMap SEM software for Scanning Electron Microscopes. 3D is obtained by tilting the specimen + photogrammetry.
- Voodoo Camera Tracker, non-commercial tool for the integration of virtual and real scenes.
Original site, archived: Laboratorium für Informationstechnologie, University of Hannover
- MetaIO Toolbox SfM for augmented reality on mobile devices.
- TacitView by 2d3 Sensing
- CMPMVS Multi-View Reconstruction Software
- University of Stuttgart's LibTSgm library