This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.

/pdf/a-survey-of-augmented-reality-17irtaxt7z.pdf

A survey of augmented reality

We present a real-time algorithm which can recover the 3D trajectory of a monocular camera, moving rapidly through a previously unknown scene. Our system, which we dub MonoSLAM, is the first successful application of the SLAM methodology from mobile robotics to the "pure vision" domain of a single uncontrolled camera, achieving real time but drift-free performance inaccessible to structure from motion approaches. The core of the approach is the online creation of a sparse but persistent map of natural landmarks within a probabilistic framework. Our key novel contributions include an active approach to mapping and measurement, the use of a general motion model for smooth camera movement, and solutions for monocular feature initialization and feature orientation estimation. Together, these add up to an extremely efficient and robust algorithm which runs at 30 Hz with standard PC and camera hardware. This work extends the range of robotic systems in which SLAM can be usefully applied, but also opens up new areas. We present applications of MonoSLAM to real-time 3D localization and mapping for a high-performance full-size humanoid robot and live augmented reality with a hand-held camera

/pdf/monoslam-real-time-single-camera-slam-ydmj8dc1vj.pdf

MonoSLAM: Real-Time Single Camera SLAM

In 1997, Azuma published a survey on augmented reality (AR). Our goal is to complement, rather than replace, the original survey by presenting representative examples of the new advances. We refer one to the original survey for descriptions of potential applications (such as medical visualization, maintenance and repair of complex equipment, annotation, and path planning); summaries of AR system characteristics (such as the advantages and disadvantages of optical and video approaches to blending virtual and real, problems in display focus and contrast, and system portability); and an introduction to the crucial problem of registration, including sources of registration error and error-reduction strategies.

https://www.cc.gatech.edu/~blair/papers/ARsurveyCGA.pdf

Recent advances in augmented reality

Ego-motion estimation for an agile single camera moving through general, unknown scenes becomes a much more challenging problem when real-time performance is required rather than under the off-line processing conditions under which most successful structure from motion work has been achieved. This task of estimating camera motion from measurements of a continuously expanding set of self-mapped visual features is one of a class of problems known as Simultaneous Localisation and Mapping (SLAM) in the robotics community, and we argue that such real-time mapping research, despite rarely being camera-based, is more relevant here than off-line structure from motion methods due to the more fundamental emphasis placed on propagation of uncertainty. We present a top-down Bayesian framework for single-camera localisation via mapping of a sparse set of natural features using motion modelling and an information-guided active measurement strategy, in particular addressing the difficult issue of real-time feature initialisation via a factored sampling approach. Real-time handling of uncertainty permits robust localisation via the creating and active measurement of a sparse map of landmarks such that regions can be re-visited after periods of neglect and localisation can continue through periods when few features are visible. Results are presented of real-time localisation for a hand-waved camera with very sparse prior scene knowledge and all processing carried out on a desktop PC.

/pdf/real-time-simultaneous-localisation-and-mapping-with-a-15tf3exidx.pdf

Real-time simultaneous localisation and mapping with a single camera

This paper presents a novel orientation algorithm designed to support a computationally efficient, wearable inertial human motion tracking system for rehabilitation applications. It is applicable to inertial measurement units (IMUs) consisting of tri-axis gyroscopes and accelerometers, and magnetic angular rate and gravity (MARG) sensor arrays that also include tri-axis magnetometers. The MARG implementation incorporates magnetic distortion compensation. The algorithm uses a quaternion representation, allowing accelerometer and magnetometer data to be used in an analytically derived and optimised gradient descent algorithm to compute the direction of the gyroscope measurement error as a quaternion derivative. Performance has been evaluated empirically using a commercially available orientation sensor and reference measurements of orientation obtained using an optical measurement system. Performance was also benchmarked against the propriety Kalman-based algorithm of orientation sensor. Results indicate the algorithm achieves levels of accuracy matching that of the Kalman based algorithm; < 0.8° static RMS error, < 1.7° dynamic RMS error. The implications of the low computational load and ability to operate at small sampling rates significantly reduces the hardware and power necessary for wearable inertial movement tracking, enabling the creation of lightweight, inexpensive systems capable of functioning for extended periods of time.

Estimation of IMU and MARG orientation using a gradient descent algorithm

Among other things, positioning a magnetic instrument on a pedestrian; positioning an inertial instrument on a foot of a pedestrian; receiving positioning signals at the pedestrian; aligning the inertial instrument based in part on the received positioning signals; calibrating the magnetic instrument using the inertial instrument; and tracking the pedestrian using the calibrated magnetic instrument and the inertial instrument.

Indoor/outdoor pedestrian navigation

This paper describes the design, development and testing of an AR system that was developed for aerospace and ground vehicles to meet stringent accuracy and robustness requirements. The system uses an optical see-through HMD, and thus requires extremely low latency, high tracking accuracy and precision alignment and calibration of all subsystems in order to avoid mis-registration and “swim”. The paper focuses on the optical/inertial hybrid tracking system and describes novel solutions to the challenges with the optics, algorithms, synchronization, and alignment with the vehicle and HMD systems. Tracker accuracy is presented with simulation results to predict the registration accuracy. A car test is used to create a through-the-eyepiece video demonstrating well-registered augmentations of the road and nearby structures while driving. Finally, a detailed covariance analysis of AR registration error is derived.

Design and Error Analysis of a Vehicular AR System with Auto-Harmonization

This paper introduces a new set of tracking devices for 3D virtual environment interaction and reviews enhancements including device miniaturization, power reduction, and ruggedization. The goal of these improvements was to reduce size and weight of head-worn tracking devices for use with passive-stereo immersive display systems, whose polarized glasses are much smaller than LCD shutter glasses and therefore require smaller tracking sensors. The major changes include a 2-fold reduction in the size and weight of the wearable sensor devices and improvement of wireless tracking capability and battery life.

Improved 3D Interactive Devices for Passive and Active Stereo Virtual Environments

Technology described in this document can be embodied in an optical sensor that includes a rectangular array of at least four photodetector cells of substantially equal size. An opaque mask is affixed over the rectangular array. The opaque mask defines a substantially rectangular aperture for admitting light onto only a portion of the surface of each of the at least four photodetector cells, where the aperture can be centrally positioned over the rectangular array.

Sensing Direction and Distance

Aspects of the present invention relate to systems, methods, and computer program products for tracking an orientation of a first object. The system includes a light emitting device located relative to a second object at a fixed predetermined position; a sensor having a photodetector array that is configured to receive incident light emitted from the light emitting device, the photodetector array being mounted on the first object; and a processor coupled to the photodetector array, the processor configured to determine the orientation of the first object relative to the second object based on an angle of incident light detected by the photodetector array from the light emitting device.

Eric Foxlin

Papers

Indoor/outdoor pedestrian navigation

Design and Error Analysis of a Vehicular AR System with Auto-Harmonization

Improved 3D Interactive Devices for Passive and Active Stereo Virtual Environments

Sensing Direction and Distance

System for light source location detection