How your phone recognizes your home: An investigation of mobile object recognition

Thanh Vu, Daniel Piros, Amir Sadovnik

Often what is effortless for a human brain challenges machines the most. Visual recognition, a fairly easy task for humans, can be surprisingly difficult for machines due to variations in angle, size, and lighting. The challenge is amplified on mobile platforms because of computational constraints. There has been a number of studies on image recognition, but few focus on algorithms that run completely on portable devices. With this work, we present an improved image retrieval method that can run on mobile devices in real time without the need to access a remote server, targeting building and poster recognition. One of its possible applications is an electronic tour guide, where users instantly gain detailed information on buildings or posters by taking pictures of them with their phones or notepads. In this work, we designed a fast and robust image matching technique using binary object descriptors. First, since no well-structured database was publicly available, we built new datasets comprised of hundreds of photographs of college buildings and academic posters, taken from combinations of distances and angles. These datasets will be made publicly available. Then, speed and accuracy of different known keypoint detectors and descriptors were studied to select the best one. Finally, we further optimized the results by filtering best matches, exploiting user location, and extending a grayscale descriptor to include colors. The experiment was done on a real mobile device using a tour guide Android application that we developed implementing the algorithm. The program successfully matched various objects to locally stored sample images with an improved accuracy of 98.5% in less than a second. The extended color descriptor in particular boosted efficiency for poster recognition significantly. Although in a campus context we focused on buildings and posters, our programs could potentially be expanded for use of general object recognition.

    

OpenCVTour is an Android app for creating and following tours using OpenCV image recognition. It performs a role similar to traditional museum audio tour guides, but scans the actual object instead of a QR code.

Code for OpenCVTour. A visual audio tour app for Android:
https://github.com/WriterOfAlicrow/OpenCVTour