Large Scale Visual Recognition

Abstract

Visual recognition remains one of the grand goals of artificial intelligence research. One major challenge is endowing machines with human ability to recognize tens of thousands of categories. Moving beyond previous work that is mostly focused on hundreds of categories, we make progress toward human scale visual recognition. Specifically, our contributions are as follows:

First, we have constructed "ImageNet," a large scale image ontology. The Fall 2011 version consists of 22 thousand categories and 14 million images; it depicts each category by an average of 650 images collected from the Internet and verified by multiple humans. To the best of our knowledge this is currently the largest human-verified dataset in terms of both the number of categories and the number of images. Given the large amount of human effort required, the traditional approach to dataset collection, involving in-house annotation by a small number of human subjects, becomes infeasible. In this dissertation we describe how ImageNet has been built through quality controlled, cost effective, large scale online crowdsourcing.

Next, we use ImageNet to conduct the first benchmarking study of state of the art recognition algorithms at the human scale. By experimenting on 10 thousand categories, we discover that the previous state of the art performance is still low (6.4%). We further observe that the confusion among categories is hierarchically structured at large scale, a key insight that leads to our subsequent contributions.

Third, we study how to efficiently classify tens of thousands of categories by exploiting the structure of visual confusion among categories. We propose a novel learning technique that scales logarithmically with the number of classes in both training and testing, improving both accuracy and efficiency of the previous state of the art while reducing training time by 31 fold on 10 thousand classes.

Fourth, we consider the problem of retrieving semantically similar images from a large database, a problem closely related to classification. We propose an indexing approach that exploits the hierarchical structure between categories. Experiments demonstrate that our approach is more efficient, scalable, and accurate than previous work. In particular, our indexing technique achieves close to 90% of the accuracy of brute force with a 1,000 times speedup.

Finally, further exploiting the hierarchy, we show how to select the appropriate level of specificity to guarantee an arbitrary classification accuracy. We propose an algorithm that is provably optimal under mild conditions and demonstrate its effectiveness on classifying 10 thousand classes. Experiments show that our algorithm guarantees a 90% accuracy while giving informative answers 83% of the time. This holds promise toward a practical large scale recognition system.