Content-based image retrieval
Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR) is the application of computer vision techniques to the image retrieval problem, that is, the problem of searching for digital images in large databases (see this survey for a recent scientific overview of the CBIR field). Content-based image retrieval is opposed to traditional concept-based approaches (see Concept-based image indexing).
"Content-based" means that the search analyzes the contents of the image rather than the metadata such as keywords, tags, or descriptions associated with the image. The term "content" in this context might refer to colors, shapes, textures, or any other information that can be derived from the image itself. CBIR is desirable because searches that rely purely on metadata are dependent on annotation quality and completeness. Having humans manually annotate images by entering keywords or metadata in a large database can be time consuming and may not capture the keywords desired to describe the image. The evaluation of the effectiveness of keyword image search is subjective and has not been well-defined. In the same regard, CBIR systems have similar challenges in defining success.
- 1 History
- 2 Technical progress
- 3 CBIR techniques
- 4 Applications
- 5 See also
- 6 References
- 7 Further reading
- 8 External links
The term "content-based image retrieval" seems to have originated in 1992 when it was used by T. Kato to describe experiments into automatic retrieval of images from a database, based on the colors and shapes present. Since then, the term has been used to describe the process of retrieving desired images from a large collection on the basis of syntactical image features. The techniques, tools, and algorithms that are used originate from fields such as statistics, pattern recognition, signal processing, and computer vision
The earliest commercial CBIR system was developed by IBM and was called QBIC (Query by Image Content).
The interest in CBIR has grown because of the limitations inherent in metadata-based systems, as well as the large range of possible uses for efficient image retrieval. Textual information about images can be easily searched using existing technology, but this requires humans to manually describe each image in the database. This can be impractical for very large databases or for images that are generated automatically, e.g. those from surveillance cameras. It is also possible to miss images that use different synonyms in their descriptions. Systems based on categorizing images in semantic classes like "cat" as a subclass of "animal" can avoid the miscategorization problem, but will require more effort by a user to find images that might be "cats", but are only classified as an "animal". Many standards have been developed to categorize images, but all still face scaling and miscategorization issues.
Initial CBIR systems were developed to search databases based on image color, texture, and shape properties. After these systems were developed, the need for user-friendly interfaces became apparent. Therefore, efforts in the CBIR field started to include human-centered design that tried to meet the needs of the user performing the search. This typically means inclusion of: query methods that may allow descriptive semantics, queries that may involve user feedback, systems that may include machine learning, and systems that may understand user satisfaction levels.
Many CBIR systems have been developed, but the problem of retrieving images on the basis of their pixel content remains largely unsolved.
Different implementations of CBIR make use of different types of user queries.
Query by example is a query technique that involves providing the CBIR system with an example image that it will then base its search upon. The underlying search algorithms may vary depending on the application, but result images should all share common elements with the provided example.
Options for providing example images to the system include:
- A preexisting image may be supplied by the user or chosen from a random set.
- The user draws a rough approximation of the image they are looking for, for example with blobs of color or general shapes.
This query technique removes the difficulties that can arise when trying to describe images with words.
Semantic retrieval starts with a user making a request like "find pictures of Abraham Lincoln". This type of open-ended task is very difficult for computers to perform - Lincoln may not always be facing the camera or in the same pose. Many CBIR systems therefore generally make use of lower-level features like texture, color, and shape. These features are either used in combination with interfaces that allow easier input of the criteria or with databases that have already been trained to match features (such as faces, fingerprints, or shape matching). However, in general, image retrieval requires human feedback in order to identify higher-level concepts.
Relevance Feedback (Human Interaction)
Combining CBIR search techniques available with the wide range of potential users and their intent can be a difficult task. An aspect of making CBIR successful relies entirely on the ability to understand the user intent. CBIR systems can make use of relevance feedback, where the user progressively refines the search results by marking images in the results as "relevant", "not relevant", or "neutral" to the search query, then repeating the search with the new information. Examples of this type of interface have been developed.
Other query methods
Other query methods include browsing for example images, navigating customized/hierarchical categories, querying by image region (rather than the entire image), querying by multiple example images, querying by visual sketch, querying by direct specification of image features, and multimodal queries (e.g. combining touch, voice, etc.)
Content comparison using image distance measures
The most common method for comparing two images in content-based image retrieval (typically an example image and an image from the database) is using an image distance measure. An image distance measure compares the similarity of two images in various dimensions such as color, texture, shape, and others. For example a distance of 0 signifies an exact match with the query, with respect to the dimensions that were considered. As one may intuitively gather, a value greater than 0 indicates various degrees of similarities between the images. Search results then can be sorted based on their distance to the queried image. Many measures of image distance (Similarity Models) have been developed.
Computing distance measures based on color similarity is achieved by computing a color histogram for each image that identifies the proportion of pixels within an image holding specific values. Examining images based on the colors they contain is one of the most widely used techniques because it can be completed without regard to image size or orientation. However, research has also attempted to segment color proportion by region and by spatial relationship among several color regions.
Texture measures look for visual patterns in images and how they are spatially defined. Textures are represented by texels which are then placed into a number of sets, depending on how many textures are detected in the image. These sets not only define the texture, but also where in the image the texture is located.
Texture is a difficult concept to represent. The identification of specific textures in an image is achieved primarily by modeling texture as a two-dimensional gray level variation. The relative brightness of pairs of pixels is computed such that degree of contrast, regularity, coarseness and directionality may be estimated. The problem is in identifying patterns of co-pixel variation and associating them with particular classes of textures such as silky, or rough.
Other methods of classifying textures include:
- Co-occurrence matrix
- Laws texture energy
- Wavelet Transform
- Orthogonal Transforms (Discrete Tchebichef moments)
Shape does not refer to the shape of an image but to the shape of a particular region that is being sought out. Shapes will often be determined first applying segmentation or edge detection to an image. Other methods use shape filters to identify given shapes of an image. Shape descriptors may also need to be invariant to translation, rotation, and scale.
Some shape descriptors include:
Image Retrieval Evaluation
Potential uses for CBIR include:
- Architectural and engineering design
- Art collections
- Crime prevention
- Geographical information and remote sensing systems
- Intellectual property
- Medical diagnosis
- Photograph archives
- Retail catalogs
- Nudity-detection filters
- Face Finding
- Textiles Industry
Commercial Systems that have been developed include:
- IBM’s QBIC
- Virage’s VIR Image Engine
- Excalibur’s Image RetrievalWare
- VisualSEEk and WebSEEk
Experimental Systems include:
- MIT’s Photobook
- Columbia University’s WebSEEk
- Carnegie-Mellon University’s Informedia
- Document classification
- Image retrieval
- List of CBIR engines
- Macroglossa Visual Search
- Multimedia Information Retrieval
- Multiple-instance learning
- Nearest neighbor search
- Content-based Multimedia Information Retrieval: State of the Art and Challenges, Michael Lew, et al., ACM Transactions on Multimedia Computing, Communications, and Applications, pp. 1–19, 2006.
- Eakins, John; Graham,Margaret. "Content-based Image Retrieval". University of Northumbria at Newcastle. Retrieved 2014-03-10.
- Rui, Yong; Thomas S. Huang, Shih-Fu Chang (1999). "Image Retrieval: Current Techniques, Promising Directions, and Open Issues".
- Shapiro, Linda; George Stockman (2001). Computer Vision. Upper Saddle River, NJ: Prentice Hall. ISBN 0-13-030796-3.
- Datta, Ritendra; Dhiraj Joshi; Jia Li; James Z. Wang (2008). "Image Retrieval: Ideas, Influences, and Trends of the New Age". ACM Computing Surveys 40 (2): 1–60. doi:10.1145/1348246.1348248.
- Bird, C.L.; P.J. Elliott, Griffiths (1996). "User interfaces for content-based image retrieval".
- Cardoso, Douglas et al. "Iterative Technique for Content-Based Image Retrieval using Multiple SVM Ensembles" (PDF). Federal University of Parana(Brazil). Retrieved 2014-03-11.
- Liam M. Mayron. "Image Retrieval Using Visual Attention" (PDF). Mayron.net. Retrieved 2012-10-18.
- Eidenberger, Horst (2011). “Fundamental Media Understanding”, atpress. ISBN 978-3-8423-7917-6.
- Tamura, Hideyuki; Mori, Shunji; Yamawaki, Takashi (1978). "Textural Features Corresponding to Visual Perception". Systems, Man and Cybernetics, IEEE Transactions on 8 (6): 460, 473. doi:10.1109/tsmc.1978.4309999.
- Tushabe, F.; M.H.F. Wilkinson (2008). "Content-based Image Retrieval Using Combined 2D Attribute Pattern Spectra". Springer Lecture Notes in Computer Science.
- Deselaers, Thomas; Keysers, Daniel; Ney, Hermann (2007). "Features for Image Retrieval: An Experimental Comparison" (PDF). RWTH Aachen University. Retrieved 11 March 2014.
- Wang, James Ze; Jia Li, Gio Wiederhold and Oscar Firschein (1998). "System for Screening Objectionable Images". Computer Communications 21 (15): 1355–1360. doi:10.1016/s0140-3664(98)00203-5.
- Relevant research papers
- Query by Image and Video Content: The QBIC System, (Flickner, 1995)
- Finding Naked People (Fleck et al., 1996)
- Virage Video Engine, (Hampapur, 1997)
- Library-based Coding: a Representation for Efficient Video Compression and Retrieval, (Vasconcelos & Lippman, 1997)
- System for Screening Objectionable Images (Wang et al., 1998)
- Content-based Image Retrieval (JISC Technology Applications Programme Report 39) (Eakins & Graham 1999)
- Windsurf: Region-Based Image Retrieval Using Wavelets (Ardizzoni, Bartolini, and Patella, 1999)
- A Probabilistic Architecture for Content-based Image Retrieval, (Vasconcelos & Lippman, 2000)
- A Unifying View of Image Similarity, (Vasconcelos & Lippman, 2000)
- Next Generation Web Searches for Visual Content, (Lew, 2000)
- Image Indexing with Mixture Hierarchies, (Vasconcelos, 2001)
- SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries (Wang, Li, and Wiederhold, 2001)
- A Conceptual Approach to Web Image Retrieval (Popescu and Grefenstette, 2008)
- FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps (Ruiz-del-Solar et al., 2002)
- Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach (Li and Wang, 2003)
- Video google: A text retrieval approach to object matching in videos (Sivic & Zisserman, 2003)
- Minimum Probability of Error Image Retrieval (Vasconcelos, 2004)
- On the Efficient Evaluation of Probabilistic Similarity Functions for Image Retrieval (Vasconcelos, 2004)
- Extending image retrieval systems with a thesaurus for shapes (Hove, 2004)
- Names and Faces in the News (Berg et al., 2004)
- Cortina: a system for large-scale, content-based web image retrieval (Quack et al., 2004)
- A new perspective on Visual Information Retrieval (Eidenberger 2004)
- Language-based Querying of Image Collections on the basis of an Extensible Ontology (Town and Sinclair, 2004)
- The PIBE Personalizable Image Browsing Engine (Bartolini, Ciaccia, and Patella, 2004)
- Costume: A New Feature for Automatic Video Content Indexing (Jaffre 2005)
- Automatic Face Recognition for Film Character Retrieval in Feature-Length Films (Arandjelovic & Zisserman, 2005)
- Meaningful Image Spaces (Rouw, 2005)
- Content-based Multimedia Information Retrieval: State of the Art and Challenges (Lew et al. 2006)
- Adaptively Browsing Image Databases with PIBE (Bartolini, Ciaccia, and Patella, 2006)
- Algorithm on which Retrievr (Flickr search) and imgSeek is based on (Jacobs, Finkelstein, Salesin)
- Imagination: Exploiting Link Analysis for Accurate Image Annotation (Bartolini and Ciaccia, 2007)
- Evaluating Use of Interfaces for Visual Query Specification. (Hove, 2007)
- From Pixels to Semantic Spaces: Advances in Content-Based Image Retrieval (Vasconcelos, 2007)
- Content-based Image Retrieval by Indexing Random Subwindows with Randomized Trees (Maree et al., 2007)
- Image Retrieval: Ideas, Influences, and Trends of the New Age (Datta et al., 2008)
- Real-Time Computerized Annotation of Pictures (Li and Wang, 2008)
- Query Processing Issues in Region-based Image Databases (Bartolini, Ciaccia, and Patella, 2010)
- Shiatsu: Semantic-based Hierarchical Automatic Tagging of Videos by Segmentation Using Cuts (Bartolini, Patella, and Romani, 2010)
- Efficient and Effective Similarity-based Video Retrieval (Bartolini and Romani, 2010)
- Multi-dimensional Keyword-based Image Annotation and Search (Bartolini and Ciaccia, 2010)
- The Windsurf Library for the Efficient Retrieval of Multimedia Hierarchical Data (Bartolini, Patella, and Stromei, 2011)
- "Pl@ntNet: Interactive plant identification based on social image data" (Joly, Alexis et al.)
- Superimage: Packing Semantic-Relevant Images for Indexing and Retrieval (Luo, Zhang, Huang, Gao, Tian, 2014)
- Indexing and searching 100M images with Map-Reduce (Moise, Shestakov, Gudmundsson, and Amsaleg, 2013)