A large number of catalogs and search engines index the plethora of documents on the World-Wide Web. For example, recent systems, such as Lycos, Alta Vista and Yahoo, index the documents by their textual content. These systems periodically scour the Web, record the text on each page and through processes of automated analysis and/or (semi-) automated classification, condense the Web into compact and searchable indexes. The user, by entering query terms and/or by selecting subjects, uses these search engines to more easily find the desired Web documents. Generally, the text-based Web search engines are evaluated on the basis of the size of the catalog, speed and effectiveness of search and ease of use [1].
However, no tools are currently available for searching for images and videos. This absence is particularly notable given the highly visual and graphical nature of the Web [2]. Visual information is published both as embedded in Web documents and as stand-alone objects. The visual information takes the form of images, graphics, bitmaps, animations and videos. As with Web documents in general, the publication of visual information is highly volatile. New images and videos are added everyday and others are replaced or removed entirely. In order to catalog the visual information, a highly efficient automated system is needed that regularly traverses the Web, detects visual information and processes it in such a way to allow for efficient and effective search and retrieval.
We recently developed a prototype visual information retrieval system
to fulfill this need. The system collects images and videos from the Web and provides tools for browsing and searching through the collection. The system is novel in that it utilizes text and visual information synergistically to provide for cataloging and searching for the images and videos. The complete system possesses several powerful functionalities, namely,
Content-based technologies have enabled recent advances in the management, search and retrieval of visual information. In particular, content-based techniques provide for the automated assessment of salient visual features such as colors, textures, shapes and spatial information contained within visual scenes. By computing the similarities between images and videos using these extracted visual features, several powerful functionalities are added to the system, which allow
Content-based tools may also be used to improve the query process by learning from the user. In one form of relevance feedback, the user selects an item from the list returned from a query and asks the system to retrieve more that are similar to it in some specified way. The similarity may be based upon visual features, such as colors, textures, spatial layout, or be based upon text or subject classifications. In a second form of relevance feedback [6], the user selects the images and videos from the current results that are most and least typical of the ones desired. From these positive and negative examples, the system automatically reformulates the query to better match the user's instruction. In this sense, the system learns from the user what visual information is desired and converges to it through successive rounds of queries.
The images and videos are catalogued using a series of automated agents that traverse the Web detecting visual data. The system builds several indexes for the images and videos based upon
Image and video subject and type information is derived from both the textual information - such as the Web address and parent document reference text - and visual features. Because the recent taxonomies of knowledge are not well suited for visual information, we developed a new working taxonomy for image and video subject matter. The taxonomy is based upon a graph structure and contains classes for sports, nature, transportation, art and so forth. The images and videos are classified into subjects using several fully- and semi- automated procedures. In the first process, we utilize a key-term dictionary which prescribes subject classes to the images and videos based upon detection of certain key-terms. In the second process, the directory portion of the image and video Web addresses is parsed and analyzed such that groups of images and videos are flagged for inspection and manual classification.
After receiving a list of images and videos in response to a query, the user may manipulate the search results list by adding and removing items using operations with previous or subsequent search results lists. By tracking the user's queries and search results, the system allows for a large variety of ways for the user to refine queries and browse. The system allows the user to concatenate, subtract and intersect search results lists. Search results list manipulation is of special advantage for visual information because the user may also manipulate and search for images and videos using visual features.
In this paper we describe the complete system for cataloging and searching for images and videos on the Web. In section 2, we describe in detail the process for automated collection of the visual information. In section 3, we describe the procedures for classifying the collected images and videos using key-term mappings and directory names. We also present and utilize a new taxonomy for visual information. In section 4, we describe the system for navigating through subject classes, searching, viewing query results and manipulating the search results lists. In section 5, we describe several content-based tools for searching, browsing and revising queries. In particular, we describe the system's utilization of color histograms for the content-based manipulation of images and videos. Finally, in section 6, we provide an initial evaluation of the system in the collected of more than one half million images and videos belonging to 16,773 sites on the World-Wide Web.