next up previous
Next: Summary and Future Work Up: Searching for Images and Previous: Content-based Techniques

Evaluation

 

In the initial trials, the system has catalogued 513,323 images and videos from 46,551 directories on 16,773 distinct Web sites. The process required several months, which was performed simultaneously with the development of the user application. Various information about the catalog process is summarized in Table 4. In all the system has catalogued over 129 Gigabytes of visual information. The local storage of information, which includes coarse versions of the data and color histogram feature vectors, requires approximately 2 Gigabytes.

 

tex2html_wrap1470
number of images/videos catalogued 513,323
number of Web sites providing the images and videos 16,773
number of distinct Web directories 46,551
% of catalog is black & white/gray-scale 14.15%
% of catalog is videos 1.05%
% of images and videos classified into subject classes 68.23%
size of subject taxonomy (# classes) 941
size of key-term dictionary (# terms) 932
number of directory name to subject mappings 1018
Table 4:   Cataloging of 513,323 images and videos from the Web, (a) number of images/videos per Web site, (b) results.

(a)(b)

Subject Classification Evaluation

  

As indicated in Table 4, the catalog process assigned 68.23% of the images and videos into subject classes using automated mapping for key-terms and semi-automated mapping for directory names. We assessed the subject classification rates for several classes, which is summarized in Table 5(a). The overall performance is excellent, tex2html_wrap_inline1474 classification precision. For this assessment, as illustrated in Table 5(a), we chose the classes at random from the subject taxonomy of 941 classes. We established the ground-truth by manually verifying the subject of each image and video in the test sets.

We observed that errors in classification result from several occurrences: (1) key-terms being used out of context by the publishers of the images or videos, (2) the system's reliance on some key-terms that have multiple meanings and contexts, i.e., ``madonna'' and (3) the system's reliance on key-terms extracted from directory names. For example, in Table 5(a), the precision of subject class ``animals/possums'' is low because five out of the nine items are not images or videos of possums. These items were classified incorrectly because the key-term ``possum'' appeared in the directory name. While some of the images in that directory depict possums, others depict only the forests to which the possum are indigenous. When viewed outside of the context of the ``possum'' web site, the images of forests should not be assigned to the class ``animals/possums.''

Type Classification Evaluation

  

We assessed the precision of the automated type classification system, which is summarized in Table 5(b). For this evaluation, both the Training and Test samples consisted of 200 images from each type class. We found the automated type assessment for these five simple classes is quite satisfactory, overall tex2html_wrap_inline1480 rate of successful classification. In future work, we will try to extend this system to include a larger number of classes, including new type classes, such as Fractal images, Cartoons, Faces, Art paintings and subject classes.

 

Subject # sites Count Rate
art/illustrations 29 1410 0.978
entertainment/humour/cartoons/daffyduck 14 23 1.000
animals/possums 2 9 0.444
science/chemistry/proteins 7 40 1.000
nature/weather/snow/frosty 9 13 1.000
food 403 2460 0.833
art/paintings/pissarro 3 54 1.000
entertainment/music/mtv 15 87 0.989
horror 366 2454 0.968
Table 5:   Rates of correct (a) Subject classification (precision) for random set of classes and (b) automated type classification.
Type Rate
Color photo 0.914
Color graphic 0.923
Gray image 0.967
B/w image 1.000

(a)(b)

Efficiency

 

Another important factor in the image and video search system is the speed at which user operations and queries are performed. In particular, as the archive grows it is imperative that queries do not take so long that they inhibit the user from effectively using the system. In the initial system, the overall efficiency of various database manipulation operations is excellent, even on the large catalog, see Table 6 (server platform = SGI Onyx). In particular, the good performance of the content-based visual query tools is given by the strategies of indexing the 166 bin color histograms described in Section 5.1. For example, the system identifies the tex2html_wrap_inline1306 most similar visual scenes in the catalog of 513,323 images and videos to a selected query scene in only 1.83 seconds.

 

Operation Time (secs)
Direct subject selection 0.09
Subject class name query 0.18
Text query 0.30
Search results lists manipulation 0.19 to 1.29
Color histogram query (catalog-wide) 1.83
Color histogram query (list specific) 1.21
Relevance Feedback query 1.93
One-way communication of query results and display5 to 15
Table 6:   Execution times of various search, browse and query operations.


next up previous
Next: Summary and Future Work Up: Searching for Images and Previous: Content-based Techniques

John Smith
Fri Aug 16 11:09:46 EDT 1996