1
Introduction

1.2.1 Digitized Audio

interMedia audio services integrate the storage, retrieval, and management of digitized audio data in Oracle databases using Oracle9i.

Audio may be produced by an audio recorder, an audio source such as a microphone, digitized audio, other specialized audio recording devices, or even by program algorithms. Audio recording devices take an analog or continuous signal, such as the sound picked up by a microphone or sound recorded on magnetic media, and convert it into digital values with specific audio characteristics. Such characteristics include format, encoding type, number of channels, sampling rate, sample size, compression type, and audio duration.

1.2.2 Audio Components

Digitized audio consists of the audio data (digitized bits) and attributes that describe and characterize the audio data. Audio applications sometimes associate application-specific information, such as the description of the audio clip, date recorded, author or artist, and so forth, with audio data by storing descriptive text in an attribute or column in the database table.

The audio data can have different formats, encoding types, compression types, numbers of channels, sampling rates, sample sizes, and playing times (duration) depending upon how the audio data was digitally recorded. interMedia audio services can store and retrieve audio data of any data format, and automatically extract metadata from audio data of a variety of popular audio formats. interMedia audio services can also extract application attributes and store them in the comments field of the object in XML form, identical to what is provided by Oracle interMedia Annotator. Supported audio attributes depend upon available hardware capabilities or processing power for any user-defined formats. See Oracle interMedia User's Guide and Reference for a list of supported data formats from which interMedia audio services can extract and store attributes and other audio features.

interMedia audio services are extensible and can support additional audio formats.

The size of digitized audio (number of bytes) tends to be large compared to traditional computer objects, such as numbers and text. Therefore, several encoding schemes are used that squeeze audio data into fewer bytes, thus putting a smaller load on storage devices and networks.

1.3.1 Digitized Media Data

interMedia media services integrate the storage, retrieval, and management of media files in Oracle databases using Oracle9i.

An interMedia OrdDoc object can store audio, image, and video data in a single database column. Instead of having separate columns for audio, image, and video objects, you can use one column of interMedia OrdDoc objects to represent all types of media.

1.3.2 Media Components

Media files consist of the media data (digitized bits) and attributes that describe and characterize the data.

interMedia OrdDoc objects can store and retrieve media data of any data format. The OrdDoc object type can be used in applications that require you to store different types of media data (such as audio, image, video, and any other kind of media file) in the same column, so you can build a common metadata index on all the different types of media files. Using this index, you can search across all the different types of files. Note that you cannot use this same search technique if the different types of media data are stored in different types of objects, in separate columns of relational tables.

interMedia can automatically extract metadata from a variety of popular audio, image, and video data formats. interMedia can also extract application attributes and store them in the comments attribute of the object, in XML form. See Appendix A of Oracle interMedia User's Guide and Reference for a list of supported data formats from which interMedia can extract and store attributes and other features. interMedia is extensible and can be made to recognize and support additional formats.

1.4.1 Digitized Images

interMedia image services integrate the storage, retrieval, and management of digitized images in Oracle databases using Oracle9i.

interMedia image services support two-dimensional, static, digitized raster images stored as binary representations of real-world objects or scenes. Images may be produced by a document or photograph scanner, a video source such as a camera or VCR connected to a video digitizer or frame grabber, other specialized image capture devices, or even by program algorithms. Capture devices take an analog or continuous signal such as the light that falls onto the film in a camera, and convert it into digital values on a two-dimensional grid of data points known as pixels. Devices involved in the capture and display of images are under application control.

1.4.2 Image Components

Digitized images consist of the image data (digitized bits) and attributes that describe and characterize the image data. Image applications sometimes associate application-specific information, such as including the name of the person pictured in a photograph, description of the image, date photographed, photographer, and so forth, with image data by storing this descriptive text in an attribute or column in the database table.

The image data (pixels) can have varying depths (bits per pixel) depending on how the image was captured, and can be organized in various ways. The organization of the image data is known as the data format.

interMedia image services can store or retrieve image data of any data format. interMedia image services can process and automatically extract properties of images of a variety of popular formats. See Oracle interMedia User's Guide and Reference for a list of supported data formats for which interMedia image services can process and extract metadata. In addition, certain foreign images (formats not natively understood by interMedia image services) have limited support for image processing.

The storage space required for digitized images can be large compared to traditional attribute data such as numbers and text. Many compression schemes are available to squeeze an image into fewer bytes, thus reducing storage device and network load. Lossless compression schemes squeeze an image so that when it is decompressed, the resulting image is bit-for-bit identical with the original. Lossy compression schemes do not result in an identical image when decompressed, but rather, one in which the changes may be imperceptible to the human eye.

Image interchange format describes a well-defined organization and use of image attributes, data, and often compression schemes, allowing different applications to create, exchange, and use images. Interchange formats are often stored in or as disk files. They may also be exchanged in a sequential fashion over a network and be referred to as a protocol. There are many application subdomains within the digitized imaging world and many applications that create or use digitized images within these. interMedia image services support storage and retrieval of all formats, as well as processing and attribute extraction of many of those formats.

1.4.3 Content-Based Retrieval Concepts

Inexpensive image-capture and storage technologies have allowed massive collections of digital images to be created. However, as a database grows, the difficulty of finding relevant images increases. Two general approaches to this problem have been developed, both of which use metadata for image retrieval:

• Using information manually entered or included in the table design, such as titles, descriptive keywords from a limited vocabulary, and predetermined classification schemes

• Using automated image feature extraction and object recognition to classify image content -- that is, using capabilities unique to content-based retrieval

With interMedia image services, you can combine both approaches in designing a table to accommodate images: use traditional text columns to describe the semantic significance of the image (for example, that the pictured automobile won a particular award, or that its engine has six or eight cylinders), and use the OrdImage object type for the image, to permit content-based queries based on intrinsic attributes of the image (for example, how closely its color and shape match a picture of a specific automobile).

As an alternative to defining image-related attributes in columns separate from the image, a database designer could create a specialized composite data type that combines interMedia image services and the appropriate text, numeric, and date attributes.

The primary benefit of using content-based retrieval is reduced time and effort required to obtain image-based information. With frequent adding and updating of images in massive databases, it is often not practical to require manual entry of all attributes that might be needed for queries, and content-based retrieval provides increased flexibility and practical value. It is also useful in providing the ability to query on attributes such as texture or shape that are difficult to represent using keywords.

Examples of database applications where content-based retrieval is useful -- where the query is semantically of the form "find objects that look like this one" -- include the following:

• Trademarks, copyrights, and logos

• Art galleries and museums

• Retailing

• Fashion and fabric design

• Interior design or decorating

For example, a Web-based interface to a retail clothing catalog might allow users to search by traditional categories (such as style or price range) and also by image properties (such as color or texture). Thus, a user might ask for formal shirts in a particular price range that are off-white with pin stripes. Similarly, fashion designers could use a database with images of fabric swatches, designs, concept sketches, and finished garments to help them in their work.

1.4.4 How Content-Based Retrieval Works

A content-based retrieval system processes the information contained in image data and creates an abstraction of that content in terms of visual attributes. Any query operations deal solely with this abstraction rather than with the image itself. Thus, every image inserted into the database is analyzed, and a compact representation of its content is stored in a feature vector, or signature.

The signature contains information about the following visual attributes:

• Color represents the distribution of colors within the entire image. This distribution includes the amounts of each color, but not the locations of colors.

• Texture represents the low-level patterns and textures within the image, such as graininess or smoothness. Unlike shape, texture is very sensitive to features that appear with great frequency in the image.

• Shape represents the shapes that appear in the image, as determined by color-based segmentation techniques. A shape is characterized by a region of uniform color.

• Location represents the positions of the shapes, color, and texture components. For example, the color blue could be located in the top half of the image. A certain texture could be located in the bottom right corner of the image.

Feature data for all these visual attributes is stored in the signature, whose size typically ranges from 3000 to 4000 bytes. For better performance with large image databases, you can create an index based on the signatures of your images.

Images in the database can be retrieved by matching them with a comparison image. The comparison image can be any image inside or outside the current database, a sketch, an algorithmically generated image, and so forth.

The matching process requires that a signature be generated for the comparison image. Image matching is based on comparing the signatures. A score, which is a weighted sum of the value of each attribute, is used to determine the degree of similarity when images are compared, with a smaller difference reflecting a closer match. The weights are positive real numbers whose values reflect how sensitive the matching process for a given attribute should be to the degree of similarity or dissimilarity between two images. If the weighted sum of the differences of the visual attributes is less than or equal to the threshold value, the images match; if the weighted sum is greater than the threshold, the images do not match.

Oracle interMedia Java Classes provides APIs for the OrdImageSignature object, which is used to store and compare image signature information in a Java object.

For more information on the visual attributes, weights, similarity calculation, threshold values, and indexing, see Oracle interMedia User's Guide and Reference.

1.4.5 Input and Output Streams

As an extension to Java, Sun has provided the Java Advanced Imaging (JAI) API. JAI lets you introduce advanced image processing operations in your Java applications. With Oracle interMedia, you can read and write images stored in the database from your JAI applications.

Oracle interMedia Java Classes provides APIs for three types of streams, which let you read data from BLOBs and BFILEs and write to BLOBs in your JAI applications. These stream objects are not meant to replace the input and output stream objects provided by Sun; these objects are included to provide an interface to image data stored in BLOBs and BFILEs in OrdImage objects that can be used by JAI without loss in performance.

See Chapter 7 for more information on these stream objects.

1.5.1 Digitized Video

interMedia video services integrate the storage, retrieval, and management of digitized video data in Oracle databases using Oracle9i.

Video may be produced by a video recorder, a video camera, digitized animation video, other specialized video recording devices, or even by program algorithms. Some video recording devices take an analog or continuous signal, such as that picked up by a video camera or video recorded on magnetic media, and convert it into digital values with specific video characteristics. Such characteristics include format, encoding type, frame rate, frame size (width and height), frame resolution, video length, compression type, number of colors, and bit rate.

1.5.2 Video Components

Digitized video consists of the video data (digitized bits) and the attributes that describe and characterize the video data. Video applications sometimes associate application-specific information, such as the description of the video training tape, date recorded, instructor's name, producer's name, and so forth, with video data by storing descriptive text in an attribute or column in the database table.

The video data can have different formats, compression types, frame rates, frame sizes, frame resolutions, playing times, compression types, numbers of colors, and bit rates, depending upon how the video data was digitally recorded. interMedia video services can store and retrieve video data of any data format, and automatically extract metadata from video data of a variety of popular video formats. interMedia video services can also extract application attributes and store them in the comments field of the object, in XML form. Supported video attributes depend upon available hardware capabilities or processing power for any user-defined formats. See Oracle interMedia User's Guide and Reference for a list of supported data formats from which interMedia video services can extract and store attributes and other video features.

interMedia video services are extensible and can support additional video formats.

The size of digitized video (number of bytes) tends to be large compared to traditional computer objects, such as numbers and text. Therefore, several encoding schemes are used that squeeze video data into fewer bytes, thus putting a smaller load on storage devices and networks.