Abstract

Media agents are a specific class of agents which perform indexing, feature extraction, retrieval tasks, playback control and visual representation tasks on audiovisual media (video, music, speech and sound recordings, photographs). An example of a media physical agent could be for instance, a software agent embodied in a stereo and able to extract a simplified score from a musical recording while it is playing it to a human listener. Media agents receive as input data flows representing documents in a given media (and requests from other agents, and output data extracted from these documents to information fusion and user interface agents). The following proposal aims at characterizing media agents and their specific needs.

Rationale: why is a specific media agent category necessary?

A document in a given medium is very different from a raw sequence or flow of visual or auditory data: it has structure, human intention in its organization and contents. It is in many cases much easier to extract meaningful information from a document in a medium than it is to extract it from, let's say, the flow of images carried by a visio-conferencing system or tele-watching camera. Providing interfaces between media carriers and repositories and human users who access documents or produce new documents is our application domain. If we want the numerous methods that have been designed by researchers for media analysis to be exploitable in agent technology-based systems, the specific needs of such types of agents need to be taken in account. Such specific needs are for instance:

• real-time or pseudo-real time analysis on coded or compressed data flows (media agents are highly computational and highly dependent on coding standards and technology),
• specific relation with fusion of information (for instance combining soundtrack and image analysis) and synchronization agents (for instance agents synchronizing playback of a document with visual display of its user interface representation or with a synthesis process).

Further illustration of the specific needs of media agents can be illustrated by listing capabilities that should be present in media physical agents (not all of them in a given agent of course):

• visual input primitives, but adapted to the specific input (human-produced video documents for instance),
• sound input primitives including music and noise analysis (not only speech),
• visual output primitives (visual representation of medium contents),
• fusion of information (see above),
• medium model and representation (could fit under {\em semantic information representation} if the definition of this capability was modified),

As a consequence of these specific needs we propose introducing a new category 1.2.d. Interacting with Media. To fully justify this, one needs to accept the fact that media are part of the physical world, which is the case since document contents are produced by human agents for other human agents to perceive them and media agents add intelligence to this relation.

A tentative list of media agents analysis task capabilities

The following list gives an idea of the type of intelligent analysis tasks than media agents are able to perform. We can provide evidence for the feasibility of these tasks using present-day or reasonably attainable methods and technologies:

• shot change and transition effect detection in video,
• camera-work detection and micro-segmentation of video,
• video sequence global structure extraction,
• shot framing and focus analysis in video shots,
• object and character motion analysis in video,
• speech, music and natural or Foley sound detection in video sound tracks and sound media (including detecting their combinations),
• locutor labelling in video sound and sound recordings,
• word labelling of sound tracks when transcripts are available,
• micro-segmentation of music,
• approximate polyphonic pitch extraction,
• rythmical, dynamical and metrical analysis of music.

Philippe Aigrain, Philippe Joly and Philippe Lepain

Media Analysis and Interaction Research Group

Institut de Recherche en Informatique de Toulouse

Universite Paul Sabatier

118, route de Narbonne

F-31062 Toulouse Cedex, France

e-mail: {aigrain,joly,lepain}@irit.fr}