Last update: 2011/08/21
A comprehensive framework to deal with interactions between real and visual worlds
Digital Media started with the digitisation of audio, video and images but more media types have been added: characters and font information, interactive 3D graphics etc. The last media type, in particular, has led to very significant examples of new businesses. An example is Second Life where people can create their own alter ego and create/buy/sell assets.
Digital media content may also need to stimulate senses other than sight and hearing involved in traditional audio-visual media, e.g. olfaction, mechanoreception, equilibrioception, or thermoception. These may act as additions to the audio-visual content (movies, games etc.), by involving other senses that enhance the feeling of being part of the media content that creates a user experience that is expected to have a greater value for the user.
The “Media context and control” standard, nicknamed MPEG-V [11], provides an architecture, illustrated in Fig. 5, and the digital representation of a wide range of data types to enable interoperability between virtual worlds – e.g., a virtual world set up by a provider of serious games and simulations – with another virtual world. Of course both virtual worlds may have interactions with the real world, via sensors and actuators.
Fig. 1 – A general model of real and virtual world interaction
In the MPEG-V model described by Fig. 1 Ultimately, the virtual world at the left hand side receives Sensor Effects that have been generated by some sensor devices and generates Sensory Effects that eventually control some sensory devices (actuators). In general, however, the format of the information entering or leaving the virtual world is different from the one used by the sensory and sensor devices, respectively. For example, the virtual world may wish to communicate a “cold” feeling but the sensory devices in the real world may be widely different. The user can specify how Sensory Effects should be mapped to Device Commands and Devices can represent their capabilities: for instance, the room temperature cannot be changed, but a fan can be activated.
Fig. 2 provides a specific example illustrating some technologies specified in MPEG-V Part 3 “Sensory Information”, namely 1) the Sensory Effect Description Language (SEDL) enabling the description of “sensory effects” (e.g. light, wind, fog, vibration, etc.) that act on human senses, 2) the Sensory Effect Vocabulary (SEV) defining the actual sensory effects and allowing for extensibility and flexibility and 3) Sensory Effect Metadata (SEM), a specific SEDL description that may be associated to any kind of multimedia content to drive sensory devices (e.g. fans, vibration chairs, lamps, etc.).
Fig. 2 – Use of MPEG-V SEDL for augmented user experience
