Computers Create Pictures And Sound

Most of what has been said so far concerns audio and visual information generated in the real world and reaching human senses either directly or through a more or less transparent communication system. However, an increasing proportion of what we perceive today through communication devices is no longer generated in that way. Images and sound from computers and game consoles, originally almost exclusively synthetically generated, have an increasing share of naturally generated media, while what we perceive from TV sets and movie theatre screens, originally almost exclusively naturally generated, is more and more complemented by synthetically generated audio and video.

Because of this trend, the laying out of the digital media scene in preparation for the MPEG-4 stage would not be complete without a ride on these other types of bit. Indeed, and in spite of its “moving picture” label MPEG has been operating in this space since 1995 trying to build bridges between business communities in disciplines considered specific to them. Purpose of this page is then to clarify the background of synthetically-generated pictures and sound to help understand the role that MPEG has been, is and plans to be playing in it.

From the beginning, computers were machines capable of connecting to and controlling all sort of devices and thus ideally suited to replace the infinite number of ad-hoc solutions that for centuries humans had conceived to make a machine respond, in a predictable way, to external stimuli or to generate events directly correlated to its internal state. Besides typical “data processing” peripherals like card readers and line printers, over the years computer mice, joysticks, game pads, track balls, plotters, scanners and more were attached to computers. Of more interest for the purpose of this page, however, are other types of device such as microphones-loudspeakers and video cameras-monitors. 

As early as the 1950s, computers were already connected to oscilloscopes used as display devices and in the 1960s direct-view storage tubes were also connected. In 1963, the Bell Labs developed a computer-generated film entitled “Simulation of a two-giro gravity attitude control system”. By the mid-1960’s, major corporations started taking an interest in this field and IBM was the first to make a commercially available graphics terminal (IBM 2250). The unstoppable drive to connect all sorts of device to computers is well demonstrated by the computer-controlled Head-Mounted Display (HMD) realised in 1966 at MIT. HMD provided a synthetically-generated stereoscopic 3D view by displaying two separate images, one for each eye. In 1975 Evans & Sutherland developed a frame buffer that could hold a picture. 

In the very same years, after my second return from Japan in October 1973, my group at CSELT developed a 1 Mbyte frame store for video coding simulation built with 16 Kbit RAM chips that was capable of capturing and storing a few monochrome and composite (PAL) video frames in real time. The video store was interfaced to a GP-16, a simple but effective minicomputer manufactured by Selenia (now Alenia, at that time a company of the STET group) and video samples could be transferred to a magnetic tape and read by a mainframe computer. Video coding algorithms were tested on the mainframe and the processed data were again loaded on a magnetic tape, transferred from tape to RAM and visualised on the simulation system. If one considers that one cycle of this process could take days, it should not be difficulty to understand why I have used the word “vestals” to describe the people running the mainframe computers in those days. 

3D Graphics (3DG) is the name given to the set of computer programming techniques that allow the  effective generation of realistic 3D images for projection on a 2D surface.  The impressive evolution of this field is paradigmatic of the way academic interests successfully morphed into commercial exploitation. The development of output devices matched to the needs of synthetic picture viewing was a necessary complement to Computer Graphic’s value. The field evolved in a matter of 15 years through a number of milestones as briefly sketched in the table below.

The establishment of commercial companies was made possible by the progress of 3DG technologies and reduced price of computing. A milestone was reached in 1988 with the Renderman format. This provided all the information required to render a 3D scene: objects, light sources, cameras, atmospheric effects, etc. 3DG developers could give the modeling system the capability of producing Renderman compatible scene descriptions and output the content on machines supporting the format. In 1990 AutoDesk introduced Studio3D, a 3D computer animation product that has achieved a leading position in 3D computer animation software. 

Already in the 1970s Computer Graphics had entered the world of television and prompted the development of hardware/software systems for scanning and manipulating artwork, e.g. making it squash, stretch, spin, fly around the screen, etc. Morphing, a technique that transforms an image of an object into the image of another object, was first demonstrated in 1983 with a video sequence showing a woman transforming herself into the shape of a lynx. In 1991 massive use of 3DG techniques in movies began with “Toy Story”, the first full-length computer-animated feature film, continuing with “Terminator 2” where the evil T-1000 robot was sometimes Robert Patrick, a the real actor, and sometimes a 3D computer animated version, and “Beauty and the Beast” that contained 3D animated objects, flat shaded with bright colors so that they would blend in with the hand-drawn characters. 

In 1994 a group of companies established a consortium called Virtual Reality Modeling Language (VRML) – now Web3D Consortium – with the goal of developing a standard format to represent 3D worlds. The first specification, issued in 1997 as VRML 97, provided the coded representation of a 3D space that defined most of the commonly used semantics such as hierarchical transformations, light sources, viewpoints, geometry, animation, fog, material properties and texture mapping. 

The need to cater to the growing 3DG researchers and users communities prompted the establishment of the Special Interest Group on Computer Graphics (SIGGRAPH) of the Association of Computing Machinery (ACM). The first SIGGRAPH conference held in 1973 was attended by 1,200 people, but today the conference has an attendance of tens of thousands of people. 

What has been described so far could be called as the high end of computer graphics, but there is another, originally low- to middle-end application domain, that has given rise to an industry that has used the same computing technologies with an identity of its own: computer games. While the 3DG field had a more traditional evolution – first academia and then exploitation – the computer game field targeted exploitation from the very beginning. However, today’s the progress in computing devices is blurring the borders between the two fields. 

This is short tracking shot of this business area over 4 decades

It took a considerable amount of time for computers of reasonable cost to provide a video output because of two heavy requirements on computers: fast CPUs to process a high amount of data and a large memory because of the need to store at least one screenful of data that could be generated asynchronously by the CPU and read out synchronously and converted to analogue form to drive a display. For this reason . 

For audio, matters were much simpler because waveform could easily be generated in real time or read from a file on a disk, and converted to analogue form to drive a loudspeaker. If the waveform corresponded to a musical score, it was rather easy to provide special hardware designed to produce different types of sound. As an example, the C-64 had a built-in analog synthesiser chip and many games had an obsessive tune accompanying the game that changed depending on the state of the game. In 1989 the first sound cards, the Adlib and Soundblaster, brought a more professional sound to the PC, replacing the original “beep” of the internal speaker. 

The Musical Instrument Digital Interface (MIDI), developed in 1983 by Sequential Circuits and Roland, is a protocol to control electronic musical devices. A MIDI message can tell a synthesiser when to start and stop playing a specific note, the volume of the note, the aftertouch (the amount of pressure applied on the keys of a given channel), the instrument desired to play on a channel, how to change sounds, master volume, modulation devices, and even how to receive information. In more advanced uses, MIDI information can indicate the starting and stopping points of a song or the metric position within a song.