Last update: 2011/08/21

Riding the media bits

Computers create pictures and sound

Today humans perceive pictures and sound that are not only created by the real world but by computers as well. Brief history of a complex business case of IT use in an extended media space.

Most of what has been said so far concerns audio and visual information generated in the real world and reaching human senses either directly or through a more or less transparent communication system. An increasing proportion, however, of what we perceive today through communication devices is no longer generated in that way. Images and sound from computers and game consoles, originally almost exclusively synthetically generated, have an increasing share of naturally generated media, while what we perceive from TV sets and movie theatre screens, originally almost exclusively naturally generated, are increasingly complemented by synthetically generated audio and video.

Because of this trend, the laying out of the digital media scene would not be complete without a ride on these other bits, particularly because since 1995 MPEG has been operating in this space trying to build bridges between business communities through disciplines proper to them. Purpose of this page is then to clarify the background of synthetically-generated pictures and sound to help understand the role that MPEG has been, is and plans on still playing in it.

From the beginning, computers were machines capable of connecting to and controlling all sort of devices and thus ideally suited to replace the infinite number of ad-hoc solutions that for centuries humans had conceived to make a machine respond, in a predictable way, to external stimuli or to generate events directly correlated to its internal state. Besides typical "data processing" peripherals like card readers and line printers, during the years computer mice, joysticks, game pads, track balls, plotters, scanners, were attached to computers. Of more interest for the purpose of this page are other types of devices such as microphones/loudspeakers and video cameras/monitors. 

As early as the 1950s, computers were already connected to oscilloscopes used as display devices and in the 1960s direct-view storage tubes were also connected. In 1963, the Bell Labs developed a computer-generated film entitled "Simulation of a two-giro gravity attitude control system". By the mid-1960's, major corporations started taking an interest in this field and IBM was the first to make a graphics terminal (IBM 2250) commercially available. The insatiable hunger for connecting all sort of devices to computers is well demonstrated by the computer controlled Head-Mounted Display (HMD) realised in 1966 at MIT. HMD provided a synthetically-generated stereoscopic 3D view by displaying two separate images, one for each eye. In 1975 Evans & Sutherland developed a frame buffer that could hold a picture. 

In the very same years, after my second return from Japan in October 1973, CSELT had developed a video simulation system with the capacity of 1 Mbyte built with 16 Kbit chips that was capable of capturing and storing a few monochrome and composite (PAL) video frames in real time. The RAM was interfaced to a GP-16, a simple but effective minicomputer manufactured by Selenia (now Alenia, at that time a company of the STET group) and video samples could be transferred to a magnetic tape and read by a mainframe computer. Video coding algorithms were tested on the mainframe and the processed data were again loaded on a magnetic tape, transferred from tape to RAM and visualised on the simulation system. If one considers that one cycle of this process could take days, it should not be difficulty to understand why I have used the word "vestals" to describe the people running the mainframe computers in those days. 

The set of techniques used to program a computer so that realistic 3D images can be generated and projected on a 2D surface, using computing resources in an effective way, is called 3D Graphics (3DG).  The impressive evolution of this field is paradigmatic of the way its academic interests successfully evolved to commercial exploitation. The development of output devices matched to the needs of viewing the synthetic pictures was a necessary complement to CG's value. The field evolved in a matter of 15 years through a number of milestones as detailed in the table below.

Algorithm	Description
Hidden-surface	Determines which surfaces are "behind" an object and thus should be "hidden" when the computer creates a 2D image representing what a viewer sees of a 3D scene.
Colour interpolation	Improves the realism of the synthetic image by interpolating across the polygons so reducing the aliasing caused by the sharp edges of the polygons.
Texture Mapping	Takes a 2D image of the surface of an object, and then applies it to a 3D object.
Z-buffer	A buffer containing the depth data for every pixel in an image (Z-buffer because Z represents the depth, Y the vertical position and X the horizontal position) to speed up the process of hidden surface removal.
Phong shading	Interpolates the colours over a polygonal surface with accurate reflective highlights and shading.
Fractal	A curve or geometric figure in a plane that covers the entire surface of the plane, to create realistic simulations of natural phenomena such as mountains, coastlines, wood grain, etc.
Ray tracing	The process of tracing every ray of light, starting from the viewer's perspective back into the 3D scene to simulate highly reflective surfaces. If an object is reflective, that ray of light is followed as it bounces off the object or until it hits other objects with an opaque non-reflective surface or leaves the scene.
Radiosity	The process of determining how light reflects between surfaces using heat propagation formulae.

This progress of 3DG technologies and the reduction in price of computers enabled the establishment of commercial companies. A milestone was reached in 1988 with the Renderman format. This provided all the information required to render a 3D scene: objects, light sources, cameras, atmospheric effects, etc. thus enabling 3DG developers to give the modeling system the capability of producing Renderman compatible scene descriptions to be able to output their content on machines supporting the format. In 1990 AutoDesk introduced Studio3D, a 3D Computer animation product that has achieved a leading position in 3D computer animation software. 

Already in the 1970s CG had entered the world of television and prompted the development of complex hardware and software systems to scan existing artwork, manipulate it, make it squash, stretch, spin, fly around the screen, etc. Morphing, a technique to transform an image of an object into the image of another object, was first demonstrated in 1983 with a video sequence showing a woman transforming herself into the shape of a lynx. In 1991 massive use of 3DG in movies began with "Toy Story", the first full-length computer-animated feature film, continuing with "Terminator 2" the evil T-1000 robot that was sometimes the real actor Robert Patrick and sometimes a 3D computer animated version, and "Beauty and the Beast" that contained 3D animated objects, flat shaded with bright colors so that they would blend in with the hand-drawn characters. 

In 1994 a group of companies established a consortium called Virtual Reality Modeling Language (VRML) - now Web3D Consortium - with the goal of developing a single format to represent 3D worlds. The first specification, issued in 1997 as VRML 97, provided the coded representation of a 3D space that defined most of the commonly used semantics such as hierarchical transformations, light sources, viewpoints, geometry, animation, fog, material properties and texture mapping. 

The need to cater for the growing community of researchers and users of 3DG prompted the establishment of the Special Interest Group on Computer Graphics (SIGGRAPH) of the Association of Computing Machinery (ACM). The first SIGGRAPH conference held in 1973 was attended by 1,200 people, but today the conference has an attendance of tens of thousands of people. 

One could call what has been described so far as the high end of computer graphics, but there is another, originally low- to middle-end application domain, that has given rise to an industry that has used the same computing technologies with an identity of its own: computer games. While the 3DG field had a more traditional evolution - first academia and then exploitation - the computer game field had a foot in exploitation from the very beginning. Today the progress in computing devices is already blurring the borders between the two fields. 

This is short tracking shot of this business area over 4 decades

1. Space War is probably the first video game, said to have been created by an MIT student in 1961 on the Digital Equipment (DEC) PDP-1. It was very successful and was even used by DEC engineers as a diagnostic program.
2. Five years later the first home video game was created. Its object was to catch spots of light with manually controlled dots. The game was licensed to Magnavox which sold the game for the consumer market under the name Odyssey.
3. In 1971 Space War became the basis for the world's first arcade video game with the name Computer Space. It was too sophisticated for the market of that time and had only limited success but the next arcade video game called Pong - from ping-pong -  released the following year was hugely successful. 
4. The first two arcade video games were developed by Atari (the name comes from a move in the Japanese game "go"), a company sold to Warner Communication in 1976.
5. The following year Atari introduced the very successful 2600 VCS, with 2Kbyte of ROM and 128 bytes of RAM. This was the first home game console with multiple games. The pattern of video games hopping from the consumer to the arcade market and vice-versa was thus set from the early years of the industry. 
6. In 1978 Taito released Space Invaders, the first blockbuster videogame installed in restaurants and corner stores.
7. In 1979 the 8-bit Atari 800 was introduced.
8. Space Invaders was translated to the Atari 2600 video home game system
9. This was followed by a number of third-party development houses: Activision, formed in 1979 by Atari developers, and in the 1980's Epyx, Broderbund, Sierra On-Line and SSI.
10. n 1980 Philips released Odyssey2, Mattel released Intellivision and Namco released Pac-Man of which more than 300,000 arcade units were sold since introduction. This was a huge hit around the world and is an unforgettable experience of many no-longer-so-young people who have been raised in an environment populated by a long list of computer game names, such as Zork, Donkey Kong, Galaxian, Centipede, Tempest, Ms. Pac-Man and Choplifter!
11. In 1981 home computers such as Apple, Atari, and TRS-80 became popular because dozens of games had been produced for those platforms. 
12. In 1981 the game industry was worth more than 6 billion USD in sales and Atari alone did 1 billion USD with Asteroids throughout its life span. 
13. In 1982 gaming companies that produce the hits of today, such as Access Software, Electronic Arts, and Lucasfilm Games (now LucasArts) were founded.
14. Microsoft tried its luck in this field with "Olympic Decathlon", a not particularly successful game, one reason why it took many years before Microsoft would publish another computer game.
15. In the first half of the 1980s home computers with game capabilities started being released: the Atari 400 and 800, the Commodore VIC-20 and the Commodore-64 (C-64) of which 20 million units were sold in 1982 alone, the year it was released. Common features of these home computers were: colour display capabilities, composite video output for television sets, and tape units, floppy disk drives and cartridges as storage devices. 
16. In 1984 the cartridge-based systems used, e.g., by the Atari 2600, became suddenly unpopular. The video game industry started losing ground, while home computers by Commodore and others were gaining ground because of the possibility to do other things besides games.
17. In 1985 the success of the Commodore 64 continued outselling Apple's and Atari's computers
18. In 1985 Commodore launched the Amiga personal computer - another unforgettable experience - with many advanced graphic features. 
19. In 1985 a new player - Nintendo - came to the fore with its Nintendo Entertainment System (NES), characterised by strict control on software, lockout chip, and the restriction to companies to 5 games/year.
20. In the following year another new player - Sega - came to the fore with its Sega Master System console, technically superior to Nintendo, but a market failure because of the lack of games as Sega did not consider third-party developers. 
21. In 1989 Sega released the 16-bit Genesis console with Electronic Arts sports titles, while Nintendo kept its 8-bits. Nintendo released Super Mario 3, the all-time best-seller, while Amiga and Atari ST died out.
22. In 1991 Nintendo launched the 16-bit Super-NES. The following year Nintendo had 7 billion USD in sales and higher profits than all U.S. movie and TV studios combined. 
23. In 1992 PC gaming exploded.
24. In 1993 Panasonic shipped the 32-bit Real console from 3DO.
25. In 1993 Sega and Nintendo consoles held 80% of the game market.
26. In 1994 Atari shipped the 64-bit Jaguar. In 1995 Sega shipped the 32-bit Saturn and Sony the 32-bit Playstation. Microsoft released Window 95 that included the Game SDK - Direct-X thus bringing major game performance into the folds of Windows.
27. In 1996 Nintendo shipped Ultra 64 and the Internet boosted the growth of multi-player gaming. 
28. In 1997 3D acceleration started to standardise on 3D-FX, and 3D acceleration became a common game feature. In the meantime Pentium II at 200 MHz started providing serious game experiences.
29. In 1998 a lot of very good PC games appeared, while Playstation ruled in the console domain. An interesting comparison with movies reveals the commonality of the two businesses: 300 games released each year with only 30 making money and 5 billion USD in PC games, about the movie industry's size. 

The first PC colour video adaptor was the 4-color Colour Graphic Adaptor (CGA). The first board for professional applications was the TARGA video adapter for the PC, capable of displaying 32 colours, released by ATT in 1985. In 1988 the 16-colour Enhanced Graphic Adaptor (EGA) graphics was used by Sierra On-Line and the following year the first game that used 256-colour Video Graphic Adaptor (VGA) graphics was published. 

Providing an output for a video signal placed two heavy requirements on computers. The first was fast CPU to process a high amount of data and the second a large memory because of the need to store at least one screenful of data that could be generated asynchronously by the CPU and read out synchronously and converted to analogue form to drive a display. For this reason it took a considerable amount of time before computers of reasonable cost could provide a video output. 

For audio, matters were much simpler because waveform generation could easily be done in real time or read from a file on a disk, and converted to analogue form to drive a loudspeaker. If the waveform corresponded to a musical score, it was rather easy to provide special hardware designed to produce different types of sound. As an example, the C-64 had a built-in analog synthesiser chip and many games had an obsessive tune accompanying the game that changed depending on the state of the game. In 1989 the first sound cards, the Adlib and Soundblaster, brought a more professional sound to the PC, replacing the original "beep" of the internal speaker. 

The Musical Instrument Digital Interface (MIDI), developed in 1983 by Sequential Circuits and Roland, is a protocol to control electronic musical devices. A MIDI message can tell a synthesiser when to start and stop playing a specific note, the volume of the note, the aftertouch (the amount of pressure applied on the keys of a given channel), the instrument desired to play on a channel, how to change sounds, master volume, modulation devices, and even how to receive information. In more advanced uses, MIDI information can indicate the starting and stopping points of a song or the metric position within a song.