Inner Space - the Final Frontier
David Leevers, BA, MIEE
Manager, Multimedia Communications, BICC Group
"From Desktop to Web-top:
Virtual Environments on the Internet, WWW and Networks"
International conference, 14-17 April 1997
Pictureville,, Bradford UK
Once upon a time the computer was clearly part of the external world, hidden behind layers of data preparation clerks and programmers. Even the recent desktop interface was reassuringly external, explicitly placed at arms length. However the networked virtual environments that have been described at this conference are not just animating the vastness of space beyond the screen, they are also punching out at us in true 3D movie fashion and starting to colonise the "inner space" of our private mental models.
The catalyst for this impertinent invasion is a hope that shared visualisations can be used to support new forms of collaboration with other people. This paper draws on a number of communications projects that are helping to define a new "Enhanced Reality" that goes way beyond shared visualisation to a seamless integration of the real world, telecommunications and virtual reality. A conceptual framework known as the "Cycle of Cognition" is being proposed as a way of disentangling the many modes of communications with individuals, groups and data, all of which may be local, remote or virtual.
Because VR applications play with the boundary between real and imagined spaces they are helping to give new insights into how thinking is dominated by the physics of the real world on one hand and by trust in colleagues on the other. Such an understanding is becoming essential as new networked services are helping to reconcile the complete range of fundamental human needs within an ever more complex global society. This society is considerably more than just an "Information Society". It is perhaps the Holy Grail of the Me Generation, an ego society or "i-society".
My own introduction to Virtual Reality was not a need to simulate a real environment but the desire to support day to day activities in manufacturing and construction. Since the workface is inherently three dimensional and teams can be distributed across a wide area this was seen as an ideal application for collaborative virtual environments.
The range of social and practical activities in manufacturing and construction is similar to that in the rest of our lives. This has encouraged speculation on how society as a whole will change as local and physical reality is augmented with telepresent and virtually present people and objects.
Most forms of Western leisure involve copious consumption of material resources in the form of possessions and travel. Any attempt to replicate such a lifestyle throughout the planet is unsustainable and would be destined to plunge the world into a new dark age of conflict and destruction. However, by applying emerging understanding of the evolution of humans as social animals, we may be able to construct a new "Enhanced Reality" in which compatible wishes are fulfilled in the real world and conflicting ones are transferred to the virtual. The route to this Enhanced Reality is still obscured by the fog function of our personal z-axis buffers. Many sci-fi writers have well developed private visions but, as the recent 2001 retrospect industry has shown, even the most competent of those private visions can be hopelessly inaccurate. HAL can tell us that, or can he?
We have attempted to build this optimistic scenario by trying to emulate the lifestyle of those who already have unlimited resources, the emperors and lottery winners amongst us, by introducing virtual surrogates for what can never be made available to all. Already the compact disc has replaced the emperor's private orchestra, the movie and TV have almost completely taken over from the theatre and agents are being programmed to provide the obsequious but stimulating conversation of a skilled courtier.
This new world of virtually unlimited resources (or unlimited virtual resources!) will only be accepted if it is satisfying and fulfilling. The enhanced real space must stimulate a new "Inner Space", a cosmology of the mind that will be far richer and more rewarding than the barrenness of outer space. It will be a space in which we can enjoy fantasy lives while remaining firmly connected to the responsibilities of the physical reality that we share with other people. The new society is effectively a multi-player mixed reality game in which everyone can win, and everyone can be virtually famous, infamous (or whatever else they want to be) not for just 15 minutes but for the whole of their lives.
Millions of years of natural selection have fine tuned the brains of our pre-human ancestors for competence in the spatial and gravitational world. However humans also share language, ideals, ethics, and logic. How could such an overwhelmingly powerful thinking capability have emerged over an incredibly small number of generations?
Current thinking in some anthropological circles is that an existing part of the brain was diverted to distinctively human activities. It is pretty obvious which part of the brain was left with little to do as proto-humans descended from the three dimensional complexity of the forest canopy to the flat two dimensional savannah. Much of our 3D spatial competence may have been co-opted to handle the increased complexity of social interactions that became possible on the ground.
The "archaeological" evidence is found not under our feet but in our words. Not only is every language replete with spatial metaphors for social and abstract concepts but, more persuasively, the metaphors are much the same in all languages. Up and down, near and far, left and right, inside and outside, on top or underneath are typical of any conversation about abstract concepts. We appear to have diverted much of our 3D capability into a richly verbal social life while retaining immediate visual awareness for navigating across the 2D surface of savannah or city.
Perhaps our heads did remain in the clouds when we descended to the ground, the new clouds of fuzzy spatial metaphors replacing the old clouds of the rain forest canopy!
The Proscenium Arch
Given that what we are actually aware of is a visual panorama with a minimal amount of depth information, awareness of the surroundings is better described as 2.5D than 3D. In a sense, the blind are the only ones that can "see" in 3D. Because we cannot see things in the round we put a lot of effort into trying to suppress the third dimension. We prefer to stand with our back to a wall and everything is pushed to the periphery of a room. Books and pictures hang on the walls, they are not placed in the centre of the room, and most abstract creative activities are recorded on a 2D paper surface.
This trend to flatten the real world is illustrated in the evolution from the "theatre in the round" of traditional tribal celebrations to the rectangular stage or screen of the most highly developed forms of narrative entertainment. The 2.5D stage with its simplified layers of scenery has a more consistent dramatic impact on every member of the audience. The fact that the rectangle of entertainment can occupy a small fraction of the field of view shows that it is not difficult to suspend disbelief, or immerse the mind, as long as the performance is good enough. The experience of physical immersion does seem to be so overpowering that it is best confined to brief dramatic moments - the theme park ride, the actor who enters by bursting out of the audience or the thunderous opening to Star Wars.
Where is the Virtual Mind?
Strangely enough, the mind-body problem can become a major issue when trying to build a useful shared virtual environment. Although we no longer try to place that mind in the heart or another even more unsuitable organ, we are still not quite sure how the mind relates to the brain. Perhaps we can take advantage of this uncertainty by using more abstract representations of remote and virtual colleagues and thus remove an annoying problem with the real world that it is not possible for two people to stand in the same place and same time and share the same physical point of view, an important step towards sharing the same conceptual point of view.
It may not be necessary to represent the immediate social group as VR avatars or live video within the shared Enhanced Reality. Since, in another sense, the network is already starting to liberate the spirit, perhaps the other minds should be represented as "free spirits" only loosely associated with the shared virtual environment. Thus the telepresent friends watching a virtual football match need not be represented realistically whereas the virtual players must remain firmly bounded by the physics of gravity, space, time - and the rules of football.
The Cycle of Cognition
Over the last decade the Multimedia Communications Group of BICC has been trying to define the architecture and benefits of what is now called a persistent multimedia communications environment. This work has been partially funded by the UK CSCW project VirtuOsi and a series of EC RACE and ACTS projects, DIMUN, BRICC, CICC, RESOLV and MICC, that have all been aimed at the manufacturing and construction sectors.
The conceptual framework for this work is known as the "Cycle of Cognition". By including the most probable path from one type of communications to another the cycle helps to define requirements not only for communicating but also for changing from one mode of communications to another. The stages in the cycle: the Home, Map, Landscape, Room, Table and Theatre, cover an idealised working day. However the metaphoric power of the cycle comes from its ability to support tele- and virtual communications and to cover a very wide range of time-scales.
The Cycle of Cognition is effectively an ecology of information and communications services. It presents a dynamic view of such services and indicates how squeezing them in one place is bound to lead to bulges of confusion or bandwidth at others.
The Home is that private place where we sleep on what we have recently learnt and subconsciously adjust our existing mental model of the world to accommodate new concepts. We rehearse the implications of new ideas, perhaps in dreams or in play, and start to formulate what needs to be done next. This metaphoric Home is more of a private study or bedroom than family residence because it is private territory, not shared with others.
When we venture out from Home we are not immediately ready to confront other people. The Map refers to passive information: the morning paper, a book, Yellow Pages, a real map or a reference book that provide a stable starting point for the tasks of the day. It is not necessarily up to date, but that is an assurance of its stability. Using the Map is a reassuringly familiar mental 'warming-up exercise' that we have probably been through many times before.
As the objective becomes clearer it becomes necessary to track down the most up-to-date information. We scavenge in an open landscape made up of people and the documents with which they are associated. Brief conversations are confined to fact finding and so it is acceptable for others to overhear them. The Landscape can be real, such as an open plan office, or virtual, say a VR visualisation relevant people and documents.
Contentious or unresolved issues need a more committed discussion than is possible in the public landscape. The walls of a real room manifest a complete security and privacy barrier that encourages occupants to share information. Similarly the walls of a shared Virtual Room displayed on each participants screen are a reminder of effective firewalls until the group have established mutual trust and understanding. Then they feel confident enough to take their eyes off each other and look down at the shared Table.
Most meetings take place round a table and there is always the problem of letting everyone get an adequate view of documents placed on the table. One advantage of a networked Virtual Table, usually known as a shared window, is that it can display identical material to all participants. However each person still has their private area; the part of their computer screen that is not shared with the others, or their real life notepad or filofax.
In our implementation the Virtual Table occupies the lower part of the computer screen. Across the top are video windows of the other participants together with shrunken copies of their own screens. This is an example of being able to break the rules of Cartesian space when representing minds rather than things. All the other minds appear to be in ideal positions on the other side of the table, an impossibility in real life.
The video windows indicate the extent to which the others are participating. They maintain rapport and encourage convergence to an agreed conclusion. The shrunken screens serves the same function as glancing across a real table to see what the others are doing. As in real life, these miniatures are not clear enough to read what is being written.
The result of a meeting is only of use if it is accepted by the relevant audience; customers, colleagues or students. A representative of the meeting room group takes on a theatrical role to broadcast the conclusions. In such a performance the narrative flow is usually decided in advance but the emotional emphasis can depend on the mood of the audience. It is important to note that the audience is prepared to accept the new story because they know that agreement has been reached round the table and, perhaps more significantly, because each member senses that the rest of the audience is also accepting the message.
Any effective performance changes the way members of the audience think, i.e. how they will react to related situations in future. The performance can be said to have killed the previous personality and given birth to a slightly different one. This is one reason for the rituals associated with becoming a member of an audience, checking reviews before buying a ticket, studying the mood of others in the foyer and keeping an eye on them during the performance.
Whereas the facts collected in the landscape are used immediately and often subsequently forgotten, the wisdom received during a performance does not trigger immediate action. It is usually necessary to retreat to the Home where any new constructs can be absorbed into long term memory, ready to guide reactions when the time is ripe.
When the potential new construct is physical rather than mental the Home refers to that part of the real world that is temporary personal territory, the location of activities such as bricklaying or manufacturing that are carried out after receiving instructions in the Theatre of the foremans instructions.
A Fractal Cycle
The Cycle of Cognition echoes our daily cycle, waking at home, setting off using a map, actively browsing the business landscape in the morning, negotiating over the midday meal and collaborating in the hazy glow of the afternoon, then taking a seat in the theatre as the sun sets to surrender the mind to the persuasive powers of the actors. Finally staggering home with new ideas teeming inside our head, ideas that will have been absorbed into and have slightly altered our "Inner Space" by the time we wake up the next morning.
Similarly the cycle reflects our journey through life: emerging from the home of the womb, spending a few months in a map of sensations until learning from the landscape of older people becomes possible, then spending adolescence in a lively meeting room of developing personalities, subsequently settling down to more focused and shared activities round the more formal table of career and family. Finally gaining the respect of the community and taking the stage to pass cultural memes to the next generation.
The Cycle has fractal properties in that cycles can be nested within other cycles almost indefinitely, from the formulation and presentation of a single on the white-board during a meeting to the complete life-cycle of a civilisation.
The Global Virtual Factory
Our first attempt to evaluate the usefulness of the Cycle of Cognition is a prototype Intranet known as the Global Virtual Factory. The business objective is to make it as easy to work with people in remote factories as with colleagues on the same shop floor. In this way a distributed global organisation can gain the economies of scale that are currently confined to massive centralised plants. The collaboration tools include:
Home pages. These personal Web pages would eventually supersede the company telephone directory. They include video and screen glances that are refreshed once a minute in order to provide the modest degree of activity awareness that is comparable with walking around an open plan office
Nearest Neighbours. Personality awareness is conveyed by asking everyone to nominate their 6 most important contacts in the organisation. This proved to be an effective and acceptable way of capturing the informal social structure of the company as opposed to its formal organisation chart
Photo Panorama. A panoramic photo taken from where a person works provides a literal way of seeing things from the other persons point of view, an essential first step towards reaching an agreement
Shared Window. This is one of the oldest PC based collaboration tools, allowing all participants in a distributed meeting to see the subject from the same point of view without having to crowd round the same physical document
Video Open Plan Office. A row of video windows below the metaphoric landscape separating the collaborating minds from the 3D subject, figure 2
Egocentric Meeting. The video glances of all those who might contribute to a meeting are placed at the top of the screen, as if the shared window is the meeting table and everyone else is on the other side of it - an egocentric seating plan that can only be enjoyed by one member in a real meeting. In addition people who are only needed for part of the meeting can continue with their other activities
Reference Factory. People in different factories do not always have the same view of what a factory is. By building a Virtual Reality reference factory we have been able to reinforce the important common factors between a group of factories that look very different in real life.
Photo Walkaround. Unique features of a particular factory can be displayed by selecting from the saturation photo set for the factory of interest. By zooming and panning between views the user gets the impression of moving round a remote factory without having to travel half way round the world to visit it.
Figure 2, The Reference Factory Landscape
This repertoire of tools have been selected in the light of workplace studies that have analysed the nature of collaboration in both construction and manufacturing. Three distinct approaches have been used:
Distributed Cognition, identifying how teams work together to solve problems, contributing to an understanding that is somehow spread across the brains of several people.
Organisational Memory. Earlier ethnographic studies of the introduction of "networked information" to such work groups has indicated that it is important to distinguish four types of information:
in peoples heads
in structured data
in the real world,
and ensure that they are all adequately supported. Only when people feel that they can choose freely from all the available sources will they trust the more rigid but also more accurate structured data.
Situation Theory. This is a new mathematical theory of information that provides a way of structuring the context of any form of communication. Context is not an issue when everyone is in the same room but is vital if misunderstandings are to be avoided during distributed meetings.
Most of the collaboration tools are emerging as part of the Internet and Web consensus. A few difficulties remain over bandwidth, response times and privacy, but there is nothing that is inherently either impossible or expensive.
Initial results are very promising, in spite of the immature state of the network and applications. By focusing on the new forms of social interaction that take place across the network we are addressing the organisational issues that are central to the effectiveness of comprehensive data repositories. The communications tools that place information at our fingertips and the IT tools that make sure the information is of value are beginning complement each other in a very encouraging way.
Initially immersive audio-visual Collaborative Virtual Environments showed great promise as a way of supporting social interaction for both business and leisure. Unfortunately most activities remain obstinately attached to the reality of our immediate surroundings - high quality touch, smell, taste and kinaesthetic sensing. We need to enhance reality, not simulate it. Virtual Reality technology may be condemned to a rather fruitless asymptotic path towards mere reality - perhaps Moore's Law should be updated to state that the difference between the virtual and the real halves every two years! A way out of this Xeno paradox is to leap-frog reality, to ignore the difficult (but challenging) problems of touch, smell and taste and go straight for a networked Enhanced Reality.
In the last few decades the silicon technologies of electron and photon have grown to dominate our lives. They are now maturing and it is becoming possible to regain control, to move on from the silicon products and start thinking about the Enhanced Reality processes that they can support. The user surface of Enhanced Reality will be made up of many different components, including more comprehensive forms of the 3D Reference Factory mentioned above. The technology itself will have become a new infrastructure as invisible and ubiquitous as traditional urban services.
The new enhancements to our local realities promise to be just what the human animal needs, a way of mediating both our social nature and our egocentric urges. Network infrastructure will support negotiation tools that reconcile each persons own need for self-fulfilment with the needs of others. If the conflict is over the location of an activity then real presence is supported by telepresence. Only one person can occupy a Formula 1 car, but the radio camera allows 1000 million people to become telepassengers. If the conflict is over physical resources then virtual resources are introduced. The arcade game replaces the Formula 1 car for the smaller number of enthusiasts who would rather occupy the driver's seat than admire their hero's skills.
The global middle class is starting to use these technologies to achieve a degree of self-fulfilment that was impossible for all but those at the top of a traditional community. Because the tools required for this cosmic sleight of hand are inherently inexpensive there is no reason why the whole world cannot eventually rise to a common plateau of real presence, telepresence and virtual presence.
Components of this new i-society include
i-nformation, the i-nfrastructure of co-operation and competition
i-nnovation, within a sustainable framework and
i-nteraction, the vital spark that brings all of us to life.
Not a moment too soon the Web is becoming the nervous system of the planet, albeit a dry silicon one rather than a wet carbon one. Perhaps this is the time to update the Gaia hypothesis. The new CyberGaia is a global web-footed amphibian who is equally at home in the world of bits as in the world of atoms, and whose own webbed feat is to provide the physical and the information infrastructure for a sustainable, balanced and fair society.
Spatial Metaphors in language: http://metaphor.uoregon.edu/metaphor.html
Evolutionary Psychology: Cosmides and Toomby, http://www.psych.ucsb.edu/research/cep/primer.htm
The new cosmology of the mind, - "Goodbye Descartes" Keith Devlin, 1997, http://www.maa.org/devlin/devlin_archives.html
The four types of information: http://www.hhdc.bicc.com/cicc/pif/pifsample.htm
The EC projects: www.hhdc.bicc.com/cicc/
The Cycle of Cognition: http://www.hhdc.bicc.com/people/dleevers/papers/cycleof.htm.
Augmented Reality Through Wearable Computing: http://www-white.media.mit.edu/tilde/testarne/TR397/main-tr397.html