Essentials
Click on the links in the shopping list to explore some of the essential elements that make a successful pervasive media project.

Technologies Essay

A Pervasive Media system consists of four different types of element working together: sensors that input data, computing that processes data, actuators that produce an audio-visual or mechanical effect, and connectivity to join them all together. Sensors read information about the user and their context, whether it’s the user’s movement or button presses. This is sent as input to the processing functions of a computer, whether a laptop, mobile phone, Arduino….. The computer processes the input from the sensors and makes it meaningful, and in response sends commands to the relevant actuators; the components that make something happen in the world. These could be a mechanical device, (a door opens), a screen effect (a movie plays), a loudspeaker (some audio is triggered). The three elements are connected in some way that enables them to talk to each other, whether it is, for example, by cable, WiFi or Bluetooth between devices, or by direct connection within the same device such as a smartphone.

Each piece of the technology in the system you use will have its own constraints and affordances. The choices you make will affect and effect the nature of your user experience. One of the more obvious examples is power supply; every component in your system will need power, if this is supplied by battery, for instance, that has an implication for the way your system is maintained – you will have to stop and recharge, or carry extra batteries. On the other hand, a system reliant on mains power will be not be portable and will only work within the length of power cable that you have.

To deliver a media experience the pervasive media system needs inputs from sensors that trigger corresponding reactions from the system. Mobile devices and embedded computing ‘sense’ events in the physical world, whether movement, gesture, heart rate or button press. Increasingly they come equipped with capabilities that emulate the human sensorium, they can ‘see’ (with cameras), ‘hear’ (with microphones), sense movement, location, and proximity. All these inputs can be used to tell the system about the user and their context in order to trigger relevant media delivery. The camera in your phone is a sensor when it is pointed at a QR code allowing it to connect you to data quickly. Limited image recognition will increasingly free the camera from its dependence on QR to find data; logos or coded brand images can connect the user to information about an image that the camera ‘sees’. Devices like Microsoft’s Kinect can be enhanced by software to interpret gesture, so movement can be a way to control or input to a computer system. Sound is another commonly used form of sensing; If Only used clapping as an input to control one of its games. Mutant Labs have used pitch and volume of voice as a way of controlling computer game play. Voice recognition is becoming an increasingly familiar way for a device to ‘sense’ and process commands.

GPS gives us a technological sense of location and direction by tracking us as we move around the outdoors. It ‘senses’ where we are. GPS devices receive signals from at least four satellites in order to generate data on the ‘map’ that your device happens to be using. GPS has been one of the most important ‘sensors’ in the development of first ‘locative’ and now Pervasive Media. It lies at the operational heart of the promise that Pervasive Media could deliver the right media in the right place at the right time. However GPS only works outdoors; it is notoriously inaccurate, giving readings that are ten or twenty metres away from their actual location; GPS availability is also poor in the canyons of the 21st Century city that cut off lines of sight to satellites. These limitations create their own affordances; GPS is better used in big open spaces where the distinctions between the different data zones can be made to dissolve into one another without the need for pinpoint accuracy.

Other sensors can provide data relevant to what the user is doing. For example, mobile devices have gyroscopes and accelerometers that perform some of these functions. The gyroscope in your Wiimote tells the computer at what angle the device is being held, allowing the computer to ‘sense’ what your hand is doing. Phones (and Wiimotes) use accelerometers that know how fast and in what direction the device is moving. This data can be fed into an application on the device or back into the network. Or it could be used to create music as in the case of the Guerilla Dance piece in the Cookbook where the movement data was used to control a soundtrack by translating it through the OSCulator software.

Sensors are relatively crude technologies. They provide the system with a very broad range of information that needs a lot of interpretation by the computing part of the system to become useful to it. This interpretation is done by the software that processes the raw data from the sensors; the more ‘fine grained’ inputs you need to the system the more processing you will need.

Actuators are more straightforward in this respect. Actuators are the ‘end point’ of the system where the user sees an effect of his or her action. A speaker or a screen responds to commands that have been issued by software running at the computing nodes in the system. In mechanical engineering a robot arm is an actuator, the point where the system creates an action in the world. In Pervasive Media systems actuation can be mechanical, visual, or audio: Charlotte Crofts’ Curzon piece features a mini cinema where the curtains, light levels and screen ‘projection’ are controlled (actuated) in response to the user accessing a website by reading a QR code within an app using a mobile phone. The media outputs in the user experiences may also take the form of text messages, tweets, emails, movies that play or audio actuated through a variety of screen and speaker configurations.

Computational processing is at the heart of the system, taking the inputs from sensors and turning them into effects via actuators. Depending on what your system wants to achieve you may find an Arduino, mobile phone, tablet, laptop or PC is needed. Pervasive Media systems often need to access media content too; this may mean connecting to a server or storing content on the device itself if it has the storage capacity. The possible media outputs are served or retrieved depending on what ‘requests’ come from the sensor inputs. Not all these processing systems need to be built from scratch; the Pervasive Media Studio for instance has used Calvium’s AppFurnace software, which is an easy to use system for making smartphone apps and integrates different sensors that can trigger media to be available at specific times or locations or in response to movement. Code for particular tasks can often be found as open source software; the Heads face detection system is an adaptation of open source code. What you can build into the heart of your system will often be determined by your budget, your team skills and the complexity of the processing you require.

Finally, all these parts of the system need to be connected. Most obviously in static installations connectivity might simply be a matter of plugging in USB cables. However in more mobile systems you will need to consider the relative merits of WiFi, SMS, 3G data, Bluetooth or Zigbee, and often you will use a combination. 3G Phone networks will constrain what kind of data you can use; the designer needs to consider end user costs and bandwidth limitations here. It is easy to fall into the trap of assuming that everyone has cutting edge connectivity which, by its nature, is always the most expensive tariff. It may be that your target audience, teenagers or the elderly for instance, only have pay-as-you-go tariffs with SMS only. Connectivity costs money. In some instances you may need to think about installing connectivity into your location. Calvium’s Escape from the Tower’ for instance deployed a WiFi beacon that they installed themselves near the box office for the Tower so that users could easily install the app without being charged for data download.