MFP Diary - Gesture Recognition in Audio-Only Games

The development and testing of my prototype gesture controlled audio-game produces issues daily. These problems relate to the selected target audience, which is primarily visually impaired users.

My discussion with visually impaired/able sighted members of the audiogames community highlighted the fact that blind people either don't have a webcam or those who do have immense difficulty getting the correct lighting for data capture. As an able sighted person it is easy to overlook the difficulty visually impaired users have in setting up a webcam to face the user whilst creating the correct light and making sure the camera's focus is suitable.So to counter this issue, the project will have to consider all possible options. Currently these are;

• Abandon camera based interfaces
• Acknowledge visually impaired users often have carers
• Wait for the release of Leap Motion
• Focus on able sighted
• Ensure that visually impaired users have the ability to setup a game in order to achieve maximum accuracy

Gesture based games in 'video' games have the ability to visually demonstrate where a gesture is needed to start, pause, select etc an aspect of the game. In audio-only games this ability is not available.  So again, questions need to be asked regarding how to control these aspects of the game in an audio game.

A series of possible ideas are currently being developed which will be tested to select a suitable method. One of which is to take a similar approach to audio-based games that use a combination of very little visuals to enhance the audio-games experience. For example, Papa Sangre (2010, 2013) allows the user to tap on the visuals on the screen to affect what they hear through the speakers/headphones. A more likely scenario could be to adopt a gesture controlled system used in Wonderbook: Book of Spells (2012), which has an area on the screen which when the player gestures (using PS Move) over the visual and clicks, the game will start, pause etc.

An attempt has been made to combine the method adopted by Wonderbook: Book of Spells (2012) with the system used in PS: Eye Toy (2003) and MS Kinect: Adventures (2010). These games use a system which acknowledges a players gestures to an area on the screen and times how long the gesture is maintained in that area. For example, after 5 seconds, the game will start.

One of the issues with using gestures in video games is the unpredictability of user gestures (Collins 2013). In an attempt to combine the gesture control methods above and minimize the issues regarding user gesture unpredictability, the system will continuously acknowledge the gesture data but the player can only control one function at a time. Thus avoiding any accidental restarts, pausing etc.

More to follow...