Natural user interface

Natural user interface, or NUI, is the common parlance used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users. The word natural is used because most computer interfaces use artificial control devices whose operation has to be learned. A NUI relies on a user being able to carry out relatively natural motions, movements or gestures that they quickly discover control the computer application or manipulate the on-screen content. The most descriptive identifier of a NUI is the lack of a physical keyboard and/or mouse.

Contents 1 Evolution 2 Early Examples of interfaces commonly referred to as NUI 3 Examples of interfaces commonly referred to as NUI 4 Towards a Definition: The Myth of the Natural Gesture 5 See also 6 References 7 External links

[edit] Evolution

In 2006 Christian Moore established an open research community with the goal to expand discussion and development related to NUI technologies.^[1] In a 2008 conference presentation "Predicting the Past," August de los Reyes, a Principal User Experience Director of Surface Computing at Microsoft described the NUI as the next evolutionary phase following the shift from the command-line interface (CLI) to the graphical user interface (GUI).^[2]

In the CLI, users had to learn an artificial means of input, the keyboard, and a series of codified inputs, that had a limited range of responses, where the syntax of those commands was strict.

Then, when the mouse enabled the GUI, users could more easily learn the mouse movements and actions, and were able to explore the interface much more. The GUI relied on metaphors for interacting with on-screen content or objects. The 'desktop' and 'drag' for example, being metaphors for a visual interface that ultimately was translated back into the strict codified language of the computer.

The natural user interface removes the metaphors, and many of the artificially learned devices, to allow users to more directly manipulate content using more natural movements, motions and gestures. Because the NUI is so fast to learn, the adjective 'intuitive' is applied by many to describe how users interact with it.

The Consumer Electronics Show in 2010 demonstrated several new products utilizing NUI interfaces. "Now a new wave of products is poised to bring "natural user interfaces"—as these methods of controlling electronics devices are called—to an even broader audience."^[3]

In 2010 Microsoft's Bill Buxton reiterated the importance of the NUI within Microsoft Corporation with a video discussing the technologies and it's future potential. ^[4]

[edit] Early Examples of interfaces commonly referred to as NUI

Multi-Touch

When Bill Buxton was asked about the iPhone's interface, he responded "Multi-touch technologies have a long history. To put it in perspective, the original work undertaken by my team was done in 1984, the same year that the first Macintosh computer was released, and we were not the first."^[5]

Multi-Touch is a technology which could enable a natural user interface. However, most UI toolkits used to construct interfaces executed with such technology are traditional GUI interfaces.

[edit] Examples of interfaces commonly referred to as NUI

Perceptive Pixel

One example is the work done by Jefferson Han on multi-touch interfaces. In a demonstration at TED in 2006, he showed a variety of means of interacting with on-screen content using both direct manipulations and gestures. For example, to shape an on-screen glutinous mass, Jeff literally 'pinches' and prods and pokes it with his fingers. In a GUI interface for a design application for example, a user would use the metaphor of 'tools' to do this, for example, selecting a prod tool, or selecting two parts of the mass that they then wanted to apply a 'pinch' action to. Han showed that user interaction could be much more intuitive by doing away with the interaction devices that we are used to and replacing them with a screen that was capable of detecting a much wider range of human actions and gestures.

Microsoft Surface

Microsoft Surface takes similar ideas on how users interact with content, but adds in the ability for the device to optically recognise objects placed on top of it. In this way, users can trigger actions on the computer through the same gestures and motions as Jeff Han's touchscreen allowed, but also objects become a part of the control mechanisms. So for example, when you place a wine glass on the table, the computer recognises it as such and displays content associated with that wine glass. Placing a wine glass on a table is a natural thing for a person to do, hence it fits as part of a natural user interface.

3D Immersive Touch

3D Immersive Touch is defined as the direct manipulation of 3D virtual environment objects using single or multi-touch surface hardware in multi-user 3D virtual environments. Coined first in 2007 to describe and define the 3D natural user interface learning principles associated with Edusim. Immersive Touch natural user interface now appears to be taking on a broader focus and meaning with the broader adaption of surface and touch driven hardware such as the IPhone, IPod Touch, IPad, and a growing list of other hardware. Apple also seems to be taking a keen interest in “Immersive Touch” 3D natural user interfaces over the past few years.

Xbox Kinect

Xbox Kinect is a project from Xbox that will be using spatial gestures for interaction instead of a game controller. According to Microsoft's page, Kinect is designed for a "a revolutionary new way to play: no controller required."^[6]

Dragon Naturally Speaking

Dragon Naturally Speaking is a speech recognition software package developed and sold by Nuance Communications for Windows personal computers.

[edit] Towards a Definition: The Myth of the Natural Gesture

The term “Natural User Interface” evokes mimicry of the physical world. A naïve designer looks only to the physical world, and copies it, in the hopes that this will create a natural user interface. Paradoxically to some, mimicking the physical world will not yield an interface that feels natural. Consider two examples to illustrate the point.

First, consider the GUI, or graphical user interface. Our conception of the GUI is based on the WIMP toolset (windows, icons, menus, and pointers). WIMP is actually based on a principle of ‘manipulation’, or physical movement of objects according to naïve physics. The mouse pointer is a disembodied finger, poking, prodding, and dragging content around the display. In short, if ‘natural’ means mimicking nature, we did that already – and it got us to the GUI.

Second, consider the development of the first generation of touch gestures. The promise of the NUI is an interface that is immediately intuitive, with minimum effort to operate. In designing early gestures, first-time designers constantly asked: “what is natural to the user?” This question is a proxy for “what gesture would the user likely perform at this point to accomplish this task?” In both cases, this is the wrong question. Here’s why:

Imagine an experiment intended to elicit the ‘natural’ gesture set – that set of gestures users would perform without any prompting. You might design it like this: show the user screen shots of your system before and after the gesture is performed. Then, ask them to perform the gesture that they believe would cause that transition. Here’s an example:

In experiments conducted by a team at both Microsoft Research, there was found almost no congruence between any user-defined gestures for even the simplest of operations (MSR work). A solution to this lies in the realization the study was intentionally done free of context: no on-screen graphics to induce a particular behaviour. Imagine trying to use a slider from a GUI without a thumb painted on the screen to show you the level, induce behaviour (drag), and give feedback during the operation so you know when to stop. As we develop NUI systems, they too will need to provide different affordances and graphics.

The goal of a Natural User Interface is in evoking a feeling of effortlessness in the user. The word "Natural" is a goal, not a method.

[edit] See also

• Kinetic user interface
• Organic user interface
• Post-WIMP
• Spatial navigation
• Touch user interface
• Eye tracking
• Edusim

[edit] References

This article includes a list of references, but its sources remain unclear because it has insufficient inline citations. Please help to improve this article by introducing more precise citations where appropriate. (April 2009)

[edit] External links

Examples and Further Reading

• http://bravenuiworld.com
• http://www.vimeo.com/channels/nui - Vimeo's NUI Channel showcasing NUI Interfaces
• http://www.billbuxton.com/ - Principal Researcher at Microsoft
• http://www.wigdor.com/daniel/ - User Experience Architect at Microsoft
• http://nuigroup.com/faq/ - NUI Group FAQ
• http://wiki.nuigroup.com/Natural_User_Interface NUI Group's Wiki Entry on Natural User Interface

Academic research

• NiCE (Natural User Interfaces for Collaborative Environments) - NUI project at the Media Interaction Lab of the Upper Austria University of Applied Sciences
• Natural user interaction in virtual/augmented reality - NUI research at the University of North Carolina at Chapel Hill Department of Computer Science
• Multimodal Human-Computer Interaction: a constructive and empirical study - Research from the University_of_Tampere

Academic conferences

• IEEE Tabletop annual conference
• CHI Computer Human Interaction annual conference