User experience evaluation
User experience (UX) evaluation means investigating how a person feels about using a system (product, service, non-commercial item, or a combination of them). It is non-trivial to evaluate user experience and come up with solid results, since user experience is subjective, context-dependent and dynamic over time. Laboratory experiments may work well for studying a specific aspect of user experience, but holistic user experience is optimally studied over a longer period of time with real users in a natural environment.
Detailed guidance for user experience evaluation is hard to give, since there are many different dimensions to consider when choosing the evaluation approach:
- Goal: Summative (score) or Formative (areas for improvement)
- Approach: Objective or Subjective
- Data: Quantitative or Qualitative
- Granularity: Momentary, episodic, or overall UX
- Setup: Lab or field
In all cases, however, there are certain aspects of user experience that evaluators are interested in (measures), and certain procedures and techniques used for collecting the data (methods). When investigating user experience evaluation methods, we can identify methods for emotion assessment and overall UX assessment. The measures and methods for these two evaluation types are described below. Episodic UX can be evaluated with either approach, depending on the case.
When investigating momentary user experiences, we can evaluate the level of positive affect, negative affect, joy, surprise, frustration, etc. The measures for emotions are bound to the methods used for emotion assessment, but typical emotion measures are e.g. valence and arousal. Objective emotion data can be collected by psychophysiological measurements or by observing expressed emotions. Subjective emotional data can be collected by using self-report methods, which can be verbal or non-verbal.
Examples of emotion evaluation methods:
- Psychophysiological emotion measurements aim to identify emotions from physiological changes in muscles (e.g. face), pupils, skin, heart, brains, etc.
- Expression observers monitor person's facial and other gestures or the tone of voice to identify emotions manually
- Think aloud protocol can be used for reporting emotions (real-time verbal self-report)
- PANAS (retrospective verbal self-report)
- Geneva emotion wheel (retrospective verbal self-report)
- Emotion Slider (continuous non-verbal self-report)
- Sensual Evaluation Instrument (snapshot non-verbal self-report)
- PrEmo, a new version of EmoCards for assessing emotion (snapshot non-verbal self-report)
Overall UX evaluation
In contrast to identifying a momentary emotion, overall UX evaluation investigates how a person feels about a system as a whole, typically after using it for a while. Many of the overall UX evaluation methods are suitable also for evaluating episodic UX, i.e., assessing how a person feels about a specific interaction episode or after executing a task.
There is no agreement on the exact measures for evaluating the overall UX with a system, largely because different products aim at different kinds of experiences. However, there are some high-level constructs of user experience that can be used as the basis for defining the user experience measures, for example:
- Utility: Does the user perceive the functions in the system as useful and fit for the purpose?
- Usability: Does the user feel that it is easy and efficient to get things done with the system?
- Aesthetics: Does the user see the system as visually attractive? Does it feel pleasurable in hand?
- Identification: Can I identify myself with the product? Do I look good when using it?
- Stimulation: Does the system give me inspiration? Or wow experiences?
- Value: Is the system important to me? What is its value for me?
Since the importance of the above user experience constructs is different to different people, it is an interesting option to define the overall UX measures together with each study participant. Another option to evaluate overall UX is to use simply a scale from positive to negative, without further consideration of the user experience constructs.
Overall UX assessment is methodologically different from objective emotion assessment, but similar to subjective emotion assessment. Generic subjective user experience evaluation methods include interviews, questionnaires, story-telling, and often, a combination of these. An individual method can collect data about a set of specific constructs of user experience, for instance usability testing is used to collect data about usability construct.
Examples of overall UX evaluation methods (excluding traditional usability methods):
- Diary methods for self-reporting experiences during field studies
- Experience Sampling Method (ESM) for self-reporting during field studies
- Day Reconstruction Method (DRM) – story-telling to reveal the meaningful experiences during field studies
- AttrakDiff questionnaire for overall UX evaluation
- User Experience Questionnaire - UEQ (available in several language versions)
- Ladder interviews e.g. to find out attitudes or values behind behaviour or experience
- HUX - Holistic User Experience identifying the relevant product factors for holistic user experience
UX in video games
A relatively new pursuit in video game playtesting is UX and usability research. An increasing number of companies including some of the world's biggest publishers have begun outsourcing UX evaluation or opening their own in-house labs. Researchers use a variety of HCI and psychological techniques to examine the effectiveness of the user experience of the games during the design process.
There are also some companies starting to use biometrics to scientifically measure the relationship between in-game events and the player's emotions and feelings (the UX), such as Player Research and Serco ExperienceLab in the UK, and Valve Software, Electronic Arts, BoltPeters, and VMC Labs in the USA and Canada. The interest in this area comes from both academia and industry, sometimes enabling collaborative work. Game UX work has been featured at professional venues, such as the Game Developers Conference (GDC)
- Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., Kort, J.: Understanding, Scoping and Defining User Experience: A Survey Approach. In Proceedings of Human Factors in Computing Systems conference, CHI’09. 4–9 April 2009, Boston, MA, USA (2009)
- Baenziger, T., Tran, V. and Scherer,K.R. (2005) ‘‘The EmotionWheel. A Tool for the Verbal Report of Emotional Reactions’’, poster presented at the conference of the International Society of Research on Emotion, Bari, Italy.
- Laurans, G., Desmet, P.M.A., & Hekkert, P.P.M. (2009). The emotion slider: a self-report device for the continuous measurement of emotion. Proceedings of the 2009 International Conference on Affective Computing and Intelligent Interaction. Amsterdam, The Netherlands.
- Isbister, K., Höök, K., Sharp, M., and Laaksolahti, J. 2006. The sensual evaluation instrument: developing an affective evaluation tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, 22–27 April 2006). CHI '06. ACM, New York, NY, 1163–1172
- Desmet, P.M.A., Overbeeke, C.J., Tax, S.J.E.T. (2001). Designing products with added emotional value: development and application of an approach for research through design. The Design Journal, 4(1), 32–47.
- Moshagen, M. & Thielsch, M. T. (2010). Facets of visual aesthetics. In: International Journal of Human-Computer Studies, 68 (10), 689–709.
- Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579–616.
- Csikszentmihalyi M, Larson R. (1987). Validity and reliability of the Experience-Sampling Method. Journal of Nervous and Mental Diseases. Sep 1987;175(9):526–536.
- Kahneman, D., Krueger, A., Schkade, D., Schwarz, N., and Stone, A. (2004). A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method. Science. 306:5702, pp. 1776–780.
- Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In J.Ziegler & G. Szwillus (Eds.), Mensch & Computer 2003. Interaktion in Bewegung (pp. 187–196). Stuttgart, Leipzig: B.G. Teubner.
- Laugwitz, B., Schrepp, M. & Held, T. (2008). Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (Ed.): USAB 2008, LNCS 5298, S. 63-76.
- Toussaint, C., Ulrich, S., Toussaint, M. (2012). HUX - Measuring Holistic User Experience. In German UPA e.V., Usability Professionals 2012 - Tagungsband (pp. 90-94).
- Halo 3: How Microsoft Labs Invented a New Science of Play. Wired.com. Retrieved on 21 October 2011.
- Bolt, Nate. (2009-01-22) Researching Video Games the UX Way – Boxes and Arrows: The design behind the design. Boxes and Arrows. Retrieved on 21 October 2011.
- THQ Chooses The Guildhall at SMU to House New Usability Lab | games industry | MCV. Mcvuk.com. Retrieved on 21 October 2011.
- Hong, T. (2008) Shoot to Thrill: Bio-Sensory Reactions to 3D Shooting Games, Game Developer Magazine, October
- GamesIndustry.biz. Player Research. Retrieved on 16 March 2013
- Game usability testing. PlayableGames. Retrieved on 21 October 2011.
- Valve. Valvesoftware.com. Retrieved on 21 October 2011.
- EA Games – Electronic Arts. Ea.com. Retrieved on 21 October 2011.
- VMC Consulting – Tailored Solutions for Your Business. Vmc.com. Retrieved on 21 October 2011.
- Bolt | Peters | Research, design, and products. Boltpeters.com. Retrieved on 21 October 2011.
- Nacke, L., Ambinder, M., Canossa, A., Mandryk, R., Stach, T. (2009). "Game Metrics and Biometrics: The Future of Player Experience Research“ Panel at Future Play 2009
- 8–9 April 2010, Seminar Presentation at Games Research Methods Seminar, "Using physiological measures in conjunction with other UX approaches for better understanding of the player’s gameplay experiences", University of Tampere, Finland
- Ambinder, M. (2011) Biofeedback in Gameplay: How Valve Measures Physiology to Enhance Gaming Experience. Game Developers Conference 2011
- Zammitto, V. (2011) The Science of Play Testing: EA's Methods for User Research. Game Developers Conference 2011