Interaction Design continues to be the standard textbook in the field. Seasoned practitioners will find it use-ful when they need a reference to best practices or to explain a concept to a colleague. Students can turn to Interaction Design for an easy-to-understand description of the basics or in-depth how-tos. From personas and disabilities to the design of UX organizations and working in Agile, if you’re going to pick one book to bring into the office, it should be this one.
Jofish Kaye, Principal Research Scientist, Mozilla, USA
This is the perfect textbook for a wide range of User interface/User experience design courses. For an undergradu-ate, it provides a variety of compelling examples that illustrate best practice in Interaction Design. For a graduate student, it provides a foundational overview of advanced topics. This book is also essential for the professional who wants to know the state of the art in Interaction design. I use this textbook and recommend it widely.
Rosa I. Arriaga, Ph.D., Senior Research Scientist, School of Interactive Computing Georgia Institute of Technology, USA
The Interaction Design book has immensely contributed to a growing Namibian HCI skilled community over the last decade. Exposing students, academics and practitioners to the basic principles and theories as well as most recent trends and technologies, with global and local case studies, in the latest edition, allows for reflective applications within very specific contexts. This book remains our number one reference in the education of future generations of interaction designers in Namibia, promoting the creation of thoughtful user experiences for respon-sible citizens.
Heike Winschiers-Theophilus, Professor, Faculty of Computing and Informatics, Namibia University of Science and Technology, Africa
Throughout my teaching of user experience and interaction design, the book by Rogers, Preece and Sharp has been an absolute cornerstone textbook for students. The authors bring together their own wealth of knowledge of academic HCI with a deep understanding of industry practice to provide what must be the most comprehensive introduction to the key areas of interaction design and user experience work, now an established field of practice. I put this book in the “essential reading” section of many of the reading lists I give to students.
Simon Attfield, Associate Professor in Human Centred Technology, Middlesex University, UK
Interaction design has gone through tremendous changes in the last few years—for example the rising importance of “big” data streams to design, and the growing prevalence of everyday ubiquitous computing issues of sensing and blending gracefully and ethically into peoples’ daily lives. This is an important and timely update to a text that’s long been considered gold standard in our field. I’m looking forward to using it with my students to help prepare them for the design challenges they will face in today’s industrial practice.
Katherine Isbister, Professor, Computational Media, University of California Santa Cruz, USA
More than ever, designing effective human-computer interactions is crucial for modern technological systems. As digital devices become smaller, faster and smarter, the interface and interaction challenges become ever more complex. Vast quantities of data are often accessed on handheld screens, or no screens at all through voice com-mands; and AI systems have interfaces that “bite back” with sophisticated dialogue structures. What are the best interaction metaphors for these technologies? What are the best tools for creating interfaces that are enjoyable and universally accessible? How do we ensure emerging technologies remain relevant and respectful of human values? In this book, you’ll find detailed analysis of these questions and much more. (It is a valuable resource for both the mature student and the reflective professional.)
Frank Vetere, Professor of Interaction Design, School of Computing and Information Systems, University of Melbourne, Australia
This is at the top of my recommended reading list for undergraduate and master’s students as well as professionals looking to change career paths. Core issues to interaction design are brought to life through compelling vignettes and contemporary case examples from leading experts. What has long been a comprehensive resource for interac-tion design now incorporates timely topics in computing, such as data at scale, artificial intelligence, and ethics, making it essential reading for anyone entering the field of interaction design.
Anne-Marie Piper, PhD, Associate Professor, Departments of Communication Studies, Electrical Engineering and Computer Science, Northwestern University, USA
I have been using Interaction Design as a textbook since its first edition for both my undergraduate and graduate introductory HCI courses. This is a must-read seminal book which provides a thorough coverage of the discipline of HCI and the practice of user-centered design. The fifth edition lives up to its phenomenal reputation by includ-ing updated content on the process of interaction design, the practice of interaction design (e.g., technical debt in UX, Lean UX), design ethics, new types of interfaces, etc. I always recommend Interaction Design to students and practitioners who want to gain a comprehensive overview of the fields of HCI and UX.
Olivier St-Cyr, Assistant Professor, Teaching Stream, University of Toronto, Canada
Interaction design is a practice that spans many domains. The authors acknowledge this by providing a tremen-dous amount of information across a wide spectrum of disciplines. This book has evolved from a simple textbook for HCI students, to an encyclopedia of design practices, examples, discussions of related topics, suggestions for further reading, exercises, interviews with practitioners, and even a bit of interesting history here and there. I see it as one of the few sources effectively bridging the gulf between theory and practice. A copy has persistently occu-pied my desk since the first edition, and I regularly find myself revisiting various sections for inspiration on how to communicate the reasoning behind my own decisions to colleagues and peers.
William R. Hazlewood, PhD, Principal Design Technologist, Retail Experience Design Concept Lab, Amazon, USA
For years Interaction Design has been my favourite book not only for supporting my classes but also as my primary source for preparing UX studies to industrial and academic settings. The chapters engage readers with easy-to-read content while presenting, harmonically, theories, examples and case studies which touch in multidisci-plinary aspects of construction and evaluation of interactive products. The fifth edition again maintains the tradi-tion of being an up-to-date book on HCI, and includes new discussions on Lean UX, emotional interaction, social and cognitive aspects, and ethics in human studies, which are certainly contemporary topics of utmost relevance for practitioners and academics in interaction design.
Luciana Zaina, Senior Lecturer, Federal University of São Carlos, Brazil
This book is always my primary recommendation for newcomers to human-computer interaction. It addresses the subject from several perspectives: understanding of human behaviour in context, the challenges of ever-changing technology, and the practical processes involved in interaction design and evaluation. The new edition again shows the authors’ dedication to keeping both the primary content and representative examples up to date.
Robert Biddle, Professor of Human–Computer Interaction, Carleton University, Ottawa, Canada
This fifth edition provides a timely update to one of the must-have classics on interaction design. The changes in our field, including how to deal with emerging sensing technology and the volumes of data it provides, are well addressed in this volume. This is a book for those new to and experienced in interaction design.
Jodi Forlizzi, Professor and Geschke Director, Human-Computer Interaction Institute, The School of Computer Science, CMU, USA
The milieu of digital life surrounds us. However, how we choose to design and create our experiences and interactions with these emerging technologies remains a significant challenge. This book provides both a road-map of essential skills and methodologies to tackle these designs confidently as well as the critical deeper history, literature, and poetry of interaction design. You will return to this book throughout your career to operationalize, ground and inspire your creative practice of interaction design.
Eric Paulos, Professor, Electrical Engineering and Computer Sciences, UC Berkeley, USA
Preece, Sharp and Rogers offer once again an engaging excursion through the world of interaction design. This series is always up-to-date and offers a fresh view on a broad range of topics needed for students in the field of interaction design, human-computer interaction, information design, web design or ubiquitous computing. The book should be the book every student should have in their backpack. It is a “survival guide”! It guides one through the jungle of information and the dark technological forests of our digital age. It also helps to develop a critical view on developing novel technologies as our computing research community needs to confront much more seriously the negative impacts of our innovations. The online resources are a great help for me to create good classes and remove some weight from the backpacks of my students.
Johannes Schöning, Professor of Computer Science, Bremen University, Germany
Nearly 20 years have passed since the release of the first edition of Interaction Design, with massive changes to technology and thus the science and practice of interaction design. The new edition combines the brilliance of the first book with the wisdom of the lessons learned in the meantime, and the excitement of new technological fron-tiers. Complex concepts are elegantly and beautifully explained, and the reader is left with little doubt as to how to put them into practice. The book is an excellent resource for those new to interaction design, or as a guidebook or reference to practitioners.
Dana McKay, UX Researcher, Practitioner and Academic, University of Melbourne, Australia
Computers are ubiquitous and embedded in virtually every new device and system, ranging from the omnipresent cellphone to the complex web of sociotechnical systems that envelop most every sphere of personal and profes-sional life. They connect our activities to ever-expanding information resources with previously unimaginable computational power. To ensure interface design respects human needs and augments our abilities is an intellectual challenge of singular importance. It involves not only complex theoretical and methodological issues of how to design effective representations and mechanisms of interaction but also confronts complex social, cultural, and political issues such as those of privacy, control of attention, and ownership of information. The new edition of Interaction Design continues to be the introductory book I recommend to my students and to anyone interested in this crucially important area.
Jim Hollan, Distinguished Professor of Cognitive Science, University of California San Diego, USA
Interaction Design continues to be my favorite textbook on HCI. Even named our undergraduate and postgradu-ate programmes at Aalborg University after it. In its fifth edition, it captures the newest developments in the field’s cumulative body of knowledge, and continues to be the most updated and accessible work available. As always, it serves as a clear pointer to emerging trends in interactive technology design and use.
Jesper Kjeldskov, Professor and Head of Department of Computer Science, Aalborg University, Denmark
I got to learn about the field of HCI and interaction design when I came across the first edition of this book at the library in my junior year of college. As an HCI researcher and educator, I have been having the pleasure of intro-ducing the subject to undergraduates and professional master’s students using the previous editions. I thank the authors for their studious efforts to update and add new contents that are relevant for students, academics, and professionals to help them learn this ever-evolving field of HCI and interaction design in a delightful manner.
Eun Kyoung Choe, Professor of Human-Computer Interaction, College of Information Studies, University of Maryland, USA
This new edition is, without competition, the most comprehensive and authoritative source in the field when it comes to modern interaction design. It is highly accessible and it is a pleasure to read. The authors of this book have once again delivered what the field needs!
Erik Stolterman, Professor in Informatics, School of Informatics and Computing, Indiana University, Bloomington, USA
This book illuminates the interaction design field like no other. Interaction design is such a vast, multidisciplinary field that you might think it would be impossible to synthesize the most relevant knowledge in one book. This book does not only that, but goes even further: it eloquently brings contemporary examples and diverse voices to make the knowledge concrete and actionable, so it is useful for students, researchers, and practitioners alike. This new edition includes invaluable discussions about the current challenges we now face with data at scale, embrac-ing the ethical design concerns our society needs so much in this era.
Simone D. J. Barbosa, Professor of Computer Science, PUC-Rio, and Co-Editor-in-Chief of ACM Interactions, Brazil
My students like this book a lot! It provides a comprehensive coverage of the essential aspects of HCI/UX, which is key to the success of any software applications. I also like many aspects of the book, particularly the examples and videos (some of which are provided as hyperlinks) because they not only help to illustrate the HCI/UX con-cepts and principles, but also relate very well to readers. I highly recommend this book to anyone who wants to learn more about HCI/UX.
Fiona Fui-Hoon Nah, Editor-in-Chief of AIS Transactions on Human-Computer Interaction, Professor of Business and Information Technology, Missouri University of Science
and Technology, Rolla, Missouri, USA
I have been using the book for several years in my Human-Computer Interaction class. It helps me, not only for teaching, but also for theses supervision. I really appreciate the authors regarding their efforts in maintaining the relevance and up-to-dateness of the Interaction Design book. For example, they put Data At Scale and AgileUX in the new edition. Really love the book!
Harry B. Santoso, PhD, Instructor of Interaction System (HCI) course at Faculty of Computer Science, Universitas Indonesia, Indonesia
During my PhD already the first edition of Interaction Design: beyond human-computer interaction in 2002 quickly became my preferred reference book. Seventeen years later, and now in its fifth edition, I commend the authors for their meticulous and consistent effort in updating and enriching what has become the field’s standard introductory textbook. Not just about objects and artefact, design today is increasingly recognized as a sophis-ticated and holistic approach for systems thinking. Similarly, Preece, Sharp, and Rogers have kept the book’s coverage with the times by providing a comprehensive, compelling, and accessible coverage of concepts, methods and cases of interaction design across many domains such as experience design, ubiquitous computing, and urban informatics.
Marcus Foth, Professor of Urban Informatics, QUT Design Lab, Brisbane, Australia
“Interaction Design” has long been my textbook of choice for general HCI courses. The latest edition has intro-duced a stronger practitioner focus that should add value for students transitioning into practice, for practitioners, and also for others interested in interaction design and its role in product development. It manages to be an engag-ing read while also being “snackable”, to cover the basics and also inspire. I still find it a great read, and believe others will too.”
Ann Blandford, Professor of Human – Computer Interaction, University College London
Very clear style, with plenty of active learning material and pointers to further reading. I found that it works very well with engineering students.
Albert Ali Salah, Professor at Utrecht University, the Netherlands
INTERACTION DESIGNbeyond human-computer
interaction
Fifth Edition
Interaction Design: beyond human-computer interaction, Fifth Edition
Published byJohn Wiley & Sons, Inc.10475 Crosspoint BoulevardIndianapolis, IN 46256www.wiley.com
Copyright © 2019 by John Wiley & Sons, Inc., Indianapolis, IndianaPublished simultaneously in CanadaISBN: 978-1-119-54725-9ISBN: 978-1-119-54735-8 (ebk)ISBN: 978-1-119-54730-3 (ebk)Manufactured in the United States of America
10 9 8 7 6 5 4 3 2 1
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make. Further, readers should be aware that Internet websites listed in this work may have changed or disappeared between when this work was written and when it is read.
For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.
Library of Congress Control Number: 2019932998
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.
The authors are senior academics with a background in teaching, researching, and consulting in the United Kingdom, United States, Canada, India, Australia, South Africa, and Europe. Having worked together on four previous editions of this book, as well as an earlier textbook on human-computer interaction, they bring considerable experience in curriculum develop-ment using a variety of media for online learning as well as face-to-face teaching. They have considerable knowledge in creating learning texts and websites that motivate and support learning for a range of students. All three authors are specialists in interaction design and human-computer interaction (HCI). In addition, they bring skills from other disciplines; for instance, Yvonne Rogers started off as a cognitive scientist, Helen Sharp is a software engi-neer, and Jenny Preece works in information systems. Their complementary knowledge and skills enable them to cover the breadth of concepts in interaction design and HCI to produce an interdisciplinary text and website.
Helen Sharp is a Professor of Software Engineering and Associate Dean in the Faculty of Science, Technology, Engineering, and Mathematics at the Open University. Originally trained as a software engineer, it was by watching the frustration of users and the clever “work-arounds” they developed that inspired her to investigate HCI, user-centered design, and the other related disciplines that now underpin the field of interaction design. Her research focuses on the study of professional software practice and the effect of human and social aspects on software development, leveraging her expertise in the intersection between inter-action design and software engineering and working closely with practitioners to support practical impact. She is active in both the software engineering and CHI communities, and she has had a long association with practitioner-related conferences. Helen is on the editorial board of several software engineering journals, and she is a regular invited speaker at aca-demic and practitioner venues.
Yvonne Rogers is the Director of the Interaction Centre at University College London, a Professor of Interaction Design, and a deputy head of department for Computer Science. She is internationally renowned for her work in HCI and ubiquitous computing and, in particular, for her pioneering approach to innovation and ubiquitous learning. Yvonne is widely published, and she is the author of two recent books: Research in the Wild (2017, co-authored with Paul Marshall) and The Secrets of Creative People (2014). She is also a regular keynote speaker at computing and HCI conferences worldwide. Former positions include Professor of Interaction Design at the Open University (2006–2011), Professor of Human-Computer Interaction at the School of Informatics and Computing at Indiana University (2003–2006), and Professor in the former School of Cognitive and Computing Sciences at Sussex University (1992–2003). She has also been a Visiting Professor at UCSC, University of Cape Town, Melbourne University, Stanford, Apple, Queensland University, and UCSD. She has been elected as a Fellow of the ACM, the British Computer Society, and the ACM’s CHI Academy.
About the Authors
A b o u t t h e A u t h o r sviii
Jennifer Preece is Professor and Dean Emerita in the College of Information Studies—Maryland’s iSchool—at the University of Maryland. Jenny’s research focuses on the intersec-tion of information, community, and technology. She is particularly interested in community participation online and offline. She has researched ways to support empathy and social support online, patterns of online participation, reasons for not participating (for example, lurking and infrequent participation), strategies for supporting online communication, devel-opment of norms, and the attributes of successful technology-supported communities. Currently, Jenny focuses on how technology can be used to educate and motivate citizens to engage and contribute quality data to citizen science projects. This research contributes to the broader need for the collection of data about the world’s flora and fauna at a time when many species are in rapid decline due to habitat loss, pollution, and climate change. She was author of one of the first books on online communities—Online Communities: Designing Usability, Supporting Sociability (2000) published by John Wiley & Sons Ltd and several other HCI texts. Jenny is also widely published, a regular keynote speaker, and a member of the ACM’s CHI Academy.
Associate PublisherJim Minatel
Editorial ManagerPete Gaughan
Production ManagerKatie Wisor
Project EditorGary Schwartz
Production EditorBarath Kumar Rajasekaran
Technical EditorsDanelle BaileyJill L. H. Reed
Copy EditorKim Wimpsett
ProofreaderNancy Bell
IndexerJohnna VanHoose Dinse
Cover DesignerWiley
Cover Image© Wiley; Jennifer Preece photo courtesy of Craig Allan Taylor
Credits
Many people have helped us over the years in writing the four previous editions of this book. We have benefited from the advice and support of our many professional colleagues across the world and from our students, friends, and families. We especially would like to thank everyone who generously contributed their ideas and time to help make all of the editions of this book successful.
These include our colleagues and students at the College of Information Studies—“Maryland’s iSchool”—at University of Maryland, the Human-Computer Interaction Laboratory (HCIL) and Center for the Advanced Study of Communities and Information (CASCI), the Open University, and University College London. We would especially like to thank (in alphabetical first name order) all of the following individuals who have helped us over the years:
Alex Quinn, Alice Robbin, Alice Siempelkamp, Alina Goldman, Allison Druin, Ana Javornik, Anijo Mathew, Ann Blandford, Ann Jones, Anne Adams, Ben Bederson, Ben Shnei-derman, Blaine Price, Carol Boston, Cathy Holloway, Clarisse Sieckenius de Souza, Connie Golsteijn, Dan Green, Dana Rotman, danah boyd, Debbie Stone, Derek Hansen, Duncan Brown, Edwin Blake, Eva Hornecker, Fiona Nah, Gill Clough, Godwin Egbeyi, Harry Brignull, Janet van der Linden, Jeff Rick, Jennifer Ferreira, Jennifer Golbeck, Jeremy Mayes, Joh Hunt, Johannes Schöning, Jon Bird, Jonathan Lazar, Judith Segal, Julia Galliers, Fiona Nah, Kent Norman, Laura Plonka, Leeann Brumby, Leon Reicherts, Mark Woodroffe, Michael Wood, Nadia Pantidi, Nick Dalton, Nicolai Marquardt, Paul Cairns, Paul Marshall, Philip “Fei” Wu, Rachael Bradley, Rafael Cronin, Richard Morris, Richie Hazlewood, Rob Jacob, Rose Johnson, Stefan Kreitmayer, Steve Hodges, Stephanie Wilson, Tamara Clegg, Tammy Toscos, Tina Fuchs, Tom Hume, Tom Ventsias, Toni Robertson, and Youn-Kyung Lim.
In addition we wish to thank the many students, instructors, researchers and practition-ers who have contacted us over the years with stimulating comments, positive feedback and provocative questions
We are particularly grateful to Vikram Mehta, Nadia Pantidi, and Mara Balestrini for filming, editing, and compiling a series of on-the-spot “talking heads” videos, where they posed probing questions to the diverse set of attendees at CHI’11, CHI’14, and CHI’18, including a variety of CHI members from across the globe. The questions included asking about the future of interaction design and whether HCI has gone too wild. There are about 75 of these videos, which can be viewed on our website at www.id-book.com. We are also indebted to danah boyd, Harry Brignull, Leah Beuchley, Albrecht Schmidt, Ellen Gottesdie-ner, and Jon Froehlich for generously contributing in-depth, text-based interviews in the book. We would like to thank Rien Sach, who has been our webmaster for several years, and Deb Yuill who did a thoughtful and thorough job of editing the old reference list.
Danelle Bailey and Jill Reed provided thoughtful critiques and suggestions on all the chapters in the fifth edition, and we thank them.
Finally, we would like to thank our editor and the production team at Wiley who have been very supportive and encouraging throughout the process of developing this fifth edition: Jim Minatel, Pete Gaughan, Gary Schwartz, and Barath Kumar Rajasekaran.
Acknowledgments
Contents
What’s Inside? xvii
1 WHAT IS INTERACTION DESIGN? 1
1.1 Introduction 11.2 Good and Poor Design 21.3 What Is Interaction Design? 91.4 The User Experience 131.5 Understanding Users 151.6 Accessibility and Inclusiveness 171.7 Usability and User Experience Goals 19Interview with Harry Brignull 34
2 THE PROCESS OF INTERACTION DESIGN 37
2.1 Introduction 372.2 What Is Involved in Interaction Design? 382.3 Some Practical Issues 55
3 CONCEPTUALIZING INTERACTION 69
3.1 Introduction 693.2 Conceptualizing Interaction 713.3 Conceptual Models 743.4 Interface Metaphors 783.5 Interaction Types 813.6 Paradigms, Visions, Theories, Models, and Frameworks 88Interview with Albrecht Schmidt 97
4 COGNITIVE ASPECTS 101
4.1 Introduction 1014.2 What Is Cognition? 1024.3 Cognitive Frameworks 123
5 SOCIAL INTERACTION 135
5.1 Introduction 1355.2 Being Social 1365.3 Face-to-Face Conversations 139
C O N T E N T Sxiv
5.4 Remote Conversations 1435.5 Co-presence 1505.6 Social Engagement 158
6 EMOTIONAL INTERACTION 165
6.1 Introduction 1656.2 Emotions and the User Experience 1666.3 Expressive Interfaces and Emotional Design 1726.4 Annoying Interfaces 1746.5 Affective Computing and Emotional AI 1796.6 Persuasive Technologies and Behavioral Change 1826.7 Anthropomorphism 187
7 INTERFACES 193
7.1 Introduction 1937.2 Interface Types 1947.3 Natural User Interfaces and Beyond 2527.4 Which Interface? 253Interview with Leah Buechley 257
8 DATA GATHERING 259
8.1 Introduction 2598.2 Five Key Issues 2608.3 Data Recording 2668.4 Interviews 2688.5 Questionnaires 2788.6 Observation 2878.7 Choosing and Combining Techniques 300
9 DATA ANALYSIS, INTERPRETATION, AND PRESENTATION 307
9.1 Introduction 3079.2 Quantitative and Qualitative 3089.3 Basic Quantitative Analysis 3119.4 Basic Qualitative Analysis 3209.5 Which Kind of Analytic Framework to Use? 3299.6 Tools to Support Data Analysis 3419.7 Interpreting and Presenting the Findings 342
C O N T E N T S xv
10 DATA AT SCALE 349
10.1 Introduction 34910.2 Approaches to Collecting and Analyzing Data 35110.3 Visualizing and Exploring Data 36610.4 Ethical Design Concerns 375
11 DISCOVERING REQUIREMENTS 385
11.1 Introduction 38511.2 What, How, and Why? 38611.3 What Are Requirements? 38711.4 Data Gathering for Requirements 39511.5 Bringing Requirements to Life: Personas and Scenarios 40311.6 Capturing Interaction with Use Cases 415Interview with Ellen Gottesdiener 418
12 DESIGN, PROTOTYPING, AND CONSTRUCTION 421
12.1 Introduction 42112.2 Prototyping 42212.3 Conceptual Design 43412.4 Concrete Design 44512.5 Generating Prototypes 44712.6 Construction 457Interview with Jon Froehlich 466
13 INTERACTION DESIGN IN PRACTICE 471
13.1 Introduction 47113.2 AgileUX 47313.3 Design Patterns 48413.4 Open Source Resources 48913.5 Tools for Interaction Design 491
14 INTRODUCING EVALUATION 495
14.1 Introduction 49514.2 The Why, What, Where, and When of Evaluation 49614.3 Types of Evaluation 50014.4 Evaluation Case Studies 50714.5 What Did We Learn from the Case Studies? 51414.6 Other Issues to Consider When Doing Evaluation 516
C O N T E N T Sxvi
15 EVALUATION STUDIES: FROM CONTROLLED TO NATURAL SETTINGS 523
15.1 Introduction 52315.2 Usability Testing 52415.3 Conducting Experiments 53315.4 Field Studies 536Interview with danah boyd 546
16 EVALUATION: INSPECTIONS, ANALYTICS, AND MODELS 549
16.1 Introduction 54916.2 Inspections: Heuristic Evaluation and Walk-Throughs 55016.3 Analytics and A/B Testing 56716.4 Predictive Models 576
References 581
Index 619
What’s Inside?
Welcome to the fifth edition of Interaction Design: beyond human-computer interaction and our interactive website at www.id-book.com. Building on the success of the previous edi-tions, we have substantially updated and streamlined the material in this book to provide a comprehensive introduction to the fast-growing and multi-disciplinary field of interaction design. Rather than let the book expand, however, we have again made a conscious effort to keep it at the same size.
Our textbook is aimed at both professionals who want to find out more about inter-action design and students from a range of backgrounds studying introductory classes in human-computer interaction, interaction design, information and communications technol-ogy, web design, software engineering, digital media, information systems, and information studies. It will appeal to practitioners, designers, and researchers who want to discover what is new in the field or to learn about a specific design approach, method, interface, or topic. It is also written to appeal to a general audience interested in design and technology.
It is called Interaction Design: beyond human-computer interaction because interaction design has traditionally been concerned with a broader scope of issues, topics, and methods than was originally the scope of human-computer interaction (HCI)—although nowadays, the two increasingly overlap in scope and coverage of topics. We define interaction design as follows:
Designing interactive products to support the way people communicate and interact in their everyday and working lives.
Interaction design requires an understanding of the capabilities and desires of people and the kinds of technology that are available. Interaction designers use this knowledge to discover requirements and develop and manage them to produce a design. Our textbook pro-vides an introduction to all of these areas. It teaches practical techniques to support develop-ment as well as discussing possible technologies and design alternatives.
The number of different types of interface and applications available to today’s interac-tion designers continues to increase steadily, so our textbook, likewise, has been expanded to cover these new technologies. For example, we discuss and provide examples of brain, smart, robotic, wearable, shareable, augmented reality, and multimodel interfaces, as well as more traditional desktop, multimedia, and web-based interfaces. Interaction design in practice is changing fast, so we cover a range of processes, issues, and examples throughout the book.
The book has 16 chapters, and it includes discussion of the different design approaches in common use; how cognitive, social, and affective issues apply to interaction design; and how to gather, analyze, and present data for interaction design. A central theme is that design and evaluation are interwoven, highly iterative processes, with some roots in theory but that rely strongly on good practice to create usable products. The book has a hands-on orienta-tion and explains how to carry out a variety of techniques used to design and evaluate the wide range of new applications coming onto the market. It has a strong pedagogical design and includes many activities (with detailed comments) and more complex activities that can form the basis for student projects. There are also “Dilemmas,” which encourage readers to weigh the pros and cons of controversial issues.
W h at ’ s I n s I d e ?xviii
The style of writing throughout the book is intended to be accessible to a range of read-ers. It is largely conversational in nature and includes anecdotes, cartoons, and case studies. Many of the examples are intended to relate to readers’ own experiences. The book and the associated website are also intended to encourage readers to be active when reading and to think about seminal issues. The goal is for readers to understand that much of interaction design needs consideration of the issues, and that they need to learn to weigh the pros and cons and be prepared to make trade-offs. There is rarely a right or wrong answer, although there is a world of difference between a good design and a poor design.
This book is accompanied by a website (www.id-book.com), which provides a variety of resources, including slides for each chapter, comments on chapter activities, and a number of in-depth case studies written by researchers and designers. There are video interviews with a wide range of experts from the field, including professional interaction designers and university professors. Pointers to respected blogs, online tutorials, YouTube videos, and other useful materials are also provided.
tasteRs
We address topics and questions about the what, why, and how of interaction design. These include the following: • Why some interfaces are good and others are poor • Whether people can really multitask • How technology is transforming the way people communicate with one another • What are users’ needs, and how we can design for them • How interfaces can be designed to change people’s behavior • How to choose between the many different kinds of interactions that are now available
(for example, talking, touching, and wearing) • What it means to design accessible and inclusive interfaces • The pros and cons of carrying out studies in the lab versus in the field and in the wild • When to use qualitative and quantitative methods • How to construct informed consent forms • How the type of interview questions posed affects the conclusions that can be drawn from
the answers given • How to move from a set of scenarios and personas to initial low-fidelity prototypes • How to visualize the results of data analysis effectively • How to collect, analyze, and interpret data at scale • Why it is that what people say can be different from what they do • The ethics of monitoring and recording people’s activities • What are Agile UX and Lean UX and how they relate to interaction design • How Agile UX can be practically integrated with interaction design throughout different
stages of the design process
W h at ’ s I n s I d e ? xix
Changes from Previous Editions
To reflect the dynamic nature of the field, the fifth edition has been thoroughly updated, and new examples, images, case studies, dilemmas, and so on, have been included to illustrate the changes. Included in this edition is a new chapter called “Data at Scale.” Collecting data has never been easier. However, knowing what to do with it when designing new user experiences is much more difficult. The chapter introduces key methods for collecting data at scale, dis-cusses how to transform data at scale to be meaningful, and reviews a number of methods for visualizing and exploring data at scale while introducing fundamental design principles for making data at scale ethical. This is positioned just after two chapters that introduce data gathering and data analysis that discuss fundamental methods.
In this edition, the chapter on the Process of Interaction Design has been re-located to Chapter 2 in order to better frame the discussion of interaction design. It has been updated with new process models and modified to fit its new location in the book structure. This means that the other chapters have been renumbered to accommodate this and the new chapter.
Chapter 13, “Interaction Design in Practice,” has been updated to reflect recent devel-opments in the use of practical UX methods. Old examples and methods no longer used in the field have been removed to make way for the new material. Some chapters have been completely rewritten, while others have been extensively revised. For example, Chapters 4, 5, and 6 have been substantially updated to reflect new developments in social media and emo-tional interaction, while also covering the new interaction design issues they raise, such as privacy and addiction. Many examples of new interfaces and technologies have been added to Chapter 7. Chapter 8 and Chapter 9 on data gathering and analysis have also been sub-stantially updated. New case studies and examples have been added to Chapters 14–16 to illustrate how evaluation methods have changed for use with the continuously evolving tech-nology that is being developed for today’s users. The interviews accompanying the chapters have been updated, and two new ones are included with leading figures involved in innova-tive research, state-of-the-art design, and contemporary practice.
We have decided to continue to provide both a print-based version of the book and an e-book. Both are in full color. The e-book supports note sharing, annotating, contextualized navigating, powerful search features, inserted videos, links, and quizzes.
W H A T I S I N T E R A C T I O N D E S I G N ?
ObjectivesThe main goals of this chapter are to accomplish the following:
• Explain the difference between good and poor interaction design. • Describe what interaction design is and how it relates to human-computer interactionand other fields.
• Explain the relationship between the user experience and usability. • Introduce what is meant by accessibility and inclusiveness in relation to human- computer interaction.
• Describe what and who is involved in the process of interaction design. • Outline the different forms of guidance used in interaction design. • Enable you to evaluate an interactive product and explain what is good and bad aboutit in terms of the goals and core principles of interaction design.
1.1 Introduction
How many interactive products are there in everyday use? Think for a minute about what you use in a typical day: a smartphone, tablet, computer, laptop, remote control, coffee machine, ticket machine, printer, GPS, smoothie maker, e-reader, smart TV, alarm clock, electric tooth-brush, watch, radio, bathroom scales, fitness tracker, game console . . . the list is endless. Now think for a minute about how usable they are. How many are actually easy, effortless, and
1.1 Introduction
1.2 Good and Poor Design
1.3 What Is Interaction Design?
1.4 The User Experience
1.5 Understanding Users
1.6 Accessibility and Inclusiveness
1.7 Usability and User Experience Goals
Chapter 1
1 W H AT I S I N T E R A C T I O N D E S I G N ?2
enjoyable to use? Some, like the iPad, are a joy to use, where tapping an app and flicking through photos is simple, smooth, and enjoyable. Others, like working out how to buy the cheapest train ticket from a ticket machine that does not recognize your credit card after completing a number of steps and then makes you start again from scratch, can be very frus-trating. Why is there a difference?
Many products that require users to interact with them, such as smartphones and fit-ness trackers, have been designed primarily with the user in mind. They are generally easy and enjoyable to use. Others have not necessarily been designed with the users in mind; rather, they have been engineered primarily as software systems to perform set functions. An example is setting the time on a stove that requires a combination of button presses that are not obvious as to which ones to press together or separately. While they may work effectively, it can be at the expense of how easily they will be learned and therefore used in a real-world context.
Alan Cooper (2018), a well-known user experience (UX) guru, bemoans the fact that much of today’s software suffers from the same interaction errors that were around 20 years ago. Why is this still the case, given that interaction design has been in existence for more than 25 years and that there are far more UX designers now in industry than ever before? He points out how many interfaces of new products do not adhere to the interaction design principles validated in the 1990s. For example, he notes that many apps do not follow even the most basic of UX principles, such as offering an “undo” option. He exclaims that it is “inexplicable and unforgivable that these violations continue to resurface in new prod-ucts today.”
How can we rectify this situation so that the norm is that all new products are designed to provide good user experiences? To achieve this, we need to be able to understand how to reduce the negative aspects (such as frustration and annoyance) of the user experience while enhancing the positive ones (for example, enjoyment and efficacy). This entails devel-oping interactive products that are easy, effective, and pleasurable to use from the users’ perspective.
In this chapter, we begin by examining the basics of interaction design. We look at the difference between good and poor design, highlighting how products can differ radically in how usable and enjoyable they are. We then describe what and who is involved in the process of interaction design. The user experience, which is a central concern of interaction design, is then introduced. Finally, we outline how to characterize the user experience in terms of usability goals, user experience goals, and design principles. An in-depth activity is presented at the end of the chapter in which you have the opportunity to put into practice what you have read by evaluating the design of an interactive product.
1.2 Good and Poor Design
A central concern of interaction design is to develop interactive products that are usable. By this we mean products that are generally easy to learn, effective to use, and provide an enjoy-able user experience. A good place to start thinking about how to design usable interactive products is to compare examples of well-designed and poorly designed ones. Through identi-fying the specific weaknesses and strengths of different interactive products, we can begin to
1 . 2 G O O D A N D P O O R D E S I G N 3
understand what it means for something to be usable or not. Here, we describe two examples of poorly designed products that have persisted over the years—a voice-mail system used in hotels and the ubiquitous remote control—and contrast these with two well-designed exam-ples of the same products that perform the same function.
1.2.1 Voice-Mail SystemImagine the following scenario. You are staying at a hotel for a week while on a business trip. You see a blinking red light on the landline phone beside the bed. You are not sure what this means, so you pick up the handset. You listen to the tone and it goes “beep, beep, beep.” Maybe this means that there is a message for you. To find out how to access the message, you have to read a set of instructions next to the phone. You read and follow the first step:
1. Touch 41.The system responds: “You have reached the Sunny Hotel voice message center. Please enter the room number for which you would like to leave a message.”You wait to hear how to listen to a recorded message. But there are no further instruc-tions from the phone. You look down at the instruction sheet again and read:2. Touch*, your room number, and #.You do so and the system replies: “You have reached the mailbox for room 106. To leave a message, type in your password.”
You type in the room number again, and the system replies: “Please enter room number again and then your password.”
You don’t know what your password is. You thought it was the same as your room num-ber, but clearly it is not. At this point, you give up and call the front desk for help. The person at the desk explains the correct procedure for listening to messages. This involves typing in, at the appropriate times, the room number and the extension number of the phone (the latter is the password, which is different from the room number). Moreover, it takes six steps to access a message. You give up.
What is problematic with this voice-mail system?
• It is infuriating. • It is confusing. • It is inefficient, requiring you to carry out a number of steps for basic tasks. • It is difficult to use. • It has no means of letting you know at a glance whether any messages have been left orhow many there are. You have to pick up the handset to find out and then go through aseries of steps to listen to them.
• It is not obvious what to do: The instructions are provided partially by the system andpartially by a card beside the phone.
Now compare it to the phone answering machine shown in Figure 1.1 The illustration shows a small sketch of a phone answering machine. Incoming messages are represented using marbles. The number of marbles that have moved into the pinball-like chute indicates the number of messages. Placing one of these marbles into a dent on the machine causes the recorded message to play. Dropping the same marble into a different dent on the phone dials the caller who left the message.
1 W H AT I S I N T E R A C T I O N D E S I G N ?4
How does the marble answering machine differ from the voice-mail system?
• It uses familiar physical objects that indicate visually at a glance how many messages have been left.
• It is aesthetically pleasing and enjoyable to use. • It requires only one-step actions to perform core tasks. • It is a simple but elegant design. • It offers less functionality and allows anyone to listen to any of the messages.
The marble answering machine is considered a design classic. It was created by Durrell Bishop while he was a student at the Royal College of Art in London (described by Crampton Smith, 1995). One of his goals was to design a messaging system that represented its basic functionality in terms of the behavior of everyday objects. To do this, he capitalized on people’s everyday knowledge of how the physical world works. In particular, he made use of the ubiq-uitous everyday action of picking up a physical object and putting it down in another place.
This is an example of an interactive product designed with the users in mind. The focus is on providing them with a pleasurable experience but one that also makes efficient the activity of receiving messages. However, it is important to note that although the marble answering machine is an elegant and usable design, it would not be practical in a hotel setting. One of the main reasons is that it is not robust enough to be used in public places; for instance, the marbles could easily get lost or be taken as souvenirs. Also, the need to identify the user before allowing the messages to be played is essential in a hotel setting.
Therefore, when considering the design of an interactive product, it is important to con-sider where it is going to be used and who is going to use it. The marble answering machine would be more suitable in a home setting—provided that there were no children around who might be tempted to play with the marbles!
Video Durrell Bishop’s answering machine: http://vimeo.com/19930744.
Figure 1.1 The marble answering machine Source: Adapted from Crampton Smith (1995)
1 . 2 G O O D A N D P O O R D E S I G N 5
1.2.2 Remote ControlEvery home entertainment system, be it the smart TV, set-top box, stereo system, and so forth, comes with its own remote control. Each one is different in terms of how it looks and works. Many have been designed with a dizzying array of small, multicolored, and double-labeled buttons (one on the button and one above or below it) that often seem arbitrarily positioned in relation to one another. Many viewers, especially when sitting in their living rooms, find it difficult to locate the right ones, even for the simplest of tasks, such as pausing or finding the main menu. It can be especially frustrating for those who need to put on their reading glasses each time to read the buttons. The remote control appears to have been put together very much as an afterthought.
In contrast, much effort and thought went into the design of the classic TiVo remote con-trol with the user in mind (see Figure 1.2). TiVo is a digital video recorder that was originally developed to enable the viewer to record TV shows. The remote control was designed with large buttons that were clearly labeled and logically arranged, making them easy to locate and use in conjunction with the menu interface that appeared on the TV screen. In terms of its physical form, the remote device was designed to fit into the palm of a hand, having a peanut shape. It also has a playful look and feel about it: colorful buttons and cartoon icons are used that are distinctive, making it easy to identify them.
Figure 1.2 The TiVo remote control Source: https://business.tivo.com/
1 W H AT I S I N T E R A C T I O N D E S I G N ?6
How was it possible to create such a usable and appealing remote device where so many others have failed? The answer is simple: TiVo invested the time and effort to follow a user-centered design process. Specifically, TiVo’s director of product design at the time involved potential users in the design process, getting their feedback on everything from the feel of the device in the hand to where best to place the batteries, making them easy to replace but not prone to falling out. He and his design team also resisted the trap of “buttonitis” to which so many other remote controls have fallen victim; that is one where buttons breed like rabbits—a button for every new function. They did this by restricting the number of control buttons embedded in the device to the essential ones. Other functions were then represented as part of the menu options and dialog boxes displayed on the TV screen, which could then be selected via the core set of physical control buttons. The result was a highly usable and pleasing device that has received much praise and numerous design awards.
DILEMMAWhat Is the Best Way to Interact with a Smart TV?
A challenge facing smart TV providers is how to enable users to interact with online content. Viewers can select a whole range of content via their TV screens, but it involves scrolling through lots of menus and screens. In many ways, the TV interface has become more like a computer interface. This raises the question of whether the remote control is the best input device to use for someone who sits on a sofa or chair that is some distance from the wide TV screen. Smart TV developers have addressed this challenge in a num-ber of ways.
An early approach was to provide an on-screen keyboard and numeric keypad that pre-sented a grid of alphanumeric characters (see Figure 1.3a), which were selected by pressing a button repeatedly on a remote control. However, entering the name of a movie or an email address and password using this method can be painstakingly slow; it is also easy to overshoot and select the wrong letter or number when holding a button down on the remote to reach a target character.
More recent remote controls, such as those provided by Apple TV, incorporate a touchpad to enable swiping akin to the control commonly found on laptops. While this form of touch control expedites skipping through a set of letters displayed on a TV screen, it does not make it any easier to type in an email address and password. Each letter, number, or special character still has to be selected. Swiping is also prone to overshoot-ing when aiming for a target letter, number, or character. Instead of providing a grid, the Apple TV interface displays two single lines of letters, numbers, and special characters to swipe across (see Figure 1.3b). While this can make it quicker for someone to reach a character, it is still tedious to select a sequence of characters in this way. For example, if you select a Y and the next letter is an A, you have to swipe all the way back to the begin-ning of the alphabet.
1 . 2 G O O D A N D P O O R D E S I G N 7
1.2.1 What to DesignDesigning interactive products requires considering who is going to be using them, how they are going to be used, and where they are going to be used. Another key concern is to understand the kind of activities people are doing when interacting with these prod-ucts. The appropriateness of different kinds of interfaces and arrangements of input and output devices depends on what kinds of activities are to be supported. For example, if the activity is to enable people to bank online, then an interface that is secure, trust-worthy, and easy to navigate is essential. In addition, an interface that allows the user to find out information about new services offered by the bank without it being intrusive would be useful.
Might there be a better way to interact with a smart TV while sitting on the sofa? An alternative is to use voice control. Remote controls, like Siri or TiVo, for example, have a speech button that when pressed allows viewers to ask for movies by name or more generally by category, for instance, “What are the best sci-fi movies on Netflix?” Smart speakers, such as Amazon Echo, can also be connected to a smart TV via an HDMI port, and, similarly, the user can ask for something general or more specific, for example, “Alexa, play Big Bang The-ory, Season 6, Episode 5, on the TV.” On recognizing the command, it will switch on the TV, switch to the right HDMI channel, open Netflix, and begin streaming the specific episode. Some TV content, however, requires the viewer to say that they are over a certain age by checking a box on the TV display. If the TV could ask the viewer and check that they are over 18, then that would be really smart! Also, if the TV needs the viewer to provide a password to access on-demand content, they won’t want to say it out aloud, character by character, espe-cially in front of others who might also be in the room with them. The use of biometrics, then, may be the answer.
(a) (b)
Figure 1.3 Typing on a TV screen (a) by selecting letters and numbers from a square matrix and (b) by swiping along a single line of letters and numbers Source: (b) https://support.apple.com/en-us/HT200107
1 W H AT I S I N T E R A C T I O N D E S I G N ?8
The world is becoming suffused with technologies that support increasingly diverse activities. Just think for a minute about what you can currently do using digital technology: send messages, gather information, write essays, control power plants, program, draw, plan, calculate, monitor others, and play games—just to name but a few. Now think about the types of interfaces and interactive devices that are available. They too are equally diverse: multitouch displays, speech-based systems, handheld devices, wearables, and large interactive displays—again, to name but a few. There are also many ways of designing how users can interact with a system, for instance, via the use of menus, commands, forms, icons, gestures, and so on. Furthermore, ever more innovative everyday artifacts are being created using novel materials, such as e-textiles and wearables (see Figure 1.4).
The Internet of Things (IoT) now means that many products and sensors can be con-nected to each other via the Internet, which enables them to talk to each other. Popular household IoT-enabled products include smart heating and lighting and home security sys-tems where users can change the controls from an app on their phone or check out who is knocking on their door via a doorbell webcam. Other apps that are being developed are meant to make life easier for people, like finding a car parking space in busy areas.
The interfaces for everyday consumer items, such as cameras, microwave ovens, toasters, and washing machines, which used to be physical and the realm of product design, are now predominantly digitally based, requiring interaction design (called consumer electronics). The move toward transforming human-human transactions into solely interface-based ones has also introduced a new kind of customer interaction. Self-checkouts at grocery stores and librar-ies are now the norm where it is commonplace for customers to check out their own goods or books themselves, and at airports, where passengers check in their own luggage. While more cost-effective and efficient, it is impersonal and puts the onus on the person to interact with the system. Furthermore, accidentally pressing the wrong button or standing in the wrong place at a self-service checkout can result in a frustrating, and sometimes mortifying, experience.
Figure 1.4 Turn signal biking jacket using e-textiles developed by Leah Beuchley Source: Used courtesy of Leah Buechley
1 . 3 W H AT I S I N T E R A C T I O N D E S I G N ? 9
What this all amounts to is a multitude of choices and decisions that interaction design-ers have to make for an ever-increasing range of products. A key question for interaction design is this: “How do you optimize the users’ interactions with a system, environment, or product so that they support the users’ activities in effective, useful, usable and pleasurable ways?” One could use intuition and hope for the best. Alternatively, one can be more prin-cipled in deciding which choices to make by basing them on an understanding of the users. This involves the following:
• Considering what people are good and bad at • Considering what might help people with the way they currently do things • Thinking through what might provide quality user experiences • Listening to what people want and getting them involved in the design • Using user-centered techniques during the design process
The aim of this book is to cover these aspects with the goal of showing you how to carry out interaction design. In particular, it focuses on how to identify users’ needs and the context of their activities. From this understanding, we move on to consider how to design usable, useful, and pleasurable interactive products.
1.3 What Is Interaction Design?
By interaction design, we mean the following:
Designing interactive products to support the way people communicate and interact in their everyday and working lives
Put another way, it is about creating user experiences that enhance and augment the way people work, communicate, and interact. More generally, Terry Winograd originally described it as “designing spaces for human communication and interaction” (1997, p. 160). John Thackara viewed it as “the why as well as the how of our daily interactions using com-puters” (2001, p. 50), while Dan Saffer emphasized its artistic aspects: “the art of facilitating interactions between humans through products and services” (2010, p. 4).
A number of terms have been used since to emphasize different aspects of what is being designed, including user interface design (UI), software design, user-centered design, product design, web design, user experience design, and interactive system design. Interaction design is generally used as the overarching term to describe the field, including its methods, theories, and approaches. UX is used more widely in industry to refer to the profession. However, the terms can be used interchangeably. Also, it depends on their ethos and brand.
1.3.1 The Components of Interaction DesignWe view interaction design as fundamental to many disciplines, fields, and approaches that are concerned with researching and designing computer-based systems for people. Figure 1.5 presents the core ones along with interdisciplinary fields that comprise one or more of these, such as cognitive ergonomics. It can be confusing to try to work out the differences between them as many overlap. The main differences between interaction design and the other approaches referred to in the figure come largely down to which methods, philosophies, and lenses they use to study, analyze, and design products. Another way they vary is in terms of
1 W H AT I S I N T E R A C T I O N D E S I G N ?10
the scope and problems they address. For example, information systems is concerned with the application of computing technology in domains such as business, health, and education, whereas ubiquitous computing is concerned with the design, development, and deployment of pervasive computing technologies (for example, IoT) and how they facilitate social inter-actions and human experiences.
BOX 1.1 Is Interaction Design Beyond HCI?
We see the main difference between interaction design (ID) and human-computer interaction (HCI) as one of scope. Historically, HCI had a narrow focus on the design and usability of computing systems, while ID was seen as being broader, concerned with the theory, research, and practice of designing user experiences for all manner of technologies, systems, and prod-ucts. That is one of the reasons why we chose to call our book Interaction Design: beyond human-computer interaction, to reflect this wider range. However, nowadays, HCI has greatly expanded in its scope (Churchill et al., 2013), so much so that it overlaps much more with ID (see Figure 1.6).
InteractionDesign
AcademicDisciplines
ErgonomicsPsychology/
Cognitive Science
InformaticsDesign
Engineering
Computer Science/Software Engineering
Social Sciences(e.g., Sociology,
Anthropology)
UbiquitousComputing
HumanFactors (HF)
CognitiveEngineering
Human-ComputerInteraction (HCI)
CognitiveErgonomics
InformationSystems
Computer-SupportedCooperativeWork (CSCW)
Film Industry
Industrial Design
Artist-Design
Product Design
Graphic Design
Design Practices
Interdisciplinary Overlapping Fields
Figure 1.5 Relationship among contributing academic disciplines, design practices, and interdisci-plinary fields concerned with interaction design (double-headed arrows mean overlapping)
1 . 3 W H AT I S I N T E R A C T I O N D E S I G N ? 11
1.3.2 Who Is Involved in Interaction Design?Figure 1.5 also shows that many people are involved in performing interaction design, rang-ing from social scientists to movie-makers. This is not surprising given that technology has become such a pervasive part of our lives. But it can all seem rather bewildering to the onlooker. How does the mix of players work together?
Designers need to know many different things about users, technologies, and the interac-tions among them to create effective user experiences. At the least, they need to understand how people act and react to events and how they communicate and interact with each other. To be able to create engaging user experiences, they also need to understand how emotions work, what is meant by aesthetics, desirability, and the role of narrative in human experi-ence. They also need to understand the business side, technical side, manufacturing side, and marketing side. Clearly, it is difficult for one person to be well versed in all of these diverse areas and also know how to apply the different forms of knowledge to the process of interac-tion design.
Interaction design is ideally carried out by multidisciplinary teams, where the skill sets of engineers, designers, programmers, psychologists, anthropologists, sociologists, marketing people, artists, toy makers, product managers, and others are drawn upon. It is rarely the case,
Figure 1.6 HCI out of the box: broadening its reach to cover more areas
1 W H AT I S I N T E R A C T I O N D E S I G N ?12
however, that a design team would have all of these professionals working together. Who to include in a team will depend on a number of factors, including a company’s design philoso-phy, size, purpose, and product line.
One of the benefits of bringing together people with different backgrounds and training is the potential of many more ideas being generated, new methods developed, and more crea-tive and original designs being produced. However, the downside is the costs involved. The more people there are with different backgrounds in a design team, the more difficult it can be to communicate and make progress with the designs being generated. Why? People with different backgrounds have different perspectives and ways of seeing and talking about the world. What one person values as important others may not even see (Kim, 1990). Similarly, a computer scientist’s understanding of the term representation is often very different from that of a graphic designer or psychologist.
What this means in practice is that confusion, misunderstanding, and communication breakdowns can surface in a team. The various team members may have different ways of talking about design and may use the same terms to mean quite different things. Other problems can arise when a group of people who have not previously worked as a team are thrown together. For example, Aruna Balakrishnan et al. (2011) found that integration across different disciplines and expertise is difficult in many projects, especially when it comes to agreeing on and sharing tasks. The more disparate the team members—in terms of culture, background, and organizational structures—the more complex this is likely to be.
ACTIVITY 1.1 In practice, the makeup of a given design team depends on the kind of interactive product being built. Who do you think should be involved in developing • A public kiosk providing information about the exhibits available in a science museum? • An interactive educational website to accompany a TV series?
CommentIdeally, each team will have a number of different people with different skill sets. For example, the first interactive product would include the following individuals: • Graphic and interaction designers, museum curators, educational advisers, software engi-neers, software designers, and ergonomists
The second project would include these types of individuals: • TV producers, graphic and interaction designers, teachers, video experts, software engi-neers, and software designers
In addition, as both systems are being developed for use by the general public, representa-tive users, such as school children and parents, should be involved.
In practice, design teams often end up being quite large, especially if they are working on a big project to meet a fixed deadline. For example, it is common to find teams of 15 or more people working on a new product like a health app. This means that a number of people from each area of expertise are likely to be working as part of the project team.
1 . 4 T H E U S E R E X P E R I E N C E 13
1.3.3 Interaction Design ConsultanciesInteraction design is now widespread in product and services development. In particular, website consultants and the computing industries have realized its pivotal role in successful interactive products. But it is not just IT companies that are realizing the benefits of having UXers on board. Financial services, retail, governments, and the public sector have realized too the value of interaction design. The presence or absence of good interaction design can make or break a company. Getting noticed in the highly competitive field of web products requires standing out. Being able to demonstrate that your product is easy, effective, and engaging to use is seen as central to this. Marketing departments are also realizing how branding, the number of hits, the customer return rate, and customer satisfaction are greatly affected by the usability of a website.
There are many interaction design consultancies now. These include established compa-nies, such as Cooper, NielsenNorman Group, and IDEO, and more recent ones that specialize in a particular area, such as job board software (for example, Madgex), digital media (think of Cogapp), or mobile design (such as CXpartners). Smaller consultancies, such as Bunnyfoot and Dovetailed, promote diversity, interdisciplinarity, and scientific user research, having psycholo-gists, researchers, interaction designers, usability, and customer experience specialists on board.
Many UX consultancies have impressive websites, providing case studies, tools, and blogs. For example, Holition publishes an annual glossy booklet as part of its UX Series (Javornik et al., 2017) to disseminate the outcomes of their in-house research to the wider community, with a focus on the implications for commercial and cultural aspects. This shar-ing of UX knowledge enables them to contribute to the discussion about the role of technol-ogy in the user experience.
1.4 The User Experience
The user experience refers to how a product behaves and is used by people in the real world. Jakob Nielsen and Don Norman (2014) define it as encompassing “all aspects of the end-user’s interaction with the company, its services, and its products.” As stressed by Jesse Gar-rett (2010, p. 10), “Every product that is used by someone has a user experience: newspapers, ketchup bottles, reclining armchairs, cardigan sweaters.” More specifically, it is about how people feel about a product and their pleasure and satisfaction when using it, looking at it, holding it, and opening or closing it. It includes their overall impression of how good it is to use, right down to the sensual effect small details have on them, such as how smoothly a switch rotates or the sound of a click and the touch of a button when pressing it. An impor-tant aspect is the quality of the experience someone has, be it a quick one, such as taking a photo; a leisurely one, such as playing with an interactive toy; or an integrated one, such as visiting a museum (Law et al., 2009).
It is important to point out that one cannot design a user experience, only design for a user experience. In particular, one cannot design a sensual experience, but only create the design features that can evoke it. For example, the outside case of a smartphone can be designed to be smooth, silky, and fit in the palm of a hand; when held, touched, looked at, and interacted with, that can provoke a sensual and satisfying user experience. Conversely, if it is designed to be heavy and awkward to hold, it is much more likely to end up providing a poor user experience—one that is uncomfortable and unpleasant.
1 W H AT I S I N T E R A C T I O N D E S I G N ?14
Designers sometimes refer to UX as UXD. The addition of the D to UX is meant to encourage design thinking that focuses on the quality of the user experience rather than on the set of design methods to use (Allanwood and Beare, 2014). As Don Norman (2004) has stressed for many years, “It is not enough that we build products that function, that are understandable and usable, we also need to build joy and excitement, pleasure and fun, and yes, beauty to people’s lives.”
ACTIVITY 1.2
The iPod PhenomenonApple’s classic (and subsequent) generations of portable music players, called iPods, including the iPod Touch, Nano, and Shuffle, released during the early 2000s were a phenomenal success. Why do you think this occurred? Has there been any other product that has matched this quality of experience? With the exception of the iPod Touch, Apple stopped production of them in 2017. Playing music via a smartphone became the norm, superseding the need for a separate device.
CommentApple realized early on that successful interaction design involves creating interactive prod-ucts that have a quality user experience. The sleek appearance of the iPod music player (see Figure 1.7), its simplicity of use, its elegance in style, its distinct family of rainbow colors, a novel interaction style that many people discovered was a sheer pleasure to learn and use, and the catchy naming of its product and content (iTunes, iPod), among many other design features, led to it becoming one of the greatest products of its kind and a must-have fashion item for teenagers, students, and adults alike. While there were many competing players on the market at the time—some with more powerful functionality, others that were cheaper and easier to use, or still others with bigger screens, more memory, and so forth—the quality of the overall user experience paled in comparison to that provided by the iPod.
Figure 1.7 The iPod Nano Source: David Paul Morris / Getty Images
1 . 5 U N D E R S TA N D I N G U S E R S 15
There are many aspects of the user experience that can be considered and many ways of taking them into account when designing interactive products. Of central importance are the usability, functionality, aesthetics, content, look and feel, and emotional appeal. In addition, Jack Carroll (2004) stresses other wide-reaching aspects, including fun, health, social capital (the social resources that develop and are maintained through social networks, shared values, goals, and norms), and cultural identity, such as age, ethnicity, race, disability, family status, occupation, and education.
Several researchers have attempted to describe the experiential aspect of a user experi-ence. Kasper Hornbæk and Morten Hertzum (2017) note how it is often described in terms of the way that users perceive a product, such as whether a smartwatch is seen as sleek or chunky, and their emotional reaction to it, such as whether people have a positive experi-ence when using it. Marc Hassenzahl’s (2010) model of the user experience is the most well-known, where he conceptualizes it in terms of pragmatic and hedonic aspects. By pragmatic, it is meant how simple, practical, and obvious it is for the user to achieve their goals. By hedonic, it is meant how evocative and stimulating the interaction is to them. In addition to a person’s perceptions of a product, John McCarthy and Peter Wright (2004) discuss the importance of their expectations and the way they make sense of their experiences when using technology. Their Technology as Experience framework accounts for the user experi-ence largely in terms of how it is felt by the user. They recognize that defining experience is incredibly difficult because it is so nebulous and ever-present to us, just as swimming in water is to a fish. Nevertheless, they have tried to capture the essence of human experience by describing it in both holistic and metaphorical terms. These comprise a balance of sensual, cerebral, and emotional threads.
How does one go about producing quality user experiences? There is no secret sauce or magical formula that can be readily applied by interaction designers. However, there are numerous conceptual frameworks, tried and tested design methods, guidelines, and relevant research findings, which are described throughout the book.
1.5 Understanding Users
A main reason for having a better understanding of people in the contexts in which they live, work, and learn is that it can help designers understand how to design interactive products that provide good user experiences or match a user’s needs. A collaborative planning tool for a space mission, intended to be used by teams of scientists working in different parts of the world, will have quite different needs from one targeted at customer and sales agents, to be used in a furniture store to draw up kitchen layout plans. Understanding individual
The nearest overall user experience that has all of the above is not so much for a product but for a physical store. The design of the Apple Store as a completely new customer experi-ence for buying technology has been very successful in how it draws people in and what they do when browsing, discovering, and purchasing goods in the store. The products are laid out in a way to encourage interaction.
1 W H AT I S I N T E R A C T I O N D E S I G N ?16
differences can also help designers appreciate that one size does not fit all; what works for one user group may be totally inappropriate for another. For example, children have different expectations than adults about how they want to learn or play. They may find having inter-active quizzes and cartoon characters helping them along to be highly motivating, whereas most adults find them annoying. Conversely, adults often like talking-head discussions about topics, but children find them boring. Just as everyday objects like clothes, food, and games are designed differently for children, teenagers, and adults, so too should interactive products be designed for different kinds of users.
Learning more about people and what they do can also reveal incorrect assumptions that designers may have about particular user groups and what they need. For example, it is often assumed that because of deteriorating vision and dexterity, old people want things to be big—be it text or graphical elements appearing on a screen or the physical controls, like dials and switches, used to control devices. This may be true for some elderly people, but studies have shown that many people in their 70s, 80s, and older are perfectly capa-ble of interacting with standard-size information and even small interfaces, for example, smartphones, just as well as those in their teens and 20s, even though, initially, some might think they will find it difficult (Siek et al., 2005). It is increasingly the case that as people get older, they do not like to consider themselves as lacking in cognitive and manual skills. Being aware of people’s sensitivities, such as aging, is as important as knowing how to design for their capabilities (Johnson and Finn, 2017). In particular, while many older adults now feel comfortable with and use a range of technologies (for instance, email, online shop-ping, online games, or social media), they may resist adopting new technologies. This is not because they don’t perceive them as being useful to their lives but because they don’t want to waste their time getting caught up by the distractions that digital life brings (Knowles and Hanson, 2018), for example, not wanting to be “glued to one’s mobile phone” like younger generations.
Being aware of cultural differences is also an important concern for interaction design, particularly for products intended for a diverse range of user groups from different countries. An example of a cultural difference is the dates and times used in different countries. In the United States, for example, the date is written as month, day, year (05/21/20), whereas in other countries, it is written in the sequence of day, month, year (21/05/20). This can cause problems for designers when deciding on the format of online forms, especially if intended for global use. It is also a concern for products that have time as a function, such as operating systems, digital clocks, or car dashboards. To which cultural group do they give preference? How do they alert users to the format that is set as default? This raises the question of how easily an interface designed for one user group can be used and accepted by another. Why is it that certain products, like a fitness tracker, are universally accepted by people from all parts of the world, whereas websites are designed differently and reacted to differently by people from different cultures?
To understand more about users, we have included three chapters (Chapters 4–6) that explain in detail how people act and interact with one another, with information, and with various technologies, together with describing their abilities, emotions, needs, desires, and what causes them to get annoyed, frustrated, lose patience, and get bored. We draw upon relevant psychological theory and social science research. Such knowledge enables designers to determine which solutions to choose from the many design alternatives available and how to develop and test these further.
1 . 6 A C C E S S I B I L I T Y A N D I N C L U S I V E N E S S 17
1.6 Accessibility and Inclusiveness
Accessibility refers to the extent to which an interactive product is accessible by as many people as possible. Companies like Google and Apple provide tools for their developers to promote this. The focus is on people with disabilities. For example, Android OS provides a range of tools for those with disabilities, such as hearing aid compatibility to a built-in screen reader, while Apple VoiceOver lets the user know what’s happening on its devices, so they can easily navigate and even know who is in a selfie just taken, by listening to the phone. Inclusiveness means being fair, open, and equal to everyone. Inclusive design is an over-arching approach where designers strive to make their products and services accommodate the widest possible number of people. An example is ensuring that smartphones are being designed for all and made available to everyone—regardless of their disability, education, age, or income.
Whether or not a person is considered to be disabled changes over time with age, or as recovery from an accident progresses throughout their life. In addition, the severity and impact of an impairment can vary over the course of a day or in different environmental conditions. Disability can result because technologies are often designed in such a way as to necessitate a certain type of interaction that is impossible for someone with an impairment. Disability in this context is viewed as the result of poor interaction design between a user and the technology, not the impairment alone. Accessibility, on the other hand, opens up experi-ences so that they are accessible to all. Technologies that are now mainstream once started out as solutions to accessibility challenges. For example, SMS was designed for hearing-impaired people before it became a mainstream technology. Furthermore, designing for accessibility inherently results in inclusive design for all.
Accessibility can be achieved in two ways: first, through the inclusive design of technology, and second, through the design of assistive technology. When designing for accessibility, it is essential to understand the types of impairments that can lead to dis-ability as they come in many forms. They are often classified by the type of impairment, for example:
• Sensory impairment (such as loss of vision or hearing) • Physical impairment (having loss of functions to one or more parts of the body, for exam-ple, after a stroke or spinal cord injury)
• Cognitive (for instance, learning impairment or loss of memory/cognitive function due toold age or a condition such as Alzheimer’s disease)
Within each type is a complex mix of people and capabilities. For example, a person might have only peripheral vision, be color blind, or have no light perception (and be regis-tered blind). All are forms of visual impairment, and all require different design approaches. Color blindness can be overcome by an inclusive design approach. Designers can choose colors that will appear as separate colors to everyone. However, peripheral vision loss or complete blindness will often need an assistive technology to be designed.
Impairment can also be categorized as follows:
• Permanent (for example, long-term wheelchair user) • Temporary (such as after an accident or illness) • Situational (for instance, a noisy environment means a person can’t hear)
1 W H AT I S I N T E R A C T I O N D E S I G N ?18
The number of people living with permanent disability increases with age. Fewer than 20 percent of people are born with a disability, whereas 80 percent of people will have a disability once they reach 85. As people age, their functional abilities diminish. For exam-ple, people older than 50 often find it difficult to hear conversations in rooms with hard surfaces and lots of background noise. This is a disability that will come to most of us at some point.
People with permanent disabilities often use assistive technology in their everyday life, which they consider to be life-essential and an extension of their self (Holloway and Dawes, 2016). Examples include wheelchairs (people now refer to “wearing their wheels,” rather than “using a wheelchair”) and augmented and alternative communication aids. Much cur-rent HCI research into disability explores how new technologies, such as IoT, wearables, and virtual reality, can be used to improve upon existing assistive technologies.
Aimee Mullens is an athlete, actor, and fashion model who has shown how prosthetics can be designed to move beyond being purely functional (and often ugly) to being desirable and highly fashionable. She became a bilateral amputee when her legs were amputated below the knee as a one-year-old. She has done much to blur the boundary between disabled and nondisabled people, and she uses fashion as a tool to achieve this. Several prosthetic compa-nies now incorporate fashion design into their products, including striking leg covers that are affordable by all (see Figure 1.8).
Figure 1.8 Fashionable leg cover designed by Alleles Design Studio Source: https://alleles.ca/. Used courtesy of Alison Andersen
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 19
1.7 Usability and User Experience Goals
Part of the process of understanding users is to be clear about the primary objective of devel-oping an interactive product for them. Is it to design an efficient system that will allow them to be highly productive in their work? Is it to design a learning tool that will be challenging and motivating? Or, is it something else? To help identify the objectives, we suggest classify-ing them in terms of usability and user experience goals. Traditionally, usability goals are concerned with meeting specific usability criteria, such as efficiency, whereas user experience goals are concerned with explicating the nature of the user experience, for instance, to be aesthetically pleasing. It is important to note, however, that the distinction between the two types of goals is not clear-cut since usability is often fundamental to the quality of the user experience and, conversely, aspects of the user experience, such as how it feels and looks, are inextricably linked with how usable the product is. We distinguish between them here to help clarify their roles but stress the importance of considering them together when designing for a user experience. Also, historically HCI was concerned primarily with usability, but it has since become concerned with understanding, designing for, and evaluating a wider range of user experience aspects.
1.7.1 Usability GoalsUsability refers to ensuring that interactive products are easy to learn, effective to use, and enjoyable from the user’s perspective. It involves optimizing the interactions people have with interactive products to enable them to carry out their activities at work, at school, and in their everyday lives. More specifically, usability is broken down into the following six goals:
• Effective to use (effectiveness) • Efficient to use (efficiency) • Safe to use (safety) • Having good utility (utility) • Easy to learn (learnability) • Easy to remember how to use (memorability)
Usability goals are typically operationalized as questions. The purpose is to provide the interaction designer with a concrete means of assessing various aspects of an interactive product and the user experience. Through answering the questions, designers can be alerted very early on in the design process to potential design problems and conflicts that they might not have considered. However, simply asking “Is the system easy to learn?” is not going to be very helpful. Asking about the usability of a product in a more detailed way—for example, “How long will it take a user to figure out how to use the most basic functions for a new smartwatch; how much can they capitalize on from their prior experience; and how long would it take the user to learn the whole set of functions?”—will elicit far more information.
The following are descriptions of the usability goals and a question for each one:
(i) Effectiveness is a general goal, and it refers to how good a product is at doing what it is supposed to do.Question: Is the product capable of allowing people to learn, carry out their work effi-
ciently, access the information that they need, or buy the goods that they want?
1 W H AT I S I N T E R A C T I O N D E S I G N ?20
(ii) Efficiency refers to the way a product supports users in carrying out their tasks. The marble answering machine described earlier in this chapter was considered efficient in that it let the user carry out common tasks, for example, listening to messages, through a minimal number of steps. In contrast, the voice-mail system was considered inefficient because it required the user to carry out many steps and learn an arbitrary set of sequences for the same common task. This implies that an efficient way of supporting common tasks is to let the user use single button or key presses. An example of where this kind of efficiency mechanism has been employed effectively is in online shopping. Once users have entered all of the necessary personal details in an online form to make a purchase, they can let the website save all of their personal details. Then, if they want to make another purchase at that site, they don’t have to re-enter all of their personal details. A highly successful mechanism patented by Amazon.com is the one-click option, which requires users to click only a single button when they want to make another purchase.Question: Once users have learned how to use a product to carry out their tasks, can they
sustain a high level of productivity? (iii) Safety involves protecting the user from dangerous conditions and undesirable situa-
tions. In relation to the first ergonomic aspect, it refers to the external conditions where people work. For example, where there are hazardous conditions—such as X-ray machines or toxic chemicals—operators should be able to interact with and control computer-based systems remotely. The second aspect refers to helping any kind of user in any kind of situation to avoid the dangers of carrying out unwanted actions acciden-tally. It also refers to the perceived fears that users might have of the consequences of making errors and how this affects their behavior. Making interactive products safer in this sense involves (1) preventing the user from making serious errors by reducing the risk of wrong keys/buttons being mistakenly activated (an example is not placing the quit or delete-file command right next to the save command on a menu) and (2) provid-ing users with various means of recovery should they make errors, such as an undo func-tion. Safe interactive systems should engender confidence and allow the user the opportunity to explore the interface to try new operations (see Figure 1.9a). Another safety mechanism is confirming dialog boxes that give users another chance to consider their intentions (a well-known example is the appearance of a dialog box after issuing the command to delete everything in the trash, saying: “Are you sure you want to remove the items in the Trash permanently?”) (see Figure 1.9b).Question: What is the range of errors that are possible using the product, and what
measures are there to permit users to recover easily from them? (iv) Utility refers to the extent to which the product provides the right kind of functionality
so that users can do what they need or want to do. An example of a product with high utility is an accounting software package that provides a powerful computational tool that accountants can use to work out tax returns. An example of a product with low utility is a software drawing tool that does not allow users to draw freehand but forces them to use a mouse to create their drawings, using only polygon shapes.Question: Does the product provide an appropriate set of functions that will enable users
to carry out all of their tasks in the way they want to do them? (v) Learnability refers to how easy a system is to learn to use. It is well known that people
don’t like spending a long time learning how to use a system. They want to get started right away and become competent at carrying out tasks without too much effort. This is
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 21
especially true for interactive products intended for everyday use (for example social media, email, or a GPS) and those used only infrequently (for instance, online tax forms). To a certain extent, people are prepared to spend a longer time learning more complex systems that provide a wider range of functionality, such as web authoring tools. In these situations, pop-up tutorials can help by providing contextualized step-by-step material with hands-on exercises. A key concern is determining how much time users are prepared to spend learn-ing a product. It seems like a waste if a product provides a range of functionality that the majority of users are unable or unprepared to spend the time learning how to use.Question: Is it possible for the user to work out how to use the product by exploring the
interface and trying certain actions? How hard will it be to learn the whole set of func-tions in this way?
(vi) Memorability refers to how easy a product is to remember how to use, once learned. This is especially important for interactive products that are used infrequently. If users haven’t used an operation for a few months or longer, they should be able to remember or at least rapidly be reminded how to use it. Users shouldn’t have to keep relearning how to carry
(a)
(b)
Figure 1.9 (a) A safe and unsafe menu. Which is which and why? (b) A warning dialog box for Mac OS X
1 W H AT I S I N T E R A C T I O N D E S I G N ?22
out tasks. Unfortunately, this tends to happen when the operations required to be learned are obscure, illogical, or poorly sequenced. Users need to be helped to remember how to do tasks. There are many ways of designing the interaction to support this. For example, users can be helped to remember the sequence of operations at different stages of a task through contextualized icons, meaningful command names, and menu options. Also, structuring options and icons so that they are placed in relevant categories of options, for example, placing all of the drawing tools in the same place on the screen, can help the user remember where to look to find a particular tool at a given stage of a task.Question: What types of interface support have been provided to help users remember
how to carry out tasks, especially for products and operations they use infrequently?
In addition to couching usability goals in terms of specific questions, they are turned into usability criteria. These are specific objectives that enable the usability of a product to be assessed in terms of how it can improve (or not improve) a user’s performance. Examples of commonly used usability criteria are time to complete a task (efficiency), time to learn a task (learnability), and the number of errors made when carrying out a given task over time (memorability). These can provide quantitative indicators of the extent to which productivity has increased, or how work, training, or learning have been improved. They are also useful for measuring the extent to which personal, public, and home-based products support leisure and information gathering activities. However, they do not address the overall quality of the user experience, which is where user experience goals come into play.
1.7.2 User Experience GoalsA diversity of user experience goals has been articulated in interaction design, which covers a range of emotions and felt experiences. These include desirable and undesirable ones, as shown in Table 1.1.
Desirable aspects
Satisfying Helpful Fun
Enjoyable Motivating Provocative
Engaging Challenging Surprising
Pleasurable Enhancing sociability Rewarding
Exciting Supporting creativity Emotionally fulfilling
Entertaining Cognitively stimulating Experiencing flow
Undesirable aspects
Boring Unpleasant
Frustrating Patronizing
Making one feel guilty Making one feel stupid
Annoying Cutesy
Childish Gimmicky
Table 1.1 Desirable and undesirable aspects of the user experience
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 23
Many of these are subjective qualities and are concerned with how a system feels to a user. They differ from the more objective usability goals in that they are concerned with how users experience an interactive product from their perspective, rather than assessing how useful or productive a system is from its own perspective. Whereas the terms used to describe usability goals comprise a small distinct set, many more terms are used to describe the mul-tifaceted nature of the user experience. They also overlap with what they are referring to. In so doing, they offer subtly different options for expressing the way an experience varies for the same activity over time, technology, and place. For example, we may describe listening to music in the shower as highly pleasurable, but consider it more apt to describe listening to music in the car as enjoyable. Similarly, listening to music on a high-end powerful music system may invoke exciting and emotionally fulfilling feelings, while listening to it on a smartphone that has a shuffle mode may be serendipitously enjoyable, especially not know-ing what tune is next. The process of selecting terms that best convey a user’s feelings, state of being, emotions, sensations, and so forth when using or interacting with a product at a given time and place can help designers understand the multifaceted and changing nature of the user experience.
The concepts can be further defined in terms of elements that contribute to making a user experience pleasurable, fun, exciting, and so on. They include attention, pace, play, interactivity, conscious and unconscious control, style of narrative, and flow. The concept of flow (Csikszentmihalyi, 1997) is popular in interaction design for informing the design of user experiences for websites, video games, and other interactive products. It refers to a state of intense emotional involvement that comes from being completely involved in an activity, like playing music, and where time flies. Instead of designing web interfaces to cater to visi-tors who know what they want, they can be designed to induce a state of flow, leading the visitor to some unexpected place, where they become completely absorbed. In an interview with Wired magazine, Mihaly Csikszentmihalyi (1996) uses the analogy of a gourmet meal to describe how a user experience can be designed to be engrossing, “starting off with the appetizers, moving on to the salads and entrées, and building toward dessert and not know-ing what will follow.”
The quality of the user experience may also be affected by single actions performed at an interface. For example, people can get much pleasure from turning a knob that has the perfect level of gliding resistance; they may enjoy flicking their finger from the bottom of a smartphone screen to reveal a new menu, with the effect that it appears by magic, or enjoy the sound of trash being emptied from the trashcan on a screen. These one-off actions can be performed infrequently or several times a day—which the user never tires of doing. Dan Saffer (2014) has described these as micro-interactions and argues that designing these moments of interaction at the interface—despite being small—can have a big impact on the user experience.
ACTIVITY 1.3 There are more desirable than undesirable aspects of the user experience listed in Table 1.1. Why do you think this is so? Should you consider all of these when designing a product?
(Continued)
1 W H AT I S I N T E R A C T I O N D E S I G N ?24
CommentThe two lists we have come up with are not meant to be exhaustive. There are likely to be more—both desirable and undesirable—as new products surface. The reason for there being more of the former is that a primary goal of interaction design is to create positive experi-ences. There are many ways of achieving this.
Not all usability and user experience goals will be relevant to the design and evaluation of an interactive product being developed. Some combinations will also be incompatible. For example, it may not be possible or desirable to design a process control system that is both safe and fun. Recognizing and understanding the nature of the relationship between usability and user experience goals is central to interaction design. It enables designers to become aware of the consequences of pursuing different combinations when designing products and high-lighting potential trade-offs and conflicts. As suggested by Jack Carroll (2004), articulating the interactions of the various components of the user’s experience can lead to a deeper and more significant interpretation of the role of each component.
BOX 1.3 Beyond Usability: Designing to Persuade
Eric Schaffer (2009) argues that we should be focusing more on the user experience and less on usability. He points out how many websites are designed to persuade or influence rather than enable users to perform their tasks in an efficient manner. For example, many online shopping sites are in the business of selling services and products, where a core strategy is to entice people to buy what they might not have thought they needed. Online shopping experi-ences are increasingly about persuading people to buy rather than being designed to make shopping easy. This involves designing for persuasion, emotion, and trust, which may or may not be compatible with usability goals.
This entails determining what customers will do, whether it is to buy a product or renew a membership, and it involves encouraging, suggesting, or reminding the user of things that they might like or need. Many online travel sites try to lure visitors to purchase additional items (such as hotels, insurance, car rental, car parking, or day trips) besides the flight they originally wanted to book, and they will add a list full of tempting graphics to the visitor’s booking form, which then has to be scrolled through before being able to complete the trans-action. These opportunities need to be designed to be eye-catching and enjoyable, in the same way that an array of products are attractively laid out in the aisles of a grocery store that one is required to walk past before reaching one’s desired product.
Some online sites, however, have gone too far, for example, adding items to the cus-tomer’s shopping basket (for example, insurance, special delivery, and care and handling) that the shopper has to deselect if not desired or start all over again. This sneaky add-on approach can often result in a negative experience. More generally, this deceptive approach
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 25
to UX has been described by Harry Brignull as dark patterns (see http://darkpatterns.org/). Shoppers often become annoyed if they notice decisions that add cost to their purchase have been made on their behalf without even being asked. For example, on clicking the unsubscribe button on the website of a car rental company, as indicated in Figure 1.10, the user is taken to another page where they have to uncheck additional boxes and then Update. They are then taken to yet another page where they are asked for their reason. The next screen says “Your email preferences have been updated. Do you need to hire a vehicle?” without letting the user know whether they have been unsubscribed from their mailing list.
(Continued)
Email preferences
y.rogers@ucl.ac.uk
Uncheck the emails you do not want to receive
Newsletters UK
* required fields
NiftyCars Partners offers About your rental
Update
Email preferences
We’d love to get some feedback on why you’re unsubscribing.
Emails were too frequent
Update
Emails were not relevant
I am no longer interested in this content
I never signed up for newsletters from NiftyCars
Figure 1.10 Dark pattern for a car rental company
1 W H AT I S I N T E R A C T I O N D E S I G N ?26
1.7.3 Design PrinciplesDesign principles are used by interaction designers to aid their thinking when designing for the user experience. These are generalizable abstractions intended to orient designers toward thinking about different aspects of their designs. A well-known example is feedback: Products should be designed to provide adequate feedback to the users that informs them about what has already been done so that they know what to do next in the interface. Another one that is important is findability (Morville, 2005). This refers to the degree to which a particular object is easy to discover or locate—be it navigating a website, moving through a building, or finding the delete image option on a digital camera. Related to this is the principle of navigability: Is it obvious what to do and where to go in an interface; are the menus structured in a way that allows the user to move smoothly through them to reach the option they want?
Design principles are derived from a mix of theory-based knowledge, experience, and com-mon sense. They tend to be written in a prescriptive manner, suggesting to designers what to provide and what to avoid at the interface—if you like, the dos and don’ts of interaction design. More specifically, they are intended to help designers explain and improve their designs (Thim-bleby, 1990). However, they are not intended to specify how to design an actual interface, for instance, telling the designer how to design a particular icon or how to structure a web portal, but to act more like triggers for designers, ensuring that they provide certain features in an interface.
A number of design principles have been promoted. The best known are concerned with how to determine what users should see and do when carrying out their tasks using an interactive product. Here we briefly describe the most common ones: visibility, feedback, constraints, consistency, and affordance.
VisibilityThe importance of visibility is exemplified by our contrasting examples at the beginning of the chapter. The voice-mail system made the presence and number of waiting messages invis-ible, while the answering machine made both aspects highly visible. The more visible functions are, the more likely it is that users will be able to know what to do next. Don Norman (1988) describes the controls of a car to emphasize this point. The controls for different operations are clearly visible, such as indicators, headlights, horn, and hazard warning lights, indicating what
The key is to nudge people in subtle and pleasant ways with which they can trust and feel comfortable. Natasha Loma (2018) points out how dark pattern design is “deception and dishonesty by design.” She describes in a TechCrunch article the many kinds of dark patterns that are now used to deceive users. A well-known example that most of us have experienced is unsubscribing from a marketing mailing list. Many sites go to great lengths to make it dif-ficult for you to leave; you think you have unsubscribed, but then you discover that you need to type in your email address and click several more buttons to reaffirm that you really want to quit. Then, just when you think you are safe, they post a survey asking you to answer a few questions about why you want to leave. Like Harry Brignull, she argues that companies should adopt fair and ethical design where users have to opt in to any actions that benefit the company at the expense of the users’ interests.
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 27
can be done. The relationship between the way the controls have been positioned in the car and what they do makes it easy for the driver to find the appropriate control for the task at hand.
In contrast, when functions are out of sight, it makes them more difficult to find and to know how to use. For example, devices and environments that have become automated through the use of sensor technology (usually for hygiene and energy-saving reasons)—like faucets, elevators, and lights—can sometimes be more difficult for people to know how to control, especially how to activate or deactivate them. This can result in people getting caught short and frustrated. Figure 1.11 shows a sign that explains how to use the automatically controlled faucet for what is normally an everyday and well-learned activity. It also states that the faucets cannot be operated if wearing black clothing. It does not explain, however, what to do if you are wearing black clothing! Increasingly, highly visible controlling devices, like knobs, buttons, and switches, which are intuitive to use, have been replaced by invisible and ambiguous activating zones where people have to guess where to move their hands, bod-ies, or feet—on, into, or in front of—to make them work.
FeedbackRelated to the concept of visibility is feedback. This is best illustrated by an analogy to what everyday life would be like without it. Imagine trying to play a guitar, slice bread using a knife, or write using a pen if none of the actions produced any effect for several seconds.
Figure 1.11 A sign in the restrooms at the Cincinnati airportSource: http://www.baddesigns.com
1 W H AT I S I N T E R A C T I O N D E S I G N ?28
There would be an unbearable delay before the music was produced, the bread was cut, or the words appeared on the paper, making it almost impossible for the person to continue with the next strum, cut, or stroke.
Feedback involves sending back information about what action has been done and what has been accomplished, allowing the person to continue with the activity. Various kinds of feedback are available for interaction design—audio, tactile, verbal, visual, and combinations of these. Deciding which combinations are appropriate for different types of activities and interactivities is central. Using feedback in the right way can also provide the necessary vis-ibility for user interaction.
ConstraintsThe design concept of constraining refers to determining ways of restricting the kinds of user inter-action that can take place at a given moment. There are various ways that this can be achieved. A common design practice in graphical user interfaces is to deactivate certain menu options by shading them gray, thereby restricting the user only to actions permissible at that stage of the activity (see Figure 1.12). One of the advantages of this form of constraining is that it prevents the user from selecting incorrect options and thereby reduces the chance of making a mistake.
The use of different kinds of graphical representations can also constrain a person’s interpretation of a problem or information space. For example, flow chart diagrams show which objects are related to which, thereby constraining the way that the information can be perceived. The physical design of a device can also constrain how it is used; for example, the
Figure 1.12 A menu showing restricted availability of options as an example of logical constraining. Gray text indicates deactivated options.Source: https://www.ucl.ac.uk
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 29
external slots in a computer have been designed to allow a cable or card to be inserted in a certain way only. Sometimes, however, the physical constraint is ambiguous, as shown in Fig-ure 1.13. The figure shows part of the back of a computer. There are two sets of connectors; the two on the right are for a mouse and a keyboard. They look identical and are physically constrained in the same way. How do you know which is which? Do the labels help?
ConsistencyThis refers to designing interfaces to have similar operations and use similar elements for achiev-ing similar tasks. In particular, a consistent interface is one that follows rules, such as using the same operation to select all objects. For example, a consistent operation is using the same input action to highlight any graphical object on the interface, such as always clicking the left mouse button. Inconsistent interfaces, on the other hand, allow exceptions to a rule. An example is where certain graphical objects (for example, email messages presented in a table) can be high-lighted only by using the right mouse button, while all other operations are highlighted using the left mouse button. The problem with this kind of inconsistency is that it is quite arbitrary, making it difficult for users to remember and making its use more prone to mistakes.
One of the benefits of consistent interfaces, therefore, is that they are easier to learn and use. Users have to learn only a single mode of operation that is applicable to all objects. This principle works well for simple interfaces with limited operations, such as a portable radio with a small number of operations mapped onto separate buttons. Here, all the user has to do is to learn what each button represents and select accordingly. However, it can be more problematic to apply the concept of consistency to more complex interfaces, especially when many different operations need to be designed. For example, consider how to design an inter-face for an application that offers hundreds of operations, such as a word-processing appli-cation. There is simply not enough space for a thousand buttons, each of which maps to an individual operation. Even if there were, it would be extremely difficult and time-consuming for the user to search through all of them to find the desired operation. A much more effec-tive design solution is to create categories of commands that can be mapped into subsets of operations that can be displayed at the interface, for instance, via menus.
Figure 1.13 Ambiguous constraints on the back of a computer Source: http://www.baddesigns.com
1 W H AT I S I N T E R A C T I O N D E S I G N ?30
AffordanceThis is a term used to refer to an attribute of an object that allows people to know how to use it. For example, a mouse button invites pushing (in so doing, activating clicking) by the way it is physically constrained in its plastic shell. At a simple level, to afford means “to give a clue”’ (Norman, 1988). When the affordances of a physical object are perceptually obvious, it is easy to know how to interact with it. For example, a door handle affords pulling, a cup handle affords grasping, and a mouse button affords pushing. The term has since been much popularized in interaction design, being used to describe how interfaces should make it obvi-ous as to what can be done when using them. For example, graphical elements like buttons, icons, links, and scrollbars are discussed with respect to how to make it appear obvious how they should be used: icons should be designed to afford clicking, scrollbars to afford moving up and down, and buttons to afford pushing.
Don Norman (1999) suggests that there are two kinds of affordance: perceived and real. Physical objects are said to have real affordances, like grasping, that are perceptually obvious and do not have to be learned. In contrast, user interfaces that are screen-based are virtual and do not have these kinds of real affordances. Using this distinction, he argues that it does not make sense to try to design for real affordances at the interface, except when designing physical devices, like control consoles, where affordances like pulling and pressing are help-ful in guiding the user to know what to do. Alternatively, screen-based interfaces are better conceptualized as perceived affordances, which are essentially learned conventions. However, watching a one-year-old swiping smartphone screens, zooming in and out on images with their finger and thumb, and touching menu options suggests that kind of learning comes naturally.
Applying Design Principles in PracticeOne of the challenges of applying more than one of the design principles in interaction design is that trade-offs can arise among them. For example, the more you try to constrain an interface, the less visible information becomes. The same can also happen when trying to apply a single design principle. For example, the more an interface is designed to afford through trying to resemble the way physical objects look, the more it can become clut-tered and difficult to use. It can also be the case that the more an interface is designed to be aesthetic, the less usable it becomes. Consistency can be a problematic design principle; trying to design an interface to be consistent with something can make it inconsistent with something else. Furthermore, sometimes inconsistent interfaces are actually easier to use than consistent interfaces. This is illustrated by Jonathan Grudin’s classic (1989) use of the analogy of where knives are stored in a house. Knives come in a variety of forms, including butter knives, steak knives, table knives, and fish knives. An easy place to put them all and subsequently locate them is in the top drawer by the sink. This makes it easy for everyone to find them and follows a simple consistent rule. But what about the knives that don’t fit or are too sharp to put in the drawer, like carving knives and bread knives? They are placed in a wooden block. And what about the best knives kept only for special occasions? They are placed in the cabinet in another room for safekeeping. And what about other knives like putty knives and paint-scraping knives used in home improvement projects (kept in the garage) and jack-knives (kept in one’s pockets or backpack)? Very quickly, the consistency rule begins to break down.
1 . 7 U S A B I L I T Y A N D U S E R E X P E R I E N C E G O A L S 31
Jonathan Grudin notes how, in extending the number of places where knives are kept, inconsistency is introduced, which in turn increases the time needed to learn where they are all stored. However, the placement of the knives in different places often makes it easier to find them because they are at hand for the context in which they are used and are also next to the other objects used for a specific task; for instance, all of the home improvement project tools are stored together in a box in the garage. The same is true when designing interfaces: introducing inconsistency can make it more difficult to learn an interface, but in the long run it can make it easier to use.
ACTIVITY 1.4 One of the main design principles for website design is simplicity. Jakob Nielsen (1999) pro-posed that designers go through all of their design elements and remove them one by one. If a design works just as well without an element, then remove it. Do you think this is a good design principle? If you have your own website, try doing this and seeing what happens. At what point does the interaction break down?
CommentSimplicity is certainly an important design principle. Many designers try to cram too much into a screenful of space, making it unwieldy for people to find the element in which they are interested. Removing design elements to see what can be discarded without affecting the over-all function of the website can be a salutary lesson. Unnecessary icons, buttons, boxes, lines, graphics, shading, and text can be stripped, leaving a cleaner, crisper, and easier-to-navigate website. However, graphics, shading, coloring, and formatting can make a site aesthetically pleasing and enjoyable to use. Plain vanilla sites consisting solely of lists of text and a few links may not be as appealing and may put certain visitors off, never to return. Good interaction design involves getting the right balance between aesthetic appeal and the optimal amount and kind of information per page.
In-Depth ActivityThis activity is intended for you to put into practice what you have studied in this chapter. Specifically, the objective is to enable you to define usability and user experience goals and to transform these and other design principles into specific questions to help evaluate an inter-active product.
Find an everyday handheld device, for example, a remote control, digital camera, or smartphone and examine how it has been designed, paying particular attention to how the user is meant to interact with it.
(Continued)
1 W H AT I S I N T E R A C T I O N D E S I G N ?32
(a) From your first impressions, write down what is good and bad about the way the device works.
(b) Give a description of the user experience resulting from interacting with it.(c) Outline some of the core micro-interactions that are supported by it. Are they pleasurable,
easy, and obvious?(d) Based on your reading of this chapter and any other material you have come across about
interaction design, compile a set of usability and user experience goals that you think will be most relevant in evaluating the device. Decide which are the most important ones and explain why.
(e) Translate each of your sets of usability and user experience goals into two or three specific questions. Then use them to assess how well your device fares.
(f) Repeat steps (c) and (d), but this time use the design principles outlined in the chapter.(g) Finally, discuss possible improvements to the interface based on the answers obtained in
steps (d) and (e).
SummaryIn this chapter, we have looked at what interaction design is and its importance when developing apps, products, services, and systems. To begin, a number of good and bad designs were pre-sented to illustrate how interaction design can make a difference. We described who and what is involved in interaction design and the need to understand accessibility and inclusiveness. We explained in detail what usability and user experience are, how they have been characterized, and how to operationalize them to assess the quality of a user experience resulting from interact-ing with an interactive product. The increasing emphasis on designing for the user experience and not just products that are usable was stressed. A number of core design principles were also introduced that provide guidance for helping to inform the interaction design process.
Key Points • Interaction design is concerned with designing interactive products to support the waypeople communicate and interact in their everyday and working lives.
• Interaction design is multidisciplinary, involving many inputs from wide-ranging disciplinesand fields.
• The notion of the user experience is central to interaction design. • Optimizing the interaction between users and interactive products requires considerationof a number of interdependent factors, including context of use, types of activity, UX goals, accessibility, cultural differences, and user groups.
• Identifying and specifying relevant usability and user experience goals can help lead to thedesign of good interactive products.
• Design principles, such as feedback and simplicity, are useful heuristics for informing, ana-lyzing, and evaluating aspects of an interactive product.
F U R T H E R R E A D I N G 33
Further Reading
Here we recommend a few seminal readings on interaction design and the user experience (in alphabetical order).
COOPER, A., REIMANN, R., CRONIN, D. AND NOESSEL, C. (2014) About Face: The Essentials of Interaction Design (4th ed.). John Wiley & Sons Inc. This fourth edition of About Face provides an updated overview of what is involved in interaction design, and it is written in a personable style that appeals to practitioners and students alike.
GARRETT, J. J. (2010) The Elements of User Experience: User-Centered Design for the Web and Beyond (2nd ed.). New Riders Press. This is the second edition of the popular coffee-table introductory book to interaction design. It focuses on how to ask the right questions when designing for a user experience. It emphasizes the importance of understanding how products work on the outside, that is, when a person comes into contact with those products and tries to work with them. It also considers a business perspective.
LIDWELL, W., HOLDEN, K. AND BUTLER, J. (2010) Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions and Teach Through Design. Rockport Publishers, Inc. This book presents classic design principles such as consistency, accessibility, and visibility in addition to some lesser-known ones, such as constancy, chunking, and symmetry. They are alphabetically ordered (for easy reference) with a diversity of examples to illustrate how they work and can be used.
NORMAN, D.A. (2013) The Design of Everyday Things: Revised and Expanded Edition. MIT Press. This book was first published in 1988 and became an international best seller, introducing the world of technology to the importance of design and psychology. It covers the design of everyday things, such as refrigerators and thermostats, providing much food for thought in relation to how to design interfaces. This latest edition is comprehensively revised showing how principles from psychology apply to a diversity of old and new technologies. The book is highly accessible with many illustrative examples.
SAFFER, D. (2014) Microinteractions: Designing with Details. O’Reilly. This highly acces-sible book provides many examples of the small things in interaction design that make a big difference between a pleasant experience and a nightmare one. Dan Saffer describes how to design them to be efficient, understandable, and enjoyable user actions. He goes into detail about their structure and the different kinds, including many examples with lots of illustra-tions. The book is a joy to dip into and enables you to understand right away why and how it is important to get the micro-interactions right.
1 W H AT I S I N T E R A C T I O N D E S I G N ?34
INTERVIEW with Harry Brignull
Harry Brignull is a user experience con-sultant based in the United Kingdom. He has a PhD in cognitive science, and his work involves building better experiences by blending user research and interaction design. In his work, Harry has consulted for companies including Spotify, Smart Pension, The Telegraph, British Airways, Vodafone, and many others. In his spare time, Harry also runs a blog on interaction design that has attracted a lot of eyeballs. It is called 90percentofeverything.com, and it is well worth checking out.
What are the characteristics of a good interaction designer?I think of interaction design, user expe-rience design, service design, and user research as a combined group of disci-plines that are tricky to tease apart. Every company has slightly different terminol-ogy, processes, and approaches. I’ll let you into a secret, though. They’re all making it up as they go along. When you see any organization portraying its design and research publicly, they’re showing you a fictionalized view of it for recruitment and marketing purposes. The reality of the work is usually very different. Research and design is naturally messy. There’s a lot of waste, false assumptions, and blind alleys you have to go down before you can define and understand a problem well enough to solve it. If an employer doesn’t understand this and they don’t give you the space and time you need, then you won’t be able to do a good job, regardless of your skills and training.
A good interaction designer has skills that work like expanding foam. You expand to fill the skill gaps in your team. If you don’t have a writer present, you need to be able to step up and do it yourself, at least to the level of a credible draft. If you don’t have a researcher, you’ll need to step up and do it yourself. The same goes for devel-oping code-based prototypes, planning the user journeys, and so on. You’ll soon learn to become used to working outside of your comfort zone and relish the new challenges that each project brings.
How has interaction design changed in the past few years?In-housing of design teams is a big trend at the moment. When I started my con-sultancy career in the mid-2000s, the main route to getting a career in industry was to get a role at an agency, like a UX consultancy, a research agency, or a full- service agency. Big organizations didn’t even know where to start with hiring and building their own teams, so they paid enormous sums to agencies to design and build their products. This turned out to be a pretty ineffective model—when the agencies finish a project, they take all the acquired expertise away with them to their next clients.
These days, digital organizations have wised up, and they’ve started building their own in-house teams. This means that a big theme in design these days is organizational change. You can’t do good design in an organization that isn’t set up for it. In fact, in old, large organizations, the political
35
structure often seems to be set up to sab-otage good design and development prac-tices. It sounds crazy, but it’s very common to walk into an organization to find a proj-ect manager brandishing a waterfall Gantt chart while ranting obsessively about Agile (which is a contradiction in terms) or to find a product owner saying in one breath they value user research yet in the next breath getting angry with researchers for bringing them bad news. As well as “leg-acy technology,” organizations naturally end up with “legacy thinking.” It’s really tricky to change it. Design used to be just a department. Nowadays it’s understood that good design requires the entire organi-zation to work together in a cohesive way.
What projects are you working on now?I’m currently head of UX at a FinTech start-up called Smart Pension in London. Pen-sions pose a really fascinating user-centered design challenge. Consumers hate thinking about pensions, but they desperately need them. In a recent research session, one of the participants said something that really stuck with me: “Planning your pension is like planning for your own funeral.” Humans are pretty terrible at long-term planning over multiple decades. Nobody likes to think about their own mortality. But this is exactly what you need to do if you want to have a happy retirement.
The pension industry is full of jar-gon and off-putting technical complexity. Even fundamental financial concepts like risk aren’t well understood by many con-sumers. In some recent research, one of our participants got really tongue-tied try-ing to understand the idea that since they were young, it would be “high risk” (in the loose nontechnical definition of the word) to put their money into a “low-risk” fund
(in the technical definition of the word) since they’d probably end up with lower returns when they got older. Investment is confusing unless you’ve had training. Then, there’s the problem that “a little knowledge can hurt.” Some consumers who think they know what they’re doing can end up suffering when they think they can beat the market by moving their money around between funds every week.
Self-service online pension (retirement plans) platforms don’t do anything to help people make the right decisions because that would count as advice, which they’re not able to give because of the way it’s reg-ulated. Giving an average person a self- service platform and telling them to go sort out their pension is like giving them a Unix terminal and telling them to sort out their own web server. A few PDF fact sheets just aren’t going to help. If consumers want advice, they have to go to a financial advi-sor, which can be expensive and doesn’t make financial sense unless you have a lot of money in the first place. There’s a gap in the market, and we’re working these sorts of challenges in my team at Smart Pension.
What would you say are the biggest chal-lenges facing you and other consultants doing interaction design these days?A career in interaction design is one of con-tinual education and training. The biggest challenge is to keep this going. Even if you feel that you’re at the peak of your skills, the technology landscape will be shifting under your feet, and you need to keep an eye on what’s coming next so you don’t get left behind. In fact, things move so quickly in interaction design that by the time you read this interview, it will already be dated.
If you ever find yourself in a “com-fortable” role doing the same thing every
(Continued)
I N T E R V I E W W I T H H A R R Y B R I G N U L L
1 W H AT I S I N T E R A C T I O N D E S I G N ?36
day, then beware—you’re doing yourself a disservice. Get out there, stretch yourself, and make sure you spend some time every week outside your comfort zone.
If you’re asked to evaluate a prototype ser-vice or product and you discover it is really bad, how do you break the news?It depends what your goal is. If you want to just deliver the bad news and leave, then by all means be totally brutal and don’t pull any punches. But if you want to build a relationship with the client, you’re going
to need to help them work out how to move forward.
Remember, when you deliver bad news to a client, you’re basically explaining to them that they’re in a dark place and it’s their fault. It can be quite embarrassing and depressing. It can drive stakeholders apart when really you need to bring them together and give them a shared vision to work toward. Discovering bad design is an opportunity for improvement. Always pair the bad news with a recommendation of what to do next.
NOTEWe use the term interactive products generically to refer to all classes of interactive systems, technologies, environments, tools, applications, services, and devices.
Chapter 2
T H E P R O C E S S O F I N T E R A C T I O N D E S I G N
ObjectivesThe main goals of this chapter are to accomplish the following:
• Reflect on what interaction design involves. • Explain some of the advantages of involving users in development. • Explain the main principles of a user-centered approach. • Introduce the four basic activities of interaction design and how they are related in a simple lifecycle model.
• Ask some important questions about the interaction design process and provide the answers.
• Consider how interaction design activities can be integrated into other development lifecycles.
2.1 Introduction
Imagine that you have been asked to design a cloud-based service to enable people to share and curate their photos, movies, music, chats, documents, and so on, in an efficient, safe, and enjoyable way. What would you do? How would you start? Would you begin by sketching how the interface might look, work out how the system architecture should be structured, or just start coding? Or, would you start by asking users about their current experiences with sharing files and examine the existing tools, for example, Dropbox and Google Drive, and based on this begin thinking about how you were going to design the new service? What would you do next? This chapter discusses the process of interaction design, that is, how to design an interactive product.
There are many fields of design, such as graphic design, architectural design, industrial design, and software design. Although each discipline has its own approach to design, there
2.1 Introduction
2.2 What Is Involved in Interaction Design?
2.3 Some Practical Issues
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N38
are commonalities. The Design Council of the United Kingdom captures these in the double-diamond of design, as shown in Figure 2.1. This approach has four phases which are iterated:
• Discover: Designers try to gather insights about the problem. • Define: Designers develop a clear brief that frames the design challenge. • Develop: Solutions or concepts are created, prototyped, tested, and iterated. • Deliver: The resulting project is finalized, produced, and launched.
Interaction design also follows these phases, and it is underpinned by the philosophy of user-centered design, that is, involving users throughout development. Traditionally, interac-tion designers begin by doing user research and then sketching their ideas. But who are the users to be researched, and how can they be involved in development? Will they know what they want or need if we just ask them? From where do interaction designers get their ideas, and how do they generate designs?
In this chapter, we raise and answer these kinds of questions, discuss user-centered design, and explore the four basic activities of the interaction design process. We also intro-duce a lifecycle model of interaction design that captures these activities and the relationships among them.
2.2 What Is Involved in Interaction Design?
Interaction design has specific activities focused on discovering requirements for the prod-uct, designing something to fulfill those requirements, and producing prototypes that are then evaluated. In addition, interaction design focuses attention on users and their goals.
Discover
Pro
ble
m
So
luti
on
Pro
ble
m D
efin
itio
nD
esig
n B
rief
Define Develop Deliverinsight into the problem the area to focus upon potential solutions solutions that work
Figure 2.1 The double diamond of design Source: Adapted from https://www.designcouncil.org.uk/news-opinion/design-process-what-double-diamond
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 39
For example, the artifact’s use and target domain are investigated by taking a user-centered approach to development, users’ opinions and reactions to early designs are sought, and users are involved appropriately in the development process itself. This means that users’ concerns direct the development rather than just technical concerns.
Design is also about trade-offs—about balancing conflicting requirements. One common form of trade-off when developing a system to offer advice, for example, is deciding how much choice will be given to the user and how much direction the system should offer. Often, the division will depend on the purpose of the system, for example, whether it is for playing music tracks or for controlling traffic flow. Getting the balance right requires experience, but it also requires the development and evaluation of alternative solutions.
Generating alternatives is a key principle in most design disciplines and one that is also central to interaction design. Linus Pauling, twice a Nobel Prize winner, once said, “The best way to get a good idea is to get lots of ideas.” Generating lots of ideas is not necessarily hard, but choosing which of them to pursue is more difficult. For example, Tom Kelley (2016) describes seven secrets for successful brainstorms, including sharpening the focus (having a well-honed problem statement), having playful rules (to encourage ideas), and getting physi-cal (using visual props).
Involving users and others in the design process means that the designs and potential solutions will need to be communicated to people other than the original designer. This requires the design to be captured and expressed in a form that allows review, revision, and improvement. There are many ways of doing this, one of the simplest being to produce a series of sketches. Other common approaches are to write a description in natural language, to draw a series of diagrams, and to build a prototype, that is, a limited version of the final product. A combination of these techniques is likely to be the most effective. When users are involved, capturing and expressing a design in a suitable format is especially important since they are unlikely to understand jargon or specialist notations. In fact, a form with which users can interact is most effective, so building prototypes is an extremely powerful approach.
ACTIvITY 2.1 This activity asks you to apply the double diamond of design to produce an innovative inter-active product for your own use. By focusing on a product for yourself, the activity deliber-ately de-emphasizes issues concerned with involving other users, and instead it emphasizes the overall process.
Imagine that you want to design a product that helps you organize a trip. This might be for a business or vacation trip, to visit relatives halfway around the world, or for a bike ride on the weekend—whatever kind of trip you like. In addition to planning the route or booking tickets, the product may help to check visa requirements, arrange guided tours, investigate the facilities at a location, and so on.1. Using the first three phases of the double diamond of design, produce an initial design
using a sketch or two, showing its main functionality and its general look and feel. This activity omits the fourth phase, as you are not expected to deliver a working solution.
2. Now reflect on how your activities fell into these phases. What did you do first? What was your instinct to do first? Did you have any particular artifacts or experiences upon which to base your design?
(Continued)
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N40
Comment1. The first phase focuses on discovering insights about the problem, but is there a problem?
If so, what is it? Although most of us manage to book trips and travel to destinations with the right visas and in comfort, upon reflection the process and the outcome can be improved. For example, dietary requirements are not always fulfilled, and the accommoda-tion is not always in the best location. There is a lot of information available to support organizing travel, and there are many agents, websites, travel books, and tourist boards that can help. The problem is that it can be overwhelming.
The second phase is about defining the area on which to focus. There are many rea-sons for travelling—both individual and family—but in my experience organizing business trips to meetings worldwide is stressful, and minimizing the complexity involved in these would be worthwhile. The experience would be improved if the product offers advice from the many possible sources of information and tailors that advice to individual preferences.
The third phase focuses on developing solutions, which in this case is a sketch of the design itself. Figure 2.2 shows an initial design. This has two versions of the product—one as an app to run on a mobile device and one to run on a larger screen. The assumptions underlying the choice to build two versions are based on my experience; I would normally plan the details of the trip at my desk, while requiring updates and local information while traveling. The mobile app has a simple interaction style that is easy to use on the go, while the larger-screen version is more sophisticated and shows a lot of information and the vari-ous choices available.
(a) (b)
Figure 2.2 Initial sketches of the trip organizer showing (a) a large screen covering the entire journey from home to Beerwah in Australia and (b) the smartphone screen available for the leg of the journey at Paris (Charles de Gaulle) airport
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 41
2.2.1 Understanding the Problem SpaceDeciding what to design is key, and exploring the problem space is one way in which to decide. This is the first phase in the double diamond, but it can be overlooked by those new to interaction design, as you may have discovered in Activity 2.1. In the process of creating an interactive product, it can be tempting to begin at the nuts and bolts level of design. By this we mean working out how to design the physical interface and what technologies and inter-action styles to use, for example, whether to use multitouch, voice, graphical user interface, heads-up display, augmented reality, gesture-based, and so forth. The problem with starting here is that potential users and their context can be misunderstood, and usability and user experience goals can be overlooked, both of which were discussed in Chapter 1, “What Is Interaction Design?”
For example, consider the augmented reality displays and holographic navigation sys-tems that are available in some cars nowadays (see Figure 2.3). They are the result of decades of research into human factors of information displays (for instance, Campbell et al., 2016), the driving experience itself (Perterer et al., 2013; Lee et al., 2005), and the suitability of dif-ferent technologies (for example, Jose et al., 2016), as well as improvements in technology. Understanding the problem space has been critical in arriving at workable solutions that are safe and trusted. Having said that, some people may not be comfortable using a holographic navigation system and choose not to have one installed.
2. Initially, it wasn’t clear that there was a problem to address, but on reflection the complex-ity of the available information and the benefit of tailoring choices became clearer. The second phase guided me toward thinking about the area on which to focus. Worldwide business trips are the most difficult, and reducing the complexity of information sources through customization would definitely help. It would be good if the product learned about my preferences, for example, recommending flights from my favorite airline and finding places to have a vegan meal.
Developing solutions (the third phase) led me to consider how to interact with the product—seeing detail on a large screen would be useful, but a summary that can be shown on a mobile device is also needed. The type of support also depends on where the meeting is being held. Planning a trip abroad requires both a high-level view to check visas, vaccinations, and travel advice, as well as a detailed view about the proximity of accom-modation to the meeting venue and specific flight times. Planning a local trip is much less complicated.
The exact steps taken to create a product will vary from designer to designer, from product to product, and from organization to organization (see Box 2.1). Capturing con-crete ideas, through sketches or written descriptions, helps to focus the mind on what is being designed, the context of the design, and what user experience is to be expected. The sketches can capture only some elements of the design, however, and other formats are needed to capture everything intended. Throughout this activity, you have been making choices between alternatives, exploring requirements in detail, and refining your ideas about what the product will do.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N42
While it is certainly necessary at some point to choose which technology to employ and decide how to design the physical aspects, it is better to make these decisions after articulat-ing the nature of the problem space. By this we mean understanding what is currently the user experience or the product, why a change is needed, and how this change will improve the user experience. In the previous example, this involves finding out what is problem-atic with existing support for navigating while driving. An example is ensuring that drivers can continue to drive safely without being distracted when looking at a small GPS display mounted on the dashboard to figure out on which road it is asking them to “turn left.” Even when designing for a new user experience, it still requires understanding the context for which it will be used and the possible current user expectations.
The process of articulating the problem space is typically done as a team effort. Invari-ably, team members will have differing perspectives on it. For example, a project manager is likely to be concerned about a proposed solution in terms of budgets, timelines, and staffing costs, whereas a software engineer will be thinking about breaking it down into specific technical concepts. The implications of pursuing each perspective need to be con-sidered in relation to one another. Although time-consuming and sometimes resulting in disagreements among the design team, the benefits of this process can far outweigh the associated costs: there will be much less chance of incorrect assumptions and unsupported
(a)
(b)
Figure 2.3 (a) Example of the holographic navigation display from WayRay which overlays GPS navigation instructions onto the road ahead and gathers and shares driver statistics (b) an aug-mented reality navigation system available in some cars todaySources: (a) Used courtesy of WayRay, (b) Used courtesy of Muhammad Saad
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 43
claims creeping into a design solution that later turn out to be unusable or unwanted. Spending time enumerating and reflecting upon ideas during the early stages of the design process enables more options and possibilities to be considered. Furthermore, designers are increasingly expected to justify their choice of problems and to be able to present clearly and convincingly their rationale in business as well as design language. Being able to think and analyze, present, and argue is valued as much as the ability to create a product (Kolko, 2011).
2.2.2 The Importance of Involving UsersChapter 1 stressed the importance of understanding users, and the previous description emphasizes the need to involve users in interaction design. Involving users in development is important because it’s the best way to ensure that the end product is usable and that it indeed will be used. In the past, it was common for developers to talk only to managers, experts, or
BOX 2.1 Four Approaches to Interaction Design
Dan Saffer (2010) suggests four main approaches to interaction design, each of which is based on a distinct underlying philosophy: User-centered design, Activity-centered design, Systems design, and Genius design.
Dan Saffer acknowledges that the purest form of any of these approaches is unlikely to be realized, and he takes an extreme view of each in order to distinguish among them. In user-centered design, the user knows best and is the guide to the designer; the designer’s role is to translate the users’ needs and goals into a design solution.
Activity-centered design focuses on the behavior surrounding particular tasks. Users still play a significant role, but it is their behavior rather than their goals and needs that is impor-tant. Systems design is a structured, rigorous, and holistic design approach that focuses on context and is particularly appropriate for complex problems. In systems design, it is the system (that is, the people, computers, objects, devices, and so on) that the center of attention, while the users’ role is to set the goals of the system.
Finally, genius design is different from the other three approaches because it relies largely on the experience and creative flair of a designer. Jim Leftwich, an experienced interaction designer interviewed by Dan Saffer (2010, pp. 44–45), prefers the term rapid expert design. In this approach, the users’ role is to validate ideas generated by the designer, and users are not involved during the design process itself. Dan Saffer points out that this is not necessarily by choice, but it may be because of limited or no resources for user involvement.
Different design problems lend themselves more easily to different approaches, and dif-ferent designers will tend to gravitate toward using the approach that suits them best. Although an individual designer may prefer a particular approach, it is important that the approach for any one design problem is chosen with that design problem in mind.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N44
proxy users, or even to use their own judgment without reference to anyone else. While oth-ers involved in designing the product can provide useful information, they will not have the same perspective as the target user who performs the activity every day or who will use the intended product on a regular basis.
In commercial projects, a role called the product owner is common. The product owner’s job is to filter user and customer input to the development cycle and to prioritize require-ments or features. This person is usually someone with business and technical knowledge, but not interaction design knowledge, and they are rarely (if ever) a direct user of the product. Although the product owner may be called upon to assess designs, they are a proxy user at best, and their involvement does not avoid the need for user involvement.
The best way to ensure that developers gain a good understanding of users’ goals, lead-ing to a more appropriate, more usable product, is to involve target users throughout devel-opment. However, two other aspects unrelated to functionality are equally as important if the product is to be usable and used: expectation management and ownership.
Expectation management is the process of making sure that the users’ expectations of the new product are realistic. Its purpose is to ensure that there are no surprises for users when the product arrives. If users feel they have been cheated by promises that have not been fulfilled, then this will cause resistance and even rejection. Marketing of the new arrival must be careful not to misrepresent the product, although it may be particularly difficult to achieve with a large and complex system (Nevo and Wade, 2007). How many times have you seen an advertisement for something that you thought would be really good to have, but when you actually see one, you discover that the marketing hype was a little exaggerated? We expect that you felt quite disappointed and let down. This is the kind of feeling that expectation management tries to avoid.
Involving users throughout development helps with expectation management because they can see the product’s capabilities from an early stage. They will also understand better how it will affect their jobs and lives and why the features are designed that way. Adequate and timely training is another technique for managing expectations. If users have the chance to work with the product before it is released through training or hands-on demonstrations of a prerelease version, then they will understand better what to expect when the final product is available.
A second reason for user involvement is ownership. Users who are involved and feel that they have contributed to a product’s development are more likely to feel a sense of ownership toward it and support its use (Bano et al., 2017).
How to involve users, in what roles, and for how long, needs careful planning, as dis-cussed in the next Dilemma box.
DIlEMMAToo Much of a Good Thing?
Involving users in development is a good thing, but what evidence is there that user involve-ment is productive? How much should users be involved and in what role(s)? Is it appropriate for users to lead a technical development project, or is it more beneficial for them to focus on evaluating prototypes?
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 45
2.2.3 Degrees of User InvolvementDifferent degrees of user involvement are possible, ranging from fully engaged throughout all iterations of the development process to targeted participation in specific activities and from small groups of individual users in face-to-face contexts to hundreds of thousands of potential users and stakeholders online. Where available, individual users may be co-opted onto the design team so that they are major contributors to the development. This has pros and cons. On the downside, full-time involvement may mean that they become out of touch with their user community, while part-time involvement might result in a high workload for them. On the positive side, having a user engaged full or part-time does mean that the input is available continually throughout development. On the other hand, users may take part in specific activities to inform the development or to evaluate designs once they are available. This is a valuable form of involvement, but the users’ input is limited to that particular activ-ity. Where the circumstances around a project limit user involvement in this way, there are techniques to keep users’ concerns uppermost in developers’ minds, such as through personas (see Chapter 11, “Discovering Requirements”).
Uli Abelein et al. (2013) performed a detailed review of the literature in this area and concluded that, overall, the evidence indicates that user involvement has a positive effect on user satisfaction and system use. However, they also found that even though the data clearly indicates this positive effect, some links have a large variation, suggesting that there is still no clear way to measure the effects consistently. In addition, they found that most studies with negative correlations involving users and system success were published more than 10 years previously.
Ramanath Subrayaman et al. (2010) investigated the impact of user participation on levels of satisfaction with the product by both developers and users. They found that for new products, developer satisfaction increased as user participation increased. On the other hand, user satisfaction was higher if their participation was low, and satisfaction dropped as their participation increased. They also identified that high levels of user involvement can generate conflicts and increased reworking. For maintenance projects, both developers and users were most satisfied with a moderate level of participation (approximately 20 percent of overall project development time). Based just on user satisfaction as an indicator of project success, then, it seems that low user participation is most beneficial.
The kind of product being developed, the kind of user involvement possible, the activities in which they are involved, and the application domain all have an impact on the effectiveness of user input (Bano and Zowghi, 2015). Peter Richard et al. (2014) investigated the effect of user involvement in transport design projects. They found that involving users at later stages of development mainly resulted in suggestions for service improvement, whereas users involved at earlier stages of innovation suggested more creative ideas.
Recent moves toward an agile way of working (see Chapter 13, “Interaction Design in Practice”) has emphasized the need for feedback from customers and users, but this also has its challenges. Kurt Schmitz et al. (2018) suggests that in tailoring their methods, teams consider the distinction between frequent participation in activities and effective engagement.
User involvement is undoubtedly beneficial, but the levels and types of involvement require careful consideration and balance.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N46
Initially, user involvement took the form of small groups or individuals taking part in face-to-face information-gathering design, or evaluation sessions, but increasing online con-nectivity has led to a situation in which many thousands of potential users can contribute to product development. There is still a place for face-to-face user involvement and in situ studies, but the range of possibilities for user involvement is now much wider. One example of this is online feedback exchange (OFE) systems, which are increasingly used to test design concepts with millions of target users before going to market (Foong et al., 2017).
In fact, design is becoming increasingly participative through crowdsourcing design ideas and examples, for instance (Yu et al., 2016). Where crowdsourcing is used, a range of differ-ent people are encouraged to contribute, and this can include any and all of the stakeholders. This wide participation helps to bring different perspectives to the process, which enhances the design itself, produces more user satisfaction with the final product, and engenders a sense of ownership. Another example of involving users at scale is citizen engagement, the goal of which is to engage a population—civic or otherwise—with the aim of promoting empowerment through technology. The underlying aim is to involve members of the public in helping them make a change in their lives where technology is often viewed as an integral part of the process.
Participatory design, also sometimes referred to as cooperative design or co-design, is an overarching design philosophy that places those for whom systems, technologies, and services are being designed, as central actors in creation activities. The idea is that instead of being passive receivers of new technological or industrial artifacts, end users and stake-holders are active participants in the design process. Chapter 12, “Design, Prototyping, and Construction,” provides more information on participatory design.
The individual circumstances of the project affect what is realistic and appropriate. If the end-user groups are identifiable, for example, the product is for a particular company, then it is easier to involve them. If, however, the product is intended for the open market, it is unlikely that users will be available to join the design team. In this case, targeted activities and online feedback systems may be employed. Box 2.2 outlines an alternative way to obtain user input from an existing product, and Box 2.5 discusses A/B testing, which draws on user feedback to choose between alternative designs.
BOX 2.2 User Involvement After Product Release
Once a product has been released, a different kind of user involvement is possible—one that captures data and user feedback based on day-to-day use of the product. The prevalence of customer reviews has grown considerably in recent years, and they significantly affect the popularity and success of a product (Harman et al., 2012). These reviews provide useful and far-ranging user feedback. For example, Hammad Khalid et al. (2015) studied reviews of mobile apps to see what reviewers complained about. They identified 12 complaint types, including privacy and ethics, interface, and feature removal. Customer reviews can provide useful insight to help improve products, but detailed analysis of feedback gathered this way is time-consuming.
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 47
2.2.4 What Is a User-Centered Approach?Throughout this book, we emphasize the need for a user-centered approach to development. By this we mean that the real users and their goals, not just technology, are the driving force behind product development. As a consequence, a well-designed system will make the most of human skill and judgment, will be directly relevant to the activity in hand, and will sup-port rather than constrain the user. This is less of a technique and more of a philosophy.
Error reporting systems (ERSs, also called online crashing analysis) automatically collect information from users that is used to improve applications in the longer term. This is done with users’ permission, but with a minimal reporting burden. Figure 2.4 shows two dialog boxes for the Windows error reporting system that is built into Microsoft operating systems. This kind of reporting can have a significant effect on the quality of applications. For example, 29 percent of the errors fixed by the Windows XP (Service Pack 1) team were based on infor-mation collected through their ERS (Kinshumann et al., 2011). While Windows XP is no longer being supported, this statistic illustrates the impact ERSs can have. The system uses a sophisticated approach to error reporting based on five strategies: automatic aggregation of error reports; progressive data collection so that the data collected (such as abbreviated or full stack and memory dumps) varies depending on the level of data needed to diagnose the error; minimal user interaction; preserving user privacy; and providing solutions directly to users where possible. By using these strategies, plus statistical analysis, effort can be focused on the bugs that have the highest impact on the most users.
Figure 2.4 Two typical dialog boxes from the Windows error reporting system
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N48
When the field of HCI was being established, John Gould and Clayton Lewis (1985) laid down three principles that they believed would lead to a “useful and easy to use computer system.” These principles are as follows:
1. Early focus on users and tasks. This means first understanding who the users will be by directly studying their cognitive, behavioral, anthropomorphic, and attitudinal character-istics. This requires observing users doing their normal tasks, studying the nature of those tasks, and then involving users in the design process.
2. Empirical measurement. Early in development, the reactions and performance of intended users to printed scenarios, manuals, and so forth, are observed and measured. Later, users interact with simulations and prototypes, and their performance and reactions are observed, recorded, and analyzed.
3. Iterative design. When problems are found in user testing, they are fixed, and then more tests and observations are carried out to see the effects of the fixes. This means that design and development are iterative, with cycles of design-test-measure-redesign being repeated as often as necessary.
These three principles are now generally accepted as the basis for a user-centered approach. When this paper was written, however, they were not accepted by most develop-ers. We discuss these principles in more detail in the following sections.
Early Focus on Users and TasksThis principle can be expanded and clarified through the following five further principles:
1. Users’ tasks and goals are the driving force behind the development.While technology will inform design options and choices, it is not the driving force. Instead of saying “Where can we deploy this new technology?” say “What technologies are available to provide better support for users’ goals?”
2. Users’ behavior and context of use are studied, and the system is designed to support them.This is not just about capturing users’ tasks and goals. How people perform their tasks is also significant. Understanding behavior highlights priorities, preferences, and implicit intentions.
3. Users’ characteristics are captured and designed for.When things go wrong with technology, people often think it is their fault. People are prone to making errors and have certain limitations, both cognitive and physical. Prod-ucts designed to support people should take these limitations into account and try to prevent mistakes from being made. Cognitive aspects, such as attention, memory, and per-ception issues are introduced in Chapter 4, “Cognitive Aspects.” Physical aspects include height, mobility, and strength. Some characteristics are general, such as color blindness, which affects about 4.5 percent of the population, but some characteristics are associated with a particular job or task. In addition to general characteristics, those traits specific to the intended user group also need to be captured.
4. Users are consulted throughout development from earliest phases to the latest.As discussed earlier, there are different levels of user involvement, and there are different ways in which to consult users.
5. All design decisions are taken within the context of the users, their activities, and their environment.This does not necessarily mean that users are actively involved in design decisions, but that is one option.
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 49
Empirical MeasurementWhere possible, specific usability and user experience goals should be identified, clearly doc-umented, and agreed upon at the beginning of the project. They can help designers choose between alternative designs and check on progress as the product is developed. Identifying specific goals up front means that the product can be empirically evaluated at regular stages throughout development.
Iterative DesignIteration allows designs to be refined based on feedback. As users and designers engage with the domain and start to discuss requirements, needs, hopes, and aspirations, then different insights into what is needed, what will help, and what is feasible will emerge. This leads to a need for iteration—for the activities to inform each other and to be repeated. No matter how good the designers are and however clear the users may think their vision is of the required artifact, ideas will need to be revised in light of feedback, likely several times. This is particu-larly true when trying to innovate. Innovation rarely emerges whole and ready to go. It takes time, evolution, trial and error, and a great deal of patience. Iteration is inevitable because designers never get the solution right the first time (Gould and Lewis, 1985).
ACTIvITY 2.2 Assume you are involved in developing a novel online experience for buying garden plants. Although many websites exist for buying plants online, you want to produce a distinct expe-rience to increase the organization’s market share. Suggest ways of applying the previous principles in this task.
CommentTo address the first three principles, you would need to find out about the tasks and goals, behavior, and characteristics of potential customers of the new experience, together with any different contexts of use. Studying current users of existing online plant shops will provide some information, and it will also identify some challenges to be addressed in the new experi-ence. However, as you want to increase the organization’s market share, consulting existing users alone would not be enough. Alternative avenues of investigation include physical shop-ping situations—for example, shopping at the market, in the local corner shop, and so on, and local gardening clubs, radio programs, or podcasts. These alternatives will help you find the advantages and disadvantages of buying plants in different settings, and you will observe different behaviors. By looking at these options, a new set of potential users and contexts can be identified.
For the fourth principle, the set of new users will emerge as investigations progress, but people who are representative of the user group may be accessible from the beginning. Work-shops or evaluation sessions could be run with them, possibly in one of the alternative shop-ping environments such as the market. The last principle could be supported through the creation of a design room that houses all of the data collected, and it is a place where the devel-opment team can go to find out more about the users and the product goals.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N50
2.2.5 Four Basic Activities of Interaction DesignThe four basic activities for interaction design are as follows:
1. Discovering requirements for the interactive product.2. Designing alternatives that meet those requirements.3. Prototyping the alternative designs so that they can be communicated and assessed.4. Evaluating the product and the user experience it offers throughout the process.
Discovering RequirementsThis activity covers the left side of the double diamond of design, and it is focused on dis-covering something new about the world and defining what will be developed. In the case of interaction design, this includes understanding the target users and the support an interactive product could usefully provide. This understanding is gleaned through data gathering and analysis, which are discussed in Chapters 8–10. It forms the basis of the product’s require-ments and underpins subsequent design and development. The requirements activity is dis-cussed further in Chapter 11.
Designing AlternativesThis is the core activity of designing and is part of the Develop phase of the double diamond: proposing ideas for meeting the requirements. For interaction design, this activity can be viewed as two subactivities: conceptual design and concrete design. Conceptual design involves pro-ducing the conceptual model for the product, and a conceptual model describes an abstraction outlining what people can do with a product and what concepts are needed to understand how to interact with it. Concrete design considers the detail of the product including the colors, sounds, and images to use, menu design, and icon design. Alternatives are considered at every point. Conceptual design is discussed in Chapter 3, and more design issues for specific interface types are in Chapter 7; more detail about how to design an interactive product is in Chapter 12.
PrototypingPrototyping is also part of the Develop phase of the double diamond. Interaction design involves designing the behavior of interactive products as well as their look and feel. The most effec-tive way for users to evaluate such designs is to interact with them, and this can be achieved through prototyping. This does not necessarily mean that a piece of software is required. There are different prototyping techniques, not all of which require a working piece of software. For example, paper-based prototypes are quick and cheap to build and are effective for identifying problems in the early stages of design, and through role-playing users can get a real sense of what it will be like to interact with the product. Prototyping is covered in Chapter 12.
EvaluatingEvaluating is also part of the Develop phase of the double diamond. It is the process of deter-mining the usability and acceptability of the product or design measured in terms of a variety of usability and user-experience criteria. Evaluation does not replace activities concerned with quality assurance and testing to make sure that the final product is fit for its intended purpose, but it complements and enhances them. Chapters 14–16 cover evaluation.
The activities to discover requirements, design alternatives, build prototypes, and evalu-ate them are intertwined: alternatives are evaluated through the prototypes, and the results are fed back into further design or to identify alternative requirements.
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 51
2.2.6 A Simple Lifecycle Model for Interaction DesignUnderstanding what activities are involved in interaction design is the first step to being able to do it, but it is also important to consider how the activities are related to one another. The term lifecycle model (or process model) is used to represent a model that captures a set of activities and how they are related. Existing models have varying levels of sophistication and complexity and are often not prescriptive. For projects involving only a few experienced developers, a simple process is adequate. However, for larger systems involving tens or hun-dreds of developers with hundreds or thousands of users, a simple process just isn’t enough to provide the management structure and discipline necessary to engineer a usable product.
Many lifecycle models have been proposed in fields related to interaction design. For example, software engineering lifecycle models include the waterfall, spiral, and V models (for more information about these models, see Pressman and Maxim [2014]). HCI has been less associated with lifecycle models, but two well-known ones are the Star (Hartson and Hix, 1989) and an international standard model ISO 9241-210. Rather than explaining the details of these models, we focus on the classic lifecycle model shown in Figure 2.5. This model shows how the four activities of interaction design are related, and it incorporates the three principles of user-centered design discussed earlier.
Many projects start by discovering requirements from which alternative designs are gen-erated. Prototype versions of the designs are developed and then evaluated. During prototyp-ing or based on feedback from evaluations, the team may need to refine the requirements or to redesign. One or more alternative designs may follow this iterative cycle in parallel. Implicit in this cycle is that the final product will emerge in an evolutionary fashion from an initial idea through to the finished product or from limited functionality to sophisticated functionality. Exactly how this evolution happens varies from project to project. However many times through the cycle the product goes, development ends with an evaluation activity that ensures that the final product meets the prescribed user experience and usability criteria. This evolutionary production is part of the Delivery phase of the double diamond.
In recent years, a wide range of lifecycle models has emerged, all of which encom-pass these activities but with different emphases on activities, relationships, and outputs. For example, Google Design Sprints (Box 2.3) emphasize problem investigation, solution development, and testing with customers all in one week. This does not result in a robust final product, but it does make sure that the solution idea is acceptable to customers. The in-the-wild approach (Box 2.4) emphasizes the development of novel technologies that are not necessarily designed for specific user needs but to augment people, places, and settings. Further models are discussed in Chapter 13.
Source: Fran / Cartoon Stock
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N52
PROTOTYPING
EVALUATING
FINAL PRODUCT
DESIGNINGALTERNATIVES
DISCOVERINGREQUIREMENTS
Figure 2.5 A simple interaction design lifecycle model
BOX 2.3 Google Design Sprints (Adapted from Knapp et al. (2016))
Google Ventures has developed a structured approach to design that supports rapid idea-tion and testing of potential solutions to a design challenge. This is called the Google Design Sprint. A sprint is divided into five phases, and each phase is completed in a day. This means that in five days, you can go from a design challenge to a solution that has been tested with customers. As the authors say, “You won’t finish with a complete, detailed, ready-to-ship product. But you will make rapid progress, and know for sure if you’re headed in the right direction” (Knapp et al., 2016, p16–17). Teams are encouraged to iterate on the last two phases and to develop and re-test prototypes. If necessary, the first idea can be thrown away and the process started again at Phase 1. There is preparation to be done before the sprint begins. This preparation and the five phases are described next (see Figure 2.6).
Google Design Sprint
Unpack Sketch Decide Prototype Test Iterate
Unpack
Sketch
Decide
Prototype
Test1 2 3 4 5
Figure 2.6 The five phases of the Google Design Sprint Source: www.agilemarketing.net/google-design-sprints. Used courtesy of Agile Marketing
2 . 2 W H AT I S I N v O lv E D I N I N T E R A C T I O N D E S I G N ? 53
Setting the StageThis time is used to choose the right design challenge, gather the right team, and organize the time and space to run the sprint (that is, full-time for everyone for five days). The sprint can help in high-stake challenges, when you’re running out of time, or if you’re just stuck. The team composition depends on the product, but it has about seven people including a decider (who chooses the design to show to the customer), customer expert, technical expert, and anyone who will bring a disruptive perspective.
UnpackDay 1 focuses on making a map of the challenge and choosing a target, that is, a part of the challenge that can be achieved in a week.
Sketch Competing SolutionsDay 2 focuses on generating solutions, with an emphasis on sketching and individual creativ-ity rather than group brainstorming.
Decide on the BestDay 3 focuses on critiquing the solutions generated on Day 1, choosing the one most likely to meet the sprint’s challenge, and producing a storyboard. Whichever solution is chosen, the decider needs to support the design.
Build a Realistic PrototypeDay 4 focuses on turning the storyboard into a realistic prototype, that is, something on which customers can provide feedback. We discuss prototyping further in Chapter 12.
Test with Target CustomersDay 5 focuses on getting feedback from five customers and learning from their reactions.
The Google Design Sprint is a process for answering critical business questions through design, prototyping, and testing ideas with customers. Marta Rey-Babarro, who works at Google as a staff UX researcher and was the cofounder of Google’s internal Sprint Academy, describes how they used a sprint to improve the experience of traveling for business.
We wanted to see if we could improve the business travel experience. We started by doing research with Googlers to find out what experiences and what needs they had when they traveled. We discovered that there were some Googlers who traveled over 300 days a year and others who traveled only once or twice a year. Their travel experiences and needs were very different. After this research, some of us did a sprint in which we explored the whole travel experience, from the planning phase to coming back home and submitting receipts. Within five days we came up with a vision of what that experience could be. On the fifth day of the sprint, we presented that vision to higher-level execs. They loved it and sponsored the creation of a new team at Google that has developed new
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N54
BOX 2.4 Research in the Wild (Adapted from Rogers and Marshall (2017))
Research in the wild (RITW) develops technology solutions in everyday living by creating and evaluating new technologies and experiences in situ. The approach supports designing prototypes in which researchers often experiment with new technological possibilities that can change and even disrupt behavior, rather than ones that fit in with existing practices. The results of RITW studies can be used to challenge assumptions about technology and human behavior in the real world and to inform the re-thinking of HCI theories. The perspective taken by RITW studies is to observe how people react to technology and how they change and integrate it into their everyday lives.
Figure 2.7 shows the framework for RITW studies. In terms of the four activities intro-duced earlier, this framework focuses on designing, prototyping, and evaluating technology and ideas and is one way in which requirements may be discovered. It also considers relevant theory since often the purpose of an RITW study is to investigate a theory, idea, concept, or observation. Any one RITW study may emphasize the elements of the framework to a differ-ent degree.Technology: Concerned with appropriating existing infrastructures/devices (e.g., Internet of
Things toolkit, mobile app) in situ or developing new ones for a given setting (e.g., a novel public display).
Design: Covers the design space of an experience (e.g., iteratively creating a collaborative travel planning tool for families to use or an augmented reality game for playing outdoors).
In situ study: Concerned with evaluating in situ an existing device/tool/service or novel research-based prototype when placed in various settings or given to someone to use over a period of time.
Theory: Investigating a theory, idea, concept or observation about a behavior, setting or other phenomenon using existing ones or developing a new one or extending an existing one.
tools and experiences for the traveling Googler. Some of those internal online experiences made it also to our external products and services outside of Google.
Marta Rey-Babarro
To see a more detailed description of the Google Design Sprint and to access a set of five videos that describe what happens on each day of the sprint, go to www.gv.com/sprint/#book.
2 . 3 S O M E P R A C T I C A l I S S u E S 55
2.3 Some Practical Issues
The discussion so far has highlighted some issues about the practical application of user-centered design and the simple lifecycle of interaction design introduced earlier. These issues are listed here:
• Who are the users? • What are the users’ needs? • How to generate alternative designs • How to choose among alternatives • How to integrate interaction design activities with other lifecycle models
2.3.1 Who Are the Users?Identifying users may seem like a straightforward activity, but it can be harder than you think. For example, Sha Zhao et al. (2016) found a more diverse set of users for smartphones than most manufacturers recognize. Based on an analysis of one month’s smartphone app
TechnologyIn SituStudies
Design
Theory
Figure 2.7 A framework for research in the wild studies Source: Rogers and Marshall (2017), p. 6. Used courtesy of Morgan & Claypool
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N56
usage, they discovered 382 distinct types of users, including Screen Checkers and Young Parents. Charlie Wilson et al. (2015) found that little is understood about who the users of smart homes in general are expected to be, beyond those focused on health-related condi-tions. In part, this is because many products nowadays are being developed for use by large sections of the population, and so it can be difficult to determine a clear description. Some products (such as a system to schedule work shifts) have more constrained user communities, for example a specific role (shop assistant) within a particular industrial sector (retail). In this case, there may be a range of users with different roles who relate to the product in differ-ent ways. Examples are those who manage direct users, those who receive outputs from the system, those who test the system, those who make the purchasing decision, and those who use competitive products (Holzblatt and Jones, 1993).
There is a surprisingly wide collection of people who all have a stake in the development of a successful product. These people are called stakeholders. Stakeholders are the individu-als or groups that can influence or be influenced by the success or failure of a project. Alan Dix et al. (2004) observed that is pertinent to a user-centered view of development: “It will frequently be the case that the formal ‘client’ who orders the system falls very low on the list of those affected. Be very wary of changes which take power, influence or control from some stakeholders without returning something tangible in its place.”
The group of stakeholders for a particular product will be larger than the group of users. It will include customers who pay for it; users who interact with it; developers who design, build, and maintain it; legislators who impose rules on the development and operation of it; people who may lose their jobs because of its introduction; and so on (Sharp et al., 1999).
Identifying the stakeholders for a project helps to decide who to involve as users and to what degree, but identifying relevant stakeholders can be tricky. Ian Alexander and Suzanne Robertson (2004) suggest using an onion diagram to model stakeholders and their involve-ment. This diagram shows concentric circles of stakeholder zones with the product being developed sitting in the middle. Soo Ling Lim and Anthony Finkelstein (2012) developed a method called StakeRare and supporting tool called StakeNet that relies on social networks and collaborative filtering to identify and prioritize relevant stakeholders.
ACTIvITY 2.3 Who are the stakeholders for an electricity smart meter for use in the home to help households control their energy consumption?
CommentFirst, there are the people who live in the house, such as older adults and young children, with a range of abilities and backgrounds. To varying degrees, they will be users of the meter, and their stake in its success and usability is fairly clear and direct. Householders want to make sure that their bills are controlled, that they can easily access suppliers if they want to, and that their electricity supply is not interrupted. On the other hand, the entire family will
2 . 3 S O M E P R A C T I C A l I S S u E S 57
2.3.2 What Are the Users’ Needs?If you had asked someone in the street in the late 1990s what they needed, their answer probably wouldn’t have included a smart TV, a ski jacket with an integrated smartphone, or a robot pet. If you presented the same person with these possibilities and asked whether they would buy them if they were available, then the answer may have been more positive. Deter-mining what product to build is not simply a question of asking people “What do you need?” and then supplying it, because people don’t necessarily know what is possible. Suzanne and James Robertson (2013) refer to “un-dreamed-of” needs, which are those that users are una-ware they might have. Instead of asking users, this is approached by exploring the problem space, investigating the users and their activities to see what can be improved, or trying out ideas with potential users to see whether the ideas are successful. In practice, a mixture of these approaches is often taken—trying ideas in order to discover requirements and decide what to build, but with knowledge of the problem space, potential users, and their activities.
If a product is a new invention, then identifying the users and representative tasks for them may be harder. This is where in-the-wild studies or rapid design sprints that provide authentic user feedback on early ideas are valuable. Rather than imagining who might want to use a product and what they might want to do with it, it’s more effective to put it out there and find out—the results might be surprising!
It may be tempting for designers simply to design what they would like to use them-selves, but their ideas would not necessarily coincide with those of the target user group, because they have different experiences and expectations. Several practitioners and com-mentators have observed that it’s an “eye-opening experience” when developers or designers see a user struggling to complete a task that seemed so clear to them (Ratcliffe and McNeill, 2012, p. 125).
Focusing on people’s goals, usability goals and user experience goals is a more promising approach to interaction design than simply expecting stakeholders to be able to articulate the requirements for a product.
want to continue to live in the house in comfort, for example, with enough heat and light. Then there are the people who install and maintain the meter. They make sure that the meter is installed correctly and that it continues to work effectively. Installers and maintainers want the meter to be straightforward to install and to be robust and reliable to reduce the need for return visits or maintenance calls. Outside of these groups are electricity suppliers and dis-tributors who also want to provide a competitive service so that the householders are satis-fied and to minimize maintenance costs. They also don’t want to lose customers and money because the meters are faulty or are providing inaccurate information. Other people who will be affected by the success of the meter include those who work on the powerlines and at electricity generation plants, those who work in other energy industries, and ultimately the government of the country that will want to maintain steady supply for its industry and population.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N58
2.3.3 How to Generate Alternative DesignsA common human tendency is to stick with something that works. While recognizing that a better solution may exist, it is easy to accept the one that works as being “good enough.” Set-tling for a solution that is good enough may be undesirable because better alternatives may never be considered, and considering alternative solutions is a crucial step in the process of design. But where do these alternative ideas come from?
One answer to this question is that they come from the individual designer’s flair and cre-ativity (the genius design described in Box 2.1). Although it is certainly true that some people are able to produce wonderfully inspired designs while others struggle to come up with any ideas at all, very little in this world is completely new. For example, the steam engine, com-monly regarded as an invention, was inspired by the observation that steam from a kettle boiling on the stove lifted the lid. An amount of creativity and engineering was needed to make the jump from a boiling kettle to a steam engine, but the kettle provided inspiration to translate this experience into a set of principles that could be applied in a different context. Innovations often arise through cross-fertilization of ideas from different perspectives, indi-viduals, and contexts; the evolution of an existing product through use and observation; or straightforward copying of other, similar products.
Cross-fertilization may result from discussing ideas with other designers, while Bill Bux-ton (2007) reports that different perspectives from users generated original ideas about alter-native designs. As an example of evolution, consider the cell phone and its descendant, the smartphone. The capabilities of the phone in your pocket have increased from the time they first appeared. Initially, the cell phone simply made and received phone calls and texts, but now the smartphone supports a myriad of interactions, can take photos and record audio, play movies and games, and record your exercise routine.
Creativity and invention are often wrapped in mystique, but a lot has been uncovered about the process and of how creativity can be enhanced or inspired (for example, see Rogers, 2014). For instance, browsing a collection of designs will inspire designers to con-sider alternative perspectives and hence alternative solutions. As Roger Schank (1982, p. 22) puts it, “An expert is someone who gets reminded of just the right prior experience to help him in processing his current experiences.” And while those experiences may be the designer’s own, they can equally well be others’.
Another approach to creativity has been adopted by Neil Maiden et al. (2007). They ran creativity workshops to generate innovative requirements in an air traffic management (ATM) application domain. Their idea was to introduce experts in different fields into the workshop and then invite stakeholders to identify analogies between their own field and this new one. For example, they have invited an Indian textile expert, a musician, a TV program scheduler, and a museum exhibit designer. Although not all obviously analogical domains, they sparked creative ideas for the air traffic management application. For example, partici-pants reported that one textile design was elegant, that is, simple, beautiful, and symmetrical. They then transferred these properties to a key area of the ATM domain—that of aircraft conflict resolution. They explored the meaning of elegance within this context and realized that elegance is perceived differently by different controllers. From this they generated the requirement that the system should be able to accommodate different air traffic controller styles during conflict resolution.
2 . 3 S O M E P R A C T I C A l I S S u E S 59
A more pragmatic answer to this question, then, is that alternatives come from seeking different perspectives and looking at other designs. The process of inspiration and creativ-ity can be enhanced by prompting a designer’s own experience and studying others’ ideas and suggestions. Deliberately seeking out suitable sources of inspiration is a valuable step in any design process. These sources may be very close to the intended new product, such as competitors’ products; they may be earlier versions of similar systems; or they may be from a completely different domain.
Under some circumstances, the scope to consider alternative designs is limited. Design is a process of balancing constraints and trading off one set of requirements with another, and the constraints may mean that there are few viable alternatives available. For example, when designing software to run under the Windows operating system, the design must conform to the Windows look and feel and to other constraints intended to make Windows programs consistent for the user. When producing an upgrade to an existing system, keeping familiar elements of it to retain the same user experience may be prioritized.
2.3.4 How to Choose Among Alternative DesignsChoosing among alternatives is mostly about making design decisions: Will the device use keyboard entry or a touch screen? Will the product provide an automatic memory function or not? These decisions will be informed by the information gathered about users and their tasks and by the technical feasibility of an idea. Broadly speaking, though, the decisions fall into two categories: those that are about externally visible and measurable features
ACTIvITY 2.4 Consider the product introduced in Activity 2.1. Reflecting on the process again, what inspired your initial design? Are there any innovative aspects to it?
CommentFor our design, existing sources of information and their flaws were influential. For example, there is so much information available about travel, destinations, hotel comparisons, and so forth, that it can be overwhelming. However, travel blogs contain useful and practical insights, and websites that compare alternative options are informative. We were also influenced by some favorite mobile and desktop applications such as the United Kingdom’s National Rail smartphone app for its real-time updating and by the Airbnb website for its mixture of sim-plicity and detail.
Perhaps you were inspired by something that you use regularly, like a particularly enjoy-able game or a device that you like to use? I’m not sure how innovative our ideas were, but the main goal was for the application to tailor its advice for the user’s preferences. There are probably other aspects that make your design unique and that may be innovative to a greater or lesser degree.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N60
and those that are about characteristics internal to the system that cannot be observed or measured without dissecting it. For example, in a photocopier, externally visible and measurable factors include the physical size of the machine, the speed and quality of copy-ing, the different sizes of paper it can use, and so on. Underlying each of these factors are
BOX 2.5 A Box Full of Ideas
The innovative product design company IDEO was mentioned in Chapter 1. Underlying some of its creative flair is a collection of weird and wonderful engineering housed in a large flatbed filing cabinet called the TechBox. The TechBox holds hundreds of gizmos and interesting materials, divided into categories such as Amazing Materials, Cool Mech-anisms, Interesting Manufacturing Processes, Electronic Technologies, and Thermal and Optical. Each item has been placed in the box because it represents a neat idea or a new process. The staff at IDEO take along a selection of items from the TechBox to brainstorming meetings. The items may be chosen because they provide useful visual props or possible solu-tions to a particular issue or simply to provide some light relief.
Each item is clearly labeled with its name and category, but further information can be found by accessing the TechBox’s online catalog. Each item has its own page detailing what the item is, why it is interesting, where it came from, and who has used it or knows more about it. Items in the box include an example of metal-coated wood and materials with and without holes that stretch, bend, and change shape or color at different temperatures.
Each of IDEO’s offices has a TechBox, and each TechBox has its own curator who is responsible for maintaining and cataloging the items and for promoting its use within the office. Anyone can submit a new item for consideration. As items become common-place, they are removed from the TechBox to make way for the next generation of fascinating curios.
2 . 3 S O M E P R A C T I C A l I S S u E S 61
other considerations that cannot be observed or studied without dissecting the machine. For example, the choice of materials used in a photocopier may depend on its friction rating and how much it deforms under certain conditions. In interaction design, the user experience is the driving force behind the design and so externally visible and measurable behavior is the main focus. Detailed internal workings are still important to the extent that they affect external behavior or features.
DIlEMMACopying for Inspiration: Is It Legal?
Designers draw on their experience of design when approaching a new project. This includes the use of previous designs that they know work—both designs that they have created them-selves and those that others have created. Others’ creations often spark inspiration that also leads to new ideas and innovation. This is well known and understood. However, the expres-sion of an idea is protected by copyright, and people who infringe on that copyright can be taken to court and prosecuted. Note that copyright covers the expression of an idea and not the idea itself. This means, for example, that while there are numerous smartphones all with similar functionality, this does not represent an infringement of copyright as the idea has been expressed in different ways and it is the expression that has been copyrighted. Copy-right is free and is automatically invested in the author, for instance, the writer of a book or a programmer who develops a program, unless they sign the copyright over to someone else. Employment contracts often include a statement that the copyright relating to anything pro-duced in the course of that employment is automatically assigned to the employer and does not remain with the employee.
Patenting is an alternative to copyright that does protect the idea rather than the expres-sion of the idea. There are various forms of patenting, each of which is designed to allow the inventor to capitalize on their idea. For example, Amazon patented its one-click purchasing process, which allows regular users simply to choose a purchase and buy it with one mouse click (US Patent No. 5960411, September 29, 1999). This is possible because the system stores its customers’ details and recognizes them when they access the Amazon site again.
In recent years, the creative commons community (https://creativecommons.org/) has suggested more flexible licensing arrangements that allow others to reuse and extend a piece of created work, thereby supporting collaboration. In the open source software development movement, for example, software code is freely distributed and can be modified, incorporated into other software, and redistributed under the same open source conditions. No royalty fees are payable on any use of open source code. These movements do not replace copyright or patent law, but they provide an alternative route for the dissemination of ideas.
So, the dilemma comes in knowing when it is OK to use someone else’s work as a source of inspiration and when you are infringing copyright or patent law. The issues are complex and detailed and well beyond the scope of this book, but Bainbridge (2014) is a good resource to understand this area better.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N62
One answer to the previous question is that choosing between alternative designs is informed by letting users and stakeholders interact with them and by discussing their expe-riences, preferences, and suggestions for improvement. To do this, the designs must be in a form that can be reasonably evaluated by users, not in technical jargon or notation that seems impenetrable to them. Documentation is one traditional way to communicate a design, for example, a diagram showing the product’s components or a description of how it works. But a static description cannot easily capture the dynamics of behavior, and for an interactive product this needs to be communicated so that users can see what it will be like to operate it.
Prototyping is often used to overcome potential client misunderstandings and to test the technical feasibility of a suggested design and its production. It involves producing a limited version of the product with the purpose of answering specific questions about the design’s feasibility or appropriateness. Prototypes give a better impression of the user experience than simple descriptions; different kinds of prototyping are suitable for different stages of development and for eliciting different kinds of feedback. When a deployable version of the product is available, another way to choose between alternative designs is to deploy two dif-ferent variations and collect data from actual use that is then used to inform the choice. This is called A/B testing, and it is often used for alternative website designs (see Box 2.5).
Another basis on how to choose between alternatives is quality, but that requires a clear understanding of what quality means, and people’s views of quality vary. Everyone has a notion of the level of quality that is expected, wanted, or needed from a product. Whether this is expressed formally, informally, or not at all, it exists and informs the choice between alternatives. For example, one smartphone design might make it easy to access a popular music channel but restrict sound settings, while another requires more complicated key sequences to access the channel but has a range of sophisticated sound settings. One user’s view of quality may lean toward ease of use, while another may lean toward sophisticated sound settings.
Most projects involve a range of different stakeholder groups, and it is common for each of them to define quality differently and to have different acceptable limits for it. For exam-ple, although all stakeholders may agree on goals for a video game such as “characters will be appealing” or “graphics will be realistic,” the meaning of these statements can vary between different groups. Disputes will arise if, later in development, it transpires that “realistic” to a stakeholder group of teenage players is different from “realistic” to a group of parent stakeholders or to developers. Capturing these different views clearly clarifies expectations, provides a benchmark against which products and prototypes can be compared, and forms a basis on which to choose among alternatives.
The process of writing down formal, verifiable—and hence measurable—usability crite-ria is a key characteristic of an approach to interaction design called usability engineering. This has emerged over many years and with various proponents (Whiteside et al., 1988; Nielsen, 1993). Most recently, it is often applied in health informatics (for example, see Kushniruk et al., 2015). Usability engineering involves specifying quantifiable measures of product performance, documenting them in a usability specification, and assessing the prod-uct against them.
2 . 3 S O M E P R A C T I C A l I S S u E S 63
BOX 2.5 A/B Testing
A/B testing is an online method to inform the choice between two alternatives. It is most commonly used for comparing different versions of web pages or apps, but the principles and mathematics behind it came about in the 1920s (Gallo, 2017). In an interaction design context, different versions of web pages or apps are released for use by users performing their everyday tasks. Typically, users are unaware that they are contributing to an evaluation. This is a powerful way to involve users in choosing between alternatives because a huge number of users can be involved and the situations are authentic.
On the one hand, it’s a simple idea—give one set of users one version and a second set the other version and see which set scores more highly against the success criteria. But dividing up the sets, choosing the success criteria, and working out the metrics to use are nontrivial (for example, see Deng and Shi, 2016). Pushing this idea further, it is common to have “multivari-ate” testing in which several options are tried at once, so you end up doing A/B/C testing or even A/B/C/D testing.
ACTIvITY 2.5 Consider your product from Activity 2.1. Suggest some usability criteria that could be applied to determine its quality. Use the usability goals introduced in Chapter 1—effectiveness, effi-ciency, safety, utility, learnability, and memorability. Be as specific as possible. Check the crite-ria by considering exactly what to measure and how to measure its performance.
Then try to do the same thing for some of the user experience goals introduced in Chapter 1. (These relate to whether a system is satisfying, enjoyable, motivating, rewarding, and so on.)
CommentFinding measurable characteristics for some of these is not easy. Here are some suggestions, but there are others. Where possible, criteria that are measurable and specific are preferable. • Effectiveness: Identifying measurable criteria for this goal is particularly difficult since it is a combination of the other goals. For example, does the system support travel organiza-tion, choosing transport routes, booking accommodation, and so on? In other words, is the product used?
• Efficiency: Is it clear how to ask for recommendations from the product? How quickly does it identify a suitable route or destination details?
(Continued)
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N64
2.3.5 How to Integrate Interaction Design Activities Within Other Lifecycle ModelsAs illustrated in Chapter 1 (Figure 1.4), many other disciplines contribute to interaction design, and some of these disciplines have lifecycles of their own. Prominent among them are those associated with software development, and integrating interaction design activities within software development has been discussed for many years; for example, see Carmelo Ardito et al. (2014) and Ahmed Seffah et al. (2005).
The latest attempts to integrate these practices focus on agile software development. Agile methods began to emerge in the late 1990s. The most well-known of these are eXtreme Programming (Beck and Andres, 2005), Scrum (Schwaber and Beedle, 2002), and Kanban (Anderson, 2010). The Dynamic Systems Development Method (DSDM) (DSDM, 2014), although established before the current agile movement, also belongs to the agile family as it adheres to the agile manifesto. These methods differ, but they all stress the importance of iteration, early and repeated user feedback, being able to handle emergent requirements, and striking a good balance between flexibility and structure. They also all emphasize collabora-tion, face-to-face communication, streamlined processes to avoid unnecessary activities, and the importance of practice over process, that is, of getting work done.
The opening statement for the Manifesto for Agile Software Development (www .agilemanifesto.org/) reads as follows:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
• Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan
• Safety: How often does data get lost or is the wrong option chosen? This may be measured, for example, as the number of times this happens per trip.
• Utility: How many functions offered are used for every trip, how many every other trip, and how many are not used at all? How many tasks are difficult to complete in a reasonable time because functionality is missing or the right subtasks aren’t supported?
• Learnability: How long does it take for a novice user to be able to do a series of set tasks, for example, to book a hotel room in Paris near the meeting venue for the meeting dates, identify appropriate flights from Sydney to Wellington, or find out whether a visa is needed to go to China?
• Memorability: If the product isn’t used for a month, how many functions can the user remember how to perform? How long does it take to remember how to perform the most frequent task?
Finding measurable characteristics for the user experience criteria is harder. How do you measure satisfaction, fun, motivation, or aesthetics? What is entertaining to one person may be boring to another; these kinds of criteria are subjective and so cannot be measured as objectively.
2 . 3 S O M E P R A C T I C A l I S S u E S 65
This manifesto is underpinned by a series of principles, which range from communica-tion with the business to excellence of coding and maximizing the amount of work done. The agile approach to development is particularly interesting from the point of view of interac-tion design because it incorporates tight iterations and feedback and collaboration with the customer. For example, in Scrum, each sprint is between one and four weeks, with a product of value being delivered at the end of each sprint. Also, eXtreme1 Programming (XP) stipu-lates that the customer should be on-site with developers. In practice, the customer role is usually taken by a team rather than by one person (Martin et al., 2009), and integration is far from straightforward (Ferreira et al., 2012). Many companies have integrated agile meth-ods with interaction design practices to produce a better user experience and business value (Loranger and Laubheimer, 2017), but it is not necessarily easy, as discussed in Chapter 13, “Interaction Design in Practice.”
In-Depth ActivityThese days, timepieces (such as clocks, wristwatches, and so on) have a variety of functions. Not only do they tell the time and date, but they can speak to you, remind you when it’s time to do something, and record your exercise habits among other things. The interface for these devices, however, shows the time in one of two basic ways: as a digital number such as 11:40 or through an analog display with two or three hands—one to represent the hour, one for the minutes, and one for the seconds.
This in-depth activity is to design an innovative timepiece. This could be in the form of a wristwatch, a mantelpiece clock, a sculpture for a garden or balcony, or any other kind of timepiece you prefer. The goal is to be inventive and exploratory by following these steps:
(a) Think about the interactive product that you are designing: What do you want it to do? Find three to five potential users, and ask them what they would want. Write a list of requirements for the clock, together with some usability criteria and user experience cri-teria based on the definitions in Chapter 1.
(b) Look around for similar devices and seek out other sources of inspiration that you might find helpful. Make a note of any findings that are interesting, useful, or insightful.
(c) Sketch some initial designs for the timepiece. Try to develop at least two distinct alterna-tives that meet your set of requirements.
(d) Evaluate the two designs by using your usability criteria and by role-playing an interac-tion with your sketches. Involve potential users in the evaluation, if possible. Does it do what you want? Is the time or other information being displayed always clear? Design is iterative, so you may want to return to earlier elements of the process before you choose one of your alternatives.
1 The method is called extreme because it pushes a key set of good practices to the limit; that is, it is good practice to test often, so in XP the development is test-driven, and a complete set of tests is executed many times a day. It is good practice to talk to people about their requirements, so rather than having weighty documentation, XP reduces documentation to a minimum, thus forcing communication, and so on.
2 T H E P R O C E S S O F I N T E R A C T I O N D E S I G N66
Further Reading
ASHMORE, S. and RUNYAN, K. (2015) Introduction to Agile Methods, Addison Wesley. This book introduces the basics of agile software development and the most popular agile methods in an accessible way. It touches on usability issues and the relationship between agile and marketing. It is a good place to start for someone new to the agile way of working.
KELLEY, T., with LITTMAN, J. (2016) The Art of Innovation, Profile Books. Tom Kelley is a partner at IDEO. In this book, Kelley explains some of the innovative techniques used at
SummaryIn this chapter, we looked at user-centered design and the process of interaction design. That is, what is user-centered design, what activities are required in order to design an interactive product, and how are these activities related? A simple interaction design lifecycle model consisting of four activities was introduced, and issues surrounding the involvement and identification of users, generating alternative designs, evaluating designs, and integrating user-centered concerns with other lifecycles were discussed.
Key Points • Different design disciplines follow different approaches, but they have commonalities that are captured in the double diamond of design.
• It is important to have a good understanding of the problem space before trying to build anything.
• The interaction design process consists of four basic activities: discover requirements, design alternatives that meet those requirements, prototype the designs so that they can be communicated and assessed, and evaluate them.
• User-centered design rests on three principles: early focus on users and tasks, empirical measurement, and iterative design. These principles are also key for interaction design.
• Involving users in the design process assists with expectation management and feelings of ownership, but how and when to involve users requires careful planning.
• There are many ways to understand who users are and what their goals are in using a prod-uct, including rapid iterations of working prototypes.
• Looking at others’ designs and involving other people in design provides useful inspiration and encourages designers to consider alternative design solutions, which is key to effec-tive design.
• Usability criteria, technical feasibility, and users’ feedback on prototypes can all be used to choose among alternatives.
• Prototyping is a useful technique for facilitating user feedback on designs at all stages. • Interaction design activities are becoming better integrated with lifecycle models from other related disciplines such as software engineering.
F u R T H E R R E A D I N G 67
IDEO, but more importantly he talks about the culture and philosophy underlying IDEO’s success. There are some useful practical hints in here as well as an informative story about building and maintaining a successful design company.
PRESSMAN, R.S. and MAXIM, B.R. (2014) Software Engineering: A Practitioner’s Approach (Int’l Ed), McGraw-Hill Education. If you are interested in pursuing the software engineering aspects of the lifecycle models section, then this book provides a useful overview of the main models and their purpose.
SIROKER. D. and KOOMEN, P. (2015) A/B Testing: The Most Powerful Way to Turn Clicks into Customers, John Wiley. This book is written by two experienced practitioners who have been using A/B testing with a range of organizations. It is particularly interesting because of the example cases that show the impact that applying A/B testing successfully can have.
ROGERS. Y. (2014) Secrets of Creative People (PDF available from www.id-book.com/). This short book summarizes the findings from a two-year research project into creativity. It emphasizes the importance of different perspectives to creativity and describes how suc-cessful creativity arises from sharing, constraining, narrating, connecting, and even sparring with others.
Chapter 3
C O N C E P T U A L I Z I N G I N T E R A C T I O N
ObjectivesThe main goals of this chapter are to accomplish the following:
• Explain how to conceptualize interaction. • Describe what a conceptual model is and how to begin to formulate one. • Discuss the use of interface metaphors as part of a conceptual model. • Outline the core interaction types for informing the development of a conceptual model.
• Introduce paradigms, visions, theories, models, and frameworks informing interaction design.
3.1 Introduction
When coming up with new ideas as part of a design project, it is important to conceptualize them in terms of what the proposed product will do. Sometimes, this is referred to as creat-ing a proof of concept. In relation to the double diamond framework, it can be viewed as an initial pass to help define the area and also when exploring solutions. One reason for needing to do this is as a reality check where fuzzy ideas and assumptions about the benefits of the proposed product are scrutinized in terms of their feasibility: How realistic is it to develop what they have suggested, and how desirable and useful will it actually be? Another reason is to enable designers to begin articulating what the basic building blocks will be when develop-ing the product. From a user experience (UX) perspective, it can lead to better clarity, forcing designers to explain how users will understand, learn about, and interact with the product.
3.1 Introduction
3.2 Conceptualizing Interaction
3.3 Conceptual Models
3.4 Interface Metaphors
3.5 Interaction Types
3.6 Paradigms, Visions, Theories, Models, and Frameworks
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N70
For example, consider the bright idea that a designer has of creating a voice-assisted mobile robot that can help waiters in a restaurant take orders and deliver meals to customers (see Figure 3.1). The first question to ask is: why? What problem would this address? The designer might say that the robot could help take orders and entertain customers by having a conversation with them at the table. They could also make recommendations that can be customized to different customers, such as restless children or fussy eaters. However, none of these addresses an actual problem. Rather, they are couched in terms of the putative benefits of the new solution. In contrast, an actual problem identified might be the following: “It is difficult to recruit good wait staff who provide the level of customer service to which we have become accustomed.”
Having worked through a problem space, it is important to generate a set of research questions that need to be addressed, when considering how to design a robot voice interface to wait on customers. These might include the following: How intelligent does it have to be? How would it need to move to appear to be talking? What would the customers think of it? Would they think it is too gimmicky and get easily tired of it? Or, would it always be a pleas-ure for them to engage with the robot, not knowing what it would say on each new visit to the restaurant? Could it be designed to be a grumpy extrovert or a funny waiter? What might be the limitations of this voice-assisted approach?
Many unknowns need to be considered in the initial stages of a design project, especially if it is a new product that is being proposed. As part of this process, it can be useful to show where your novel ideas came from. What sources of inspiration were used? Is there any theory or research that can be used to inform and support the nascent ideas?
Asking questions, reconsidering one’s assumptions, and articulating one’s concerns and standpoints are central aspects of the early ideation process. Expressing ideas as a set of con-cepts greatly helps to transform blue-sky and wishful thinking into more concrete models of
Figure 3.1 A nonspeaking robot waiter in Shanghai. What would be gained if it could also talk with customers? Source: ZUMA Press / Alamy Stock Photo
3 . 2 C O N C E P T U A L I Z I N G I N T E R A C T I O N 71
how a product will work, what design features to include, and the amount of functionality that is needed. In this chapter, we describe how to achieve this through considering the dif-ferent ways of conceptualizing interaction.
3.2 Conceptualizing Interaction
When beginning a design project, it is important to be clear about the underlying assump-tions and claims. By an assumption, we mean taking something for granted that requires fur-ther investigation; for example, people now want an entertainment and navigation system in their cars. By a claim, we mean stating something to be true when it is still open to question. For instance, a multimodal style of interaction for controlling this system—one that involves speaking or gesturing while driving—is perfectly safe.
Writing down your assumptions and claims and then trying to defend and support them can highlight those that are vague or wanting. In so doing, poorly constructed design ideas can be reformulated. In many projects, this process involves identifying human activities and interactivities that are problematic and working out how they might be improved through being supported with a different set of functions. In others, it can be more speculative, requir-ing thinking through how to design for an engaging user experience that does not exist.
Box 3.1 presents a hypothetical scenario of a team working through their assumptions and claims; this shows how, in so doing, problems are explained and explored and leads to a specific avenue of investigation agreed on by the team.
BOX 3.1 Working Through Assumptions and Claims
This is a hypothetical scenario of early design highlighting the assumptions and claims (itali-cized) made by different members of a design team.
A large software company has decided that it needs to develop an upgrade of its web browser for smartphones because its marketing team has discovered that many of the com-pany’s customers have switched over to using another mobile browser. The marketing people assume that something is wrong with their browser and that their rivals have a better product. But they don’t know what the problem is with their browser.
The design team put in charge of this project assumes that they need to improve the usability of a number of the browser’s functions. They claim that this will win back users by making features of the interface simpler, more attractive, and more flexible to use.
The user researchers on the design team conduct an initial user study investigating how people use the company’s web browser on a variety of smartphones. They also look at other mobile web browsers on the market and compare their functionality and usability. They observe and talk to many different users. They discover several things about the usability of their web browser, some of which they were not expecting. One revelation is that many of their customers have never actually used the bookmarking tool. They present their findings to the rest of the team and have a long discussion about why each of them thinks it is not being used.
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N72
Explaining people’s assumptions and claims about why they think something might be a good idea (or not) enables the design team as a whole to view multiple perspectives on the problem space and, in so doing, reveals conflicting and problematic ones. The following framework is intended to provide a set of core questions to aid design teams in this process:
• Are there problems with an existing product or user experience? If so, what are they? • Why do you think there are problems? • What evidence do you have to support the existence of these problems? • How do you think your proposed design ideas might overcome these problems?
ACTIVITY 3.1 Use the framework in the previous list to guess what the main assumptions and claims were behind 3D TV. Then do the same for curved TV, which was designed to be bendy so as to make the viewing experience more immersive. Are the assumptions similar? Why were they problematic?
CommentThere was much hype and fanfare about the enhanced user experience 3D and curved TVs would offer, especially when watching movies, sports events, and dramas (see Figure 3.2).
One member claims that the web browser’s function for organizing bookmarks is tricky and error-prone, and she assumes that this is the reason why many users do not use it. Another member backs her up, saying how awkward it is to use this method when wanting to move bookmarks between folders. One of the user experience architects agrees, noting how several of the users with whom he spoke mentioned how difficult and time-consuming they found it when trying to move bookmarks between folders and how they often ended up accidentally putting them into the wrong folders.
A software engineer reflects on what has been said, and he makes the claim that the book-mark function is no longer needed since he assumes that most people do what he does, which is to revisit a website by flicking through their history of previously visited pages. Another member of the team disagrees with him, claiming that many users do not like to leave a trail of the sites they have visited and would prefer to be able to save only the sites that they think they might want to revisit. The bookmark function provides them with this option. Another option discussed is whether to include most-frequently visited sites as thumbnail images or as tabs. The software engineer agrees that providing all of the options could be a solution but worries how this might clutter the small screen interface.
After much discussion on the pros and cons of bookmarking versus history lists, the team decides to investigate further how to support effectively the saving, ordering, and retrieving of websites using a mobile web browser. All agree that the format of the existing web browser’s structure is too rigid and that one of their priorities is to see how they can create a simpler way of revisiting websites on a smartphone.
3 . 2 C O N C E P T U A L I Z I N G I N T E R A C T I O N 73
However, both never really took off. Why was this? One assumption for 3D TV was that people would not mind wearing the glasses that were needed to see in 3D, nor would they mind paying a lot more for a new 3D-enabled TV screen. A claim was that people would really enjoy the enhanced clarity and color detail provided by 3D, based on the favorable feedback received worldwide when viewing 3D films, such as Avatar, at a cinema. Similarly, an assump-tion made about curved TV was that it would provide more flexibility for viewers to optimize the viewing angles in someone’s living room.
The unanswered question for both concepts was this: Could the enhanced cinema view-ing experience that both claimed become an actual desired living room experience? There was no existing problem to overcome—what was being proposed was a new way of experiencing TV. The problem they might have assumed existed was that the experience of viewing TV at home was inferior to that of the cinema. The claim could have been that people would be prepared to pay more for a better-quality viewing experience more akin to that of the cinema.
But were people prepared to pay extra for a new TV because of this enhancement? A number of people did. However, a fundamental usability problem was overlooked—many people complained of motion sickness when watching 3D TV. The glasses were also easily lost. Moreover, wearing them made it difficult to do other things such as flicking through multiple channels, texting, and tweeting. (Many people simultaneously use additional devices, such as smartphones and tablets while watching TV.) Most people who bought 3D TVs stopped watching them after a while because of these usability problems. While curved TV didn’t require viewers to wear special glasses, it also failed because the actual benefits were not that significant relative to the cost. While for some the curve provided a cool aesthetic look and an improved viewing angle, for others it was simply an inconvenience.
Figure 3.2 A family watching 3D TV Source: Andrey Popov/Shutterstock
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N74
Making clear what one’s assumptions are about a problem and the claims being made about potential solutions should be carried out early on and throughout a project. Design teams also need to work out how best to conceptualize the design space. Primarily, this involves articulating the proposed solution as a conceptual model with respect to the user experience. The benefits of conceptualizing the design space in this way are as follows:
Orientation Enabling the design team to ask specific kinds of questions about how the conceptual model will be understood by the targeted users.
Open-Mindedness Allowing the team to explore a range of different ideas to address the problems identified.
Common Ground Allowing the design team to establish a set of common terms that all can understand and agree upon, reducing the chance of misunderstandings and confusion arising later.
Once formulated and agreed upon, a conceptual model can then become a shared blue-print leading to a testable proof of concept. It can be represented as a textual description and/or in a diagrammatic form, depending on the preferred lingua franca used by the design team. It can be used not just by user experience designers but also to communicate ideas to busi-ness, engineering, finance, product, and marketing units. The conceptual model is used by the design team as the basis from which they can develop more detailed and concrete aspects of the design. In doing so, design teams can produce simpler designs that match up with users’ tasks, allow for faster development time, result in improved customer uptake, and need less training and customer support (Johnson and Henderson, 2012).
3.3 Conceptual Models
A model is a simplified description of a system or process that helps describe how it works. In this section, we look at a particular kind of model used in interaction design intended to articulate the problem and design space—the conceptual model. In a later section, we describe more generally how models have been developed to explain phenomena in human-computer interaction.
Jeff Johnson and Austin Henderson (2002) define a conceptual model as “a high-level description of how a system is organized and operates” (p. 26). In this sense, it is an abstrac-tion outlining what people can do with a product and what concepts are needed to under-stand how to interact with it. A key benefit of conceptualizing a design at this level is that it enables “designers to straighten out their thinking before they start laying out their widg-ets” (p. 28).
In a nutshell, a conceptual model provides a working strategy and a framework of gen-eral concepts and their interrelations. The core components are as follows:
• Metaphors and analogies that convey to people how to understand what a product is used for and how to use it for an activity (for example browsing and bookmarking).
• The concepts to which people are exposed through the product, including the task-domain objects they create and manipulate, their attributes, and the operations that can be per-formed on them (such as saving, revisiting, and organizing).
• The relationships between those concepts (for instance, whether one object contains another).
3 . 3 C O N C E P T U A L M O d E L s 75
• The mappings between the concepts and the user experience the product is designed to support or invoke (for example, one can revisit a page through looking at a list of visited sites, most-frequently visited, or saved websites).
How the various metaphors, concepts, and their relationships are organized determines the user experience. By explaining these, the design team can debate the merits of providing different methods and how they support the main concepts, for example, saving, revisiting, categorizing, reorganizing, and their mapping to the task domain. They can also begin dis-cussing whether a new overall metaphor may be preferable that combines the activities of browsing, searching, and revisiting. In turn, this can lead the design team to articulate the kinds of relationships between them, such as containership. For example, what is the best way to sort and revisit saved pages, and how many and what types of containers should be used (for example, folders, bars, or panes)? The same enumeration of concepts can be repeated for other functions of the web browser—both current and new. In so doing, the design team can begin to work out systematically what will be the simplest and most effective and memorable way of supporting users while browsing the Internet.
The best conceptual models are often those that appear obvious and simple; that is, the operations they support are intuitive to use. However, sometimes applications can end up being based on overly complex conceptual models, especially if they are the result of a series of upgrades, where more and more functions and ways of doing something are added to the original conceptual model. While tech companies often provide videos showing what new features are included in an upgrade, users may not pay much attention to them or skip them entirely. Furthermore, many people prefer to stick to the methods they have always used and trusted and, not surprisingly, become annoyed when they find one or more have been removed or changed. For example, when Facebook rolled out its revised newsfeed a few years back, many users were unhappy, as evidenced by their post-ings and tweets, preferring the old interface that they had gotten used to. A challenge for software companies, therefore, is how best to introduce new features that they have added to an upgrade—and explain their assumed benefits to users—while also justifying why they removed others.
BOX 3.2 Design Concept
Another term that is sometimes used is a design concept. Essentially, it is a set of ideas for a design. Typically, it is composed of scenarios, images, mood boards, or text-based documents. For example, Figure 3.3 shows the first page of a design concept developed for an ambient display that was aimed at changing people’s behavior in a building, that is, to take the stairs instead of the elevator. Part of the design concept was envisioned as an animated pattern of twinkly lights that would be embedded in the carpet near the entrance of the building with the intention of luring people toward the stairs (Hazlewood et al., 2010).
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N76
Most interface applications are actually based on well-established conceptual models. For example, a conceptual model based on the core aspects of the customer experience when at a shopping mall underlies most online shopping websites. These include the placement of items that a customer wants to purchase into a shopping cart or basket and proceeding to checkout when they’re ready to make the purchase. Collections of patterns are now readily available to help design the interface for these core transactional processes, together with many other aspects of a user experience, meaning interaction designers do not have to start from scratch every time they design or redesign an application. Examples include patterns for online forms and navigation on mobile phones.
It is rare for completely new conceptual models to emerge that transform the way daily and work activities are carried out at an interface. Those that did fall into this category include the following three classics: the desktop (developed by Xerox in the late 1970s), the digital spreadsheet (developed by Dan Bricklin and Bob Frankston in the late 1970s), and the World Wide Web (developed by Tim Berners Lee in the early 1980s). All of these inno-vations made what was previously limited to a few skilled people accessible to all, while greatly expanding what is possible. The graphical desktop dramatically changed how office tasks could be performed (including creating, editing, and printing documents). Perform-ing these tasks using the computers prevalent at the time was significantly more arduous, having to learn and use a command language (such as DOS or UNIX). Digital spreadsheets made accounting highly flexible and easier to accomplish, enabling a diversity of new com-putations to be performed simply through filling in interactive boxes. The World Wide Web allowed anyone to browse a network of information remotely. Since then, e-readers and digi-tal authoring tools have introduced new ways of reading documents and books online, sup-porting associated activities such as annotating, highlighting, linking, commenting, copying,
Figure 3.3 The first page of a design concept for an ambient display
3 . 3 C O N C E P T U A L M O d E L s 77
and tracking. The web has also enabled and made many other kinds of activities easier, such as browsing for news, weather, sports, and financial information, as well as banking, shop-ping, and learning online among other tasks. Importantly, all of these conceptual models were based on familiar activities.
BOX 3.3 A Classic Conceptual Model: The Xerox Star
The Star interface, developed by Xerox in 1981 (see Figure 3.4), revolutionized the way that interfaces were designed for personal computing (Smith et al., 1982; Miller and Johnson, 1996) and is viewed as the forerunner of today’s Mac and Windows desktop interfaces. Origi-nally, it was designed as an office system, targeted at workers not interested in computing per se, and it was based on a conceptual model that included the familiar knowledge of an office. Paper, folders, filing cabinets, and mailboxes were represented as icons on the screen and were designed to possess some of the properties of their physical counterparts. Dragging a docu-ment icon across the desktop screen was seen as equivalent to picking up a piece of paper in the physical world and moving it (but this, of course, is a very different action). Similarly, dragging a digital document into a digital folder was seen as being analogous to placing a physical document into a physical cabinet. In addition, new concepts that were incorporated as part of the desktop metaphor were operations that could not be performed in the physical world. For example, digital files could be placed onto an icon of a printer on the desktop, resulting in the computer printing them out.
Figure 3.4 The Xerox Star Source: Used courtesy of Xerox
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N78
3.4 Interface Metaphors
Metaphors are considered to be a central component of a conceptual model. They provide a structure that is similar in some way to aspects of a familiar entity (or entities), but they also have their own behaviors and properties. More specifically, an interface metaphor is one that is instantiated in some way as part of the user interface, such as the desktop metaphor. Another well-known one is the search engine, originally coined in the early 1990s to refer to a software tool that indexed and retrieved files remotely from the Internet using various algorithms to match terms selected by the user. The metaphor invites comparisons between a mechanical engine, which has several working parts, and the everyday action of looking in different places to find something. The functions supported by a search engine also include other features besides those belonging to an engine that searches, such as listing and prior-itizing the results of a search. It also does these actions in quite different ways from how a mechanical engine works or how a human being might search a library for books on a given topic. The similarities implied by the use of the term search engine, therefore, are at a general level. They are meant to conjure up the essence of the process of finding relevant information, enabling the user to link these to less familiar aspects of the functionality provided.
Interface metaphors are intended to provide familiar entities that enable people read-ily to understand the underlying conceptual model and know what to do at the interface. However, they can also contravene people’s expectations about how things should be, such
Video The history of the Xerox Star at http://youtu.be/Cn4vC80Pv6Q.
ACTIVITY 3.2 Go to a few online stores and see how the interface has been designed to enable the customer to order and pay for an item. How many use the “add to shopping cart/basket” followed by the “checkout” metaphor? Does this make it straightforward and intuitive to make a purchase?
CommentMaking a purchase online usually involves spending money by inputting one’s credit/debit card details. People want to feel reassured that they are doing this correctly and do not get frustrated with lots of forms to fill in. Designing the interface to have a familiar metaphor (with an icon of a shopping cart/basket, not a cash register) makes it easier for people to know what to do at the different stages of making a purchase. Most important, placing an item in the basket does not commit the customer to purchase it there and then. It also enables them to browse further and select other items, as they might in a physical store.
3 . 4 I N T E R F A C E M E TA P h O R s 79
as the recycle bin (trash can) that sits on the desktop. Logically and culturally (meaning, in the real world), it should be placed under the desk. But users would not have been able to see it because it would have been hidden by the desktop surface. So, it needed to go on the desk-top. While some users found this irksome, most did not find it to be a problem. Once they understood why the recycle bin icon was on the desktop, they simply accepted it being there.
An interface metaphor that has become popular in the last few years is the card. Many of the social media apps, such as Facebook, Twitter, and Pinterest, present their content on cards. Cards have a familiar form, having been around for a long time. Just think of how many kinds there are: playing cards, business cards, birthday cards, credit cards, and post-cards to name a few. They have strong associations, providing an intuitive way of organizing limited content that is “card sized.” They can easily be flicked through, sorted, and themed. They structure content into meaningful chunks, similar to how paragraphs are used to chunk a set of related sentences into distinct sections (Babich, 2016). In the context of the smart-phone interface, the Google Now card provides short snippets of useful information. This appears on and moves across the screen in the way people would expect a real card to do—in a lightweight, paper-based sort of way. The elements are also structured to appear as if they were on a card of a fixed size, rather than, say, in a scrolling web page (see Figure 3.5).
Figure 3.5 Google Now card for restaurant recommendation in Germany Source: Used courtesy of Johannes Schöning
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N80
In many cases, new interface metaphors rapidly become integrated into common par-lance, as witnessed by the way people talk about them. For example, parents talk about how much screen time children are allowed each day in the same way they talk more generally about spending time. As such, the interface metaphors are no longer talked about as familiar terms to describe less familiar computer-based actions; they have become everyday terms in their own right. Moreover, it is hard not to use metaphorical terms when talking about technology use, as they have become so ingrained in the language that we use to express our-selves. Just ask yourself or someone else to describe Twitter and Facebook and how people use them. Then try doing it without using a single metaphor.
Albrecht Schmidt (2017) suggests a pair of glasses as a good metaphor for thinking about future technologies, helping us think more about how to amplify human cognition. Just as they are seen as an extension of ourselves that we are not aware of most of the time (except when they steam up!), he asks can we design new technologies that enable users to do things without having to think about how to use them? He contrasts this “amplify” meta-phor with the “tool” metaphor of a pair of binoculars that is used for a specific task—where someone consciously has to hold them up against their eyes while adjusting the lens to bring what they are looking at into focus. Current devices, like mobile phones, are designed more like binoculars, where people have to interact with them explicitly to perform tasks.
BOX 3.4 Why Are Metaphors So Popular?
People frequently use metaphors and analogies (here we use the terms interchangeably) as a source of inspiration for understanding and explaining to others what they are doing, or trying to do, in terms that are familiar to them. They are an integral part of human language (Lakoff and Johnson, 1980). Metaphors are commonly used to explain something that is unfamiliar or hard to grasp by way of comparison with something that is familiar and easy to grasp. For example, they are frequently employed in education, where teachers use them to introduce something new to students by comparing the new material with something they already understand. An example is the comparison of human evolution with a game. We are all familiar with the properties of a game: there are rules, each player has a goal to win (or lose), there are heuristics to deal with situations where there are no rules, there is the propen-sity to cheat when the other players are not looking, and so on. By conjuring up these proper-ties, the analogy helps us begin to understand the more difficult concept of evolution—how it happens, what rules govern it, who cheats, and so on.
It is not surprising, therefore, to see how widely metaphors have been used in interaction design to conceptualize abstract, hard-to-imagine, and difficult-to-articulate computer-based concepts and interactions in more concrete and familiar terms and as graphical visualizations at the interface level. Metaphors and analogies are used in these three main ways: • As a way of conceptualizing what we are doing (for instance, surfing the web) • As a conceptual model instantiated at the interface level (for example, the card metaphor) • As a way of visualizing an operation (such as an icon of a shopping cart into which items
are placed that users want to purchase on an online shopping site)
3 . 5 I N T E R A C T I O N T Y P E s 81
3.5 Interaction Types
Another way of conceptualizing the design space is in terms of the interaction types that will underlie the user experience. Essentially, these are the ways a person interacts with a product or application. Originally, we identified four main types: instructing, conversing, manipulating, and exploring (Preece et al., 2002). A fifth type has since been proposed by Christopher Lueg et al. (2019) that we have added to ours, which they call responding. This refers to proactive systems that initiate a request in situations to which a user can respond, for example, when Netflix pauses a person’s viewing to ask them whether they would like to continue watching.
Deciding upon which of the interaction types to use, and why, can help designers formu-late a conceptual model before committing to a particular interface in which to implement them, such as speech-based, gesture-based, touch-based, menu-based, and so on. Note that we are distinguishing here between interaction types (which we discuss in this section) and interface types (which are discussed in Chapter 7, “Interfaces”). While cost and other product constraints will often dictate which interface style can be used for a given application, consid-ering the interaction type that will best support a user experience can highlight the potential trade-offs, dilemmas, and pros and cons.
Here, we describe in more detail each of the five types of interaction. It should be noted that they are not meant to be mutually exclusive (for example, someone can interact with a system based on different kinds of activities); nor are they meant to be definitive. Also, the label used for each type refers to the user’s action even though the system may be the active partner in initiating the interaction.
• Instructing: Where users issue instructions to a system. This can be done in a number of ways, including typing in commands, selecting options from menus in a windows environ-ment or on a multitouch screen, speaking aloud commands, gesturing, pressing buttons, or using a combination of function keys.
• Conversing: Where users have a dialog with a system. Users can speak via an interface or type in questions to which the system replies via text or speech output.
• Manipulating: Where users interact with objects in a virtual or physical space by manipu-lating them (for instance, opening, holding, closing, and placing). Users can hone their familiar knowledge of how to interact with objects.
• Exploring: Where users move through a virtual environment or a physical space. Virtual environments include 3D worlds and augmented and virtual reality systems. They enable users to hone their familiar knowledge by physically moving around. Physical spaces that use sensor-based technologies include smart rooms and ambient environments, also ena-bling people to capitalize on familiarity.
• Responding: Where the system initiates the interaction and the user chooses whether to respond. For example, proactive mobile location-based technology can alert people to points of interest. They can choose to look at the information popping up on their phone or ignore it. An example is the Google Now Card, shown in Figure 3.5, which pops up a restaurant recommendation for the user to contemplate when they are walking nearby.
Besides these core activities of instructing, conversing, manipulating, exploring, and responding, it is possible to describe the specific domain and context-based activities in which users engage, such as learning, working, socializing, playing, browsing, writing, problem- solving, decision-making, and searching—just to name but a few. Malcolm McCullough (2004)
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N82
suggests describing them as situated activities, organized by work (for example, presenting to groups), home (such as resting), in town (for instance, eating), and on the road (for example, walking). The rationale for classifying activities in this way is to help designers be more sys-tematic when thinking about the usability of technology-modified places in the environment. In the following sections we illustrate in more detail the five core interaction types and how to design applications for them.
3.5.1 InstructingThis type of interaction describes how users carry out their tasks by telling the system what to do. Examples include giving instructions to a system to perform operations such as tell the time, print a file, and remind the user of an appointment. A diverse range of products has been designed based on this model, including home entertainment systems, consumer electronics, and computers. The way in which the user issues instructions can vary from pressing buttons to typing in strings of characters. Many activities are readily supported by giving instructions.
In Windows and other graphical user interfaces (GUIs), control keys or the selection of menu options via a mouse, touch pad, or touch screen are used. Typically, a wide range of functions are provided from which users have to select when they want to do something to the object on which they are working. For example, a user writing a report using a word processor will want to format the document, count the number of words typed, and check the spelling. The user instructs the system to do these operations by issuing appropriate commands. Typically, commands are carried out in a sequence, with the system responding appropriately (or not) as instructed.
One of the main benefits of designing an interaction based on issuing instructions is that the interaction is quick and efficient. It is particularly fitting where there is a frequent need to repeat actions performed on multiple objects. Examples include the repetitive actions of saving, deleting, and organizing files.
ACTIVITY 3.3 There are many different kinds of vending machines in the world. Each offers a range of goods, requiring users to part with some of their money. Figure 3.6 shows photos of two dif-ferent types of vending machines: one that provides soft drinks and the other that delivers a range of snacks. Both machines use an instructional mode of interaction. However, the way they do so is quite different.
What instructions must be issued to obtain a soda from the first machine and a bar of chocolate from the second? Why has it been necessary to design a more complex mode of inter-action for the second vending machine? What problems can arise with this mode of interaction?
CommentThe first vending machine has been designed using simple instructions. There is a small num-ber of drinks from which to choose, and each is represented by a large button displaying the label of each drink. The user simply has to press one button, and this will have the effect of delivering the selected drink. The second machine is more complex, offering a wider range of snacks. The trade-off for providing more options, however, is that the user can no longer
3 . 5 I N T E R A C T I O N T Y P E s 83
3.5.2 ConversingThis form of interaction is based on the idea of a person having a conversation with a system, where the system acts as a dialogue partner. In particular, the system is designed to respond in a way that another human being might when having a conversation. It differs from the activ-ity of instructing insofar as it encompasses a two-way communication process, with the sys-tem acting like a partner rather than a machine that obeys orders. It has been most commonly used for applications where the user needs to find out specific kinds of information or wants to discuss issues. Examples include advisory systems, help facilities, chatbots, and robots.
instruct the machine using a simple one-press action but is required to follow a more complex process involving (1) reading off the code (for example, C12) under the item chosen, then (2) keying this into the number pad adjacent to the displayed items, and finally (3) checking the price of the selected option and ensuring that the amount of money inserted is the same or greater (depending on whether the machine provides change). Problems that can arise from this type of interaction are the customer misreading the code and/or incorrectly keying the code, resulting in the machine not issuing the snack or providing the wrong item.
A better way of designing an interface for a large number of options of variable cost might be to continue to use direct mapping but use buttons that show miniature versions of the snacks placed in a large matrix (rather than showing actual versions). This would use the available space at the front of the vending machine more economically. The customer would need only to press the button of the object chosen and put in the correct amount of money. There is a lower chance of error resulting from pressing the wrong code or keys. The trade-off for the vending company, however, is that the machine is less flexible in terms of which snacks it can sell. If a new product line comes out, they will also need to replace part of the physical interface to the machine, which would be costly.
Figure 3.6 Two different types of vending machine
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N84
The kinds of conversation that are currently supported range from simple voice- recognition, menu-driven systems, to more complex natural language–based systems that involve the system parsing and responding to queries typed in or spoken by the user. Examples of the former include banking, ticket booking, and train-time inquiries, where the user talks to the system in single-word phrases and numbers, that is, yes, no, three, and so on, in response to prompts from the system. Examples of the latter include help systems, where the user types in a specific query, such as “How do I change the margin widths?” to which the system responds by giving various answers. Advances in AI during the last few years have resulted in a significant improvement in speech recognition to the extent that many companies now routinely employ speech-based and chatbot-based interaction for their customer queries.
A main benefit of developing a conceptual model that uses a conversational style of inter-action is that it allows people to interact with a system in a way that is familiar to them. For example, Apple’s speech system, Siri, lets you talk to it as if it were another person. You can ask it to do tasks for you, such as make a phone call, schedule a meeting, or send a message. You can also ask it indirect questions that it knows how to answer, such as “Do I need an umbrella today?” It will look up the weather for where you are and then answer with some-thing like, “I don’t believe it’s raining” while also providing a weather forecast (see Figure 3.7).
Figure 3.7 Siri’s response to the question “Do I need an umbrella today?”
3 . 5 I N T E R A C T I O N T Y P E s 85
A problem that can arise from using a conversational-based interaction type is that certain kinds of tasks are transformed into cumbersome and one-sided interactions. This is especially true for automated phone-based systems that use auditory menus to advance the interaction. Users have to listen to a voice providing several options, then make a selection, and repeat through further layers of menus before accomplishing their goal, for example, reaching a real human or paying a bill. Here is the beginning of a dialogue between a user who wants to find out about car insurance and an insurance company’s phone reception system:
<user dials an insurance company>“Welcome to St. Paul’s Insurance Company. Press 1 if you are a new customer; 2 if you are an existing customer.”<user presses 1>“Thank you for calling St. Paul’s Insurance Company. If you require house insurance, say 1; car insurance, say 2; travel insurance, say 3; health insurance, say 4; other, say 5.”<user says 2>“You have reached the car insurance division. If you require information about fully comprehensive insurance, say 1; third-party insurance, say 2. …”
3.5.3 ManipulatingThis form of interaction involves manipulating objects, and it capitalizes on users’ knowledge of how they do so in the physical world. For example, digital objects can be manipulated by moving, selecting, opening, and closing. Extensions to these actions include zooming in and out, stretching, and shrinking—actions that are not possible with objects in the real world. Human actions can be imitated through the use of physical controllers (for example, the Wii) or gestures made in the air, such as the gesture control technology now used in some cars. Physical toys and robots have also been embedded with technology that enable them to act and react in ways depending on whether they are squeezed, touched, or moved. Tagged physical objects (such as balls, bricks, or blocks) that are manipulated in a physical world (for example, placed on a surface) can result in other physical and digital events occurring, such as a lever moving or a sound or animation being played.
A framework that has been highly influential (originating from the early days of HCI) in guiding the design of GUI applications is direct manipulation (Shneiderman, 1983). It pro-poses that digital objects be designed at the interface level so that they can be interacted with in ways that are analogous to how physical objects in the physical world are manipulated.
Source: © Glasbergen. Reproduced with permission of Glasbergen Cartoon Service
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N86
In so doing, direct manipulation interfaces are assumed to enable users to feel that they are directly controlling the digital objects represented by the computer. The three core principles are as follows:
• Continuous representation of the objects and actions of interest • Rapid reversible incremental actions with immediate feedback about the object of interest • Physical actions and button pressing instead of issuing commands with complex syntax
According to these principles, an object on the screen remains visible while a user per-forms physical actions on it, and any actions performed on it are immediately visible. For example, a user can move a file by dragging an icon that represents it from one part of the desktop to another. The benefits of direct manipulation include the following:
• Helping beginners learn basic functionality rapidly • Enabling experienced users to work rapidly on a wide range of tasks • Allowing infrequent users to remember how to carry out operations over time • Preventing the need for error messages, except rarely • Showing users immediately how their actions are furthering their goals • Reducing users’ experiences of anxiety • Helping users gain confidence and mastery and feel in control
Many apps have been developed based on some form of direct manipulation, includ-ing word processors, video games, learning tools, and image editing tools. However, while direct manipulation interfaces provide a versatile mode of interaction, they do have their drawbacks. In particular, not all tasks can be described by objects, and not all actions can be undertaken directly. Some tasks are also better achieved through issuing commands. For example, consider how you edit a report using a word processor. Suppose that you had ref-erenced work by Ben Shneiderman but had spelled his name as Schneiderman throughout. How would you correct this error using a direct manipulation interface? You would need to read the report and manually select the c in every Schneiderman, highlight it, and then delete it. This would be tedious, and it would be easy to miss one or two. By contrast, this opera-tion is relatively effortless and also likely to be more accurate when using a command-based interaction. All you need to do is instruct the word processor to find every Schneiderman and replace it with Shneiderman. This can be done by selecting a menu option or using a combination of command keys and then typing the changes required into the dialog box that pops up.
3.5.4 ExploringThis mode of interaction involves users moving through virtual or physical environments. For example, users can explore aspects of a virtual 3D environment, such as the interior of a building. Physical environments can also be embedded with sensing technologies that, when they detect the presence of someone or certain body movements, respond by triggering cer-tain digital or physical events. The basic idea is to enable people to explore and interact with an environment, be it physical or digital, by exploiting their knowledge of how they move and navigate through existing spaces.
Many 3D virtual environments have been built that comprise digital worlds designed for people to move between various spaces to learn (for example, virtual campuses) and fantasy worlds where people wander around different places to socialize (for instance, virtual parties)
3 . 5 I N T E R A C T I O N T Y P E s 87
or play video games (such as Fortnite). Many virtual landscapes depicting cities, parks, build-ings, rooms, and datasets have also been built, both realistic and abstract, that enable users to fly over them and zoom in and out of different parts. Other virtual environments that have been built include worlds that are larger than life, enabling people to move around them, experiencing things that are normally impossible or invisible to the eye (see Figure 3.8a); highly realistic representations of architectural designs, allowing clients and customers to imagine how they will use and move through planned buildings and public spaces; and visu-alizations of complex datasets that scientists can virtually climb inside and experience (see Figure 3.8b).
3.5.5 RespondingThis mode of interaction involves the system taking the initiative to alert, describe, or show the user something that it “thinks” is of interest or relevance to the context the user is pres-ently in. It can do this through detecting the location and/or presence of someone in a vicinity (for instance, a nearby coffee bar where friends are meeting) and notifying them about it on their phone or watch. Smartphones and wearable devices are becoming increasingly proactive in initiating user interaction in this way, rather than waiting for the user to ask, command, explore, or manipulate. An example is a fitness tracker that notifies the user of a milestone they have reached for a given activity, for example, having walked 10,000 steps in a day. The fitness tracker does this automatically without any requests made by the user; the user responds by looking at the notification on their screen or listening to an audio announce-ment that is made. Another example is when the system automatically provides some funny or useful information for the user, based on what it has learned from their repeated behaviors when carrying out particular actions in a given context. For example, after taking a photo
(a) (b)
Figure 3.8 (a) A CAVE that enables the user to stand near a huge insect, for example, a beetle, be swallowed, and end up in its abdomen; and (b) NCSA’s CAVE being used by a scientist to move through 3D visualizations of the datasets Source: (a) Used courtesy of Alexei Sharov (b) Used courtesy of Kalev Leetaru, National Center for Supercom-puting Applications, University of Illinois.
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N88
of a friend’s cute dog in the park, Google Lens will automatically pop up information that identifies the breed of the dog (see Figure 3.9).
For some people, this kind of system-initiated interaction—where additional informa-tion is provided which has not been requested—might get a bit tiresome or frustrating, espe-cially if the system gets it wrong. The challenge is knowing when the user will find it useful and interesting and how much and what kind of contextual information to provide without overwhelming or annoying them. Also, it needs to know what to do when it gets it wrong. For example, if it thinks the dog is a teddy bear, will it apologize? Will the user be able to correct it and tell it what the photo actually is? Or will the system be given a second chance?
3.6 Paradigms, Visions, Theories, Models, and Frameworks
Other sources of conceptual inspiration and knowledge that are used to inform design and guide research are paradigms, visions, theories, models, and frameworks (Carroll, 2003). These vary in terms of their scale and specificity to a particular problem space. A paradigm refers to a general approach that has been adopted by a community of researchers and designers for carrying out their work in terms of shared assumptions, concepts, values, and practices. A vision is a future scenario that frames research and development in interaction design—often depicted in the form of a film or a narrative. A theory is a well-substantiated explana-tion of some aspect of a phenomenon; for example, the theory of information processing that explains how the mind, or some aspect of it, is assumed to work. A model is a simplification of some aspect of human-computer interaction intended to make it easier for designers to predict and evaluate alternative designs. A framework is a set of interrelated concepts and/or
Figure 3.9 Google Lens in action, providing pop-up information about Pembroke Welsh Corgi having recognized the image as oneSource: https://lens.google.com
3 . 6 PA R A d I G M s , V I s I O N s , T h E O R I E s , M O d E L s , A N d F R A M E w O R k s 89
a set of specific questions that are intended to inform a particular domain area (for example, collaborative learning), or an analytic method (for instance, ethnographic studies).
3.6.1 ParadigmsFollowing a particular paradigm means adopting a set of practices upon which a community has agreed. These include the following:
• The questions to be asked and how they should be framed • The phenomena to be observed • The way in which findings from studies are to be analyzed and interpreted (Kuhn, 1972)
In the 1980s, the prevailing paradigm in human-computer interaction was how to design user-centered applications for the desktop computer. Questions about what and how to design were framed in terms of specifying the requirements for a single user interacting with a screen-based interface. Task analytic and usability methods were developed based on an individual user’s cognitive capabilities. Windows, Icons, Menus, and Pointers (WIMP) was used as a way of characterizing the core features of an interface for a single user. This was later superseded by the graphical user interface (GUI). Now many interfaces have touch screens that users tap, press and hold, pinch, swipe, slide, and stretch.
A big influence on the paradigm shift that took place in HCI in the 1990s was Mark Weiser’s (1991) vision of ubiquitous technology. He proposed that computers would become part of the environment, embedded in a variety of everyday objects, devices, and displays. He envisioned a world of serenity, comfort, and awareness, where people were kept perpetu-ally informed of what was happening around them, what was going to happen, and what had just happened. Ubiquitous computing devices would enter a person’s center of attention when needed and move to the periphery of their attention when not, enabling the person to switch calmly and effortlessly between activities without having to figure out how to use a computer when performing their tasks. In essence, the technology would be unobtrusive and largely disappear into the background. People would be able to get on with their everyday and working lives, interacting with information and communicating and collaborating with others without being distracted or becoming frustrated with technology.
This vision was successful at influencing the computing community’s thinking; inspiring them especially regarding what technologies to develop and problems to research (Abowd, 2012). Many HCI researchers began to think beyond the desktop and design mobile and perva-sive technologies. An array of technologies was developed that could extend what people could do in their everyday and working lives, such as smart glasses, tablets, and smartphones.
The next big paradigm shift that took place in the 2000s was the emergence of Big Data and the Internet of Things (IoT). New and affordable sensor technologies enabled masses of data to be collected about people’s health, well-being, and real-time changes happening in the environment (for example, air quality, traffic congestion, and business). Smart buildings were also built, where an assortment of sensors were embedded and experimented with in homes, hospitals, and other public buildings. Data science and machine-learning algorithms were developed to analyze the amassed data to draw new inferences about what actions to take on behalf of people to optimize and improve their lives. This included introducing variable speed limits on highways, notifying people via apps of dangerous pollution levels, crowds at an airport, and so on. In addition, it became the norm for sensed data to be used to automate mundane operations and actions—such as turning lights or faucets on and off or flushing toilets automatically—replacing conventional knobs, buttons, and other physical controls.
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N90
3.6.2 VisionsVisions of the future, like Mark Weiser’s vision of ubiquitous technology, provide a powerful driving force that can lead to a paradigm shift in terms of what research and development is carried out in companies and universities. A number of tech companies have produced videos about the future of technology and society, inviting audiences to imagine what life will be like in 10, 15, or 20 years’ time. One of the earliest was Apple’s 1987 Knowledge Navigator, which presented a scenario of a professor using a touchscreen tablet with a speech-based intelligent assistant reminding him of what he needed to do that day while answering the phone and helping him prepare his lectures. It was 25 years ahead of its time—set in 2011—the actual year that Apple launched its speech system, Siri. It was much viewed and discussed, inspiring widespread research into and development of future interfaces.
A current vision that has become pervasive is AI. Both utopian and dystopian visions are being bandied about on how AI will make our lives easier on the one hand and how it will take our jobs away on the other. This time, it is not just computer scientists who are extolling the benefits or dangers of AI advances for society but also journalists, social commentators, policy-makers, and bloggers. AI is now replacing the user interface for an increasing number of applications where the user had to make choices, for example, smart-phones learning your music preferences and home heating systems deciding when to turn the heating on and off and what temperature you prefer. One objective is to reduce the stress of people having to make decisions; another is to improve upon what they would choose. For example, in the future instead of having to agonize over which clothes to buy, or vacation to select, a personal assistant will be able to choose on your behalf. Another example depicts what a driverless car will be like in a few years, where the focus is not so much on current concerns with safety and convenience but more on improving comfort and life quality in terms of the ultimate personalized passenger experience (for example, see VW’s video). More and more everyday tasks will be transformed through AI learning what choices are best in a given situation.
You can watch a video about the Apple Knowledge Navigator here: http://youtu .be/hGYFEI6uLy0.
VW’s vision of its future car can be seen in this video: https://youtu.be/AyihacflLto.
Video IBM’s Internet of Things: http://youtu.be/sfEbMV295kk.
3 . 6 PA R A d I G M s , V I s I O N s , T h E O R I E s , M O d E L s , A N d F R A M E w O R k s 91
While there are many benefits of letting machines make decisions for us, we may feel a loss of control. Moreover, we may not understand why the AI system chose to drive the car along a particular route or why our voice-assisted home robot keeps ordering too much milk. There are increasing expectations that AI researchers find ways of explaining the rationale behind the decisions that AI systems make on the user’s behalf. This need is often referred to as transparency and accountability—which we discuss further in Chapter 10. It is an area that is of central concern to interaction design researchers, who have started conducting user studies on transparency and developing explanations that are meaningful and reassuring to the user (e.g., Radar et al., 2018).
Another challenge is to develop new kinds of interfaces and conceptual models that can support the synergy of humans and AI systems, which will amplify and extend what they can do currently. This could include novel ways of enhancing group collaboration, creative problem-solving, forward planning, policy-making, and other areas that can become intrac-table, complex, and messy, such as divorce settlements.
Science fiction has also become a source of inspiration in interaction design. By this, we mean in movies, writing, plays, and games that envision what role technology may play in the future. Dan Russell and Svetlana Yarosh (2018) discuss the pros and cons of using dif-ferent kinds of science fiction for inspiration in HCI design, arguing that they can provide a good grounding for debate but are often not a great source of accurate predictions of future technologies. They point out how, although the visions can be impressively futuristic, their embellishments and what they actually look like are often limited by the author’s ability to extend and build upon the ideas and the cultural expectations associated with the current era. For example, the holodeck portrayed in the Star Trek TV series had 3D-bubble indicator lights and push-button designs on its bridge with the sound of a teletype in the background. This is the case to such an extent that Russell and Yarosh even argue that the priorities and concerns of the author’s time and their cultural upbringing can bias the science fiction toward telling narratives from the perspective of the present, rather than providing new insights and paving the way to future designs.
The different kinds of future visions provide concrete scenarios of how society can use the next generation of imagined technologies to make their lives more comfortable, safe, informative, and efficient. Furthermore, they also raise many questions concerning privacy, trust, and what we want as a society. They provide much food for thought for research-ers, policy-makers, and developers, challenging them to consider both positive and negative implications.
Many new challenges, themes, and questions have been articulated through such visions (see, for example, Rogers, 2006; Harper et al., 2008; Abowd, 2012), including the following:
• How to enable people to access and interact with information in their work, social, and everyday lives using an assortment of technologies
• How to design user experiences for people using interfaces that are part of the environ-ment but where there are no obvious controlling devices
• How and in what form to provide contextually relevant information to people at appropri-ate times and places to support them while on the move
• How to ensure that information that is passed around via interconnected displays, devices, and objects is secure and trustworthy
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N92
3.6.3 TheoriesOver the past 30 years, numerous theories have been imported into human-computer interac-tion, providing a means of analyzing and predicting the performance of users carrying out tasks for specific types of computer interfaces and systems (Rogers, 2012). These have been primarily cognitive, social, affective, and organizational in origin. For example, cognitive theories about human memory were used in the 1980s to determine the best ways of repre-senting operations, given people’s memory limitations. One of the main benefits of applying such theories in interaction design is to help identify factors (cognitive, social, and affective) relevant to the design and evaluation of interactive products. Some of the most influential theories in HCI, including distributed cognition, will be covered in the next chapter.
3.6.4 ModelsWe discussed earlier why a conceptual model is important and how to generate one when designing a new product. The term model has also been used more generally in interaction design to describe, in a simplified way, some aspect of human behavior or human-computer interaction. Typically, it depicts how the core features and processes underlying a phenom-enon are structured and related to one another. It is usually abstracted from a theory coming from a contributing discipline, like psychology. For example, Don Norman (1988) developed a number of models of user interaction based on theories of cognitive processing, arising out of cognitive science, which were intended to explain the way users interacted with interactive technologies. These include the seven stages of action model that describes how users move from their plans to executing physical actions that they need to perform to achieve them to evaluating the outcome of their actions with respect to their goals. More recent models developed in interaction design are user models, which predict what information users want in their interactions and models that characterize core components of the user experience, such as Marc Hassenzahl’s (2010) model of experience design.
3.6.5 FrameworksNumerous frameworks have been introduced in interaction design to help designers con-strain and scope the user experience for which they are designing. In contrast to a model, a framework offers advice to designers as to what to design or look for. This can come in a variety of forms, including steps, questions, concepts, challenges, principles, tactics, and dimensions. Frameworks, like models, have traditionally been based on theories of human behavior, but they are increasingly being developed from the experiences of actual design practice and the findings arising from user studies.
Many frameworks have been published in the HCI/interaction design literature, cover-ing different aspects of the user experience and a diversity of application areas. For example, there are frameworks for helping designers think about how to conceptualize learning, work-ing, socializing, fun, emotion, and so on, and others that focus on how to design particular kinds of technologies to evoke certain responses, for example, persuasive technologies (see Chapter 6, “Emotional Interaction”). There are others that have been specifically developed to help researchers analyze the qualitative data they collect in a user study, such as Dis-tributed Cognition (Rogers, 2012). One framework, called DiCoT (Furniss and Blandford, 2006), was developed to analyze qualitative data at the system level, allowing researchers to understand how technologies are used by teams of people in work or home settings. (Chapter 9, “Data Analysis,” describes DiCoT in more detail.)
3 . 6 PA R A d I G M s , V I s I O N s , T h E O R I E s , M O d E L s , A N d F R A M E w O R k s 93
A classic example of a conceptual framework that has been highly influential in HCI is Don Norman’s (1988) explanation of the relationship between the design of a conceptual model and a user’s understanding of it. The framework comprises three interacting compo-nents: the designer, the user, and the system. Behind each of these are the following:
Designer’s Model The model the designer has of how the system should workSystem Image How the system actually works, which is portrayed to the user through the
interface, manuals, help facilities, and so onUser’s Model How the user understands how the system works
The framework makes explicit the relationship between how a system should function, how it is presented to users, and how it is understood by them. In an ideal world, users should be able to carry out activities in the way intended by the designer by interacting with the sys-tem image that makes it obvious what to do. If the system image does not make the designer’s model clear to the users, it is likely that they will end up with an incorrect understanding of the system, which in turn will increase the likelihood of their using the system ineffectively and making errors. This has been found to happen often in the real world. By drawing atten-tion to this potential discrepancy, designers can be made aware of the importance of trying to bridge the gap more effectively.
To summarize, paradigms, visions, theories, models, and frameworks are not mutually exclusive, but rather they overlap in their way of conceptualizing the problem and design space, varying in their level of rigor, abstraction, and purpose. Paradigms are overarching approaches that comprise a set of accepted practices and framing of questions and phe-nomena to observe; visions are scenarios of the future that set up challenges, inspirations, and questions for interaction design research and technology development; theories tend to be comprehensive, explaining human-computer interactions; models are an abstraction that simplify some aspect of human-computer interaction, providing a basis for designing and evaluating systems; and frameworks provide a set of core concepts, questions, or principles to consider when designing for a user experience or analyzing data from a user study.
dILEMMAWho Is in Control?
A recurrent theme in interaction design, especially in the current era of AI-based systems, is who should be in control at the interface level. The different interaction types vary in terms of how much control a user has and how much the computer has. While users are primarily in control for instructing direct manipulation interfaces, they are less so in responding type interfaces, such as sensor-based and context-aware environments where the system takes the initiative to act. User-controlled interaction is based on the premise that people enjoy mastery and being in control. It assumes that people like to know what is going on, be involved in the action, and have a sense of power over the computer.
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N94
In contrast, autonomous and context-aware control assumes that having the environ-ment monitor, recognize, and detect deviations in a person’s behavior can enable timely, help-ful, and even critical information to be provided when considered appropriate (Abowd and Mynatt, 2000). For example, elderly people’s movements can be detected in the home and emergency or care services alerted if something untoward happens to them that might other-wise go unnoticed, for instance, if they fall over and are unable to sound the alarm. But what happens if a person chooses to take a rest in an unexpected area (on the carpet), which the system detects as a fall? Will the emergency services be called out unnecessarily and cause care givers undue worry? Will the person who triggered the alarm be mortified at triggering a false alarm? And how will it affect their sense of privacy, knowing that their every move is constantly being monitored?
Another concern is what happens when the locus of control switches between user and system. For example, consider who is in control when using a GPS for vehicle navigation. At the beginning, the driver is very much in control, issuing instructions to the system as to where to go and what to include, such as highways, gas stations, and traffic alerts. However, once on the road, the system takes over and is in control. People often find themselves slavishly follow-ing what the GPS tells them to do, even though common sense suggests otherwise.
To what extent do you need to be in control in your everyday and working life? Are you happy to let technology monitor and decide what you need or might be interested in know-ing, or do you prefer to tell it what you want to do? What do you think of autonomous cars that drive for you? While it might be safer and more fuel-efficient, will it take the pleasure out of driving?
A tongue-in-cheek video made by Superflux, called Uninvited Guests, shows who is very much in control when a man is given lots of smart gadgets by his children for his birthday to help him live more healthily: https://vimeo.com/128873380.
Source: Cluff / Cartoon Stock
3 . 6 PA R A d I G M s , V I s I O N s , T h E O R I E s , M O d E L s , A N d F R A M E w O R k s 95
In-depth ActivityThe aim of this in-depth activity is for you to think about the appropriateness of differ-ent kinds of conceptual models that have been designed for similar physical and digital information artifacts.Compare the following: • A paperback book and an ebook • A paper-based map and a smartphone map app
What are the main concepts and metaphors that have been used for each? (Think about the way time is conceptualized for each of them.) How do they differ? What aspects of the paper-based artifact have informed the digital app? What is the new functionality? Are any aspects of the conceptual model confusing? What are the pros and cons?
summaryThis chapter explained the importance of conceptualizing the problem and design spaces before trying to build anything. It stressed throughout the need to be explicit about the claims and assumptions behind design decisions that are suggested. It described an approach to formu-lating a conceptual model and explained the evolution of interface metaphors that have been designed as part of the conceptual model. Finally, it considered other ways of conceptualizing interaction in terms of interaction types, paradigms, visions, theories, models, and frameworks.
Key Points • A fundamental aspect of interaction design is to develop a conceptual model. • A conceptual model is a high-level description of a product in terms of what users can do with it and the concepts they need to understand how to interact with it.
• Conceptualizing the problem space in this way helps designers specify what it is they are doing, why, and how it will support users in the way intended.
• Decisions about conceptual design should be made before commencing physical design (such as choosing menus, icons, dialog boxes).
• Interface metaphors are commonly used as part of a conceptual model. • Interaction types (for example, conversing or instructing) provide a way of thinking about how best to support the activities users will be doing when using a product or service.
• Paradigms, visions, theories, models, and frameworks provide different ways of framing and informing design and research.
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N96
Further Reading
Here we recommend a few seminal readings on interaction design and the user experience (in alphabetical order).
DOURISH, P. (2001) Where the Action Is. MIT Press. This book presents an approach for thinking about the design of user interfaces and user experiences based on the notion of embodied interaction. The idea of embodied interaction reflects a number of trends that have emerged in HCI, offering new sorts of metaphors.
JOHNSON, J. and HENDERSON, A. (2012) Conceptual Models: Core to Good Design. Morgan and Claypool Publishers. This short book, in the form of a lecture, provides a com-prehensive overview of what is a conceptual model using detailed examples. It outlines how to construct one and why it is necessary to do so. It is cogently argued and shows how and where this design activity can be integrated into interaction design.
JU, W. (2015) The Design of Implicit Interactions. Morgan and Claypool Publishers. This short book, in the form of a lecture, provides a new theoretical framework to help design smart, automatic, and interactive devices by examining the small interactions that we engage in our everyday lives, often without any explicit communication. It puts forward the idea of implicit interaction as a central concept for designing future interfaces.
97
INTERVIEw with Albrecht schmidt
Albrecht Schmidt is professor of human- centered ubiquitous media in the com-puter science department of the Ludwig- Maximilians-Universität München in Ger-many. He studied computer science in Ulm and Manchester and received a PhD from Lancaster University, United Kingdom, in 2003. He held several prior academic positions at different universities, includ-ing Stuttgart, Cambridge, Duisburg-Essen, and Bonn. He also worked as a researcher at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS) and at Microsoft Research in Cambridge. In his research, he investigates the inherent complexity of human-computer interaction in ubiquitous computing environments, particularly in view of increasing computer intelligence and system autonomy. Albrecht has actively contributed to the scientific discourse in human-computer interaction through the development, deployment, and study of functional prototypes of interac-tive systems and interface technologies in different real world domains. Most recently, he focuses on how information technol-ogy can provide cognitive and perceptual support to amplify the human mind.
How do you think future visions inspire research in interaction design? Can you give an example from your own work?
Envisioning the future is key to research in human-computer interaction. In contrast to traditional fields that discover phenom-ena (such as physics or sociology), research in interaction design is constructive and creates new things that potentially change our world. Research in interaction design also analyzes the world and aims to under-stand phenomena but mainly as a means to inspire and guide innovations. A major aspect of research is then to create concrete designs, build concepts and prototypes, and evaluate them.
Future visions are an excellent way to describe the big picture of a future where we still have to invent and implement the details. A vision enables us to communicate the overall goal for which we are aiming. In formulating the vision, we have to con-textualize our ideas, link them to practices in our lives, and describe the anticipated impact on individuals and society. A pre-requisite for formulating a coherent future vision is a good understanding of the prob-lems that we want to address. If formulated well, the vision shows a clear direction, but it leaves room for researchers in the community to make their own interpreta-tion. A well-formulated future vision also leaves room for individuals to align their research efforts with the goal or to criticize it fundamentally through their research.
I N T E R V I E w w I T h A L B R E C h T s C h M I d T
3 C O N C E P T U A L I Z I N G I N T E R A C T I O N98
We have proposed the vision of ampli-fying human perception and cognition through digital technologies (see Schmidt, 2017a; 2017b). This vision emerged from our various concrete research prototypes over the last 10 years. We realized that many of the prototypes and technologies we developed were pointed in a similar direction: enabling superhuman abilities through devices and applications. At the same time, we demonstrate why amplify-ing human abilities is a timely endeavor, in particular given the current advances in artificial intelligence, in sensing technol-ogies, as well as in personal display devices. For our group and for colleagues with whom we work, this vision has become a means for inspiring new ideas, for investi-gating relevant areas for potential innova-tion systematically, and for assessing ideas early on.
Why do metaphors persist in HCI?Good metaphors allow people to transfer their understanding and skills from another domain in the real world to interaction with computers and data. Good metaphors are abstract enough to persist over time, but concrete enough to simplify the use of computers. Early metaphors included computers as an advanced typewriter and computers as intelligent assistants. Such metaphors help in the design process to create understandable interaction concepts and user interfaces. A designer can take their idea for a user interface or an inter-action concept and evaluate it in the light of the metaphor. They can assess if the interaction is understandable for people familiar with the concept or technologies on which the metaphor is based. Meta-phors often suggest interaction styles and hence can help to create interfaces that are more consistent and interaction designs
that can be used without explanation using intuition (which in this case is the implicit understanding of the metaphor).
Metaphors for which the underlying concept has disappeared from everyday usage may still persist. In many cases, users will not know the original concept from their own experience (for example, a type-writer), but have grown up with technol-ogies using the metaphor. For a metaphor to persist, it must remain conducive and helpful over time for new users as well as for experienced ones.
What do you think of the rise of AI and automation? Do you think there is a role for HCI, and if so, what?Advancements in AI and in automation are exciting. They have the potential to empower humans to do things, to think things, and to experience things we cannot even imagine right now. However, the key to unlocking this potential is to create effi-cient ways for interacting with artificial intelligence. Meaningful automation and intelligent systems always have boundaries and intersections with human action. For example, an autonomous car will transport a human, a drone will deliver a parcel to a person, an automated kitchen will prepare a meal for a family, and large-scale data analytics in companies will lead to better services for their customers. With intelli-gent systems and smart services taking a more active role through artificial intelli-gence, the way in which interaction and interfaces are designed becomes even more crucial. Creating a positive user experience in the presence of artificial intelligence is a challenge where new visions and meta-phors are required.
One concept that we suggested is the notion of intervention user interfaces (Schmidt and Herrmann, 2017). The basic
99
expectation is that in the future many intelligent systems in our environment will work just fine, without any human inter-action. However, to stay in control and to tailor the system to current and unforesee-able needs, as well as to customize the user experience, human interventions should be easily possible. Designing interaction con-cepts for interventions and user interfaces that empower humans to make the most of a system driven by artificial intelligence is a huge challenge and includes many ba-sic research questions. Getting the interac-tion with AI right, basically finding ways for humans to harness the power of AI for what they want to do, is as important as developing the underlying algorithms. One without the other is of very limited value.
What do you see are the challenges ahead for HCI at scale?There are many challenges ahead in the context of autonomous systems and artificial intelligence as outlined earlier. Closely related to this is human data inter-action beyond interactive visualization. How can we empower humans to work with big and unstructured data? Here is a concrete example: I had a discussion with medical professionals this morning. For a specific cancer type, there are several thou-sand publications readily available. Many of them may have similar results, and others may have conflicting ones. Reading all of the publications in their current form is an intractable problem for a human reader, as it would take too long and would overload a person’s working mem-ory. The simple question that resulted from
the discussion is: what would a system and interface look like that uses AI to prepro-cess 10,000 papers, allows interactive pre-sentation of relevant content, and enables humans to make sense of the state of the art and come up with its own hypotheses? Preferably, the interface would support the person to do this in a few hours, rather than in their entire lifetime.
Another challenge at the societal scale is to understand the long-term impact of inter-active systems that we create. So far, this was very much trial and error. Providing unlim-ited and easy-to-use mass communication to individuals without journalistic training has changed how we read news. Personal communication devices and instant mes-saging have altered communication pat-terns in families and classrooms. Working in the office using a computer in order to create texts is reducing our physical move-ments. The way that we design interactive systems, the things we make easy or hard to use, and the modalities that we choose in our interaction design have inevitably resulted in long-term impacts on people. With the current methods and tools in HCI, we are well equipped to do a great job in developing easy-to-use systems with an amazing short-term user experience for the individual. However, looking at upcoming major innovations in mobility and health-care technologies, the interfaces we design may have many more consequences. One major challenge at scale is to design for a longer-term user experience (months to years) on a societal scale. Here we still first have to research and invent methods and tools.
I N T E R V I E w w I T h A L B R E C h T s C h M I d T
Chapter 4
C O G N I T I V E A S P E C T S
ObjectivesThe main goals of the chapter are to accomplish the following:
• Explain what cognition is and why it is important for interaction design. • Discuss what attention is and its effects on our ability to multitask. • Describe how memory can be enhanced through technology aids. • Show the difference between various cognitive frameworks that have been applied to HCI.
• Explain what are mental models. • Enable you to elicit a mental model and understand what it means.
4.1 Introduction
Imagine that it is getting late in the evening and you are sitting in front of your laptop. You have a report to complete by tomorrow morning, but you are not getting very far with it. You begin to panic and start biting your nails. You see two text messages flash up on your smartphone. You instantly abandon your report and cradle your smartphone to read them. One is from your mother, and the other is from your friend asking if you want to go out for a drink. You reply immediately to both of them. Before you know it, you’re back on Facebook to see whether any of your friends have posted anything about the party you wanted to go to but had to say no. Your phone rings, and you see that it’s your dad calling. You answer it, and he asks if you have been watching the football game. You say that you are too busy working toward a deadline, and he tells you that your team has just scored. You chat with him for a while and then say you have to get back to work. You realize 30 minutes have passed, and you return your attention to your report. But before you realize it, you click your favorite sports site to check the latest score of the football game and discover that your team has just scored again. Your phone starts buzzing. Two new WhatsApp messages are waiting for you. And on it goes. You glance at the time on your laptop. It is midnight. You really are in a panic now and finally close everything down except your word processor.
4.1 Introduction
4.2 What Is Cognition?
4.3 Cognitive Frameworks
4 C O G N I T I V E A S P E C T S102
In the past 10–15 years, it has become increasingly common for people to be switching their attention constantly among multiple tasks. The study of human cognition can help us understand the impact of multitasking on human behavior. It can also provide insights into other types of digital behaviors, such as decision-making, searching, and designing when using computer technologies by examining human abilities and limitations.
This chapter covers these aspects by examining the cognitive aspects of interaction design. It considers what humans are good and bad at, and it shows how this knowledge can inform the design of technologies that both extend human capabilities and compensate for human weaknesses. Finally, relevant cognitive theories, which have been applied in HCI to inform technology design, are described. (Other ways of conceptualizing human behavior that focus on the social and emotional aspects of interaction are presented in the following two chapters.)
4.2 What Is Cognition?
There are many different kinds of cognition, such as thinking, remembering, learning, day-dreaming, decision-making, seeing, reading, writing, and talking. A well-known way of dis-tinguishing between different modes of cognition is in terms of whether it is experiential or reflective (Norman, 1993). Experiential cognition is a state of mind where people perceive, act, and react to events around them intuitively and effortlessly. It requires reaching a certain level of expertise and engagement. Examples include driving a car, reading a book, hav-ing a conversation, and watching a video. In contrast, reflective cognition involves mental effort, attention, judgment, and decision-making, which can lead to new ideas and creativity. Examples include designing, learning, and writing a report. Both modes are essential for eve-ryday life. Another popular way of describing cognition is in terms of fast and slow thinking ( Kahneman, 2011). Fast thinking is similar to Don Norman’s experiential mode insofar as it is instinctive, reflexive, and effortless, and it has no sense of voluntary control. Slow thinking, as the name suggests, takes more time and is considered to be more logical and demanding, and it requires greater concentration. The difference between the two modes is easy to see when asking someone to give answers to the following two arithmetic equations:
2 2 21 19
The former can be done by most adults in a split second without thinking, while the lat-ter requires much mental effort; many people need to externalize the task to be able to com-plete it by writing it down on paper and using the long multiplication method. Nowadays, many people simply resort to fast thinking by typing the numbers to be added or multiplied into a calculator app on a smartphone or computer.
Other ways of describing cognition are in terms of the context in which it takes place, the tools that are employed, the artifacts and interfaces that are used, and the people involved (Rogers, 2012). Depending on when, where, and how it happens, cognition can be distrib-uted, situated, extended, and embodied. Cognition has also been described in terms of spe-cific kinds of processes (Eysenck and Brysbaert, 2018). These include the following:
• Attention • Perception
4 . 2 W h AT I S C O G N I T I O N ? 103
• Memory • Learning • Reading, speaking, and listening • Problem-solving, planning, reasoning, and decision-making
It is important to note that many of these cognitive processes are interdependent: several may be involved for a given activity. It is rare for one to occur in isolation. For example, when reading a book one has to attend to the text, perceive and recognize the letters and words, and try to make sense of the sentences that have been written.
In the following sections we describe the main kinds of cognitive processes in more detail, followed by a summary box highlighting the core design implications for each. The most relevant for interaction design are attention and memory, which we describe in the greatest detail.
4.2.1 AttentionAttention is central to everyday life. It enables us to cross the road without being hit by a car or bicycle, notice when someone is calling our name, and be able to text while at the same time watching TV. It involves selecting things on which to concentrate, at a point in time, from the range of possibilities available, allowing us to focus on information that is relevant to what we are doing. The extent to which this process is easy or difficult depends on (1) whether someone has clear goals and (2) whether the information they need is salient in the environment.
4.2.1.1 Clear GoalsIf someone knows exactly what they want to find out, they try to match this with the infor-mation that is available. For example, when someone has just landed at an airport after a long flight, which did not have Wi-Fi onboard, and they want to find out who won the World Cup, they might scan the headlines on their smartphone or look at breaking news on a public TV display inside the airport. When someone is not sure exactly what they are looking for, they may browse through information, allowing it to guide their attention to interesting or salient items. For example, when going to a restaurant, someone may have the general goal of eating a meal but only a vague idea of what they want to eat. They peruse the menu to find things that whet their appetite, letting their attention be drawn to the imaginative descrip-tions of various dishes. After scanning through the possibilities and imagining what each dish might be like, as well as considering other factors, such as cost, who they are with, what are the specials, what the waiter recommends, and whether they want a two- or three-course meal, and so on), they then decide.
4.2.1.2 Information PresentationThe way information is displayed can also greatly influence how easy or difficult it is to comprehend appropriate pieces of information. Look at Figure 4.1, and try the activity (based on Tullis, 1997). Here, the information-searching tasks are precise, requiring spe-cific answers.
4 C O G N I T I V E A S P E C T S104
South Carolina
City
CharlestonCharlestonCharlestonCharlestonCharlestonCharlestonCharleston
Best WesternDays InnHoliday Inn NHoliday Inn SWHoward JohnsonsRamada InnSheraton Inn
803803803803803803803
$126$118$136$133$131$133$134
$130$124$146$147$136$140$142
747-0961881-1000744-1621556-7100524-4148774-8281744-2401
ColumbiaColumbiaColumbiaColumbiaColumbiaColumbiaColumbiaColumbia
Best WesternCarolina InnDays InnHoliday Inn NWHoward JohnsonsQuality InnRamada InnVagabond Inn
803803803803803803803803
$129$142$123$132$125$134$136$127
$134$148$127$139$127$141$144$130
796-9400799-8200736-0000794-9440772-7200772-0270796-2700796-6240
Phone elbuoDelgniSletoH/letoMAreacode
Rates
PennsylvaniaBedford Motel/Hotel: Crinaline Courts (814) 623-9511 S: $118 D: $120Bedford Motel/Hotel: Holiday Inn (814) 623-9006 S: $129 D: $136Bedford Motel/Hotel: Midway (814) 623-8107 S: $121 D: $126Bedford Motel/Hotel: Penn Manor (814) 623-8177 S: $119 D: $125Bedford Motel/Hotel: Quality Inn (814) 623-5189 S: $123 D: $128Bedford Motel/Hotel: Terrace (814) 623-5111 S: $122 D: $124Bradley Motel/Hotel: De Soto (814) 362-3567 S: $120 D: $124Bradley Motel/Hotel: Holiday House (814) 362-4511 S: $122 D: $125Bradley Motel/Hotel: Holiday Inn (814) 362-4501 S: $132 D: $140Breezewood Motel/Hotel: Best Western Plaza (814) 735-4352 S: $120 D: $127Breezewood Motel/Hotel: Motel 70 (814) 735-4385 S: $116 D: $118
(a)
(b)
Figure 4.1 Two different ways of structuring the same information at the interface level. One makes it much easier to find information than the other. Source: Used courtesy of Dr. Tom Tullis
4 . 2 W h AT I S C O G N I T I O N ? 105
4.2.1.3 Multitasking and AttentionAs mentioned in the introduction to this chapter, many people now multitask, frequently switching their attention among different tasks. For example, in a study of teenage multitask-ing, it was found that the majority of teenagers were found to multitask most or some of the time while listening to music, watching TV, using a computer, or reading (Rideout et al., 2010). It is probably even higher now, considering their use of smartphones while walking, talking, and studying. While attending a presentation at a conference, we witnessed some-one deftly switch between four ongoing instant message chats (one at the conference, one at school, one with friends, and one at her part-time job), read, answer, delete, and place all new messages in various folders of her two email accounts, and check and scan her Facebook and her Twitter feeds, all while appearing to listen to the talk, take some notes, conduct a search on the speaker’s background, and open up their publications. When she had a spare moment, she played the game Patience. It was exhausting just watching her for a few minutes. It was as if she were capable of living in multiple worlds simultaneously while not letting a moment go to waste. But how much did she really take in of the presentation?
Is it possible to perform multiple tasks without one or more of them being detrimentally affected? There has been much research on the effects of multitasking on memory and attention (Burgess, 2015). The general finding is that it depends on the nature of the tasks and how much attention each demands. For example, listening to gentle music while working can help people tune out background noise, such as traffic or other people talking, and help them concentrate on what they are doing. However, if the music is loud, like heavy metal, it can be distracting.
Individual differences have also been found. For example, the results of a series of experiments comparing heavy with light multitaskers showed that heavy media multitaskers (such as the person described above) were more prone to being distracted by the multiple streams of media they are viewing than those who infrequently multitask. The latter were found to be better at allocating their attention when faced with competing distractions
ACTIVITY 4.1 Look at the top screen of Figure 4.1 and (1) find the price for a double room at the Quality Inn in Columbia, South Carolina, and (2) find the phone number of the Days Inn in Charles-ton, South Carolina. Then look at the bottom screen in Figure 4.1 and (1) find the price of a double room at the Holiday Inn in Bradley, Pennsylvania, and (2) find the phone number of the Quality Inn in Bedford, Pennsylvania. Which took longer to do?
In an early study, Tullis found that the two screens produced quite different results: It took an average of 3.2 seconds to search the top screen, while it took an average of 5.5 sec-onds to find the same kind of information in the bottom screen. Why is this so, considering that both displays have the same density of information relative to the background?
CommentThe primary reason for the disparity is the way that the characters are grouped in the display. In the top screen, they are grouped into vertical categories of information (that is, place, type of accommodation, phone number, and rates), and this screen has space in between the col-umns of information. In the bottom screen, the information is bunched together, making it much more difficult to search.
4 C O G N I T I V E A S P E C T S106
(Ophir et al., 2009). This suggests that people who are heavy multitaskers are likely to be those who are easily distracted and find it difficult to filter out irrelevant information. However, a more recent study by Danielle Lotteridge et al. (2015) found that it may be more complex. They found that while heavy multitaskers are easily distracted, they can also put this to good use if the distracting sources are relevant to the task in hand. Lotteridge et al. conducted a study that involved writing an essay under two conditions—either with relevant or irrelevant information. They found that if the information sources are relevant, they don’t affect the essay writing. The condition where irrelevant information was provided was found to nega-tively impact task performance. In summary, they found that multitasking can be both good and bad—it depends on what you are distracted by and how relevant it is to the task at hand.
The reason why multitasking is thought to be detrimental for human performance is that it overloads people’s capacity to focus. Having switched attention from what someone is working on to another piece of information requires additional effort to get back into the other task and to remember where they were in the ongoing activity. Thus, the time to complete a task can be significantly increased. A study of completion rates of coursework found that students who were involved in instant messaging took up to 50 percent longer to read a passage from a textbook com-pared with those who did not instant message while reading (Bowman et al., 2010). Multitasking can also result in people losing their train of thought, making errors, and needing to start over.
Nevertheless, many people are expected to multitask in the workplace nowadays, such as in hospitals, as a result of the introduction of ever more technology (for example, multiple screens in an operating room). The technology is often introduced to provide new kinds of real-time and changing information. However, this usually requires the constant attention of clinicians to check whether any of the data is unusual or unexpected. Managing the ever-increasing information load requires professionals, like clinicians, to develop new attention and scanning strategies, looking out for anomalies in data visualizations and listening for audio alarms alerting them to potential dangers. Interaction designers have tried to make this easier by including the use of ambient displays that come on when something needs atten-tion—flashing arrows to direct attention to a particular type of data or history logs of recent actions that can be quickly examined to refresh one’s memory of what has just happened on a given screen. However, how well clinicians manage to switch and divide their attention among different tasks in tech-rich environments has barely been researched (Douglas et al., 2017).
Source: Chris Wildt / Cartoon Stock
4 . 2 W h AT I S C O G N I T I O N ? 107
DILEMMAIs It OK to Use a Phone While Driving?
There has been considerable debate about whether drivers should be able to talk or text on their phones at the same time as driving (see Figure 4.2). People talk on their phones while walking, so why not be able to do the same thing when driving? The main reasons are that driving is more demanding, drivers are more prone to being distracted, and there is a greater chance of causing accidents (however, it is also the case that some people, when using their phones, walk out into a road without looking to see whether any cars are coming).
A meta-review of research that has investigated mobile phone use in cars has found that drivers’ reaction times are longer to external events when engaged in phone conversations (Caird et al., 2018). Drivers who use phones have also been found to be much poorer at stay-ing in their lane and maintaining the correct speed (Stavrinos et al., 2013). The reason for this is that drivers on a phone rely more on their expectations about what is likely to happen next and, as a result, respond much more slowly to unexpected events, such as the car in front of them stopping (Briggs et al., 2018). Moreover, phone conversations cause the driver visually to imagine what is being talked about. The driver may also imagine the facial expression of the person to whom they are speaking. The visual imagery involved competes for the processing resources also needed to enable the driver to notice and react to what is in front of them on the road. The idea that using a hands-free device is safer than actually holding the phone to carry out a conversation is false, as the same type of cognitive processing takes place both ways.
(Continued)
Figure 4.2 How distracting is it to be texting the phone while driving? Source: Tetra Images / Alamy Stock Photo
4 C O G N I T I V E A S P E C T S108
In several contexts, therefore, multitasking can be detrimental to performance, such as text-ing or speaking on the phone when driving. The cost of switching attention varies from person to person and which information resources are being switched between. When developing new technology to provide more information for people in their work settings, it is important to consider how best to support them so that they can easily switch their attention back and forth
Design ImplicationsAttention
• Consider context. Make information salient when it requires attention at a given stage of a task.
• Use techniques to achieve this when designing visual interfaces, such as animated graphics, color, underlining, ordering of items, sequencing of different information, and spac-ing of items.
• Avoid cluttering visual interfaces with too much information. This applies especially to the use of color and graphics: It is tempting to use lots of these attributes, which results in a mishmash of media that is distracting and annoying rather than helping the user attend to relevant information.
• Consider designing different ways of supporting effective switching and returning to a particular interface. This could be done subtly, such as the use of pulsing lights gradually getting brighter, or abruptly, such as the use of alerting sounds or voice. How much com-peting visual information or ambient sound is present also needs to be considered.
It has also been found that drivers who engage in conversation with their passengers expe-rienced similar negative effects. However, there is a difference between having a conversation with a passenger sitting next to the driver and one with a person located remotely. The driver and front-seat passenger can observe jointly what is happening in front of them on the road and will moderate or cease their conversation in order to switch their full attention to a potential or actual hazard. Someone on the other end of a phone, however, is not privy to what the driver is seeing and will carry on the conversation. They might have just asked “Where did you leave the spare set of keys?” and caused the driver mentally to search for them in their home, making it more difficult for them to switch their full attention back to what is happening on the road.
Because of these hazardous problems, many countries have banned the use of phones while driving. To help drivers resist the temptation to answer a phone that rings or glance at an incoming notification that pings, smartphone device manufacturers have been asked by some governments to introduce a driver mode akin to the airplane mode that could automatically lock down a smartphone, preventing access to apps, while disabling the phone’s keyboard when it detects a person who is driving. For example, the iPhone now has implemented this option.
4 . 2 W h AT I S C O G N I T I O N ? 109
among the multiple displays or devices and be able to return readily to what they were doing after an interruption (for instance, the phone ringing or people entering their space to ask questions).
4.2.2 PerceptionPerception refers to how information is acquired from the environment via the five sense organs (vision, hearing, taste, smell, and touch) and transformed into experiences of objects, events, sounds, and tastes (Roth, 1986). In addition, we have the additional sense of kines-thesia, which relates to the awareness of the position and movement of the parts of the body through internal sensory organs (known as proprioceptors) located in the muscles and joints. Perception is complex, involving other cognitive processes such as memory, attention, and language. Vision is the most dominant sense for sighted individuals, followed by hearing and touch. With respect to interaction design, it is important to present information in a way that can be readily perceived in the manner it was intended.
As was demonstrated in Activity 4.1, grouping items together and leaving spaces between them can aid attention because it breaks up the information. Having chunks of informa-tion makes it easier to scan, rather than one long list of text that is all the same. In addi-tion, many designers recommend using blank space (more commonly known as white space) when grouping objects, as it helps users to perceive and locate items more easily and quickly (Malamed, 2009). In a study comparing web pages displaying the same amount of informa-tion but structured using different graphical methods (see Figure 4.3), it was found that people took less time to locate items from information that was grouped using a border than
Design ImplicationsPerception
Representations of information need to be designed to be perceptible and recognizable across different media. • Design icons and other graphical representations so that users can readily distinguish
between them. • Obvious separators and white space are effective visual methods for grouping information
that make it easier to perceive and locate items. • Design audio sounds to be readily distinguishable from one another so that users can per-
ceive how they differ and remember what each one represents. • Research proper color contrast techniques when designing an interface, especially when
choosing a color for text so that it stands out from the background. For example, it is okay to use yellow text on a black or blue background, but not on a white or green background.
• Haptic feedback should be used judiciously. The kinds of haptics used should be easily distinguishable so that, for example, the sensation of squeezing is represented in a tactile form that is different from the sensation of pushing. Overuse of haptics can cause confu-sion. Apple iOS suggests providing haptic feedback in response to user-initiated actions, such as when the action of unlocking a vehicle using a smartwatch has been completed.
4 C O G N I T I V E A S P E C T S110
Black Hills ForestCheyenne RiverSocial ScienceSouth San JoseBadlands ParkJuvenile Justice
Peters LandingPublic HealthSan BernardinoMoreno ValleyAltamonte SpringsPeach Tree City
Jefferson FarmsPsychophysicsPolitical ScienceGame ScheduleSouth AddisionCherry Hills Village
Devlin HallPositionsHubard HallFernadino BeachCouncil BluffsClassical Lit
SociologyGreekWallace HallConcert TicketsPublic Radio FMChildren's Museum
Creative WritingLake Havasu CityEngineering BldgSports StudiesLakewood VillageRock Island
Highland ParkMachesney ParkVallecito Mts.Rock FallsFreeportSlaughter Beach
Results and StatsThousand OaksPromotionsNorth PalermoCredit UnionWilner Hall
Deerfield BeachArlington HillPreview GameRichland HillsExperts GuidsNeff Hall
Writing CenterTheater AuditionsDelaware CityScholarshipsHendricksvilleKnights Landing
Rocky MountainsLatinPleasant HillsObservatoryPublic AffairsHeskett Center
Performing ArtsItalianCoachesMckees RocksGlenwood SpringsUrban Affairs
Grand Wash CliffsIndian Well ValleyOnline CoursesLindquist HallFisk HallLos Padres Forest
Modern LiteratureStudio ArtsHugher ComplexCumberland FlatsCentral VillageHoffman Estates
BrunswickEast MillinocketWomen's StudiesVacantNews TheatreCandlewood Isle
McLeansboroExperimental LinksGraduationEmory LindquistClinton HallSan Luis Obispo
WebmasterRussianAthleticsGo ShockersDegree OptionsNewsletter
CurriculumEmergency (EMS)StatisticsAward DocumentsLanguage CenterFuture Shockers
Student LifeAccountancyMc Knight CenterCouncil of WomenCommuteSmall Business
DanceGerontlogeMarketingCollege BylawsWhy Wichita?Tickets
Educational MapPhysical PlantGraphic DesignNon Credit ClassMedia RelationsAdvertising
Beta Alpha PsiLiberal ArtsCounselingBiological ScienceDuerksen Fine ArtEMT Program
StaffAerospaceChoral Dept.Alberg HallFrenchSpanish
Softball, Men'sMcKinley HallEmailDental HygieneTenurePersonnel Policies
EnglishGraduate ComplexMusic EducationAdvising CenterMedical SchoolLevitt Arena
ReligionArt CompositionPhysicsEntrepreneurshipKoch ArenaRoster
ParentsWrestlingPhilosopyWichita LyceumFairmount CenterWomen's Museum
InstrmentalNrsingOperaSports HistoryAthletic Dept.Health Plan
GelogyManufacturingManagementUCATSAlumni NewsSaso
IntercollegiateBowlingWichita GatewayTransfer DayJob OpeningsLive Radio
Thinker & MoversAlumniFoundationsCorbin CenterJardine HallHugo Wall School
Career ServicesDoers & ShockersCore ValuesGrace Wilkie HallStrategic PlanMedical Tech
Figure 4.3 Two ways of structuring information on a web page Source: Weller (2004)
4 . 2 W h AT I S C O G N I T I O N ? 111
when using color contrast (Weller, 2004). The findings suggest that using contrasting colors in this manner may not be a good way to group information on a screen, but that using bor-ders is more effective (Galitz, 1997).
4.2.3 MemoryMemory involves recalling various kinds of knowledge that allow people to act appropri-ately. For example, it allows them to recognize someone’s face, remember someone’s name, recall when they last met them, and know what they said to them last.
It is not possible for us to remember everything that we see, hear, taste, smell, or touch, nor would we want to, as our brains would get overloaded. A filtering process is used to decide what information gets further processed and memorized. This filtering process, how-ever, is not without its problems. Often, we forget things that we would like to remember and conversely remember things that we would like to forget. For example, we may find it difficult to remember everyday things, like people’s names, or scientific knowledge such as mathematical formulae. On the other hand, we may effortlessly remember trivia or tunes that cycle endlessly through our heads.
How does this filtering process work? Initially, encoding takes place, determining which information is paid attention to in the environment and how it is interpreted. The extent to which it takes place affects people’s ability to recall that information later. The more attention that is paid to something and the more it is processed in terms of thinking about it and comparing it with other knowledge, the more likely it is to be remembered. For example, when learning about a topic, it is much better to reflect on it, carry out exercises, have discussions with others about it, and write notes rather than pas-sively reading a book or watching a video about it. Thus, how information is interpreted when it is encountered greatly affects how it is represented in memory and how easy it is to retrieve subsequently.
Another factor that affects the extent to which information can be subsequently retrieved is the context in which it is encoded. One outcome is that sometimes it can be difficult for people to recall information that was encoded in a different context from the one in which they are at present. Consider the following scenario:
You are on a train and someone comes up to you and says hello. You don’t recognize this person for a few moments, but then you realize it is one of your neighbors. You are only used to seeing them in the hallway of your apartment building and seeing them out of context makes this person initially difficult to recognize.
Another well-known memory phenomenon is that people are much better at recognizing things than recalling things. Furthermore, certain kinds of information is easier to recognize than others. In particular, people are good at recognizing thousands of pictures even if they have only seen them briefly before. In contrast, people are not as good at remembering details about the things they photograph when visiting places, such as museums. It seems that they remember less about objects when they have photographed them than when they observe them with the naked eye (Henkel, 2014). The reason for this is that the study participants appeared to be focusing more on framing the photo and less on the details of the object being photographed. Consequently, people don’t process as much information about an object when taking photos of it compared with when they are actually looking at it; hence, they are unable to remember as much about it later.
4 C O G N I T I V E A S P E C T S112
Increasingly, people rely on the Internet and their smartphones to act as cognitive pros-theses. Smartphones with Internet access have become an indispensable extension of the mind. Sparrow et al. (2011) showed how expecting to have readily available Internet access reduces the need and hence the extent to which people attempt to remember the information itself, while enhancing their memory for knowing where to find it online. Many people will whip out a smartphone to find out who acted in a movie, the name of a book, or what year a pop song was first released, and so on. Besides search engines, there are a number of other cognitive prosthetic apps that instantly help people find out or remember something, such as Shazam.com, the popular music recognition app.
4.2.3.1 Personal Information ManagementThe number of documents written, images created, music files recorded, videoclips down-loaded, emails with attachments saved, URLs bookmarked, and so on, increases every day. A common practice is for people to store these files on a phone, on a computer, or in the cloud with a view to accessing them later. This is known as personal information management (PIM). The design challenge here is deciding which is the best way of helping users organize their content so that it can be easily searched, for example, via folders, albums, or lists. The solution should help users readily access specific items at a later date, for example, a par-ticular image, video, or document. This can be difficult, however, especially when there are thousands or hundreds of thousands of pieces of information available. How does someone find that photo they took of their dog spectacularly jumping into the sea to chase a seagull, which they believe was taken two or three years ago? It can take them ages to wade through the hundreds of folders they have catalogued by date, name, or tag. Do they start by homing in on folders for a given year, looking for events, places, or faces, or typing in a search term to find the specific photo?
ACTIVITY 4.2 Try to remember the birthdays of all the members of your family and closest friends. How many can you remember? Then try to describe the image/graphic of the latest app you downloaded.
CommentIt is likely that you remembered the image, the colors, and the name of the app you down-loaded much better than the birthdays of your family and friends—most people now rely on Facebook or other online app to remind them about such special dates. People are good at remembering visual cues about things, for example, the color of items, the location of objects (for example, a book being on the top shelf), and marks on an object (like a scratch on a watch, a chip on a cup, and so on). In contrast, people find other kinds of information per-sistently difficult to learn and remember, especially arbitrary material like phone numbers.
4 . 2 W h AT I S C O G N I T I O N ? 113
It can become frustrating if an item is not easy to locate, especially when users have to spend lots of time opening numerous folders when searching for a particular image or an old document, simply because they can’t remember what they called it or where they stored it. How can we improve upon this cognitive process of remembering?
Naming is the most common means of encoding content, but trying to remember a name someone created some time back can be difficult, especially if they have tens of thousands of named files, images, videos, emails, and so forth. How might such a process be facilitated, considering individual memory abilities? Ofer Bergman and Steve Whittaker (2016) have proposed a model for helping people manage their “digital stuff” based on curation. The model involves three interdependent processes: how to decide what personal information to keep, how to organize that information when storing it, and which strategies to use to retrieve it later. The first stage can be assisted by the system they use. For example, email, texts, music, and photos are stored as default by many devices. Users have to decide whether to place these in folders or delete them. In contrast, when browsing the web, they have to make a conscious decision as to whether a site they are visiting is worth bookmarking as one they might want to revisit later.
A number of ways of adding metadata to documents have been developed, includ-ing time stamping, categorizing, tagging, and attribution (for example color, text, icon, sound, or image). Surprisingly, however, the majority of people still prefer the old- fashioned way of using folders for holding their files and other digital content. One reason is that folders provide a powerful metaphor (see Chapter 3, “Conceptualizing Interaction”) that people can readily understand—placing things that have something in common into a container.
A folder that is often seen on many users’ desktop is one simply labeled “stuff.” This is where documents, images, and so forth, that don’t have an obvious place to go are often placed but that people still want to keep somewhere. It has also been found that there is a strong preference for scanning across and within folders when looking for something rather than simply typing a term into a search engine (Ofer and Whittaker, 2016). Part of the problem with using search engines is that it can be difficult to recall the name of the file someone is seeking. This process requires more cognitive effort than navigating through a set of folders.
To help users with searching, a number of search and find tools, such as Apple’s Spot-light, now enable them to type a partial name or even the first letter of a file that it then searches for throughout the entire system, including the content inside documents, apps, games, emails, contacts, images, calendars, and applications. Figure 4.4 shows a partial list of files that Spotlight matched to the word cognition, categorized in terms of documents, mail and text messages, PDF documents, and so on.
4.2.3.2 Memory Load and PasswordsPhone, online, and mobile banking allow customers to carry out financial transactions, such as paying bills and checking the balance of their accounts, at their convenience. One of the problems confronting banks that provide these capabilities, however, is how to manage secu-rity concerns, especially preventing fraudulent transactions.
4 C O G N I T I V E A S P E C T S114
Figure 4.4 Apple’s Spotlight search tool
BOX 4.1 The Problem with the Magical Number Seven, Plus or Minus Two
Perhaps the best-known finding in psychology (certainly the one that nearly all students remember many years after they have finished their studies) is George Miller’s (1956) theory that seven, plus or minus two, chunks of information can be held in short-term memory at any one time. However, it is also one that has been misapplied in interaction design because several designers assume that it means they should design user interfaces only to have seven, plus or minus two, widgets on a screen, such as menus. In fact, however, this is a misapplication of the phenomenon, as explained here.
4 . 2 W h AT I S C O G N I T I O N ? 115
By short-term memory, Miller meant a memory store in which information was assumed to be processed when first perceived. By chunks of information, Miller meant a range of items such as numbers, letters, or words. According to Miller’s theory, therefore, people’s immediate mem-ory capacity is very limited. They are able to remember only a few words or numbers that they have heard or seen. If you are not familiar with this phenomenon, try the following exercise:
Read the first set of numbers here (or get someone to read them to you), cover it up, and then try to recall as many of the items as possible. Repeat this for the other sets. • 3, 12, 6, 20, 9, 4, 0, 1, 19, 8, 97, 13, 84 • cat, house, paper, laugh, people, red, yes, number, shadow, broom, rain, plant, lamp,
chocolate, radio, one, coin, jet • t, k, s, y, r, q, x, p, a, z, l, b, m, e
How many did you correctly remember for each set? Between five and nine, as suggested by Miller’s theory?
Chunks of information can also be combined items that are meaningful. For example, it is possible to remember the same number of two-word phrases like hot chocolate, banana split, cream cracker, rock music, cheddar cheese, leather belt, laser printer, tree fern, fluffy duckling, or cold rain. When these are all jumbled up (that is, split belt, fern crackers, banana laser, printer cream, cheddar tree, rain duckling, or hot rock), however, it is much harder to remem-ber as many chunks. This is mainly because the first set contains all meaningful two-word phrases that have been heard before and that require less time to be processed in short-term memory, whereas the second set is made up of completely novel phrases that don’t exist in the real world. You need to spend time linking the two parts of the phrase together while trying to memorize them. This takes more time and effort to achieve. Of course, it is possible to do if you have time to spend rehearsing them, but if you are asked to do it having heard them only once in quick succession, it is most likely that you will remember only a few.
So, how might people’s ability to remember only 7 ± 2 chunks of information that they have just read or heard be usefully applied to interaction design? According to a survey by Bob Bailey (2000), several designers have been led to believe the following guidelines and have created interfaces based on them: • Have only seven options on a menu. • Display only seven icons on a menu bar. • Never have more than seven bullets in a list. • Place only seven tabs at the top of a website page. • Place only seven items on a pull-down menu.
He points out how this is not how the principle should be applied. The reason is that these are all items that can be scanned and rescanned visually and hence do not have to be recalled from short-term memory. They don’t just flash up on the screen and disappear, requiring the user to remember them before deciding which one to select. If you were asked to find an item of food most people crave in the set of single words listed earlier, would you have any problem? No, you would just scan the list until you recognized the one (chocolate) that matched the task and then select it—just as people do when interacting with menus, lists, and tabs, regardless of whether they consist of three or 30 items. What users are required to do here is not remember as many items as possible, having only heard or seen them once in a sequence, but instead scan through a set of items until they recognize the one they want. This is a quite different task.
4 C O G N I T I V E A S P E C T S116
One solution has been to develop rigorous security measures whereby customers must provide multiple pieces of information before gaining access to their accounts. This is called multifactor authentication (MFA). The method requires a user to provide two or more pieces of evidence that only they know, such as the following:
• Their ZIP code or postal code • Their mother’s maiden name • Their birthplace • The last school they attended • The first school they attended • A password of between five and ten letters • A memorable address (not their home) • A memorable date (not their birthday)
Many of these are relatively easy to remember and recall since they are familiar to the specific user. But consider the last two. How easy is it for someone to come up with such memorable information and then be able to recall it readily? Perhaps the customer can give the address and birthday of another member of their family as a memorable address and date. But what about the request for a password? Suppose a customer selects the word inter-action as a password—fairly easy to remember, yes? The problem is that banks do not ask for the full password because of the danger that someone in the vicinity might overhear or oversee. Instead, they ask the customer to provide specific letters or numbers from it, like the seventh followed by the fifth. Certainly, such information does not spring readily to mind. Instead, it requires mentally counting each alphanumeric character of the password until the desired one is reached. How long does it take you to determine the seventh letter of the password interaction? How did you do it?
To make things harder, banks also randomize the questions they ask. Again, this is to pre-vent someone else who is nearby from memorizing the sequence of information. However, it also means that the customers themselves cannot learn the sequence of information required, meaning that they have to generate different information each time.
This requirement to remember and recall such information puts a big memory load on cus-tomers. Some people find such a procedure quite nerve-racking and are prone to forget certain pieces of information. As a coping strategy, they write down their details on a sheet of paper. Having such an external representation at hand makes it much easier for them to read off the necessary information rather than having to recall it from memory. However, it also makes them vulnerable to the fraud the banks are trying to prevent should anyone else get ahold of that piece of paper! Software companies have also developed password managers to help reduce memory load. An example is LastPass (https://www.lastpass.com/), which is designed to remember all of your passwords, meaning that you only have to remember one master password.
ACTIVITY 4.3 How can banks overcome the problem of providing a secure system while making the mem-ory load easier for people wanting to use online and mobile phone banking?
4 . 2 W h AT I S C O G N I T I O N ? 117
Much research has been conducted into how to design technology to help people suffer-ing from memory loss (for instance those with Alzheimer’s disease). An early example was the SenseCam, which was originally developed by Microsoft Research Labs in Cambridge (UK) to enable people to remember everyday events. The device they developed was a wearable camera that intermittently took photos, without any user intervention, while it was worn (see Figure 4.5). The camera could be set to take pictures at particular times, for example, every 30 seconds, or based on what it sensed (for example, acceleration). The camera employed a fish-eye lens, enabling nearly everything in front of the wearer to be captured. The digital images for each day were stored, providing a record of the events that a person experienced. Several studies were conducted on patients with various forms of memory loss using the device. For example, Steve Hodges et al. (2006) describe how a patient, Mrs. B, who had amnesia
CommentAdvances in computer vision and biometrics technology means that it is now possible to replace the need for passwords to be typed in each time. For example, facial and touch ID can be configured on newer smartphones to enable password-free mobile banking. Once these are set up, a user simply needs to put their face in front of their phone’s camera or their finger on the fingerprint sensor. These alternative approaches put the onus on the phone to recognize and authenticate the person rather than the person having to learn and remember a password.
BOX 4.2 Digital Forgetting
Much of the research on memory and interaction design has focused on developing cogni-tive aids that help people to remember, for example, reminders, to-do lists, and digital photo collections. However, there are times when we want to forget a memory. For example, when someone breaks up with their partner, it can be emotionally painful to be reminded of them through shared digital images, videos, and Facebook friends. How can technology be designed to help people forget such memories? How could social media, such as Facebook, be designed to support this process?
Corina Sas and Steve Whittaker (2013) suggest designing new ways of harvesting digital materials connected to a broken relationship through using various automatic methods, such as facial recognition, which dispose of them without the person needing to go through them personally and be confronted with painful memories. They also suggest that during a separa-tion, people could create a collage of their digital content connected to the ex, so as to trans-form them into something more abstract, thereby providing a means for closure and helping with the process of moving on.
4 C O G N I T I V E A S P E C T S118
was given a SenseCam to wear. The images that were collected were uploaded to a computer at the end of each day. For the next two weeks, Mrs. B and her husband looked through these and talked about them. During this period, Mrs. B’s recall of an event nearly tripled, to a point where she could remember nearly everything about that event. Prior to using the SenseCam, Mrs. B would have typically forgotten the little that she could initially remember about an event within a few days.
Since this seminal research, a number of digital memory apps have been developed for people with dementia. For example, RemArc has been designed to trigger long-term memo-ries in people with dementia using BBC Archive material such as old photos, videos, and sound clips.
Figure 4.5 The SenseCam device and a digital image taken with it Source: Used courtesy of Microsoft Research Cambridge
Design ImplicationsMemory
• Reduce cognitive load by avoiding long and complicated procedures for carrying out tasks. • Design interfaces that promote recognition rather than recall by using familiar interaction
patterns, menus, icons, and consistently placed objects. • Provide users with a variety of ways of labeling digital information (for example files,
emails, and images) to help them easily identify it again through the use of folders, cate-gories, color, tagging, time stamping, and icons.
4 . 2 W h AT I S C O G N I T I O N ? 119
4.2.4 LearningLearning is closely connected with memory. It involves the accumulation of skills and knowl-edge that would be impossible to achieve without memory. Likewise, people would not be able to remember things unless they had learned them. Within cognitive psychology, learning is thought to be either incidental or intentional. Incidental learning occurs without any inten-tion to learn. Examples include learning about the world such as recognizing faces, streets, and objects, and what you did today. In contrast, intentional learning is goal-directed with the goal of being able to remember it. Examples include studying for an exam, learning a foreign language, and learning to cook. This is much harder to achieve. Software develop-ers, therefore, cannot assume that users will simply be able to learn how to use an app or a product. It often requires much conscious effort.
Moreover, it is well known that people find it hard to learn by reading a set of instructions in a manual. Instead, they much prefer to learn through doing. GUIs and direct manipulation interfaces are good environments for supporting this kind of active learning by supporting exploratory interaction and, importantly, allowing users to undo their actions, that is, return to a previous state if they make a mistake by clicking the wrong option.
There have been numerous attempts to harness the capabilities of different technologies to support intentional learning. Examples include online learning, multimedia, and virtual reality. They are assumed to provide alternative ways of learning through interacting with information that is not possible with traditional technologies, for example, books. In so doing, they have the potential of offering learners the ability to explore ideas and concepts in different ways. For example, multimedia simulations, wearables, and augmented reality (see Chapter 7, “Interfaces”) have been designed to help teach abstract concepts (such as mathe-matical formulae, notations, laws of physics, biological processes) that students find difficult to grasp. Different representations of the same process (for instance, a graph, formula, sound, or simulation) are displayed and interacted with in ways that make their relationship with each other clearer to the learner.
People often learn effectively when collaborating together. Novel technologies have also been designed to support sharing, turn-taking, and working on the same documents. How these can enhance learning is covered in the next chapter.
Design ImplicationsLearning
• Design interfaces that encourage exploration. • Design interfaces that constrain and guide users to select appropriate actions when initially
learning.
4 C O G N I T I V E A S P E C T S120
4.2.5 Reading, Speaking, and ListeningReading, speaking, and listening are three forms of language processing that have some similar and some different properties. One similarity is that the meaning of sentences or phrases is the same regardless of the mode in which it is conveyed. For example, the sen-tence “Computers are a wonderful invention.” essentially has the same meaning whether one reads it, speaks it, or hears it. However, the ease with which people can read, listen, or speak differs depending on the person, task, and context. For example, many people find listening easier than reading. Specific differences between the three modes include the following:
• Written language is permanent while listening is transient. It is possible to re-read informa-tion if not understood the first time around. This is not possible with spoken information that is being broadcast unless it is recorded.
• Reading can be quicker than speaking or listening, as written text can be rapidly scanned in ways not possible when listening to serially presented spoken words.
• Listening requires less cognitive effort than reading or speaking. Children often prefer to listen to narratives provided in multimedia or web-based learning material rather than to read the equivalent text online. The popularity of audiobooks suggests adults also enjoy listening to novels, and so forth.
• Written language tends to be grammatical, while spoken language is often ungrammati-cal. For example, people often start talking and stop in midsentence, letting someone else start speaking.
• Dyslexics have difficulties understanding and recognizing written words, making it hard for them to write grammatical sentences and spell correctly.
Many applications have been developed either to capitalize on people’s reading, writing, and listening skills, or to support or replace them where they lack or have difficulty with them. These include the following:
• Interactive books and apps that help people to read or learn foreign languages. • Speech-recognition systems that allow people to interact with them by using spoken com-mands (for example, Dragon Home, Google Voice Search, and home devices, such as Ama-zon Echo, Google Home, and Home Aware that respond to vocalized requests).
• Speech-output systems that use artificially generated speech (for instance, written text-to-speech systems for the blind).
• Natural-language interfaces that enable people to type in questions and get written responses (for example, chatbots).
• Interactive apps that are designed to help people who find it difficult to read, write, or speak. Customized input and output devices that allow people with various disabilities to have access to the web and use word processors and other software packages.
• Tactile interfaces that allow people who are visually impaired to read graphs (for example, Designboom’s braille maps for the iPhone).
4 . 2 W h AT I S C O G N I T I O N ? 121
4.2.6 Problem-Solving, Planning, Reasoning, and Decision-MakingProblem-solving, planning, reasoning, and decision-making are processes involving reflective cognition. They include thinking about what to do, what the available options are, and what the consequences might be of carrying out a given action. They often involve conscious pro-cesses (being aware of what one is thinking about), discussion with others (or oneself), and the use of various kinds of artifacts (for example, maps, books, pens, and paper). Reasoning involves working through different scenarios and deciding which is the best option or solu-tion to a given problem. For example, when deciding on where to go on a vacation, people may weigh the pros and cons of different locations, including cost, weather at the location, availability and type of accommodation, time of flights, proximity to a beach, the size of the local town, whether there is nightlife, and so forth. When weighing all of the options, they reason through the advantages and disadvantages of each before deciding on the best one.
There has been a growing interest in how people make decisions when confronted with information overload, such as when shopping on the web or at a store (Todd et al., 2011). How easy is it to decide when confronted with an overwhelming choice? Classical rational theories of decision-making (for instance, von Neumann and Morgenstern, 1944) posit that making a choice involves weighing up the costs and benefits of different courses of action. This is assumed to involve exhaustively processing the information and making trade-offs between features. Such strategies are very costly in computational and informational terms—not the least because they require the decision-maker to find a way of comparing the differ-ent options. In contrast, research in cognitive psychology has shown how people tend to use simple heuristics when making decisions (Gigerenzer et al., 1999). A theoretical explanation is that human minds have evolved to act quickly, making just good enough decisions by using fast and frugal heuristics. We typically ignore most of the available information and rely only on a few important cues. For example, in the supermarket, shoppers make snap judg-ments based on a paucity of information, such as buying brands that they recognize, that are low-priced, or that offer attractive packaging—seldom reading other package information. This suggests that an effective design strategy is to make key information about a product
Design ImplicationsReading, Speaking, and Listening
• Keep the length of speech-based menus and instructions to a minimum. Research has shown that people find it hard to follow spoken menus with more than three or four options. Likewise, they are bad at remembering sets of instructions and directions that have more than a few parts.
• Accentuate the intonation of artificially generated speech voices, as they are harder to understand than human voices.
• Provide opportunities for making text large on a screen, without affecting the formatting, for people who find it hard to read small text.
4 C O G N I T I V E A S P E C T S122
highly salient. However, what exactly is salient will vary from person to person. It may depend on the user’s preferences, allergies, or interests. For example, one person might have a nut allergy and be interested in food miles, while another may be more concerned about the farming methods used (such as organic, FairTrade, and so on) and a product’s sugar content.
Thus, instead of providing ever more information to enable people to compare products when making a choice, a better strategy is to design technological interventions that provide just enough information, and in the right form, to facilitate good choices. One solution is to exploit new forms of augmented reality and wearable technology that enable information-frugal decision-making and that have glanceable displays that can represent key information in an easy-to-digest form (Rogers et al., 2010b). The interface for an AR or wearable app could be designed to provide certain “food” or other information filters, which could be switched on or off by the user to match their preferences.
DILEMMACan You Make Up Your Mind Without an App?
In their book The App Generation (Yale University Press, 2014), Howard Gardner and Katie Davis note how some young people find it hard to make their own decisions because they are becoming more and more risk averse. The reason for this is that they now rely on using an increasing number of mobile apps to help them in their decision-making, removing the risk of having to decide for themselves. Often, they will first read what others have said on social media sites, blogs, and recommender apps before choosing where to eat or go, what to do or listen to, and so on. However, relying on a multitude of apps means that young people are becom-ing increasingly unable to make decisions by themselves. For many, their first big decision is choosing which college or university to attend. This has become an agonizing and prolonged experience where both parents and apps play a central role in helping them out. They will read countless reviews, go on numerous visits to colleges and universities with their parents over several months, study university rankings that apply different measures, read up on what others say on social networking sites, and so on. In the end, however, they may finally choose the insti-tution where their friends attend or the one they liked the look of in the first place.
Design ImplicationsProblem-Solving, Planning, Reasoning, and Decision-Making
• Provide information and help pages that are easy to access for people who want to under-stand more about how to carry out an activity more effectively (for example, web searching).
• Use simple and memorable functions to support rapid decision-making and planning. Enable users to set or save their own criteria or preferences.
4 . 3 C O G N I T I V E F r A M E W O r k S 123
4.3 Cognitive Frameworks
A number of conceptual frameworks have been developed to explain and predict user behav-ior based on theories of cognition. In this section, we outline three that focus primarily on mental processes and three others that explain how humans interact and use technologies in the context in which they occur. These are mental models, gulfs of execution and evaluation, information processing, distributed cognition, external cognition, and embodied interaction.
4.3.1 Mental ModelsMental models are used by people when needing to reason about a technology, in particular, to try to fathom what to do when something unexpected happens with it or when encounter-ing unfamiliar products for the first time. The more someone learns about a product and how it functions, the more their mental model develops. For example, broadband engineers have a deep mental model of how Wi-Fi networks work that allows them to work out how to set them up and fix them. In contrast, an average citizen is likely to have a reasonably good mental model of how to use the Wi-Fi network in their home but a shallow mental model of how it works.
Within cognitive psychology, mental models have been postulated as internal construc-tions of some aspect of the external world that are manipulated, enabling predictions and inferences to be made (Craik, 1943). This process is thought to involve the fleshing out and the running of a mental model (Johnson-Laird, 1983). This can involve both unconscious and conscious mental processes, where images and analogies are activated.
ACTIVITY 4.4 To illustrate how we use mental models in our everyday reasoning, imagine the following two scenarios: • You arrive home from a vacation on a cold winter’s night to a cold house. You have a small baby, and you need to get the house warm as quickly as possible. Your house is centrally heated, but it does not have a smart thermostat that can be controlled remotely. Do you set the thermostat as high as possible or turn it to the desired temperature (for instance, 70°F)?
• You arrive home after being out all night and you’re starving hungry. You look in the freezer and find all that is left is a frozen pizza. The instructions on the package say heat the oven to 375°F and then place the pizza in the oven for 20 minutes. Your oven is electric. How do you heat it up? Do you turn it to the specified temperature or higher?
CommentMost people when asked the first question imagine the scenario in terms of what they would do in their own house and choose the first option. A typical explanation is that setting the tempera-ture to be as high as possible increases the rate at which the room warms up. While many people may believe this, it is incorrect. Thermostats work by switching on the heat and keeping it going at a constant speed until the desired set temperature is reached, at which point it cuts out. They cannot control the rate at which heat is given out from a heating system. Left at a given setting, thermostats will turn the heat on and off as necessary to maintain the desired temperature.
(Continued)
4 C O G N I T I V E A S P E C T S124
Why do people use erroneous mental models? It seems that in the previous two scenarios, they are using a mental model based on a general valve theory of the way something works (Kempton, 1986). This assumes the underlying principle of more is more: the more you turn or push something, the more it causes the desired effect. This principle holds for a range of phys-ical devices, such as faucets, where the more you turn them, the more water that comes out. However, it does not hold for thermostats, which instead function based on the principle of an on-off switch. What seems to happen is that in everyday life, people develop a core set of abstractions about how things work and apply these to a range of devices, irrespective of whether they are appropriate.
Using incorrect mental models to guide behavior is surprisingly common. Just watch people at a pedestrian crossing or waiting for an elevator. How many times do they press the button? A lot of people will press it at least twice. When asked why, a common reason is that they think it will make the lights change faster or ensure the elevator arrives.
Many people’s understanding of how technologies and services work is poor, for instance, the Internet, wireless networking, broadband, search engines, computer viruses, the cloud, or AI. Their mental models are often incomplete, easily confusable, and based on inappropri-ate analogies and superstition (Norman, 1983). As a consequence, they find it difficult to identify, describe, or solve a problem, and they lack the words or concepts to explain what is happening.
How can user experience (UX) designers help people to develop better mental models? A major obstacle is that people are resistant to spending much time learning about how things work, especially if it involves reading manuals or other documentation. An alterna-tive approach is to design technologies to be more transparent, which makes them easier to understand in terms of how they work and what to do when they don’t. This includes provid-ing the following:
• Clear and easy-to-follow instructions • Appropriate online help, tutorials, and context-sensitive guidance for users in the form of online videos and chatbot windows, where users can ask how to do something
• Background information that can be accessed to let people know how something works and how to make the most of the functionality provided
• Affordances of what actions an interface allows (for example, swiping, clicking, or selecting).
The concept of transparency has been used to refer to making interfaces intuitive to use so that people can simply get on with their tasks, such as taking photos, sending messages, or talking to someone remotely without having to worry about long sequences of buttons to
When asked the second question, most people say they would turn the oven to the speci-fied temperature and put the pizza in when they think it is at the right temperature. Some people answer that they would turn the oven to a higher temperature in order to warm it up more quickly. Electric ovens work on the same principle as central heating, so turning the heat up higher will not warm it up any quicker. There is also the problem of the pizza burning if the oven is too hot!
4 . 3 C O G N I T I V E F r A M E W O r k S 125
press or options to select. An ideal form of transparency is where the interface simply disap-pears from the focus of someone’s attention. Imagine if every time you had to give a presenta-tion that all you had to do was say, “Upload and start my slides for the talk I prepared today,” and they would simply appear on the screen for all to see. That would be bliss! Instead, many AV projector systems persist in being far from transparent, requiring many counterintuitive steps for someone to get their slides to show. This can include trying to find the right dongle, setting up the system, typing in a password, setting up audio controls, and so forth, all of which seems to take forever, especially when there is an audience waiting.
4.3.2 Gulfs of Execution and EvaluationThe gulf of execution and the gulf of evaluation describe the gaps that exist between the user and the interface (Norman, 1986; Hutchins et al., 1986). The gulfs are intended to show how to design the latter to enable the user to cope with them. The first one, the gulf of execution, describes the distance from the user to the physical system while the second one, the gulf of eval-uation, is the distance from the physical system to the user (see Figure 4.6). Don Norman and his colleagues suggest that designers and users need to concern themselves with how to bridge the gulfs to reduce the cognitive effort required to perform a task. This can be achieved, on the one hand, by designing usable interfaces that match the psychological characteristics of the user (for example, taking into account their memory limitations) and, on the other hand, by the user learning to create goals, plans, and action sequences that fit with how the interface works.
The conceptual framework of the gulfs is still considered useful today, as it can help designers consider whether their proposed interface design is increasing or decreasing cog-nitive load and whether it makes it obvious as to which steps to take for a given task. For example, Kathryn Whitenton (2018), who is a digital strategy manager, describes how the gulfs prevented her from understanding and why she could not get her Bluetooth headset to connect with her computer despite following the steps in the manual. She wasted a whole hour repeating the steps and getting more and more frustrated and not making any progress. Eventually, she discovered that the system she thought was toggled “on” was actually show-ing her that it was “off” (see Figure 4.7). She found this out by searching the web to see
Gulf ofEvaluation
What’s the currentsystem state?
TheUser
TheWorld
Gulf ofExecutionHow do I usethis system?
Figure 4.6 Bridging the gulfs of execution and evaluation Source: https://www.nngroup.com/articles/two-ux-gulfs-evaluation-execution. Used courtesy of the Nielsen Norman Group
4 C O G N I T I V E A S P E C T S126
whether someone else could help her. She found a site that showed a screenshot of what the settings switch looks like when turned on. There was an inconsistency between the labels of two similar-looking switches, one showing the current status of the interaction (off) and the other showing what would happen if the interaction were engaged (Add Bluetooth Or Other Device).
This inconsistency of similar functions illustrated how the gulfs of execution and evalu-ation were poorly bridged, making it confusing and difficult for the user to know what the problem was or why they could not get their headset to connect with their computer despite many attempts. In the article, she explains how the gulfs could be easily bridged by designing all sliders to give the same information as to what happens when they are moved from one side to the other. For more details about this situation, see https://www.nngroup.com/articles/two-ux-gulfs-evaluation-execution/.
4.3.3. Information ProcessingAnother approach to conceptualizing how the mind works has been to use metaphors and analogies to describe cognitive processes. Numerous comparisons have been made, including conceptualizing the mind as a reservoir, a telephone network, a digital computer, and a deep learning network. One prevalent metaphor from cognitive psychology is the idea that the mind is an information processor. Information is thought to enter and exit the mind through a series of ordered processing stages (see Figure 4.8). Within these stages, various processes are assumed to act upon mental representations. Processes include comparing and matching. Mental representations are assumed to comprise images, mental models, rules, and other forms of knowledge.
The information processing model provides a basis from which to make predictions about human performance. Hypotheses can be made about how long someone will take to perceive and respond to a stimulus (also known as reaction time) and what bottlenecks occur if a person is overloaded with too much information. One of the first HCI models to
Figure 4.7 An example where the gulfs helped explain how a seemingly trivial design decision led to much user frustration Source: https://www.nngroup.com/articles/two-ux-gulfs-evaluation-execution. Used courtesy of the Nielsen Norman Group
4 . 3 C O G N I T I V E F r A M E W O r k S 127
be derived from the information processing theory was the human processor model, which modeled the cognitive processes of a user interacting with a computer (Card et al., 1983). Cognition was conceptualized as a series of processing stages, where perceptual, cognitive, and motor processors are organized in relation to one another. The model predicts which cognitive processes are involved when a user interacts with a computer, enabling calcula-tions to be made of how long a user will take to carry out various tasks. In the 1980s, it was found to be a useful tool for comparing different word processors for a range of editing tasks. Even though it is not often used today to inform interaction design, it is considered to be a HCI classic.
The information processing approach was based on modeling mental activities that hap-pen exclusively inside the head. Nowadays, it is more common to understand cognitive activ-ities in the context in which they occur, analyzing cognition as it happens in the wild (Rogers, 2012). A central goal has been to look at how structures in the environment can both aid human cognition and reduce cognitive load. The three external approaches we consider next are distributed cognition, external cognition, and embodied cognition.
4.3.4 Distributed CognitionMost cognitive activities involve people interacting with external kinds of representations, such as books, documents, and computers and also with each other. For example, when someone goes home from wherever they have been, they do not need to remember the details of the route because they rely on cues in the environment (for instance, they know to turn left at the red house, right when the road comes to a T-junction, and so on). Similarly, when they are at home, they do not have to remember where everything is because information is available as needed. They decide what to eat and drink by scanning the items in the fridge, look out the window to see whether it is raining or not, and so on. Likewise, they are always creating external representations for a number of reasons, not only to help reduce memory load and the cognitive cost of computational tasks, but also, importantly, to extend what they can do and allow people to think more powerfully (Kirsh, 2010).
The distributed cognition approach was developed to study the nature of cognitive phe-nomena across individuals, artifacts, and internal and external representations (Hutchins, 1995). Typically, it involves describing a cognitive system, which entails interactions among people, the artifacts they use, and the environment in which they are working. An example of a cognitive system is an airline cockpit, where the top-level goal is to fly the plane (see Figure 4.9). This involves all of the following:
• The pilot, captain, and air traffic controller interacting with one another • The pilot and captain interacting with the instruments in the cockpit • The pilot and captain interacting with the environment in which the plane is flying (that is, the sky, runway, and so on)
Figure 4.8 Human information processing model Source: P. Barber (1998). Applied Cognitive Psychology. London: Methuen. Used courtesy of Taylor & Francis
4 C O G N I T I V E A S P E C T S128
A primary objective of the distributed cognition approach is to describe these interactions in terms of how information is propagated through different media. By this we mean how information is represented and re-represented as it moves across individuals and through the array of artifacts that are used (for example, maps, instrument readings, scribbles, and spo-ken word) during activities. These transformations of information are referred to as changes in representational state.
This way of describing and analyzing a cognitive activity contrasts with other cognitive approaches, such as the information processing model, in that it focuses not on what is hap-pening inside the head of an individual but on what is happening across a system of individu-als and artifacts. For example, in the cognitive system of the cockpit, a number of people and artifacts are involved in the activity of flying at a higher altitude. The air traffic controller initially tells the pilot when it is safe to ascend to a higher altitude. The pilot then alerts the captain, who is flying the plane, by moving a knob on the instrument panel in front of them, confirming that it is now safe to fly.
Hence, the information concerning this activity is transformed through different media (over the radio, through the pilot, and via a change in the position of an instrument). This kind of analysis can be used to derive design recommendations, suggesting how to change or
Figure 4.9 A cognitive system in which information is propagated through different media
4 . 3 C O G N I T I V E F r A M E W O r k S 129
redesign an aspect of the cognitive system, such as a display or a socially mediated practice. In the previous example, distributed cognition could draw attention to the importance of any new design needing to keep shared awareness and redundancy in the system so that both the pilot and the captain can be kept aware and also know that the other is aware of the changes in altitude that are occurring. It is also the basis for the DiCOT analytic framework that has been developed specifically for understanding healthcare settings and has also been used for software team interactions (see Chapter 9, “Data Analysis”).
4.3.5 External CognitionPeople interact with or create information by using a variety of external representations, including books, multimedia, newspapers, web pages, maps, diagrams, notes, drawings, and so on. Furthermore, an impressive range of tools has been developed throughout history to aid cognition, including pens, calculators, spreadsheets, and software workflows. The com-bination of external representations and physical tools has greatly extended and supported people’s ability to carry out cognitive activities (Norman, 2013). Indeed, they are such an integral part of our cognitive activities that it is difficult to imagine how we would go about much of our everyday life without them.
External cognition is concerned with explaining the cognitive processes involved when we interact with different external representations such as graphical images, multimedia, and virtual reality (Scaife and Rogers, 1996). A main goal is to explain the cognitive benefits of using different representations for different cognitive activities and the processes involved. The main ones include the following:
• Externalizing to reduce memory load • Computational offloading • Annotating and cognitive tracing
4.3.5.1 Externalizing to Reduce Memory LoadNumerous strategies have been developed for transforming knowledge into external rep-resentations to reduce memory load. One such strategy is externalizing things that we find difficult to remember, such as birthdays, appointments, and addresses. Diaries, personal reminders, and calendars are examples of cognitive artifacts that are commonly used for this purpose, acting as external reminders of what we need to do at a given time, such as buy a card for a relative’s birthday.
Other kinds of external representations that people frequently employ are notes, such as sticky notes, shopping lists, and to-do lists. Where these are placed in the environment can also be crucial. For example, people often place notes in prominent positions, such as on walls, on the side of computer screens, by the front door, and sometimes even on their hands in a deliberate attempt to ensure that they do remind them of what needs to be done or remembered. People also place things in piles in their offices and by the front door, indicating what needs to be done urgently versus what can wait for a while.
Externalizing, therefore, can empower people to trust that they will be reminded without having to remember themselves, thereby reducing their memory burden in the following ways:
• Reminding them to do something (for example, get something for mother’s birthday) • Reminding them of what to do (such as buy a card) • Reminding them of when to do something (for instance, send it by a certain date)
4 C O G N I T I V E A S P E C T S130
This is an obvious area where technology can be designed to help remind. Indeed, many apps have been developed to reduce the burden on people to remember things, including to-do and alarm-based lists. These can also be used to help improve people’s time manage-ment and work-life balance.
4.3.5.2 Computational OffloadingComputational offloading occurs when we use a tool or device in conjunction with an exter-nal representation to help us carry out a computation. An example is using pen and paper to solve a math problem as mentioned in the introduction of the chapter where you were asked to multiply 21 × 19 in your head versus using a pen and paper. Now try doing the sum again but using roman numerals: XXI × XVIIII. It is much harder unless you are an expert in using roman numerals—even though the problem is equivalent under both conditions. The reason for this is that the two different representations transform the task into one that is easy and one that is more difficult, respectively. The kind of tool used also can change the nature of the task to being easier or more difficult.
4.3.5.3 Annotating and Cognitive TracingAnother way in which we externalize our cognition is by modifying representations to reflect changes that are taking place that we want to mark. For example, people often cross things off a to-do list to indicate tasks that have been completed. They may also reorder objects in the environment by creating different piles as the nature of the work to be done changes. These two types of modification are called annotating and cognitive tracing.
• Annotating involves modifying external representations, such as crossing off or under-lining items.
• Cognitive tracing involves externally manipulating items into different orders or structures.
Annotating is often used when people go shopping. People usually begin their shop-ping by planning what they are going to buy. This often involves looking in their cupboards and fridge to see what needs stocking up. However, many people are aware that they won’t remember all this in their heads, so they often externalize it as a written shopping list. The act of writing may also remind them of other items that they need to buy, which they may not have noticed when looking through the cupboards. When they actually go shopping at the store, they may cross off items on the shopping list as they are placed in the shopping basket or cart. This provides them with an annotated externalization, allowing them to see at a glance what items are still left on the list that need to be bought.
There are a number of digital annotation tools that allow people to use pens, styluses, or their fingers to annotate documents, such as circling data or writing notes. The annotations can be stored with the document, enabling the users to revisit theirs or others’ externaliza-tions at a later date.
Cognitive tracing is useful in conditions where the current situation is in a state of flux and the person is trying to optimize their position. This typically happens when playing games, such as the following:
• In a card game, when the continuous rearrangement of a hand of cards into suits, in ascending order, or collecting same numbers together helps to determine what cards to keep and which to play as the game progresses and tactics change
4 . 3 C O G N I T I V E F r A M E W O r k S 131
• In Scrabble, where shuffling letters around in the tray helps a person work out the best word given the set of letters (Maglio et al., 1999)
Cognitive tracing has also been used as an interactive function, for example, letting stu-dents know what they have studied in an online learning package. An interactive diagram can be used to highlight all of the nodes visited, exercises completed, and units still to be studied.
A general cognitive principle for interaction design based on the external cognition approach is to provide external representations at an interface that reduce memory load, support creativity, and facilitate computational offloading. Different kinds of information visualizations can be developed that reduce the amount of effort required to make inferences about a given topic (for example, financial forecasting or identifying programming bugs). In so doing, they can extend or amplify cognition, allowing people to perceive and do activi-ties that they couldn’t do otherwise. For example, information visualizations (discussed in chapter 10) are used to represent big data in a visual form that can make it easier to make cross-comparisons across dimensions and see patterns and anomalies. Workflow and contex-tual dialog boxes can also pop up at appropriate times to guide users through their interac-tions, especially where there are potentially hundreds and sometimes thousands of options available. This reduces memory load significantly and frees up more cognitive capacity for enabling people to complete desired tasks.
4.3.6 Embodied InteractionAnother way of describing our interactions with technology and the world is to conceive of it as embodied. By this we mean the practical engagement with the social and physical environ-ment (Dourish, 2001). This involves creating, manipulating, and making meaning through our engaged interaction with physical things, including mundane objects such as cups and spoons, and technological devices, such as smartphones and robots. Artifacts and technolo-gies that indicate how they are coupled to the world make it clear how they should be used. For example, a physical artifact, like a book when left opened on someone’s desk, can remind them to complete an unfinished task the next day (Marshall and Hornecker, 2013).
Eva Hornecker et al. (2017) further explain embodied interaction in terms of how our bodies and active experiences shape how we perceive, feel, and think. They describe how our ability to think abstractly is thought to be a result of our sensorimotor experiences with the world. This enables us to learn how to think and talk using abstract concepts, such as inside-outside, up-down, on top of, and behind. Our numerous experiences of moving through and manipulating the world since we were born (for example, climbing, walking, crawling, stepping into, holding, or placing) is what enables us to develop a sense of the world at both a concrete and abstract level.
Within HCI, the concept of embodied interaction has been used to describe how the body mediates our various interactions with technology (Klemmer et al., 2006) and also our emo-tional interactions (Höök, 2018). By theorizing about embodied interactions in these ways has helped researchers uncover problems that can arise in the use of existing technologies while also informing the design of new technologies in the context in which they will be used.
David Kirsh (2013) suggests that a theory of embodiment can provide HCI practition-ers and theorists with new ideas about interaction and new principles for better designs. He explains how interacting with tools changes the way people think and perceive of their envi-ronments. He also argues that a lot of times we think with our bodies and not just with our brains. He studied choreographers and dancers and observed that they often partially model
4 C O G N I T I V E A S P E C T S132
a dance (known as marking) through using abbreviated moves and small gestures rather than doing a full workout or mentally simulating the dance in their heads. This kind of marking was found to be a better method of practice than the other two methods. The reason for doing it this way is not that it is saving energy or preventing dancers from getting exhausted emotionally, but that it enables them to review and explore particular aspects of a phrase or movement without the mental complexity involved in a full work out. The implication of how people use embodiment in their lives is that learning new procedures and skills might be better taught by a process like marking, where learners create little models of things or use their own bodies to act out. For example, rather than developing fully fledged virtual real-ity simulations for learning golf, tennis, skiing, and so on, it might be better to teach sets of abbreviated actions, using augmented reality, as a form of embodied marking.
In-Depth ActivityThe aim of this in-depth activity is for you to try to elicit mental models from people. In par-ticular, the goal is for you to understand the nature of people’s knowledge about an interactive product in terms of how to use it and how it works.1. First, elicit your own mental model. Write down how you think contactless cards (see
Figure 4.10) work—where customers place their debit or credit card over a card reader. If you are not familiar with contactless cards, do the same for a smartphone app like Apple Pay or Google Pay. Then answer the following questions:
Figure 4.10 A contactless debit card indicated by symbol
133
Further Reading
BERGMAN, O. and WHITTAKER, S. (2016). The Science of Managing Our Digital Stuff. MIT Press. This very readable book provides a fascinating account of how we manage all of our digital stuff that increases by the bucket load each day. It explains why we persist with seemingly old-fashioned methods when there are alternative, seemingly better approaches that have been designed by software companies.
• What information is sent between the card/smartphone and the card reader when it is placed in front of it?
• What is the maximum amount you can pay for something using a contactless card, or Apple/Google Pay?
• Why is there an upper limit? • How many times can you use a contactless card or Apple/Google Pay in a day? • What happens if you have two contactless cards in the same wallet/purse? • What happens when your contactless card is stolen and you report it to the bank?
Next, ask two other people the same set of questions.2. Now analyze your answers. Do you get the same or different explanations? What do the
findings indicate? How accurate are people’s mental models about the way contactless cards and smartphone Apple/Google Pay work?
SummaryThis chapter explained the importance of understanding the cognitive aspects of interaction. It described relevant findings and theories about how people carry out their everyday activities and how to learn from these to help in designing interactive products. It provided illustrations of what happens when you design systems with the user in mind and what happens when you don’t. It also presented a number of conceptual frameworks that allow ideas about cognition to be generalized across different situations.
Key points • Cognition comprises many processes, including thinking, attention, memory, perception, learning, decision-making, planning, reading, speaking, and listening.
• The way in which an interface is designed can greatly affect how well people can perceive, attend, learn, and remember how to carry out their tasks.
• The main benefits of conceptual frameworks based on theories of cognition are that they can explain user interaction, inform design, and predict user performance.
F U r T h E r r E A D I N G
4 C O G N I T I V E A S P E C T S134
ERICKSON, T. D. and MCDONALD, D. W. (2008) HCI Remixed: Reflections on Works That Have Influenced the HCI Community. MIT Press. This collection of essays from more than 50 leading HCI researchers describes the accessible prose papers, books, and software that influenced their approach to HCI and shaped its history. They include some of the classic papers on cognitive theories, including the psychology of HCI and the power of external representations.
EYSENCK, M. and BRYSBAERT, M. (2018) Fundamentals of Cognition (3rd ed.). Rout-ledge. This introductory textbook about cognition provides a comprehensive overview of the fundamentals of cognition. In particular, it describes the processes that allow us to make sense of the world around us and to enable us to make decisions about how to manage our everyday lives. It also covers how technology can provide new insights into how the mind works, for example, revealing how CAPTCHAs tell us more about perception.
GIGERENZER, G. (2008) Gut Feelings. Penguin. This provocative paperback is written by a psychologist and behavioral expert in decision-making. When confronted with choice in a variety of contexts, he explains how often “less is more.” He explains why this is so in terms of how people rely on fast and frugal heuristics when making decisions, which are often unconscious rather than rational. These revelations have huge implications for interaction design that are only just beginning to be explored.
JACKO, J. (ed.) (2012) The Human-Computer Interaction Handbook: Fundamentals, Evolv-ing Technologies and Emerging Applications (3rd ed). CRC Press. Part 1 is about human aspects of HCI and includes in-depth chapters on information processing, mental models, decision-making, and perceptual motor interaction.
KAHNEMAN, D. (2011) Thinking, Fast and Slow. Penguin. This bestseller presents an over-view of how the mind works, drawing on aspects of cognitive and social psychology. The focus is on how we make judgments and choices. It proposes that we use two ways of think-ing: one that is quick and based on intuition and one that is slow and more deliberate and challenging. The book explores the many facets of life and how and when we use each.
Chapter 5
S O C I A L I N T E R A C T I O N
5.1 Introduction
5.2 Being Social
5.3 Face-to-Face Conversations
5.4 Remote Conversations
5.5 Co-presence
5.6 Social Engagement
ObjectivesThe main goals of the chapter are to accomplish the following:
• Explain what is meant by social interaction. • Describe the social mechanisms that people use to communicate and collaborate. • Explain what social presence means. • Give an overview of new technologies intended to facilitate collaboration and group participation.
• Discuss how social media has changed how we keep in touch, make contacts, and man-age our social and working lives.
• Outline examples of new social phenomena that are a result of being able to connect online.
5.1 Introduction
People are inherently social: we live together, work together, learn together, play together, interact and talk with each other, and socialize. A number of technologies have been developed specifically to enable us to persist in being social when physically apart from one another, many of which have now become part of the fabric of society. These include the widespread use of smartphones, video chat, social media, gaming, messaging, and telepresence. Each of these afford different ways of supporting how people connect.
There are many ways to study what it means to be social. In this chapter, we focus on how people communicate and collaborate face-to-face and remotely in their social, work,
5 S O C I A L I N T E R A C T I O N136
and everyday lives—with the goal of providing models, insights, and guidelines to inform the design of “social” technologies that can better support and extend them. A diversity of communication technologies is also examined that have changed the way people live—how they keep in touch, make friends, and coordinate their social and work networks. The con-versation mechanisms that have conventionally been used in face-to-face interactions are described and discussed in relation to how they have been adapted for the various kinds of computer-based conversations that now take place at a distance. Examples of social phenom-ena that have emerged as a result of social engagement at scale are also presented.
5.2 Being Social
A fundamental aspect of everyday life is being social, and that entails interacting with each other. People continually update each other about news, changes, and developments on a given project, activity, person, or event. For example, friends and families keep each other posted on what’s happening at work, at school, at a restaurant or club, next door, in reality shows, and in the news. Similarly, people who work together keep each other informed about their social lives and everyday events, as well as what is happening at work, for instance when a project is about to be completed, plans for a new project, problems with meeting deadlines, rumors about closures, and so on.
While face-to-face conversations remain central to many social interactions, the use of social media has dramatically increased. People now spend several hours a day communicat-ing with others online—texting, emailing, tweeting, Facebooking, Skyping, instant messaging, and so on. It is also common practice for people at work to keep in touch with each other via WhatsApp groups and other workplace communication tools, such as Slack, Yammer, or Teams.
The almost universal adoption of social media in mainstream life has resulted in most people now being connected in multiple ways over time and space—in ways that were unimaginable 25 or even 10 years ago. For example, adults average about 338 Facebook friends, while it is increasingly common for people to have more than 1,000 connections on LinkedIn—many more than those made through face-to-face networking. The way that people make contact, how they stay in touch, who they connect to, and how they maintain their social networks and family ties have irrevocably changed. During the last 20 or so years, social media, teleconferencing, and other social-based technologies (often referred to as social computing) have also transformed how people collaborate and work together globally—including the rise of flexible and remote working, the widespread use of shared calendars and collaboration tools (for example Slack, Webex, Trello, and Google Docs), and professional networking platforms (such as LinkedIn, Twitter, and WhatsApp).
A key question that the universal adoption of social media and other social computing tools in society raises is how it has affected people’s ability to connect, work, and interact with one another. Have the conventions, norms, and rules established in face-to-face interac-tions to maintain social order been adopted in social media interactions, or have new norms emerged? In particular, are the established conversational rules and etiquette, whose function it is to let people know how they should behave in social groups, also applicable to online social behavior? Or, have new conversational mechanisms evolved for the various kinds of social media? For example, do people greet each other in the same way, depending on whether they are chatting online, Skyping, or at a party? Do people take turns when online
5 . 2 B E I N g S O C I A L 137
chatting in the way they do when talking with each other face to face? How do they choose which technology or app to use from the variety available today for their various work and social activities, such as SnapChat, text messaging, Skype, or phone calls? Answering these questions can help us understand how existing tools support communication and collabora-tive work while helping to inform the design of new ones.
When planning and coordinating social activities, groups often switch from one mode to another. Most people send texts in preference to calling someone up, but they may switch to calling or mobile group messaging (such as WhatsApp, GroupMe) at different stages of planning to go out (Schuler et al., 2014). However, there can be a cost as conversations about what to do, where to meet, and who to invite multiply across people. Some people might get left off or others might not reply, and much time can be spent to-ing and fro-ing across the different apps and threads. Also, some people may not look at their notifications in a timely manner, while further developments in the group planning have evolved. This is compounded by the fact that often people don’t want to commit until close to the time of the event, in case an invitation to do something from another friend appears that is more interesting to them. Teenagers, especially, often leave it until the last minute to micro-coordinate their arrange-ments with their friends before deciding on what to do. They will wait and see if a better offer comes their way rather than deciding for themselves a week in advance, say, to see a movie with a friend and sticking to it. This can make it frustrating for those who initiate the plan-ning and are waiting to book tickets before they sell out.
A growing concern that is being raised within society is how much time people spend looking at their phones—whether interacting with others, playing games, tweeting, and so forth—and its consequences on people’s well-being (see Ali et al., 2018). A report on the impact of the “decade of a smartphone” notes that on average a person in the United King-dom spends more than a day a week online (Ofcom, 2018). Often, it is the first thing they do upon waking and the last thing they do before going to bed. Moreover, lots of people cannot go for long without checking their phone. Even when sitting together, they resort to being in their own digital bubbles (see Figure 5.1). Sherry Turkle (2015) bemoans the negative impact that this growing trend is having on modern life, especially how it is affecting everyday con-versation. She points out that many people will admit to preferring texting to talking to oth-ers, as it is easier, requires less effort, and is more convenient. Furthermore, her research has shown that when children hear adults talking less, they likewise talk less. This in turn reduces opportunities to learn how to empathize. She argues that while online communication has its place in society, it is time to reclaim conversation, where people put down their phones more often and (re)learn the art and joy of spontaneously talking to each other.
On the other hand, it should be stressed that several technologies have been designed to encourage social interaction to good effect. For example, voice assistants that come with smart speakers, such as Amazon’s Echo devices, provide a large number of “skills” intended to support multiple users taking part at the same time, offering the potential for families to play together. An example skill is “Open the Magic Door,” which allows group members (such as families) to choose their path in a story by selecting different options through the narrative. Social interaction may be further encouraged by the affordance of a smart speaker when placed on a surface in the home, such as a kitchen counter or mantelpiece. In particular, its physical presence in this shared location affords joint ownership and use—similar to other domestic devices, such as the radio or TV. This differs from other virtual voice assistants that are found on phones or laptops that support individual use.
5 S O C I A L I N T E R A C T I O N138
Figure 5.1 A family sits together, but they are all in their own digital bubbles—including the dog!Source: Helen Sharp
ACTIVITY 5.1 Think of a time where you enjoyed meeting up with friends to catch up in a cafe. Compare this social occasion with the experience that you have when texting with them on your smart-phone. How are the two kinds of conversations different?
CommentThe nature of the conversations is likely to be very different with pros and cons for each. Face-to-face conversations ebb and flow unpredictably and spontaneously from one topic to the next. There can be much laughing, gesturing, and merriment among those taking part in the conversation. Those present pay attention to the person speaking, and then when someone else starts talking, all eyes move to them. There can be much intimacy through eye contact, facial expressions, and body language, in contrast to when texters send intermittent messages back and forth in bursts of time. Texting is also more premeditated; people decide what to say and can review what they have written. They can edit their message or decide even not to send it, although sometimes people press the Send button without much thought about its impact on the interlocutor that can lead to regrets afterward.
Emoticons are commonly used as a form of expressivity to compensate for nonverbal communication. While they can enrich a message by adding humor, affection, or a personal touch, they are nothing like a real smile or a wink shared at a key moment in a conversation. Another difference is that people say things and ask each other things in conversations that they would never do via text. On the one hand, such confiding and directness may be more engaging and enjoyable, but on the other hand, it can sometimes be embarrassing. It depends on the context as to whether conversing face-to-face versus texting is preferable.
5 . 3 F A C E – T O – F A C E C O N V E R S AT I O N S 139
5.3 Face-to-Face Conversations
Talking is something that is effortless and comes naturally to most people. And yet holding a conversation is a highly skilled collaborative achievement, having many of the qualities of a musical ensemble. In this section we examine what makes up a conversation. Understanding how conversations start, progress, and finish is useful when designing dialogues that take place with chatbots, voice assistants, and other communication tools. In particular, it helps research-ers and developers understand how natural it is, how comfortable people are when conversing with digital agents, and the extent to which it is important to follow conversation mechanisms that are found in human conversations. We begin by examining what happens at the beginning.
A: Hi there.B: Hi!C: Hi.A: All right?C: Good. How’s it going?A: Fine, how are you?C: Good.B: OK. How’s life treating you?
Such mutual greetings are typical. A dialogue may then ensue in which the participants take turns asking questions, giving replies, and making statements. Then, when one or more of the participants wants to draw the conversation to a close, they do so by using either implicit or explicit cues. An example of an implicit cue is when a participant looks at their watch, signal-ing indirectly to the other participants that they want the conversation to draw to a close. The other participants may choose to acknowledge this cue or carry on and ignore it. Either way, the first participant may then offer an explicit signal, by saying, “Well, I have to go now. I got a lot of work to do” or, “Oh dear, look at the time. I gotta run. I have to meet someone.” Fol-lowing the acknowledgment by the other participants of such implicit and explicit signals, the conversation draws to a close, with a farewell ritual. The different participants take turns say-ing, “Goodbye,” “Bye,” “See you,” repeating themselves several times until they finally separate.
ACTIVITY 5.2 How do you start and end a conversation when (1) talking on the phone and (2) chatting online? Do you use the same conversational mechanisms that are used in face-to-face conversations?
CommentThe person answering the call will initiate the conversation by saying “hello” or, more formally, the name of their company/department. Most phones (landline and smartphones) have the facil-ity to display the name of the caller (Caller ID) so the receiver can be more personal when answer-ing, for example “Hello, John. How are you doing?” Phone conversations usually start with a mutual greeting and end with a mutual farewell. In contrast, conversations that take place when chatting online have evolved new conventions. The use of opening and ending greetings when joining and leaving is rare; instead, most people simply start their message with what they want to talk about and then stop when they have gotten an answer, as if in the middle of a conversation.
5 S O C I A L I N T E R A C T I O N140
Many people are now overwhelmed by the number of emails they receive each day and find it difficult to reply to them all. This has raised the question of which conversational techniques to use to improve the chances of getting someone to reply. For example, can the way people compose their emails, especially the choice of opening and ending a conversation, increase the likelihood that the recipient will respond to it? A study by Boomerang (Brendan G, 2017) of 300,000 emails taken from mailing list archives of more than 20 different online communities examined whether the opening or closing phrase that was used affected the reply rate. They found that the most common opening phrase used “hey” (64 percent), fol-lowed by “hello” (63 percent), and then “hi” (62 percent) were the ones that got the highest rate of reply, in the region of 63–64 percent. This was found to be higher than emails that opened with more formal phrases, like “Dear” (57 percent) or “Greetings” (56 percent). The most popular form of sign-off was found to be “thanks” (66 percent), “regards” (63 per-cent), and “cheers” (58 percent), with “best” being used less (51 percent). Again, they found that emails that used closings with a form of “thank you” got the highest rate of responses. Hence, which conversational mechanism someone uses to address the recipient can deter-mine whether they will reply to it.
Conversational mechanisms enable people to coordinate their talk with one another, allowing them to know how to start and stop. Throughout a conversation, further turn- taking rules are followed that enable people to know when to listen, when it is their cue to speak, and when it is time for them to stop again to allow the others to speak. Sacks et al. (1978), famous for their work on conversation analysis, describe these in terms of three basic rules.
Rule 1 The current speaker chooses the next speaker by asking a question, inviting an opinion, or making a request.
Rule 2 Another person decides to start speaking.Rule 3 The current speaker continues talking.
The rules are assumed to be applied in this order so that whenever there is an opportu-nity for a change of speaker to occur, for instance, someone comes to the end of a sentence, rule 1 is applied. If the listener to whom the question or request is addressed does not accept the offer to take the floor, rule 2 is applied, and someone else taking part in the conversation may take up the opportunity and offer a view on the matter. If this does not happen, then rule 3 is applied, and the current speaker continues talking. The rules are cycled through recursively until someone speaks again.
To facilitate rule following, people use various ways of indicating how long they are going to talk and on what topic. For example, a speaker might say right at the beginning of his turn in the conversation that he has three things to say. A speaker may also explic-itly request a change in speaker by saying to the listeners, “OK, that’s all I want to say on that matter. So, what do you think?” More subtle cues to let others know that their turn in the conversation is coming to an end include the lowering or raising of the voice to indicate the end of a question or the use of phrases like “You know what I mean?” or simply “OK?” Back channeling (uh-huh, mmm), body orientation (such as moving away from or closer to someone), gaze (staring straight at someone or glancing away), and gesturing (for example, raising of arms) are also used in different combinations when talking in order to signal to others when someone wants to hand over or take up a turn in the conversation.
5 . 3 F A C E – T O – F A C E C O N V E R S AT I O N S 141
Another way in which conversations are coordinated and given coherence is through the use of adjacency pairs (Schegloff and Sacks, 1973). Utterances are assumed to come in pairs in which the first part sets up an expectation of what is to come next and directs the way in which what does come next is heard. For example, A may ask a question to which B responds appropriately.
A: So, shall we meet at 8:00?B: Um, can we make it a bit later, say 8:30?
Sometimes adjacency pairs get embedded in each other, so it may take some time for a person to get a reply to their initial request or statement.
A: So, shall we meet at 8:00?B: Wow, look at them.A: Yes, what a funny hairdo!B: Um, can we make it a bit later, say 8:30?
For the most part, people are not aware of following conversational mechanisms and would be hard-pressed to articulate how they can carry on a conversation. Furthermore, people don’t necessarily abide by the rules all the time. They may interrupt each other or talk over each other, even when the current speaker has clearly indicated a desire to hold the floor for the next two minutes to finish an argument. Alternatively, a listener may not take up a cue from a speaker to answer a question or take over the conversation but instead continue to say nothing even though the speaker may be making it glaringly obvious that it is the listener’s turn to say something. Oftentimes, a teacher will try to hand over the conversation to a student in a seminar by staring at them and asking a specific question, only to see the student look at the floor and say nothing. The outcome is an embarrassing silence, followed by either the teacher or another student picking up the conversation again.
Other kinds of breakdowns in conversation arise when someone says something that is ambiguous, and the interlocutor misinterprets it to mean something else. In such situations, the participants will collaborate to overcome the misunderstanding by using repair mecha-nisms. Consider the following snippet of conversation between two people:
A: Can you tell me the way to get to the Multiplex Ranger cinema?B: Yes, you go down here for two blocks and then take a right (pointing to the right), proceed
until you get to the light, and then it’s on the left.A: Oh, so I go along here for a couple of blocks and then take a right, and the cinema is at the
light (pointing ahead of him)?B: No, you go down this street for a couple of blocks (gesturing more vigorously than before to
the street to the right of him while emphasizing the word this).A: Ahhhh! I thought you meant that one: so it’s this one (pointing in the same direction as the
other person).B: Uh-hum, yes, that’s right: this one.
Detecting breakdowns in conversation requires that the speaker and listener both pay attention to what the other says (or does not say). Once they have understood the nature of the failure, they can then go about repairing it. As shown in the previous example, when the listener misunderstands what has been communicated, the speaker repeats what they said earlier, using a stronger voice intonation and more exaggerated gestures. This allows the speaker to repair the mistake and be more explicit with the listener, allowing them to understand and follow better what they are saying. Listeners may also signal when they don’t
5 S O C I A L I N T E R A C T I O N142
understand something or want further clarification by using various tokens, like “Huh?” or “What?” (Schegloff, 1981), together with giving a puzzled look (usually frowning). This is especially the case when the speaker says something that is vague. For example, they might say “I want it” to their partner, without saying what it is they want. The partner may reply using a token or, alternatively, explicitly ask, “What do you mean by it?” Nonverbal com-munication also plays an important role in augmenting face-to-face conversation, involving the use of facial expressions, back channeling, voice intonation, gesturing, and other kinds of body language.
Taking turns also provides opportunities for the listener to initiate repair or request clarification or for the speaker to detect that there is a problem and initiate repair. The lis-tener will usually wait for the next turn in the conversation before interrupting the speaker in order to give the speaker the chance to clarify what is being said by completing the utterance.
ACTIVITY 5.3 How do people repair breakdowns when conversing via email? Do they do the same when texting?
CommentAs people usually cannot see each other when communicating by email or text, they have to rely on other means of repairing the conversation when things are left unsaid or are unclear. For example, when someone proposes an ambiguous meeting time, where the date and day given don’t match up for the month, the person receiving the message may begin their reply by asking politely, “Did you mean this month or June?” rather than baldly stating the other person’s error, for example, “the 13th May is not a Wednesday!”
When someone does not reply to an email or text when the sender is expecting them to do so, it can put them in a quandary as to what to do next. If someone does not reply to an email within a few days, then the sender might send them a gentle nudge message that dimin-ishes any blame, for example, “I am not sure if you got my last email as I was using a different account” rather than explicitly asking them why they have not answered the email they sent. When texting, it depends on whether it is a dating, family, or business-related text that has been sent. When starting to date, some people will deliberately wait a while before replying to a text as a form of “playing games” and trying not to appear to be overly keen. If they don’t reply at all, it is a generally accepted notion that they are not interested, and no further texts should be sent. In contrast, in other contexts, double-texting has become an acceptable social norm as a way of reminding someone, without sounding too rude, to reply. It implicitly implies that the sender understands that the recipient has overlooked the first text because they were too busy or doing something else at the time, thereby saving face.
Emails and texts can also appear ambiguous, especially when things are left unsaid. For example, the use of an ellipsis (…) at the end of a sentence can make it difficult to work out what the sender intended when using it. Was it to indicate something was best left unsaid, the sender is agreeing to something but their heart is not in it, or simply that the sender did not know what to say? This email or text convention puts the onus on the receiver to decide what is meant by the ellipsis and not on the sender to explain what they meant.
5 . 4 R E m O T E C O N V E R S AT I O N S 143
5.4 Remote Conversations
The telephone was invented in the nineteenth century by Alexander Graham Bell, enabling two people to talk to one another at a distance. Since then, a number of other technologies have been developed that support synchronous remote conversations, including videophones that were developed in the 1960s–1970s (see Figure 5.2). In the late 1980s and 1990s, a range of “media spaces” were the subjects of experimentation—audio, video, and computer systems were com-bined to extend the world of desks, chairs, walls, and ceilings (Harrison, 2009). The goal was to see whether it was possible for people, distributed over space and different time zones, could communicate and interact with one another as if they were actually physically present.
An example of an early media space was the VideoWindow (Bellcore, 1989) that was developed to enable people in different locations to carry on a conversation as they would do if they were drinking coffee together in the same room (see Figure 5.3). Two lounge areas that were 50 miles apart were connected via a 3-foot by 5-foot picture window onto which video images of each location were projected. The large size enabled viewers to see a room of peo-ple roughly the same size as themselves. A study of its use showed that many of the conver-sations that took place between the remote conversants were indeed indistinguishable from similar face-to-face interactions, with the difference being that they spoke a bit louder and constantly talked about the video system (Kraut et al., 1990). Other research on how people interact when using videoconferencing has shown that they tend to project themselves more, take longer conversational turns, and interrupt each other less (O’Connaill et al., 1993).
Since this early research, videoconferencing has come of age. The availability of cheap webcams and cameras now embedded as a default in tablets, laptops, and phones has
Figure 5.2 One of British Telecom’s early videophonesSource: British Telecommunications Plc
5 S O C I A L I N T E R A C T I O N144
greatly helped make videoconferencing mainstream. There are now numerous platforms available from which to choose, both free and commercial. Many videoconferencing apps (for example, Zoom or Meeting Owl) also allow multiple people at different sites to connect synchronously. To indicate who has the floor, screen effects are often used, such as enlarging the person who is talking to take up most of the screen or highlighting their portal when they take the floor. The quality of the video has also improved, making it possible for people to appear more life-like in most setups. This is most noticeable in high-end telepresence rooms that use multiple high-definition cameras with eye-tracking features and directional micro-phones (see Figure 5.4). The effect can be to make remote people appear more present by projecting their body movements, actions, voice, and facial expressions to the other location.
Another way of describing this development is in terms of the degree of telepresence. By this we mean the perception of being there when physically remote. Robots, for example, have been built with telepresence in mind to enable people to attend events and communicate with others by controlling them remotely. Instead of sitting in front of a screen from their
Figure 5.3 Diagram of VideoWindow system in use
Figure 5.4 A telepresence roomSource: Cisco Systems, Inc.
5 . 4 R E m O T E C O N V E R S AT I O N S 145
location and seeing the remote place solely through a fixed camera at the other place, they can look around the remote place by controlling the “camera’s” eyes, which are placed on the robot and are physically moving it around. For example, telepresence robots have been devel-oped to enable children who are in a hospital to attend school by controlling their assigned robots to roam around the classroom (Rae et al., 2015).
Telepresence robots are also being investigated to determine whether they can help peo-ple who have developmental difficulties visit places remotely, such as museums. Currently, several of the activities that are involved in going on such a visit, such as buying a ticket or using public transport, are cognitively challenging, preventing them from going on such trips. Natalie Friedman and Alex Cabral (2018) conducted a study with six participants with developmental difficulties to see whether providing them each with a telepresence robot would increase their physical and social self-efficacy and well-being. The participants were taken on a remote tour of two museum exhibits and then asked to rate their experience after-ward. Their responses were positive, suggesting that this kind of telepresence can open doors to social experiences that were previously denied to those with disabilities.
BOX 5.1 Facebook Spaces: How Natural Is It to Socialize in a 3D World?
Facebook’s vision of social networking is to immerse people in 3D, where they interact with their friends in virtual worlds. Figure 5.5 shows what it might look like: Two avatars (Jack and Diane) are talking at a virtual table beside a lake and with some mountains in the back-ground. Users experience this by wearing virtual reality (VR) headsets. The goal is to provide users with a magical feeling of presence, one where they can feel as if they are together, even
(Continued)
Figure 5.5 Facebook’s vision of socializing in a 3D worldSource: Facebook
5 S O C I A L I N T E R A C T I O N146
Telepresence robots have also become a regular feature at conferences, including the ACM CHI conference, enabling people to attend who cannot travel. They are typically about 5-feet tall, have a display at the top that shows the remote person’s head, and have a base at the bottom holding wheels allowing the robot to move forward, move backward, or turn around. A commercial example is the Beam+ (https://suitabletech.com/). To help the robot navigate in its surroundings, two cameras are embedded in the display, one facing outward to provide the remote person with a view of what is in front of them and the other facing downward to provide a view of the floor. The robots also contain a microphone and speak-ers to enable the remote person to be heard and to hear what is being said locally. Remote users connect via Wi-Fi to the remote site and steer their Beam+ robot using a web interface.
A PhD student from the University College London (UCL) attended her first CHI confer-ence remotely, during which time she gave a demo of her research every day by talking to the attendees using the Beam+ robot (see Figure 5.6). Aside from a time difference of eight hours (meaning that she had to stay up through the night to attend), it was an enriching experience for her. She met lots of new people who not only were interested in her demo but who also learned how she felt about attending the conference remotely. Her colleagues at the conference also dressed up her robot to make her appear more like her, giving the robot a set of foam-cutout arms with waving hands, and they put a university T-shirt on the robot. However, she could not see how she appeared to others at the conference, so local attendees took photos of her Beam+ robot to show her how she looked. She also commented how she could not gauge the volume of her voice, and on one occasion she accidentally set the volume control to be at its highest setting. When speaking to someone, she did not realize how loud
though they are apart in the physical world. To make the experience appear more life-like, users can move their avatar’s arms through controls provided by the VR OculusTouch.
While big strides have been made toward improving social presence, there is still a way to go before the look and feel of socializing with virtual avatars becomes more like the real thing. For a start, the facial expressions and skin tone of virtual avatars still appear to be cartoon-like.
Similar to the term telepresence, social presence refers to the feeling of being there with a real person in virtual reality. Specifically, it refers to the degree of awareness, feeling, and reaction to other people who are virtually present in an online environment. The term differs from telepresence, which refers to one party being virtually present with another party who is present in a physical space, such as a meeting room (note that it is possible for more than one telepresence robot to be in the same physical space). Imagine if avatars become more convinc-ing in their appearance to users. How many people would switch from their current use of 2D media to catch up and chat with friends in this kind of immersive 3D Facebook page? Do you think it would enhance the experience of how they would interact and communicate with oth-ers remotely?
How many people would don a VR headset 10 times a day or more to teleport to meet their friends virtually? (The average number of times that someone looks at Facebook on their phones is now 14 times each day.) There is also the perennial problem of motion sickness that 25–40 percent of people say that they have experienced in VR (Mason, 2017).
5 . 4 R E m O T E C O N V E R S AT I O N S 147
she was until another person across the room told her that she was yelling. (The person she was talking to was too polite to tell her to lower her voice.)
Another navigation problem that can occur is when the remote person wants to move from one floor to another in a building. They don’t have a way of pressing the elevator but-ton to achieve this. Instead, they have to wait patiently beside the elevator for someone to come along to help them. They also lack awareness of others who are around them. For example, when moving into a room to get a good spot to see a presentation, they may not realize that they have obscured the view of people sitting behind them. It can also be a bit sur-real when their image starts breaking up on the robot “face” as the Wi-Fi signal deteriorates. For example, Figure 5.7 shows Johannes Schöning breaking up into a series of pixels that makes him look a bit like David Bowie!
Despite these usability problems, a study of remote users trying a telepresence robot for the first time at a conference found the experience to be positive (Neustaedter et al., 2016). Many felt that it provided them with a real sense of being at the conference—quite different from the experience of watching or listening to talks online—as happens when connecting via a livestream or a webinar. Being able to move around the venue also ena-bled them to see familiar faces and to bump into people during coffee breaks. For the
Figure 5.6 Susan Lechelt’s Beam+ robot given a human touch with cut-out foam arms and a uni-versity logo T-shirtSource: Used courtesy of Susan Lechelt
5 S O C I A L I N T E R A C T I O N148
conference attendees, the response was also largely positive, enabling them to chat with those who could not make the conference. However, sometimes the robot’s physical pres-ence obstructed their view in a room when watching a speaker, and that could be frustrat-ing. It is difficult to know how to tell a telepresence robot discreetly to move out of the way while a talk is in progress and for the remote person to know where to move that is out of the way as they have been told.
Figure 5.7 The image of Johannes Schöning breaking up on the Beam+ robot video display when the Wi-Fi signal deterioratedSource: Yvonne Rogers
ACTIVITY 5.4 Watch these two videos about Beam and Cisco’s telepresence. How does the experi-ence of being at a meeting using a telepresence robot compare with using a telepres-ence videoconferencing system?
VideosBeamPro overview of how robotic telepresence works—https://youtu.be/SQCigphfSvcCisco TelePresence Room EndPoints MX700 and MX800—https://youtu.be/52lgl0kh0FI
CommentThe BeamPro allows the remote person to move around a workplace as well as sit in on meet-ings. They can also have one-on-one conversations with someone at their desk. When moving
5 . 4 R E m O T E C O N V E R S AT I O N S 149
around, the remote individual can even bump into other remote workers, in the corridor, for example, who are also using a BeamPro. Hence, it supports a range of informal and formal social interactions. Using a BeamPro also allows someone to feel as if they are at work while still being at home.
In contrast, the Cisco telepresence room has been designed specifically to support meet-ings between small remote groups to make them feel more natural. When someone is speak-ing, the camera zooms in on them to have them fill the screen. From the video, it appears effortless and allows the remote groups to focus on their meeting rather than worry about the technology. However, there is limited flexibility—conducting one-on-one meetings, for example.
BOX 5.2 Simulating Human Mirroring Through Artificial Smiles
A common phenomenon that occurs during face-to-face conversations is mirroring, where people mimic each other’s facial expressions, gestures, or body movements. Have you ever noticed that when you put your hands behind your head, yawn, or rub your face during a conversation with someone that they follow suit? These kinds of mimicking behaviors are assumed to induce empathy and closeness between those conversing (Stel and Vonk, 2010). The more people engage in mimicry, the more they view each other as being similar, which in turn increases the rapport between them (Valdesolo and DeSteno, 2011). Mimicry doesn’t always occur during a conversation, however—sometimes it requires a conscious effort, while in other situations it does not occur. Might the use of technology increase its occurrence in conversations?
One way would be to use special video effects. Suppose that an artificial smile could be superimposed on the video face of someone to make them appear to smile. What might hap-pen? Would they both begin to smile and in doing so feel closer to each other? To investigate this possibility of simulating smiling mimicry, Keita Suzuki et al. (2017) developed a technique called FaceShare. The system was developed so that it could deform the image of someone’s face to make it appear to smile—even though they were not—whenever their partner’s face began smiling. The mimicry method used 3D modeling of key feature points of the face, including the contours, eyes, nose, and mouth to detect where to place the smile. The smile was created by raising the lower eyelids and both ends of the mouth in conjunction with the cheeks. The findings from this research showed that FaceShare was effective at making con-versations appear smoother and that the pseudo smiles appearing on someone’s video face were judged to be natural.
5 S O C I A L I N T E R A C T I O N150
5.5 Co-presence
Together with telepresence, there has been much interest in enhancing co-presence, that is, supporting people in their activities when interacting in the same physical space. A number of technologies have been developed to enable more than one person to use them at the same time. The motivation is to enable co-located groups to collaborate more effectively when working, learning, and socializing. Examples of commercial products that support this kind of parallel interaction are Smartboards and Surfaces, which use multitouch, and Kinect, which uses gesture and object recognition. To understand how effective they are, it is important to consider the coordination and awareness mechanisms already in use by people in face-to-face interactions and then to see how these have been adapted or replaced by the technology.
5.5.1 Physical CoordinationWhen people are working closely together, they talk to each other, issuing commands and letting others know how they are progressing. For example, when two or more people are collaborating, as when moving a piano, they shout instructions to each other, like “Down a bit, left a touch, now go straight forward,” to coordinate their actions. A lot of nonverbal communication is also used, including nods, shakes, winks, glances, and hand-raising in com-bination with such coordination talk in order to emphasize and sometimes replace it.
For time-critical and routinized collaborative activities, especially where it is difficult to hear others because of the physical conditions, people frequently use gestures (although radio-controlled communication systems may also be used). Various types of hand signals have evolved, with their own set of standardized syntax and semantics. For example, the arm and baton movements of a conductor coordinate the different players in an orchestra, while the arm and orange baton movements of ground personnel at an airport signal to a pilot how to bring the plane into its assigned gate. Universal gestures, such as beckoning, waving, and halting hand movement, are also used by people in their everyday settings.
The use of physical objects, such as wands and batons, can also facilitate coordination. Group members can use them as external thinking props to explain a principle, an idea, or a plan to the others (Brereton and McGarry, 2000). In particular, the act of waving or holding up a physical object in front of others is very effective at commanding attention. The per-sistence and ability to manipulate physical artifacts may also result in more options being explored in a group setting (Fernaeus and Tholander, 2006). They can help collaborators gain a better overview of the group activity and increase awareness of others’ activities.
5.5.2 AwarenessAwareness involves knowing who is around, what is happening, and who is talking with whom (Dourish and Bly, 1992). For example, when attending a party, people move around the physi-cal space, observing what is going on and who is talking to whom, eavesdropping on oth-ers’ conversations, and passing on gossip to others. A specific kind of awareness is peripheral awareness. This refers to a person’s ability to maintain and constantly update a sense of what is going on in the physical and social context, by keeping an eye on what is happening in the periphery of their vision. This might include noticing whether people are in a good or bad mood by the way they are talking, how fast the drink and food is being consumed, who has entered or left the room, how long someone has been absent, and whether the lonely person in
5 . 5 C O – p R E S E N C E 151
the corner is finally talking to someone—all while we are having a conversation with someone else. The combination of direct observations and peripheral monitoring keeps people informed and updated on what is happening in the world.
Another form of awareness that has been studied is situational awareness. This refers to being aware of what is happening around you in order to understand how information, events, and your own actions will affect ongoing and future events. Having good situational awareness is critical in technology-rich work domains, such as air traffic control or an oper-ating theater, where it is necessary to keep abreast of complex and continuously changing information.
People who work closely together also develop various strategies for coordinating their work, based on an up-to-date awareness of what the others are doing. This is especially so for interdependent tasks, where the outcome of one person’s activity is needed for others to be able to carry out their tasks. For example, when putting on a show, the performers will con-stantly monitor what each other is doing in order to coordinate their performance efficiently. The metaphorical expression close-knit teams exemplifies this way of collaborating. People become highly skilled in reading and tracking what others are doing and the information to which they are paying attention.
A classic study of this phenomenon is of two controllers working together in a control room in the London Underground subway system (Heath and Luff, 1992). An overriding observation was that the actions of one controller were tied closely to what the other was doing. One of the controllers (controller A) was responsible for the movement of trains on the line, while the other (controller B) was responsible for providing information to passen-gers about the current service. In many instances, it was found that controller B overheard what controller A was saying and doing and acted accordingly, even though controller A had not said anything explicitly to him. For example, on overhearing controller A discussing a problem with a train driver over the in-cab intercom system, controller B inferred from the conversation that there was going to be a disruption in the service and so started to announce this to the passengers on the platform before controller A had even finished talking with the train driver. At other times, the two controllers kept a lookout for each other, monitoring the environment for actions and events that they might not have noticed but that could have been important for them to know about so that they could act appropriately.
ACTIVITY 5.5 What do you think happens when one person in a close-knit team does not see or hear some-thing, or misunderstands what has been said, while the others in the group assume that person has seen, heard, or understood what has been said?
CommentThe person who has noticed that someone has not acted in the manner expected may use one of a number of subtle repair mechanisms, say coughing or glancing at something that needs to be attended to. If this doesn’t work, they may then resort to stating explicitly aloud what
(Continued)
5 S O C I A L I N T E R A C T I O N152
5.5.3 Shareable InterfacesA number of technologies have been designed to capitalize on existing forms of coordination and awareness mechanisms. These include whiteboards, large touch screens, and multitouch tables that enable groups of people to collaborate while interacting at the same time with content on the surfaces. Several studies have investigated whether different arrangements of shared technologies can help co-located people work better together (for example, see Müller-Tomfelde, 2010). An assumption is that shareable interfaces provide more opportuni-ties for flexible kinds of collaboration compared with single-user interfaces, through enabling co-located users to interact simultaneously with digital content. The use of fingers or pens as input on a public display is observable by others, increasing opportunities for building situational and peripheral awareness. The sharable surfaces are also considered to be more natural than other technologies, enticing people to touch them without feeling intimidated or embarrassed by the consequences of their actions. For example, small groups found it more comfortable working together around a tabletop compared with sitting in front of a PC or standing in a line in front of a vertical display (Rogers and Lindley, 2004).
BOX 5.3 Playing Together in the Same Place
Augmented reality (AR) sandboxes have been developed for museum visitors to interact with a landscape, consisting of mountains, valleys, and rivers. The sand is real, while the landscape is virtual. Visitors can sculpt the sand into different-shaped contours that change their appear-ance to look like a river or land, depending on the height of the sand piles. Figure 5.8 shows a AR sandbox that was installed at the V&A museum in London. On observing two young children playing at the sandbox, this author overheard one say to the other while flattening a pile of sand, “Let’s turn this land into sea.” The other replied “OK, but let’s make an island on that.” They continued to talk about how and why they should change their landscape. It was a pleasure to watch this dovetailing of explaining and doing.
The physical properties of the sand, together with the real-time changing superimposed landscape, provided a space for children (and adults) to collaborate in creative ways.
had previously been signaled implicitly. Conversely, the unaware person may wonder why the event hasn’t happened and, likewise, look over at the other team members, cough to get their attention, or explicitly ask them a question. The kind of repair mechanism employed at a given moment will depend on a number of factors, including the relationship among the participants, for instance, whether one is more senior than the others. This determines who can ask what, the perceived fault or responsibility for the breakdown, and the severity of the outcome of not acting there and then on the new information.
5 . 5 C O – p R E S E N C E 153
Often in meetings, some people dominate while others say very little. While this is OK in certain settings, in others it is considered more desirable for everyone to have a say. Is it possible to design shareable technologies so that people can participate around them more equally? Much research has been conducted to investigate whether this is possible. Of primary importance is whether the interface invites people to select, add, manipulate, or remove digi-tal content from the displays and devices. A user study showed that a tabletop that allowed group members to add digital content by using physical tokens resulted in more equitable participation than if only digital input was allowed via touching icons and menus at the tabletop (Rogers et al., 2009). This suggests that it was easier for people who are normally shy in groups to contribute to the task. Moreover, people who spoke the least were found to make the largest contribution to the design task at the tabletop, in terms of selecting, adding, moving, and removing options. This reveals how changing the way people can interact with a surface can affect group participation. It shows that it is possible for more reticent members to contribute without feeling under pressure to speak more.
Figure 5.8 Visitors creating together using an Augmented Reality Sandbox at the V&A Museum in LondonSource: Helen Sharp
5 S O C I A L I N T E R A C T I O N154
Experimentation with real-time feedback presented via ambient displays has also been shown to provide a new form of awareness for co-located groups. LEDs glowing in tabletops and abstract visualizations on handheld and wall displays have been designed to represent how different group members are performing, such as turn-taking. The assumption is that this kind of real-time feedback can promote self and group regulation and in so doing modify group members’ contributions to make them more equitable. For example, the Reflect Table was designed based on this assumption (Bachour et al., 2008). The table monitors and analyzes ongoing conversations using embedded microphones in front of each person and represents this in the form of increasing numbers of colored LEDs (see Figure 5.9). A study investigated whether students became more aware of how much they were speaking during a group meet-ing when their relative levels of talk were displayed in this manner and, if so, whether they regulated their levels of participation more effectively. In other words, would the girl in the bottom right reduce her contributions (as she clearly has been talking the most) while the boy in the bottom left increase his (as he has been talking the least)? The findings were mixed: Some participants changed their level to match the levels of others, while others became frus-trated and chose simply to ignore the LEDs. Specifically, those who spoke the most changed their behavior the most (that is, reduced their level) while those who spoke the least changed theirs the least (in other words, did not increase their level). Another finding was that participants who believed that it was beneficial to contribute equally to the conversation took more notice of the LEDs and regulated their conversation level accordingly. For example, one participant said that she “refrained from talking to avoid having a lot more lights than the others” (Bachour et al., 2010). Conversely, participants who thought it was not important took less notice. How do you think you would react?
An implication from the various user studies on co-located collaboration around tab-letops is that designing shareable interfaces to encourage more equitable participation isn’t straightforward. Providing explicit real-time feedback on how much someone is speaking in a group may be a good way of showing everyone who is talking too much, but it may be intimidating for those who are talking too little. Allowing discreet and accessible ways for adding and manipulating content to an ongoing collaborative task at a shareable surface may
Figure 5.9 The Reflect TableSource: Used courtesy of Pierre Dillenbourg
5 . 5 C O – p R E S E N C E 155
be more effective at encouraging greater participation from people who normally find it dif-ficult or who are simply unable to contribute verbally to group settings (for example, those on the autistic spectrum, those who stutter, or those who are shy or are non-native speakers).
How best to represent the activity of online social networks in terms of who is taking part has also been the subject of much research. A design principle that has been influential is social translucence (Erickson and Kellogg, 2000). This refers to the importance of designing communication systems to enable participants and their activities to be visible to one another. This idea was very much behind the early communication tool, Babble, developed at IBM by David Smith (Erickson et al., 1999), which provided a dynamic visualization of the par-ticipants in an ongoing chat room. A large 2D circle was depicted using colored marbles on each user’s monitor. Marbles inside the circle conveyed those individuals active in the current conversation. Marbles outside the circle showed users involved in other conversations. The more active a participant was in the conversation, the more the corresponding marble moved toward the center of the circle. Conversely, the less engaged a person was in the ongoing con-versation, the more the marble moved toward the periphery of the circle.
Since this early work on visualizing social interactions, there have been a number of virtual spaces developed that provide awareness about what people are doing, where they are, and their availability, with the intention of helping them feel more connected. Work-ing in remote teams can be isolating, especially if they rarely get to see their colleagues face to face. When teams are not co-located, they also miss out on in-person collaboration and valuable informal conversations that build team alignment. This is where the concept of the “online office” comes in. For example, Sococo (https://www.sococo.com/) is an online office platform that is bridging the gap between remote and co-located work. It uses the spatial metaphor of a floor plan of an office to show where people are situated, who is in a meeting, and who is chatting with whom. The Sococo map (see Figure 5.10) provides a
Search for colleaguesacross theworkspace to seestatus or instantly chat
See a team in a meeting,sharing screens and viewing
documents in a room
Name a room to reflectthe topic of a meeting inprogress
Knock on a door to join ameeting or just pop in
Send a link for a guest tojoin you in your Sococo office
Blinking avatars arecolleagues collaborating
Share documents or linkson a desk for immediateaccess to anyone in theroom
Instantly “Get”colleagues tospontaneouslycollaborate
Scale instantly withunlimited floors to yourSococo space
Figure 5.10 Sococo floor plan of a virtual office, showing who is where and who is meeting with whomSource: Used courtesy of Leeann Brumby
5 S O C I A L I N T E R A C T I O N156
bird’s-eye view of a team’s online office, giving everyone at-a-glance insight into teammates’ availability and what’s happening organizationally. Sococo also provides the sense of pres-ence and virtual “movement” that you get in a physical office—anyone can pop into a room, turn on their microphone and camera, and meet with another member of their team face to face. Teams can work through projects, get feedback from management, and collaborate ad hoc in their online office regardless of physical location. This allows organizations to take advantage of the benefits of the distributed future of work while still providing a central, online office for their teams.
BOX 5.4 Can Technologies Be Designed to Help People Break the Ice and Socialize?
Have you ever found yourself at a party, wedding, conference, or other social gathering, stand-ing awkwardly by yourself, not knowing who to talk to or what to talk about? Social embar-rassment and self-consciousness affect most of us at such moments, and such feelings are most acute when one is a newcomer and by oneself, such as a first-time attendee at a conference. How can conversation initiation be made easier and less awkward for people who do not know each other?
A number of mechanisms have been employed by organizers of social events, such as asking old-timers to act as mentors and the holding of various kinds of ice-breaking activities. Badge-wearing, the plying of drink and food, and introductions by others are also common ploys. While many of these methods can help, engaging in ice-breaking activities requires people to act in a way that is different from the way they normally socialize and which they may find equally uncomfortable or painful to do. They often require people to agree to join in a collaborative game, which they may find embarrassing. This can be exac-erbated by the fact that once people have agreed to take part, it is difficult for them to drop out because of the perceived consequences that it will have for the others and themselves (such as being seen by the others as a spoilsport or party-pooper). Having had one such embarrassing experience, most people will shy away from any further kinds of ice-breaking activities.
An alternative approach is to design a physical space where people can enter and exit a conversation with a stranger in subtler ways, that is, one where people do not feel threatened or embarrassed and that does not require a high level of commitment. The classic Opinionizer system was designed along these lines, with the goal of encouraging people in an informal gathering to share their opinions visually and anonymously (Brignull and Rogers, 2003). The collective creation of opinions via a public display was intended to provide a talking point for the people standing beside it. Users submitted their opinions by typing them in at a public keyboard. To add color and personality to their opinions, a selection of small cartoon avatars and speech bubbles were available. The screen was also divided into four labeled quadrants representing different backgrounds, such as techie, softie, designer, or student, to provide a factor on which people could comment (see Figure 5.11).
5 . 5 C O – p R E S E N C E 157
A range of other ambient-based displays have been developed and placed in physical work settings with the purpose of encouraging people to socialize and talk more with each other. For example, the Break-Time Barometer was designed to persuade people to come out of their offices for a break to meet others they might not talk with otherwise (Kirkham et al., 2013). An ambient display, based on a clock metaphor, shows how many people are currently in the common room; if there are people present, it also sends an alert that it would be a good time to join them for a break. While the system nudged some people to go for a break in the staff room, it also had the opposite effect on others who used it to determine when breaks weren’t happening so that they could take a break without their colleagues being around for company.
When the Opinionizer was placed in various social gatherings, a honey-pot effect was observed. By this it is meant the creation of a sociable buzz in the immediate vicinity of the Opinionizer as a result of an increase in the number of people moving into the area. Further-more, by standing in this space and showing an interest, for example visibly facing the screen or reading the text, people gave off a tacit signal to others that they were open to discussion and interested in meeting new people.
There are now a number of commercial ice-breaking phone apps available that use arti-ficial intelligence (AI) matchmaking algorithms to determine which preferences and interests shared among people make them suitable conversational partners. Wearable technology is also being developed as a new form of digital ice-breaker. Limbic Media (https://limbicmedia .ca/social-wearables/), for example, has developed a novel pendant device colored with LED lights for this purpose. When two people touch their pendants together, the effect is for them to vibrate. This coming together action can break the ice in a fun and playful way.
(a) (b)
Figure 5.11 (a) The Opinionizer interface and (b) a photo of it being used at a book launch partySource: Helen Sharp
This video features Limbic Media’s novel type of social wearable being used at the 2017 BCT Tech Summit: https://vimeo.com/216045804.
5 S O C I A L I N T E R A C T I O N158
5.6 Social Engagement
Social engagement refers to participation in the activities of a social group (Anderson and Binstock, 2012). Often it involves some form of social exchange where people give or receive something from others. Another defining aspect is that it is voluntary and unpaid. Increas-ingly, different forms of social engagement are mediated by the Internet. For example, there are many websites now that support pro-social behavior by offering activities intended to help others. One of the first websites of this ilk was GoodGym (www.goodgym.org/), which connects runners with isolated older people. While out running, the runners stop for a chat with an older person who has signed up to the service, and the runner helps them with their chores. The motivation is to help others in need while getting fit. There is no obliga-tion, and anyone is welcome to join. Another website that was set up is conservation vol-unteers (https://www.tcv.org.uk/). The website brings together those who want to help out with existing conservation activities. By bringing different people together, social cohesion is also promoted.
Not only has the Internet enabled local people to meet who would not have otherwise, it has proven to be a powerful way of connecting millions of people with a common interest in ways unimaginable before. An example is retweeting a photo that resonates with a large crowd who finds it amusing and wants to pass it on further. For example, in 2014, the most retweeted selfie was one taken by Ellen DeGeneres (an American comedian and television host) at the Oscar Academy Awards of her in front of a star-studded, smiling group of actors and friends. It was retweeted more than 2 million times (more than three-quarters of a mil-lion in the first half hour of being tweeted)—far exceeding the one taken by Barack Obama at Nelson Mandela’s funeral the previous year.
There has even been an “epic Twitter battle.” A teenager from Nevada, Carter Wilker-son, asked Wendy’s fast-food restaurant how many retweets were needed for him to receive a whole year’s supply of free chicken nuggets. The restaurant replied “18 million” (see Figure 5.12). From that moment on, his quest became viral with his tweet being retweeted more than 2 million times. Ellen’s record was suddenly put in jeopardy, and she intervened, putting out a series of requests on her show for people to continue to retweet her tweet so her record would be upheld. Carter, however, surpassed her record at the 3.5 million mark. During the Twitter battle, he used his newly found fame to create a website that sold T-shirts promoting his chicken nugget challenge. He then donated all of the proceeds from the sales toward a charity that was close to his heart. The restaurant also gave him a year’s supply of free chicken nuggets—even though he didn’t reach the target of 18 million. Not only that, it also donated $100,000 to the same charity in honor of Carter achieving a new record. It was a win-win situation (except maybe for Ellen).
Another way that Twitter connects people rapidly and at scale is when unexpected events and disasters happen. Those who have witnessed something unusual may upload an image that they have taken of it or retweet what others have posted to inform others about it. Those who like to reach out in this way are sometimes called digital volunteers. For example, while writing this chapter, there was a massive thunderstorm overhead that was very dramatic. I checked out the Twitter hashtag #hove (I was in the United Kingdom) and found that hundreds of people had uploaded photos of the hailstones, flooding, and minute-by-minute updates of how public transport and traffic were being affected. It was easy to get a sense
5 . 6 S O C I A L E N g A g E m E N T 159
of the scale of the storm before it was picked up by the official media channels, which then used some of the photos and quotes from Twitter in their coverage (see Figure 5.13). Relying on Twitter for breaking news has increasingly become the norm. When word came of a huge explosion in San Bruno, California, the chief of the Federal Emergency Management Agency in the United States logged on to Twitter and searched for the word explosion. Based on the tweets coming from that area, he was able to discern that the gas explosion and ensuing fire was a localized event that would not spread to other communities. He noted how he got bet-ter situational awareness more quickly from reading Twitter than by hearing about it from official sources.
Clearly, the immediacy and global reach of Twitter provides an effective form of com-munication, providing first responders and those living in the affected areas with up-to-the-minute information about how a wildfire, storm, or gas plume is spreading. However, the reliability of the tweeted information can sometimes be a problem. For example, some people end up obsessively checking and posting, sometimes without realizing that this can start or fuel rumors by adding news that is old or incorrect. Regulars can go into a frenzy, constantly adding new tweets about an event, as witnessed when an impending flood was announced (Starbird et al., 2010). While such citizen-led dissemination and retweeting of information from disparate sources is well intentioned, it can also flood the Twitter streams, making it difficult to know what is old, actual, or hearsay.
Figure 5.12 Carter Wilkerson’s tweet that went viral
5 S O C I A L I N T E R A C T I O N160
Figure 5.13 A weather warning photo tweeted and retweeted about a severe storm in Hove, United Kingdom
BOX 5.5 Leveraging Citizen Science and Engagement Through Technology
The growth and success of citizen science and citizen engagement has been made possible by the Internet and mobile technology, galvanizing and coordinating the efforts of millions of people throughout the world. Websites, smartphone apps, and social media have been instrumental in leveraging the reach and impact of a diversity of citizen science projects across time and geographical zones (Preece et al., 2018). Citizen science involves local peo-ple helping scientists carry out a scientific project at scale. Currently, thousands of such projects have been set up all over the world, whereby volunteers help out in a number of research areas, including biodiversity, air quality, astronomy, and environmental issues. They do so by engaging in scientific activities such as monitoring plants and wildlife, col-lecting air and water samples, categorizing galaxies, and analyzing DNA sequences. Citizen engagement involves people helping governments, rather than scientists, to improve pub-lic services and policies in their communities. Examples include setting up and oversee-ing a website that offers local services for community disasters and creating an emergency response team when a disaster occurs.
Why would anyone want to volunteer their time for the benefit of science or government? Many people want to learn more about a domain, while others want to be recognized for their contributions (Rotman et al., 2014). Some citizen science apps have developed online mecha-nisms to support this. For example, iNaturalist (https://www.inaturalist.org/) enables volun-teers to comment on and help classify others’ contributions.
5 . 6 S O C I A L E N g A g E m E N T 161
DILEmmAIs It OK to Talk with a Dead Person Using a Chatbot?
Eugenia Kuyda, an AI researcher, lost a close friend in a car accident. He was only in his 20s. She did not want to lose his memory, so she gathered all of the texts he had sent over the course of his life and made a chatbot from them. The chatbot is programmed to respond automatically to text messages so that Eugenia can talk to her friend as if he were still alive. It responds to her questions using his own words.
Do you think this kind of interaction is creepy or comforting to someone who is grieving? Is it disrespectful of the dead, especially if the dead person has not given their consent? What if the friend had agreed to having their texts mashed up in this way in a “pre-death digital agreement”? Would that be more socially acceptable?
In-Depth ActivityThe goal of this activity is to analyze how collaboration, coordination, and communication are supported in online video games involving multiple players.
The video game Fortnite arrived in 2017 to much acclaim. It is an action game designed to encourage teamwork, cooperation, and communication. Download the game from an app store (it is free) and try it. You can also watch an introductory video about it at https://youtu.be/_U2JbFhUPX8.
Answer the following questions.1. Social issues
(a) What is the goal of the game?(b) What kinds of conversations are supported?(c) How is awareness of the others in the game supported?(d) What kinds of social protocols and conventions are used?(e) What types of awareness information are provided?(f) Does the mode of communication and interaction seem natural or awkward?(g) How do players coordinate their actions in the game?
2. Interaction design issues(a) What form of interaction and communication is supported, for instance, text, audio,
and/or video?(b) What other visualizations are included? What information do they convey?(c) How do users switch between different modes of interaction, for example, exploring
and chatting? Is the switch seamless?(d) Are there any social phenomena that occur specific to the context of the game that
wouldn’t happen in face-to-face settings?3. Design issues
• What other features might you include in the game to improve communication, coordi-nation, and collaboration?
5 S O C I A L I N T E R A C T I O N162
Further Reading
boyd, d. (2014) It’s Complicated: The Social Lives of Networked Teens. Yale. Based on a series of in-depth interviews with a number of teenagers, danah boyd offers new insights into how teenagers across the United States, who have only ever grown up in a world of apps and media, navigate, use, and appropriate them to grow up and develop their identities. A number of topics are covered that are central to what it means to grow up in a networked world, including bullying, addiction, expressiveness, privacy, and inequality. It is insightful and covers much ground.
CRUMLISH, C. and MALONE, E. (2009) Designing Social Interfaces. O’Reilly. This is a col-lection of design patterns, principles, and advice for designing social websites, such as online communities.
GARDNER, H. and DAVIS, K. (2013) The App Generation: How Today’s Youth Navigate Identity, Intimacy, and Imagination in a Digital World. Yale. This book explores the impact of apps on the young generation, examining how they affect their identity, intimacy, and imagination. It focuses on what it means to be app-dependent versus app-empowered.
SummaryHuman beings are inherently social. People will always need to collaborate, coordinate, and communicate with one another, and the diverse range of applications, web-based services, and technologies that have emerged enable them to do so in more extensive and diverse ways. In this chapter, we looked at some core aspects of sociality, namely, communication and col-laboration. We examined the main social mechanisms that people use in different conversa-tional settings when interacting face to face and at a distance. A number of collaborative and telepresence technologies designed to support and extend these mechanisms were discussed, highlighting core interaction design concerns.
Key Points • Social interaction is central to our everyday lives. • Social mechanisms have evolved in face-to-face and remote contexts to facilitate conversation, coordination, and awareness.
• Talk and the way it is managed are integral to coordinating social interaction. • Many kinds of technologies have been developed to enable people to communicate remotely with one another.
• Keeping aware of what others are doing and letting others know what you are doing are important aspects of collaboration and socializing.
• Social media has brought about significant changes in the way people keep in touch and manage their social lives.
163
ROBINSON, S., MARSDEN, G. and JONES, M. (2015) There’s Not an App for That: Mobile User Experience Design for Life. Elsevier. This book offers a fresh approach for designers, students, and researchers to dare to think differently by moving away from the default framing of technological design in terms of yet another “looking down” app. It asks the reader instead to look up and around them—to be inspired by how we actually live our lives when “out there” app-less. They also explore what it means to design technologies to be more mindful.
TURKLE, S. (2016) Reclaiming Conversation: The Power of Talk in a Digital Age. Penguin. Sherry Turkle has written extensively about the positive and negative effects of digital tech-nology on everyday lives—at work, at home, at school, and in relationships. This book is a very persuasive warning about the negative impacts of perpetual use of smartphones. Her main premise is that as people—both adults and children—become increasingly glued to their phones instead of talking to one another, they lose the skill of empathy. She argues that we need to reclaim conversation to relearn empathy, friendship, and creativity.
F U R T H E R R E A D I N g
Chapter 6
E M O T I O N A L I N T E R A C T I O N
6.1 Introduction
6.2 Emotions and the User Experience
6.3 Expressive Interfaces and Emotional Design
6.4 Annoying Interfaces
6.5 Affective Computing and Emotional AI
6.6 Persuasive Technologies and Behavioral Change
6.7 Anthropomorphism
ObjectivesThe main goals of this chapter are to accomplish the following:
• Explain how our emotions relate to behavior and the user experience. • Explain what are expressive and annoying interfaces and the effects they can have on people.
• Introduce the area of emotion recognition and how it is used. • Describe how technologies can be designed to change people’s behavior. • Provide an overview on how anthropomorphism has been applied in interaction design.
6.1 Introduction
When you receive some bad news, how does it affect you? Do you feel upset, sad, angry, or annoyed—or all of these? Does it put you in a bad mood for the rest of the day? How might technology help? Imagine a wearable technology that could detect how you were feeling and provide a certain kind of information and suggestions geared toward helping to improve your mood, especially if it detected that you were having a real downer of a day. Would you find such a device helpful, or would you find it unnerving that a machine was trying to cheer you up? Designing technology to detect and recognize someone’s emotions automatically from sensing aspects of their facial expressions, body movements, gestures, and so forth,
6 E M O T I O N A L I N T E R A C T I O N166
is a growing area of research often called emotional AI or affective computing. There are many potential applications for using automatic emotion sensing, other than those intended to cheer someone up, including health, retail, driving, and education. These can be used to determine if someone is happy, angry, bored, frustrated, and so on, in order to trigger an appropriate technology intervention, such as making a suggestion to them to stop and reflect or recommending a particular activity for them to do.
In addition, emotional design is a growing area relating to the design of technology that can engender desired emotional states, for example, apps that enable people to reflect on their emotions, moods, and feelings. The focus is on how to design interactive prod-ucts to evoke certain kinds of emotional responses in people. It also examines why people become emotionally attached to certain products (for instance, virtual pets), how social robots might help reduce loneliness, and how to change human behavior through the use of emotive feedback.
In this chapter, we include emotional design and affective computing using the broader term, emotional interaction, to cover both aspects. We begin by explaining what emotions are and how they shape behavior and everyday experiences. We then consider how and whether an interface’s appearance affects usability and the user experience. In particu-lar, we look at how expressive and persuasive interfaces can change people’s emotions or behaviors. How technology can detect human emotions using voice and facial recognition is then covered. Finally, the way anthropomorphism has been used in interaction design is discussed.
6.2 Emotions and the User Experience
Consider the different emotions one experiences throughout a common everyday activity—shopping online for a product, such as a new laptop, a sofa, or a vacation. First, there is the realization of needing or wanting one and then the desire and anticipation of purchasing it. This is followed by the joy or frustration of finding out more about what products are avail-able and deciding which to choose from potentially hundreds or even thousands of them by visiting numerous websites, such as comparison sites, reviews, recommendations, and social media sites. This entails matching what is available with what you like or need and whether you can afford it. The thrill of deciding on a purchase may be quickly followed by the shock of how much it costs and the disappointment that it is too expensive. The process of having to revise your decision may be accompanied by annoyance if you discover that nothing is as good as the first choice. It can become frustrating to keep looking and revisiting sites. Finally, when you make your decision, a sense of relief is often experienced. Then there is the process of clicking through the various options (such as color, size, warranty, and so forth) until the online payment form pops up. This can be tedious, and the requirement to fill in the many details raises the possibility of making a mistake. Finally, when the order is complete, you can let out a big sigh. However, doubts can start to creep in—maybe the other one was better after all… .
This rollercoaster set of emotions is what many of us experience when shopping online, especially for big-ticket items where there is a myriad of options from which to choose and where you want to be sure that you make the right choice.
6 . 2 E M O T I O N s A N d T h E U s E R E x p E R I E N C E 167
Emotional interaction is concerned with what makes people feel happy, sad, annoyed, anxious, frustrated, motivated, delirious, and so on, and then using this knowledge to inform the design of different aspects of the user experience. However, it is not straightforward. Should an interface be designed to try to keep a person happy when it detects that they are smiling, or should it try to change them from being in a negative mood to a positive one when it detects that they are scowling? Having detected an emotional state, a decision has to
ACTIVITY 6.1 Have you seen one of the terminals shown in Figure 6.1 at an airport after you have gone through security? Were you drawn toward it, and did you respond? If so, which smiley button did you press?
CommentThe act of pressing one of the buttons can be very satisfying—providing a moment for you to reflect upon your experience. It can even be pleasurable to express how you feel in this physi-cal manner. Happyornot designed the feedback terminals that are now used in many airports throughout the world. The affordances of the large, colorful, slightly raised buttons laid out in a semicircle, with distinct smileys, makes it easy to know what is being asked of the passerby, enabling them to select among feeling happy, angry, or something in between.
The data collected from the button presses provides statistics for an airport as to when and where people are happiest and angriest after going through security. Happyornot has found that it also makes travelers feel valued. The happiest times to travel, from the data they have collected at various airports, are at 8 a.m. and 9 a.m. The unhappiest times recorded are in the early hours of the morning, presumably because people are tired and grumpier.
Figure 6.1 A Happyornot terminal located after security at Heathrow Airport Source: https://www.rsrresearch.com/research/why-metrics-matter. Used courtesy of Retail Systems Research
6 E M O T I O N A L I N T E R A C T I O N168
be made as to what or how to present information to the user. Should it try to “smile” back through using various interface elements, such as emojis, feedback, and icons? How expres-sive should it be? It depends on whether a given emotional state is viewed as desirable for the user experience or the task at hand. A happy state of mind might be considered optimal for when someone goes to shop online if it is assumed that this will make them more willing to make a purchase.
Advertising agencies have developed a number of techniques to influence people’s emo-tions. Examples include showing a picture of a cute animal or a child with hungry, big eyes on a website that “pulls at the heartstrings.” The goal is to make people feel sad or upset at what they observe and make them want to do something to help, such as by making a dona-tion. Figure 6.2, for example, shows a web page that has been designed to trigger a strong emotional response in the viewer.
Our moods and feelings are also continuously changing, making it more difficult to predict how we feel at different times. Sometimes, an emotion can descend upon us but disappear shortly afterward. For example, we can become startled by a sudden, unexpected loud noise. At other times, an emotion can stay with us for a long time; for example, we can remain annoyed for hours when staying in a hotel room that has a noisy air conditioning unit. An emotion like jealousy can keep simmering for a long period of time, manifesting itself on seeing or hearing something about the person or thing that triggered it.
Figure 6.2 A webpage from Crisis (a UK homelessness charity)Source: https://www.crisis.org.uk
In a series of short videos, Kia Höök talks about affective computing, explaining how emotion is formed and why it is important to consider when designing user experiences with technology. See www.interaction-design.org/encyclopedia/affective_computing.html.
6 . 2 E M O T I O N s A N d T h E U s E R E x p E R I E N C E 169
A good place to start understanding how emotions affect behavior and how behavior affects emotions is to examine how people express themselves and read each other’s expres-sions. This includes understanding the relationship between facial expressions, body lan-guage, gestures, and tone of voice. For example, when people are happy, they typically smile, laugh, and relax their body posture. When they are angry, they might shout, gesticulate, tense their hands, and screw up their face. A person’s expressions can trigger emotional responses in others. When someone smiles, it can cause others to feel good and smile back.
Emotional skills, especially the ability to express and recognize emotions, are central to human communication. Most people are highly skilled at detecting when someone is angry, happy, sad, or bored by recognizing their facial expressions, way of speaking, and other body signals. They also usually know what emotions to express in a given situation. For example, when someone has just heard they have failed an exam, it is not a good time to smile and be happy for them. Instead, people try to empathize and show that they feel sad, too.
There is an ongoing debate about whether and how emotion causes certain behaviors. For example, does being angry make us concentrate better? Or does being happy make us take more risks, such as spending too much money or vice versa or neither? It could be that we can just feel happy, sad, or angry, and that this does not affect our behavior. Roy Baumeister et al. (2007) argue that the role of emotion is more complicated than a simple cause-and-effect model.
Many theorists, however, argue that emotions cause behavior, for example that fear brings about flight and that anger initiates the fight perspective. A widely accepted expla-nation, derived from evolutionary psychology, is that when something makes someone frightened or angry, their emotional response is to focus on the problem at hand and try to overcome or resolve the perceived danger. The physiological responses that accompany this state usually include a rush of adrenalin through the body and the tensing of muscles. While the physiological changes prepare people to fight or flee, they also give rise to unpleasant experiences, such as sweating, butterflies in the stomach, quick breathing, heart pounding, and even feelings of nausea.
Nervousness is a state of being that is often accompanied by several emotions, includ-ing apprehension and fear. For example, many people get worried and some feel terrified before speaking at a public event or a live performance. There is even a name for this kind of nervousness—stage fright. Andreas Komninos (2017) suggests that it is the autonomous system “telling” people to avoid these kinds of potentially humiliating or embarrassing expe-riences. But performers or professors can’t simply run away. They have to cope with the negative emotions associated with having to be in front of an audience. Some are able to turn their nervous state to their advantage, using the increase in adrenalin to help them focus on their performance. Others are only too glad when the performance is over and they can relax again.
As mentioned earlier, emotions can be simple and short-lived or complex and long-lasting. To distinguish between the two types of emotion, researchers have described them in terms of being either automatic or conscious. Automatic emotions (also knowns as affect) happen rapidly, typically within a fraction of a second and, likewise, may dissipate just as quickly. Conscious emotions, on the other hand, tend to be slow to develop and equally slow to dis-sipate, and they are often the result of a conscious cognitive behavior, such as weighing the odds, reflection, or contemplation.
6 E M O T I O N A L I N T E R A C T I O N170
Understanding how emotions work provides a way of considering how to design for user experiences that can trigger affect or reflection. For example, Don Norman (2005) suggests that being in a positive state of mind can enable people to be more creative as they are less focused. When someone is in a good mood, it is thought to help them make decisions more quickly. He also suggests that when people are happy, they are more likely to overlook and cope with minor problems that they are experiencing with a device or interface. In contrast, when someone is anxious or angry, they are more likely to be less tolerant. He also suggests that designers pay special attention to the information required to do the task at hand, but especially in the case when designing apps or devices for serious tasks, such as monitor-ing a process control plant or driving a car. The interface needs to be clearly visible with
BOx 6.1 How Does Emotion Affect Driving Behavior?
Research investigating the influence of emotions on driving behavior has been extensively reviewed (Pêcher et al., 2011; Zhang and Chan, 2016). One major finding is that when driv-ers are angry, their driving becomes more aggressive, they take more risks such as dangerous overtaking, and they are prone to making more errors. Driving performance has also been found to be negatively affected when drivers are anxious. People who are depressed are also more prone to accidents.
What are the effects of listening to music while driving? A study by Christelle Pêcher et al. (2009) found that people slowed down while driving in a car simulator when they listened to either happy or sad music, as compared to neutral music. This effect is thought to be due to the drivers focusing their attention on the emotions and lyrics of the music. Listening to happy music was also found not only to slow drivers down, but to distract them more by reducing their ability to stay in their lane. This did not happen with the sad music.
Source: Jonny Hawkins / Cartoon Stock
6 . 2 E M O T I O N s A N d T h E U s E R E x p E R I E N C E 171
unambiguous feedback. The bottom line is “things intended to be used under stressful situ-ations require a lot more care, with much more attention to detail” (Norman, 2005, p. 26).
Don Norman and his colleagues (Ortony et al., 2005) have also developed a model of emotion and behavior. It is couched in terms of different “levels” of the brain. At the lowest level are parts of the brain that are prewired to respond automatically to events happening in the physical world. This is called the visceral level. At the next level are the brain processes that control everyday behavior. This is called the behavioral level. At the highest level are brain processes involved in contemplating. This is called the reflective level (see Figure 6.3). The visceral level responds rapidly, making judgments about what is good or bad, safe or dangerous, pleasurable or abhorrent. It also triggers the emotional responses to stimuli (for instance fear, joy, anger, and sadness) that are expressed through a combination of physi-ological and behavioral responses. For example, many people will experience fear on seeing a very large hairy spider running across the floor of the bathroom, causing them to scream and run away. The behavioral level is where most human activities occur. Examples include well-learned routine operations such as talking, typing, and swimming. The reflective level entails conscious thought where people generalize across events or step back from their daily routines. An example is switching between thinking about the narrative structure and spe-cial effects used in a horror movie and becoming scared at the visceral level when watching the movie.
One way of using the model is to think about how to design products in terms of the three levels. Visceral design refers to making products look, feel, and sound good. Behavio-ral design is about use and equates to the traditional values of usability. Reflective design is about considering the meaning and personal value of a product in a particular culture. For example, the design of a Swatch watch (see Figure 6.4) can be viewed in terms of the three levels. The use of cultural images and graphical elements is designed to appeal to users
Sensory Motor
Control
Reflective
Behavioral
Visceral
Control
Figure 6.3 Anthony Ortony et al.’s (2005) model of emotional design showing three levels: visceral, behavioral, and reflective Source: Adapted from Norman (2005), Figure 1.1
6 E M O T I O N A L I N T E R A C T I O N172
at the reflective level; its affordances of use at the behavioral level, and the brilliant colors, wild designs, and art attract users’ attention at the visceral level. They are combined to create the distinctive Swatch trademark, and they are what draw people to buy and wear their watches.
6.3 Expressive Interfaces and Emotional Design
Designers use a number of features to make an interface expressive. Emojis, sounds, colors, shapes, icons, and virtual agents are used to (1) create an emotional connection or feel-ing with the user (for instance, warmth or sadness) and/or (2) elicit certain kinds of emo-tional responses in users, such as feeling at ease, comfort, and happiness. In the early days, emotional icons were used to indicate the current state of a computer or a phone, notably when it was waking up or being rebooted. A classic from the 1980s was the happy Mac icon that appeared on the screen of the Apple computer whenever the machine was booted (see Figure 6.5a). The smiling icon conveyed a sense of friendliness, inviting the user to feel at ease and even smile back. The appearance of the icon on the screen was also meant to be
Figure 6.4 A Swatch watch called Dip in Color Source: http://store.swatch.com/suop103-dip-in-color.html
6 . 3 E x p R E s s I V E I N T E R F A C E s A N d E M O T I O N A L d E s I g N 173
reassuring, indicating that the computer was working. After being in use for nearly 20 years, the happy and sad Mac icons were laid to rest. Apple now uses more impersonal but aestheti-cally pleasing forms of feedback to indicate a process for which the user needs to wait, such as “starting up,” “busy,” “not working,” or “downloading.” These include a spinning colorful beach ball (see Figure 6.5b) and a moving clock indicator. Similarly, Android uses a spinning circle to show when a process is loading.
Other ways of conveying expressivity include the following:
• Animated icons (for example, a recycle bin expanding when a file is placed in it and paper disappearing in a puff of smoke when emptied)
• Sonifications indicating actions and events (such as whoosh for a window closing, “schlook” for a file being dragged, or ding for a new email arriving)
• Vibrotactile feedback, such as distinct smartphone buzzes that represent specific messages from friends or family
The style or brand conveyed by an interface, in terms of the shapes, fonts, colors, and graphical elements used, and the way they are combined, also influence its emotional impact. Use of imagery at the interface can result in more engaging and enjoyable experiences (Mullet and Sano, 1995). A designer can also use a number of aesthetic techniques such as clean lines, balance, simplicity, white space, and texture.
The benefits of having aesthetically pleasing interfaces have become more acknowl-edged in interaction design. Noam Tractinsky (2013) has repeatedly shown how the aesthet-ics of an interface can have a positive effect on people’s perception of the system’s usability. When the look and feel of an interface is pleasing and pleasurable—for example through beautiful graphics or a nice feel or the way that the elements have been put together—peo-ple are likely to be more tolerant and prepared to wait a few more seconds for a website to download. Furthermore, good-looking interfaces are generally more satisfying and pleasur-able to use.
(a) (b)
Figure 6.5 (a) Smiling and sad Apple icons depicted on the classic Mac and (b) the spinning beach ball shown when an app freezes
Source: (b) https://www.macobserver.com/tmo/article/frozen-how-to-force-quit-an-os-x-app-showing-a- spinningbeachball-of-death
6 E M O T I O N A L I N T E R A C T I O N174
6.4 Annoying Interfaces
In many situations, interfaces may inadvertently elicit negative emotional responses, such as anger. This typically happens when something that should be simple to use or set turns out to be complex. The most common examples are remote controls, printers, digital alarm clocks, and digital TV systems. Getting a printer to work with a new digital camera, trying to switch from watching a DVD to a TV channel, and changing the time on a digital alarm clock in
For more information about the design of other Nest products, see https://www.wired.com/story/inside-the-second-coming-of-nest/.
BOx 6.2 The Design of the Nest Thermostat Interface
The popular Nest thermostat provides an automatic way of controlling home heating that is personalized to the habits and needs of the occupants. Where possible, it also works out how to save money by reducing energy consumption when not needed. The wall-mounted device does this by learning what temperature the occupants prefer and when to turn the heating on and off in each room by learning their routines.
The Nest thermostat is more than just a smart meter, however. It was also designed to have a minimalist and aesthetically pleasing interface (see Figure 6.6a). It elegantly shows the tempera-ture currently on its round face and to which temperature it has been set. This is very different from earlier generations of automatic thermostats, which were utilitarian box-shaped designs with lots of complicated buttons and a dull screen that provided feedback about the setting and temperature (see Figure 6.6b). It is little wonder that the Nest thermostat has been a success.
(a) (b)
Figure 6.6 (a) The Nest thermostat and (b) A traditional thermostat Source: Nest
6 . 4 A N N O Y I N g I N T E R F A C E s 175
a hotel can be very trying. Also, complex actions such as attaching the ends of cables between smartphones and laptops, or inserting a SIM card into a smartphone, can be irksome, espe-cially if it is not easy to see which way is correct to insert them.
This does not mean that developers are unaware of such usability problems. Several methods have been devised to help the novice user get set up and become familiarized with a technology. These methods include pop-up help boxes and contextual videos. Another approach to helping users has been to make an interface appear friendlier as a way of reas-suring users—especially those who were new to computers or online banking. One technique that was first popularized in the 1990s was the use of cartoon-like companions. The assump-tion was that novices would feel more at ease with a “friend” appearing on the screen and would be encouraged to try things out after listening, watching, following, and interacting with it. For example, Microsoft pioneered a class of agent-based software, Bob, aimed at new computer users (many of whom were viewed as computer-phobic). The agents were pre-sented as friendly characters, including a pet dog and a cute bunny. An interface metaphor of a warm, cozy living room, replete with fire and furniture, was also provided (see Figure 6.7), again intended to convey a comfortable feeling. However, Bob never became a commercial product. Why do you think that was?
Contrary to the designers’ expectations, many people did not like the idea of Bob, finding the interface too cute and childish. However, Microsoft did not give up on the idea of making its interfaces friendlier and developed other kinds of agents, including the infamous Clippy (a paper clip that had human-like qualities), as part of their Windows 98 operating environment. Clippy typically appeared at the bottom of a user’s screen whenever the system thought the user needed help carrying out a particular task (see Figure 6.8a). It, too, was depicted as a cartoon character, with a warm personality. This time, Clippy was released as a commercial product, but it was not a success. Many Microsoft users found it too intrusive, distracting them from their work.
Figure 6.7 “At home with Bob” software developed for Windows 95Source: Microsoft Corporation
6 E M O T I O N A L I N T E R A C T I O N176
A number of online stores and travel agencies also began including automated virtual agents in the form of cartoon characters who acted as sales agents on their websites. The agents appeared above or next to a textbox where the user could type in their query. To make them appear as if they were listening to the user, they were animated in a semi human-like way. An example of this was Anna from IKEA (see Figure 6.8b) who occasionally nodded, blinked her eyes, and opened her mouth. These virtual agents, however, have now largely disappeared from our screens, being replaced by virtual assistants who talk in speech bubbles that have no physical appearance, or static images of real agents who the user can talk to via LiveChat.
Interfaces, if designed poorly, can make people sometimes feel insulted, stupid, or threat-ened. The effect can be to annoy them to the point of losing their temper. There are many situations that cause such negative emotional responses. These include the following:
• When an application doesn’t work properly or crashes • When a system doesn’t do what the user wants it to do • When a user’s expectations are not met • When a system does not provide sufficient information to let the user know what to do • When error messages pop up that are vague or obtuse • When the appearance of an interface is too noisy, garish, gimmicky, or patronizing
(a) (b)
Figure 6.8 Defunct virtual agents: (a) Microsoft’s Clippy and (b) IKEA’s Anna Source: Microsoft Corporation
6 . 4 A N N O Y I N g I N T E R F A C E s 177
• When a system requires users to carry out too many steps to perform a task, only to dis-cover a mistake was made somewhere along the line and they need to start all over again
• Websites that are overloaded with text and graphics, making it difficult to locate desired information and resulting in sluggish performance
• Flashing animations, especially flashing banner ads and pop-up ads that cover the user view and which require them to click in order to close them
• The overuse or automatic playing of sound effects and music, especially when selecting options, carrying out actions, running tutorials, or watching website demos
• Featuritis—an excessive number of operations, such as an array of buttons on remote controls • Poorly laid-out keyboards, touchpads, control panels, and other input devices that cause users to press the wrong keys or buttons persistently
ACTIVITY 6.2 Most people are familiar with the “404 error” message that pops up now and again when a web page does not upload for the link they have clicked or when they have typed or pasted an incorrect URL into a browser. What does it mean and why the number 404? Is there a better way of letting users know when a link to a website is not working? Might it be better for the web browser to say that it was sorry rather than presenting an error message?
CommentThe number 404 comes from the HTML language. The first 4 indicates a client error. The server is telling the user that they have done something wrong, such as misspelling the URL or requesting a page that no longer exists. The middle 0 refers to a general syntax error, such as a spelling mistake. The last 4 indicates the specific nature of the error. For the user, however, it is an arbitrary number. It might even suggest that there are 403 other errors they could make!
Early research by Byron Reeves and Clifford Nass (1996) suggested that computers should be courteous to users in the same way that people are to one another. They found that people are more forgiving and understanding when a computer says that it’s sorry after mak-ing a mistake. A number of companies now provide alternative and more humorous “error” landing pages that are intended to make light of the embarrassing situation and to take the blame away from the user (see Figure 6.9).
(Continued)
6 E M O T I O N A L I N T E R A C T I O N178
Figure 6.9 An alternative 404 error message Source: https://www.creativebloq.com/web-design/best-404-pages-812505
dILEMMAShould Voice Assistants Teach Kids Good Manners?
Many families now own a smart speaker, such as an Amazon Echo, with a voice assistant like Alexa running on it. One observation is that young children will often talk to Alexa as if she was their friend, asking her all sorts of personal questions, such as “Are you my friend?” and “What is your favorite music?” and “What is your middle name?” They also learn that is not necessary to say “please” when asking their questions or “thank you” on receiving a response, similar to how they talk to other display-based voice assistants, such as Siri or Cortana. Some parents, however, are worried that this lack of etiquette could develop into a new social norm that could transfer over to how they talk to real human beings. Imagine the scenario where Aunt Emma and Uncle Liam come over to visit their young niece for her 5th birthday, and the first thing that they hear is, “Aunty Emma, get me my drink” or “Uncle Liam, where is my birthday present?” with nary a “please” uttered. How would you feel if you were treated like that?
One would hope that parents would continue to teach their children good manners and the difference between a real human and a voice assistant. However, it is also possible to configure Alexa and other voice assistants to reward children when they are polite to them,
6 . 5 A F F E C T I V E C O M p U T I N g A N d E M O T I O N A L A I 179
6.5 Affective Computing and Emotional AI
Affective computing is concerned with how to use computers to recognize and express emotions in the same way as humans do (Picard, 1998). It involves designing ways for people to communicate their emotional states, through using novel, wearable sensors and creating new techniques to evaluate frustration, stress, and moods by analyzing people’s expressions and conversations. It also explores how affect influences personal health (Jacques et al., 2017). More recently, emotional AI has emerged as a research area that seeks to automate the measurement of feelings and behaviors by using AI technologies that can analyze facial expressions and voice in order to infer emotions. A number of sens-ing technologies can be used to achieve this and, from the data collected, predict aspects of a user’s behavior, for example, forecasting what someone is most likely to buy online when feeling sad, bored, or happy. The main techniques and technologies that have been used to do this are as follows:
• Cameras for measuring facial expressions • Biosensors placed on fingers or palms to measure galvanic skin response (which is used to infer how anxious or nervous someone is as indicated by an increase in their sweat)
• Affective expression in speech (voice quality, intonation, pitch, loudness, and rhythm) • Body movement and gestures, as detected by motion capture systems or accelerometer sen-sors placed on various parts of the body
The use of automated facial coding is gaining popularity in commercial settings, espe-cially in marketing and e-commerce. For example, Affdex emotion analytics software from Affectiva (www.affectiva.com) employs advanced computer vision and machine learning algorithms to catalog a user’s emotional reactions to digital content, as captured through a webcam, to analyze how engaged the user is with digital online content, such as movies, online shopping sites, and advertisements.
Six fundamental emotions are classified based on the facial expressions that Aff-dex collects.
for example, by saying “By the way, thanks for asking so nicely.” Voice assistants could also be programmed to be much more forceful in how they teach good manners, for example, saying, “I won’t answer you unless you say ‘please’ each time you ask me a question.” Would this be taking the role of parenting too far? Mike Elgon (2018) cogently argues why voice assistants should not do this. He questions whether by extending human social norms to voice assistants, we are teaching children that technology can have sensibilities and hence should be thought about in the same way that we consider human feelings. In particular, he wonders whether by being polite to a voice assistant, children might begin to think that they are capa-ble of feeling appreciated or unappreciated and that they have rights just like humans. Do you agree with him, or do you think that there is no harm in developing virtual assistants to teach children good manners and that children will learn? Or, do you believe that children will instinctively know voice assistants don’t have rights or feelings?
6 E M O T I O N A L I N T E R A C T I O N180
• Anger • Contempt • Disgust • Fear • Joy • Sadness
These emotions are indicated as a percentage of what was detected beside the emotion labels above the person’s face appearing on a display. For example, Figure 6.10 shows a label of 100 percent happiness and 0 percent for all the other categories above the woman’s head on the smartphone display. The white dots overlaying her face are the markers used by the app when modeling a face. They provide the data that determines the type of facial expres-sion being shown, in terms of detecting the presence or absence of the following:
• Smiling • Eye widening • Brow raising • Brow furrowing • Raising a cheek • Mouth opening • Upper-lip raising • Wrinkling of the nose
If a user screws up their face when an ad pops up, this suggests that they feel disgust, whereas if they start smiling, it suggests that they are feeling happy. The website can then adapt its ad, movie storyline, or content to what it perceives the person needs at that point in their emotional state.
Figure 6.10 Facial coding using Affdex softwareSource: Affectiva, Inc.
6 . 5 A F F E C T I V E C O M p U T I N g A N d E M O T I O N A L A I 181
Affectiva has also started to analyze drivers’ facial expressions when on the road with the goal of improving driver safety. The emotional AI software perceives if a driver is angry and then suggests an intervention. For example, a virtual agent in the car might suggest to the driver to take a deep breath and play soothing music to help relax them. In addition to identifying particular emotions through facial expressions (for example, joy, anger, and sur-prise), Affectiva uses particular markers to detect drowsiness. These are eye closure, yawning, and blinking rate. Again, upon detecting when a threshold has been reached for these facial expressions, the software might trigger an action, such as getting a virtual agent to suggest to the driver that they pull over where it is safe to do so.
Other indirect methods that are used to reveal the emotional state of someone include eye-tracking, finger pulse, speech, and the words/phrases they use when tweeting, chatting online, or posting to Facebook (van den Broek, 2013). The level of affect expressed by users, the language they use, and the frequency with which they express themselves when using social media can all indicate their mental state, well-being, and aspects of their personality (for instance, whether they are an extrovert or introvert, neurotic or calm, and so on). Some companies may try to use a combination of these measures, such as facial expressions and the language that people use when online, while others may focus on just one aspect, such as the tone of their voice when answering questions over the phone. This type of indirect emotion detection is beginning to be used to help infer or predict someone’s behavior, for example, determining their suitability for a job or how they will vote in an election.
Another application of biometric data is being used in streaming video games where spectators watch players, known as streamers, play video games. The most popular site is Twitch; millions of viewers visit it each day to watch others compete in games, such as Fort-nite. The biggest streamers have become a new breed of celebrity, like YouTubers. Some even have millions of dedicated fans. Various tools have been developed to enhance the viewers’ experience. One is called All the Feels, which provides an overlay of biometric and webcam-derived data of a streamer onto the screen interface (Robinson et al., 2017). A dashboard provides a visualization of the streamer’s heart rate, skin conductance, and emotions. This additional layer of data has been found to enhance the spectator experience and improve the connection between the streamer and spectators. Figure 6.11 shows the emotional state of a streamer using the All the Feels interface.
Figure 6.11 All the Feels app showing the biometric data of a streamer playing a videogame Source: Used courtesy of Katherine Isbister
6 E M O T I O N A L I N T E R A C T I O N182
6.6 Persuasive Technologies and Behavioral Change
A diversity of techniques has been used at the interface level to draw people’s attention to certain kinds of information in an attempt to change what they do or think. Pop-up ads, warning messages, reminders, prompts, personalized messages, and recommendations are some of the methods that are being deployed on a computer or smartphone interface. Examples include Amazon’s one-click mechanism that makes it easy to buy something on its online store and recommender systems that suggest specific books, hotels, restaurants, and so forth, that a reader might want to try based on their previous purchases, choices, and taste. The various techniques that have been developed have been referred to as persuasive design (Fogg, 2009). They include enticing, cajoling, or nudging someone into doing something through the use of persuasive technology.
Technology interventions have also been developed to change people’s behaviors in other domains besides commerce, including safety, preventative healthcare, fitness, personal rela-tionships, energy consumption, and learning. Here the emphasis is on changing someone’s habits or doing something that will improve an individual’s well-being through monitoring their behavior. An early example was Nintendo’s Pokémon Pikachu device (see Figure 6.12) that was designed to motivate children into being more physically active on a consistent basis. The owner of the digital pet that lives in the device was required to walk, run, or jump each day to keep it alive. The wearer received credits for each step taken—the currency being watts that could be used to buy Pikachu presents. Twenty steps on the pedometer rewarded the player with 1 watt. If the owner did not exercise for a week, the virtual pet became angry and refused to play anymore. This use of positive rewarding and sulking can be a powerful means of persuasion, given that children often become emotionally attached to their virtual pets, especially when they start to care for them.
BOx 6.3 Is It OK for Technology to Work Out How You Are Feeling?
Do you think it is ethical that technology is trying to read your emotions from your facial expressions or from what you write in your tweets and, based on its analysis, filter the online content that you are browsing, such as ads, news, or a movie to match your mood? Might some people think it is an invasion of their privacy?
Human beings will suggest things to each other, often based on what they think the other is feeling. For example, they might suggest a walk in the park to cheer them up. They might also suggest a book to read or a movie to watch. However, some people may not like the idea that an app can do the same, for example, suggesting what you should eat, watch, or do based on how it analyzes your facial expressions.
6 . 6 p E R s U A s I V E T E C h N O L O g I E s A N d B E h A V I O R A L C h A N g E 183
HAPIfork is a device that was developed to help someone monitor and track their eating habits (see Figure 6.13). If it detects that they are eating too quickly, it will vibrate (similar to the way a smartphone does when on silent mode), and an ambient light will appear at the end of the fork, providing the eater with real-time feedback intended to slow them down. The assumption is that eating too fast results in poor digestion and poor weight control and that making people aware that they are gobbling their food down can help them think about
Figure 6.12 Nintendo’s Pokémon Pikachu device Source: http://nintendo.wikia.com/wiki/File:Pok%C3%A9mon_Pikachu_2_GS_(Device).png
ACTIVITY 6.3 Watch these two videos:
The Piano Staircase: http://youtu.be/2lXh2n0aPywThe Outdoor Bin: http://youtu.be/cbEKAwCoCKwDo you think that such playful methods are effective at changing people’s behavior?
CommentVolkswagen sponsored an open competition, called the fun theory, asking people to transform mundane artifacts into novel enjoyable user experiences in an attempt to change people’s behavior for the better. The idea was to encourage a desired behavior by making it more fun. The Piano Staircase and the Outdoor Bin are the most well-known examples; the stairs sounded like piano keys being played as they were climbed, while the bin sounded like a well echoing when something was thrown into it. Research has shown that using these kinds of playful methods is very engaging, and they can help people overcome their social inhibition of taking part in an activity in a public place (Rogers et al., 2010a).
6 E M O T I O N A L I N T E R A C T I O N184
how to eat more slowly at a conscious level. Other data is collected about how long it took them to finish their meal, the number of fork servings per minute, and the time between them. These are turned into a dashboard of graphs and statistics so that the user can see each week whether their fork behavior is improving.
Nowadays, there are many kinds of mobile apps and personal tracking devices available that are intended to help people monitor various behaviors and change them based on the data collected and displayed back to them. These devices include fitness trackers, for exam-ple, Fitbit, and weight trackers, such as smart scales. Similar to HAPIfork, these devices are designed to encourage people to change their behavior by displaying dashboards of graphs showing how much exercise they have done or weight they have lost over a day, week, or longer period, compared with what they have done in the previous day, week, or month. These results can also be compared, through online leaderboards and charts, with how well they have done versus their peers and friends. Other techniques employed to encourage peo-ple to exercise more or to move when sedentary include goal setting, reminders, and rewards for good behavior. A survey of how people use such devices in their everyday lives revealed that people often bought them simply to try them or were given one as a present, rather than specifically trying to change a particular behavior (Rooksby et al., 2014). How, what, and when they tracked depended on their interests and lifestyles; some used them as a way of showing how fast they could run during a marathon or cycle on a course or how they could change their lifestyle to sleep or eat better.
An alternative approach to collecting quantified data about a behavior automatically is to ask people to write down manually how they are feeling now or to rate their mood and for them to reflect upon how they felt about themselves in the past. A mobile app called Echo, for example, asked people to write a subject line, rate their happiness at that moment, and add a description, photos, and/or videos if they wanted to (Isaacs et al., 2013). Sporadically, the app then asked them to reflect on previous entries. An assumption was that this type
Figure 6.13 Someone using the HAPIfork in a restaurantSource: Helen Sharp
6 . 6 p E R s U A s I V E T E C h N O L O g I E s A N d B E h A V I O R A L C h A N g E 185
of technology-mediated reflection could increase well-being and happiness. Each reflection was shown as a stacked card with the time and a smiley happiness rating. People who used the Echo app reported on the many positive effects of doing so, including reliving positive experiences and overcoming negative experiences by writing them down. The double act of recording and reflecting enabled them to generalize from the positive experiences and draw positive lessons from them.
The global concern about climate change has also led a number of HCI researchers to design and evaluate various energy-sensing devices that display real-time feedback. One goal is to find ways of helping people reduce their energy consumption, and it is part of a larger research agenda called sustainable HCI: see Mankoff et al., 2008; DiSalvo et al., 2010; Hazas et al., 2012. The focus is to persuade people to change their everyday habits with respect to environmental concerns, such as reducing their own carbon footprint, their community’s footprint (for example, a school or workplace), or an even larger organization’s carbon foot-print (such as a street, town, or country).
Extensive research has shown that domestic energy use can be reduced by providing households with feedback on their consumption (Froehlich et al., 2010). The frequency of feedback is considered important; continuous or daily feedback on energy consumption has been found to yield higher savings results than monthly feedback. The type of graphical representation also has an effect. If the image used is too obvious and explicit (for instance, a finger pointing at the user), it may be perceived as too personal, blunt, or “in your face,” resulting in people objecting to it. In contrast, simple images (for example, an infographic or emoticon) that are more anonymous but striking and whose function is to get people’s attention may be more effective. They may encourage people to reflect more on their energy use and even promote public debate about what is represented and how it affects them. However, if the image used is too abstract and implicit, other meanings may be attributed to it, such as simply being an art piece (such as an abstract painting with colored stripes that change in response to the amount of energy used), resulting in people ignoring it. The ideal may be somewhere in between. Peer pressure can also be effective, where peers, parents, or children chide or encourage one another to turn lights off, take a shower instead of a bath, and so on.
Another influencing factor is social norms. In a classic study by P. Wesley Schultz et al., (2007), households were shown how their energy consumption compared with their neigh-borhood average. Households above the average tended to decrease their consumption, but those using less electricity than average tended to increase their consumption. The study found that this “boomerang” effect could be counteracted by providing households with an emoticon along with the numerical information about their energy usage: households using less energy than average continued to do so if they received a smiley icon; households using more than average decreased their consumption even more if they were given a sad icon.
In contrast to the Schultz study, where each household’s energy consumption was kept private, the Tidy Street project (Bird and Rogers, 2010) that was run in Brighton in the United Kingdom created a large-scale visualization of the street’s electricity usage by spray-ing a stenciled display on the road surface using chalk (see Figure 6.14). The public display was updated each day to represent how the average electricity usage of the street compared to the city of Brighton’s average. The goal was to provide real-time feedback that all of the homeowner’s and the general public could see change each day over a period of three weeks. The street graph also proved to be very effective in getting people who lived on Tidy Street
6 E M O T I O N A L I N T E R A C T I O N186
to talk to each other about their electricity consumption and habits. It also encouraged them to talk with the many passersby who walked up and down the street. The outcome was to reduce electricity consumption in the street by 15 percent, which was considerably more than other projects in this area have been able to achieve.
BOx 6.4 The Darker Side: Deceptive Technology
Technology is increasingly being used to deceive people into parting with their personal details, which allows Internet fraudsters to access their bank accounts and draw money from them. Authentic-looking letters, appearing to be sent from eBay, PayPal, and various leading banks, are spammed across the world, ending up in people’s email in-boxes with messages such as “During our regular verification of accounts, we couldn’t confirm your information. Please click here to update and verify your information.” Given that many people have an account with one of these corporations, there is a good chance that they will be misled and unwittingly believe what is being asked of them, only to discover a few days later that they are several thousand dollars worse off. Similarly, letters from supposedly super-rich individuals in far-away countries, offering a share of their assets if the email recipient provides them with their bank details, have persistently been spammed worldwide. While many people are becom-ing increasingly wary of what are known as phishing scams, there are still many vulnerable individuals who are gullible to such tactics.
The term phishing is a play on the term fishing, which refers to the sophis-ticated way of luring users’ financial information and passwords. Internet fraudsters are becoming smarter and are constantly changing their tactics. While the art of deception is centuries old, the increasing, pervasive, and often ingenious use of the web to trick people into divulging personal infor-mation can have catastrophic effects on society as a whole.
Figure 6.14 Aerial view of the Tidy Street public electricity graphSource: Helen Sharp
6 . 7 A N T h R O p O M O R p h I s M 187
6.7 Anthropomorphism
Anthropomorphism is the propensity people have to attribute human qualities to animals and objects. For example, people sometimes talk to their computers as if they were humans, treat their robot cleaners as if they were their pets, and give all manner of cute names to their mobile devices, routers, and so on. Advertisers are well aware of this phenomenon and often create human-like and animal-like characters out of inanimate objects to promote their prod-ucts. For example, breakfast cereals, butter, and fruit drinks have all been transmogrified into characters with human qualities (they move, talk, have personalities, and show emotions), enticing the viewer to buy them. Children are especially susceptible to this kind of magic, as witnessed by their love of cartoons where all manner of inanimate objects are brought to life with human-like qualities.
The finding that people, especially children, have a propensity to accept and enjoy objects that have been given human-like qualities has led many designers to capitalize on it, most notably in the design of virtual agents and interactive dolls, robots, and cuddly toys. Early commercial products like ActiMates were designed to encourage children to learn by playing with them. One of the first—Barney (a dinosaur)—attempted to motivate play in children by using human-based speech and movement (Strommen, 1998). The toys were programmed to react to the child and make comments while watching TV or working together on a computer-based task. In particular, Barney was programmed to congratulate the child when-ever they produced a right answer and also to react to the content on-screen with appropriate emotions, for instance, cheering at good news and expressing concern at bad news. Interac-tive dolls have also been designed to talk, sense, and understand the world around them, using sensor-based technologies, speech recognition, and various mechanical servos embed-ded in their bodies. For example, the interactive doll Luvabella exhibits facial expressions, such as blinking, smiling, and making baby cooing noises in response to how her owner plays and looks after her. The more a child plays with her, the doll will learn to speak, transforming her babble into words and phrases.
Furnishing technologies with personalities and other human-like attributes can make them more enjoyable and fun to interact with. They can also motivate people to carry out various activities, such as learning. Being addressed in the first person (for instance, “Hello, Noah! Nice to see you again. Welcome back. Now what were we doing last time? Oh yes, Exercise 5. Let’s start again.”) is more appealing than being addressed in the impersonal third person (“User 24, commence Exercise 5.”), especially for children. It can make them feel more at ease and reduce their anxiety. Similarly, interacting with screen characters like tutors and wizards can be more engaging than interacting with a dialog box.
A YouTube video (https://youtu.be/au2Vg9xRZZ0) shows Luvabella in action and asks viewers to decide whether the interactive doll is creepy or cool. What do you think?
6 E M O T I O N A L I N T E R A C T I O N188
ACTIVITY 6.4 A Robot or a Cuddly Pet?Early robot pets, such as Sony’s AIBO, were made of hard materials that made them look shiny and clunky. In contrast, a more recent trend has been to make them look and feel more like real pets by covering them up in fur and making them behave in more cute, pet-like ways. Two con-trasting examples are presented in Figure 6.15a and 6.15b. Which do you prefer and why?
CommentMost people like stroking pets, so they may prefer a soft pet robot that they can also stroke, such as the one shown in Figure 6.15b. A motivation for making robot pets cuddly is to enhance the emotional experience people receive through using their sense of touch. For example, the Haptic Creature on the right is a robot that mimics a pet that might sit in your lap, such as a cat or a rabbit (Yohanan and MacLean, 2008). It is made up of a body, head, and two ears, as well as mechanisms that simulate breathing, a vibrating purr, and the warmth of a living creature. The robot “detects” the way it is touched by means of an array of (roughly 60) touch sensors laid out across its entire body and an accelerometer. When the Haptic Creature is stroked, it responds accordingly, using the ears, breathing, and purring to communicate its emotional state through touch. On the other hand, the sensors are also used by the robot to detect the human’s emotional state through touch. Note how the robot has no eyes, nose, or mouth. Facial expressions are the most common way humans communicate emotional states. Since the Haptic Creature communicates and senses emotional states solely through touch, the face was deliberately left off to prevent people from trying to “read” emo-tion from it.
(a) (b)
Figure 6.15 Robot pets: (a) Aibo and (b) The Haptic Creature Source: (a) Jennifer Preece, (b) Used courtesy of Steve Yohanan. Photo by Martin Dee
6 . 7 A N T h R O p O M O R p h I s M 189
A number of commercial physical robots have been developed specifically to support care giving for the elderly. Early ones were designed to be about 2 feet tall and were made from white plastic with colored parts that represented clothing or hair. An example was Zora (see Figure 6.16), developed in Belgium, that was marketed as a social robot for healthcare. One was bought by a nursing home in France. Many of the patients developed an emotional attachment to their Zora robot, holding it, cooing, and even giving it kisses on the head. However, some people found this kind of robot care a little demeaning. Certainly, it can never match the human touch and warmth that patients need, but there is no harm in it playing an entertaining and motivating role alongside human caregivers.
This video demonstrates how the Zora robot was used to entertain seniors and to help them get some exercise: https://youtu.be/jcMNY5EnQNQ.
Figure 6.16 The Zora robot Source: http://zorarobotics.be/
In-depth ActivityThis in-depth activity requires you to try one of the emotion recognition apps available and to see how well it fares in recognizing different people’s facial expressions. Download the AffdexMe app or Age Emotion Detector for Apple or Android. Take a photo of yourself looking natural and see what emotion it suggests.
(Continued)
6 E M O T I O N A L I N T E R A C T I O N190
1. How many emotions does it recognize?2. Try to make a face for each of the following: sadness, anger, joy, fear, disgust, and surprise.
After making a face for each, see how well the app detects the emotion you were expressing.3. Ask a couple of other people to try it. See whether you can find someone with a beard and
ask them to try, too. Does facial hair make it more difficult for the app to recognize an emotion?
4. What other application areas do you think these kinds of apps could be used for besides advertising?
5. What ethical issues does facial recognition raise? Has the app provided sufficient informa-tion as to what it does with the photos taken of people’s faces?
6. How well would the recognition software work when used in a more natural setting where the user is not making a face for the camera?
summaryThis chapter described the different ways that interactive products can be designed (both delib-erately and inadvertently) to make people respond in certain ways. The extent to which users will learn, buy a product online, quit a bad habit, or chat with others depends on the believ-ability of the interface, how comfortable they feel when using a product, and/or how much they can trust it. If the interactive product is frustrating to use, annoying, or patronizing, users will easily become angry and despondent and often they stop using it. If, on the other hand, the product is pleasurable, is enjoyable to use, and makes people feel comfortable and at ease, then they will continue to use it, make a purchase, return to the website, or continue to learn.
This chapter also described various interaction mechanisms that can be used to elicit positive emotional responses in users and ways of avoiding negative ones. Further, it described how new technology has been developed to detect emotional states.
Key Points • Emotional aspects of interaction design are concerned with how to facilitate certain states (for example, pleasure) or avoid certain reactions (such as frustration) in user experiences.
• Well-designed interfaces can elicit good feelings in people. • Aesthetically pleasing interfaces can be a pleasure to use. • Expressive interfaces can provide reassuring feedback to users as well as be informative and fun.
• Badly designed interfaces often make people frustrated, annoyed, or angry. • Emotional AI and affective computing use AI and sensor technology for detecting people’s emotions by analyzing their facial expressions and conversations.
• Emotional technologies can be designed to persuade people to change their behaviors or attitudes.
• Anthropomorphism is the attribution of human qualities to objects. • Robots are being used in a variety of settings, including households and assisted-living homes.
191
Further Reading
CALVO, R.A and PETERS, D. (2014) Positive Computing. MIT. This book discusses how to design technology for well-being to make a happier and healthier world. As the title sug-gests, it is positive in its outlook. It covers the psychology of well-being, including empathy, mindfulness, joy, compassion, and altruism. It also describes the opportunities and chal-lenges facing interaction designers who want to develop technology that can improve peo-ple’s well-being.
HÖÖK, K. (2018) Designing with the Body. MIT. This book proposes that interaction design should consider the experiential, felt, and aesthetic stance that encompasses the design and use cycle. The approach suggested by the author is called soma design, where body and movements are viewed as very much part of the design process, and where a slow, thoughtful process is promoted that considers fundamental human values. It is argued that adopting this stance can yield better products and create healthier, more sustainable companies.
LEDOUX, J. E. (1998) The Emotional Brain: The Mysterious Underpinnings of Emotional Life. Simon & Schuster. This book explains what causes us to feel fear, love, hate, anger, and joy, and it explores whether we control our emotions versus them controlling us. The book also covers the origins of human emotions and explains that many evolved to enable us to survive.
McDUFF, D. & CZERWINSKI, M. (2018) Designing Emotionally Sentient Agents. Com-munications of the ACM, Vol. 61 No. 12, pages 74–83. This article provides an accessible overview of the burgeoning area of emotional agents. It presents the challenges, opportuni-ties, dilemmas, concerns, and current applications that are now being developed, including bots, robots, and agents.
NORMAN, D. (2005) Emotional Design: Why We Love (or Hate) Everyday Things. Basic Books. This book is an easy read while at the same time being thought-provoking. We get to see inside Dan Norman’s kitchen and learn about the design aesthetics of his collection of teapots. The book also includes essays on the emotional aspects of robots, computer games, and a host of other pleasurable interfaces.
WALTER, A. (2011) A Book Apart: Designing for Emotion. Zeldman, Jeffrey. This short book is targeted at web designers who want to understand how to design websites that users will enjoy and want to return to. It covers the classic literature on emotions, and it proposes practical approaches to emotional web design.
F U R T h E R R E A d I N g
Chapter 7
I N T E R F A C E S
7.1 Introduction
7.2 Interface Types
7.3 Natural User Interfaces and Beyond
7.4 Which Interface?
ObjectivesThe main goals of the chapter are to accomplish the following:
• Provide an overview of the many different kinds of interfaces. • Highlight the main design and research considerations for each of the interfaces. • Discuss what is meant by a natural user interface (NUI). • Consider which interface is best for a given application or activity.
7.1 Introduction
When considering how to solve a user problem, the default solution that many developers choose to design is an app that can run on a smartphone. Making this easier still are many easy-to-use app developer tools that can be freely downloaded. It is hardly surprising, there-fore, to see just how many apps there are in the world. In December 2018, Apple, for exam-ple, had a staggering 2 million apps in its store, many of which were games.
Despite the ubiquity of the smartphone app industry, the web continues to proliferate in offering services, content, resources, and information. A central concern is how to design them to be interoperable across different devices and browsers, which takes into account the varying form factors, size, and shape of smart watches, smartphones, laptops, smart TVs, and computer screens. Besides the app and the web, many other kinds of interfaces have been developed, including voice interfaces, touch interfaces, gesture interfaces, and multimodal interfaces.
The proliferation of technological developments has encouraged different ways of think-ing about interaction design and UX. For example, input can be via mice, touchpads, pens, remote controllers, joysticks, RFID readers, gestures, and even brain-computer interaction. Output is equally diverse, appearing in the form of graphical interfaces, speech, mixed reali-ties, augmented realities, tangible interfaces, wearable computing, and more.
7 I N T E R F A C E S194
The goal of this chapter is to give you an overview of the diversity of interfaces that can be developed for different environments, people, places, and activities. We present a catalog of 20 interface types, starting with command-based and ending with smart ones. For each interface, we present an overview and outline the key research and design considerations. Some are only briefly touched upon, while others, which are more established in interaction design, are described in greater depth.
7.2 Interface Types
Numerous adjectives have been used to describe the different types of interfaces that have been developed, including graphical, command, speech, multimodal, invisible, ambient, affec-tive, mobile, intelligent, adaptive, smart, tangible, touchless, and natural. Some of the inter-face types are primarily concerned with a function (for example, to be intelligent, to be adaptive, to be ambient, or to be smart), while others focus on the interaction style used (such as command, graphical, or multimedia), the input/output device used (for instance, pen-based, speech-based, or gesture-based), or the platform being designed for (for example, tablet, mobile, PC, or wearable). Rather than cover every possible type that has been devel-oped or described, we have chosen to select the main types of interfaces that have emerged over the past 40 years. The interface types are loosely ordered in terms of when they were developed. They are numbered to make it easier to find a particular one. (See the following list for the complete set.) It should be noted, however, that this classification is for con-venience of reference. The interface entries are not mutually exclusive since some products can appear in two or more categories. For example, a smartphone can be considered to be mobile, touch, or wearable.
The types of interfaces covered in this chapter include the following:
1. Command2. Graphical3. Multimedia4. Virtual reality5. Web6. Mobile7. Appliance8. Voice9. Pen
10. Touch11. Gesture12. Haptic13. Multimodal14. Shareable
NOTEThis chapter is not meant to be read from beginning to end; rather, it should be dipped into as needed to find out about a particular type of interface.
7 . 2 I N T E R F A C E T y p E S 195
15. Tangible16. Augmented reality17. Wearables18. Robots and drones19. Brain-computer interaction20. Smart
7.2.1 Command-Line InterfacesEarly interfaces required the user to type in commands that were typically abbreviations (for example, ls) at the prompt symbol appearing on the computer display, to which the system responded (for example, by listing current files). Another way of issuing commands is by pressing certain combinations of keys (such as Shift+Alt+Ctrl). Some commands are also a fixed part of the keyboard, such as delete, enter, and undo, while other function keys can be programmed by the user as specific commands (for instance, F11 commanding print action).
Command-line interfaces were largely superseded by graphical interfaces that incor-porated commands such as menus, icons, keyboard shortcuts, and pop-up/predictable text commands as part of an application. Where command-line interfaces continue to have an advantage is when users find them easier and faster to use than equivalent menu-based systems (Raskin, 2000). Users also prefer command-line interfaces for performing certain operations as part of a complex software package, such as for CAD environments (such as Rhino3D and AutoCAD), to allow expert designers to interact rapidly and precisely with the software. They also provide scripting for batch operations, and they are being increasingly used on the web, where the search bar acts as a general-purpose command-line facility, for example, www.yubnub.org.
System administrators, programmers, and power users often find that it is much more efficient and quicker to use command languages such as Microsoft’s PowerShell. For exam-ple, it is much easier to delete 10,000 files in one go by using one command rather than scrolling through that number of files and highlighting those that need to be deleted. Com-mand languages have also been developed for visually impaired people to allow them to interact in virtual worlds, such as Second Life (see Box 7.1).
Here is a selection of classic HCI videos on the Internet that demonstrate pioneer-ing interfaces:The Sketchpad: Ivan Sutherland (1963) describes the first interactive graphical
interface: https://youtu.be/6orsmFndx_o.The Mother of All Demos: Douglas Engelbart (1968) describes the first WIMP:
http://youtu.be/yJDv-zdhzMy.put that there (1979): MIT demonstrates the first speech and gesture interface:
https://youtu.be/RyBEUyEtxQo.Unveiling the genius of multitouch interface design: Jeff Han gives a TED talk
(2007): http://youtu.be/ac0E6deG4AU.Intel’s Future Technology Vision (2012): See http://youtu.be/g_cauM3kccI.
7 I N T E R F A C E S196
Watch a video demonstration of TextSL at http://youtu.be/0Ba_w7u44MM.
Research and Design ConsiderationsIn the 1980s, much research investigated ways of optimizing command interfaces. The form of the commands (including use of abbreviations, full names, and familiar names), syntax (such as how best to combine different commands), and organization (for instance, how to structure options) are examples of some of the main areas that have been investigated (Shneiderman, 1998). A further concern was which command names would be the easiest to remember. A number of variables were tested, including how familiar users were with the chosen names. Findings from a number of studies, however, were inconclusive; some found specific names were better remem-bered than general ones (Barnard et al., 1982), others showed that names selected by users them-selves were preferable (see Ledgard et al., 1981; Scapin, 1981), while yet others demonstrated that high-frequency words were better remembered than low-frequency ones (Gunther et al., 1986).
The most relevant design principle is consistency (see Chapter 1, “What Is Interaction Design?”). Therefore, the method used for labeling/naming the commands should be chosen to be as consistent as possible; for example, always use the first letters of the operation when using abbreviations.
BOX 7.1 Command Interfaces for Virtual Worlds
Virtual worlds, such as Second Life, have become popular places for learning and social-izing. Unfortunately, people who are visually impaired cannot interact in a visual capacity. A command-based interface, called TextSL, was developed to enable them to participate using a screen reader (Folmer et al., 2009). Commands can be issued to enable the user to move their avatar around, interact with others, and find out about the environment in which they are located. Figure 7.1 shows that the user has issued the command for their avatar to smile and say hello to other avatars who are sitting by a log fire.
Figure 7.1 Second Life command-based interface for visually impaired users
Source: Used courtesy of Eelke Folmer
7 . 2 I N T E R F A C E T y p E S 197
7.2.2 Graphical User InterfacesThe Xerox Star interface (described in Chapter 3, “Conceptualizing Interaction”) led to the birth of the graphical user interface (GUI), opening up new possibilities for users to inter-act with a system and for information to be presented and represented within a graphical interface. Specifically, new ways of visually designing the interface became possible, which included the use of color, typography, and imagery (Mullet and Sano, 1995). The original GUI was called a WIMP (windows, icons, menus, pointer) and consisted of the following:
• Windows: Sections of the screen that can be scrolled, stretched, overlapped, opened, closed, and moved using a mouse
• Icons: Pictograms that represent applications, objects, commands, and tools that are opened or activated when clicked on
• Menus: Lists of options that can be scrolled through and selected in the way a menu is used in a restaurant
• Pointing device: A mouse controlling the cursor as a point of entry to the windows, menus, and icons on the screen
The first generation of WIMP interfaces were primarily boxy in design; user interaction took place through a combination of windows, scroll bars, checkboxes, panels, palettes, and dialog boxes that appeared on the screen in various forms (see Figure 7.2). Develop-ers were largely constrained by the set of widgets available to them, of which the dialog box was most prominent. (A widget is a standardized display representation of a control, like a button or scroll bar, that can be manipulated by the user.) Nowadays, GUIs have been adapted for mobile and touchscreens. Instead of using a mouse and keyboard as input, the default action for most users is to swipe and touch using a single finger when browsing and interacting with digital content. (For more on this subject, see the sections on touch and mobile interfaces.)
Figure 7.2 The boxy look of the first generation of GUIs
7 I N T E R F A C E S198
The basic building blocks of the WIMP are still part of the modern GUI used as part of a display, but they have evolved into a number of different forms and types. For example, there are now many different types of icons and menus, including audio icons and audio menus, 3D animated icons, and even tiny icon-based menus that can fit onto a smartwatch screen (see Figure 7.3). Windows have also greatly expanded in terms of how they are used and what they are used for; for example, a variety of dialog boxes, interactive forms, and feedback/error message boxes have become pervasive. In addition, a number of graphical elements that were not part of the WIMP interface have been incorporated into the GUI. These include toolbars and docks (a row or column of available applications and icons of other objects such as open files) and rollovers (where text labels appear next to an icon or part of the screen as the cursor is rolled over it). Here, we give an overview of the design considerations concerning the basic building blocks of the WIMP/GUI: windows, menus, and icons.
Window DesignWindows were invented to overcome the physical constraints of a computer display, ena-bling more information to be viewed and tasks to be performed on the same screen. Multi-ple windows can be opened at any one time, for example, web browsers, word processing documents, photos, and slideshows, enabling the user to switch between them when needing to look at or work on different documents, files, and apps. They can also enable multiple instances of one app to be opened, such as when using a web browser.
Scrolling bars within windows also enable more information to be viewed than is possi-ble on one screen. Scroll bars can be placed vertically and horizontally in windows to enable upward, downward, and sideway movements through a document and can be controlled using a touchpad, mouse, or arrow keys. Touch interfaces enable users to scroll content sim-ply by swiping the screen to the left or right or up or down.
Figure 7.3 Simple smartwatch menus with one, two, or three options Source: https://developer.apple.com/design/human-interface-guidelines/watchos/interface-elements/menus/
7 . 2 I N T E R F A C E T y p E S 199
One of the problems of having multiple windows open is that it can be difficult to find specific ones. Various techniques have been developed to help users locate a particular win-dow, a common one being to provide a list as part of an app menu. macOS also provides a function that shrinks all windows that are open for a given application so that they can be seen side by side on one screen. The user needs only to press one function key and then move the cursor over each one to see what they are called in addition to a visual preview. This tech-nique enables users to see at a glance what they have in their workspace, and it also allows them easily to select one to bring forward. Another option is to display all of the windows open for a particular application, for example, Microsoft Word. Web browsers, like Firefox, also show thumbnails of the top sites visited and a selection of sites that you have saved or visited, which are called highlights (see Figure 7.4).
A particular kind of window that is commonly used is the dialog box. Confirmations, error messages, checklists, and forms are presented through dialog boxes. Information in the dialog boxes is often designed to guide user interaction, with the user following the sequence of options provided. Examples include a sequenced series of forms (such as Wizards) present-ing the necessary and optional choices that need to be filled in when choosing a PowerPoint presentation or an Excel spreadsheet. The downside of this style of interaction is that there is a tendency to cram too much information or data entry fields into one box, making the interface confusing, crowded, and difficult to read (Mullet and Sano, 1995).
Figure 7.4 Part of the home page for the Firefox browser showing thumbnails of top sites visited and suggested highlight pages (bottom rows)
7 I N T E R F A C E S200
BOX 7.2 The Joys of Filling In Forms on the Web
For many of us, shopping on the Internet is generally an enjoyable experience. For exam-ple, choosing a book on Amazon or flowers from Interflora can be done at our leisure and convenience. The part that we don’t enjoy, however, is filling in the online form to give the company the necessary details to pay for the selected items. This can often be a frustrating and time-consuming experience, especially as there is much variability between sites. Sometimes, it requires users to create an account and a new password. At other times, guest checkout is enabled. However, if the site has a record of your email address in its database, it won’t allow you to use the guest option. If you have forgotten your password, you need to reset it, and this requires switching from the form to your email account. Once past this hurdle, different kinds of interactive forms pop up for you to enter your mailing address and credit card details. The form may provide the option of finding your address by allowing you to enter a postal or ZIP code. It may also have asterisks that denote fields that must be filled in.
Having so much inconsistency can frustrate the user, as they are unable to use the same mental model for filling in checkout forms. It is easy to overlook or miss a box that needs to be filled in, and after submitting the page, an error message may come back from the system saying it is incomplete. This may require the user to have to enter sensitive information again, as it will have been removed in the data processing stage (for example, the user’s credit card number and the three or four-digit security code on the back or front of the card, respectively).
To add to the frustration, many online forms often accept only fixed data formats, mean-ing that, for some people whose information does not fit within its constraints, they are unable to complete the form. For example, one kind of form will accept only a certain type of mailing address format. The boxes are provided for: address line 1 and address line 2, providing no extra lines for addresses that have more than two lines; a line for the town/city; and a line for the ZIP code (if the site is based in the United States) or other postal code (if based in another country). The format for the codes is different, making it difficult for non-U.S. residents (and U.S. residents for other country sites) to fill in this part.
Another gripe about online registration forms is the country of residence box that opens up as a never-ending menu, listing all of the countries in the world in alphabetical order. Instead of typing in the country in which they reside, users are required to select the one they are from, which is fine if you happen to live in Australia or Austria but not if you live in Venezuela or Zambia (see Figure 7.5).
This is an example of where the design principle of recognition over recall (see Chapter 4, “Cognitive Aspects”) does not apply and where the converse is true. A better design is to have a predictive text option, where users need only to type in the first one or two letters of their country to cause a narrowed-down list of choices to appear from which they can select within the interface. Or, one smart option is for the form to preselect the user’s country of origin by using information shared from the user’s computer or stored in the cloud. Automating the filling in of online forms, through providing prestored information about a user (for example, their address and credit card details), can obviously help reduce usability problems—provided they are OK with this.
7 . 2 I N T E R F A C E T y p E S 201
Figure 7.5 A scrolling menu of country names Source: https://www.jollyflorist.com
ACTIVITy 7.1 Go to the Interflora site in the United Kingdom, click the international delivery option, and then click “select a country.” How are the countries ordered? Is it an improvement over the scrolling pop-up menu?
CommentEarlier versions of the full list of countries to which flowers could be sent by interflora.co.uk listed eight countries at the top, starting with the United Kingdom and then the United States, France, Germany, Italy, Switzerland, Austria, and Spain. This was followed by the remaining set of countries listed in alphabetical order. The reason for having this particular ordering is likely to have been because the top eight are the countries that have most customers, with the U.K. residents using the service the most. The website has changed now to show top countries by national flag followed by a table format, grouping all of the countries in alphabetical order using four columns across the page (see Figure 7.6). Do you think this is an improvement over the use of a single scrolling list of country names shown in Figure 7.5? The use of letter head-ings and shading makes searching quicker.
7 I N T E R F A C E S202
Menu DesignInterface menus are typically ordered across the top row or down the side of a screen using category headers as part of a menu bar. The contents of the menus are also for the large part invisible, only dropping down when the header is selected or rolled over with a mouse.
Research and Design ConsiderationsA key research concern is window management—finding ways of enabling users to move flu-idly between different windows (and displays) and to be able to switch their attention rapidly between windows to find the information they need or to work on the document/task within each window without getting distracted. Studies of how people use windows and multiple dis-plays have shown that window activation time (that is, the time a window is open and with which the user interacts with it) is relatively short—an average of 20 seconds—suggesting that people switch frequently between different documents and applications (Hutchings et al., 2004). Widgets like the taskbar are often used for switching between windows.
Another technique is the use of tabs that appear at the top of the web browser that show the name and logo of the web pages that have been visited. This mechanism enables users to rapidly scan and switch among the web pages they have visited. However, the tabs can quickly multiply if a user visits a number of sites. To accommodate new ones, the web browser reduces the size of the tabs by shortening the information that appears on each. The downside of doing this, however, is it can make it more difficult to read and recognize web pages when looking at the smaller tabs. It is possible to reverse this shrinking by removing unwanted tabs by clicking the delete icon for each one. This has the effect of making more space available for the remaining tabs.
There are multiple ways that an online form can be designed to obtain details from someone. It is not surprising, therefore, that there are so many different types that are in use. Design guidelines are available to help decide which format and widgets are best to use. For example, see https://www.smashingmagazine.com/printed-books/form-design-patterns/. Another option is to automate form completion by asking the user to store their personal details on their machine or in a company’s database, requiring them only to enter security information. However, many people are becoming leery of storing their personal data in this way—fearful because of the number of data breaches that are often reported in the news.
Figure 7.6 An excerpt of the listing of countries in alphabetical order from interflora.co.uk Source: https://www.interflora.co.uk
7 . 2 I N T E R F A C E T y p E S 203
The various options under each menu are typically ordered from top to bottom in terms of most frequently used options and grouped in terms of their similarity with one another; for example, all formatting commands are placed together.
There are numerous menu interface styles, including flat lists, drop-down, pop-up, contex-tual, collapsible, mega, and expanding ones, such as cascading menus. Flat menus are good at displaying a small number of options at the same time or where the size of the display is small, for example on smartphones, cameras, and smartwatches. However, they often have to nest the lists of options within each, requiring several steps to be taken by a user to get to the list with the desired option. Once deep down in a nested menu, the user then has to take the same number of steps to get back to the top of the menu. Moving through previous screens can be tedious.
Expanding menus enable more options to be shown on a single screen than is possible with a single flat menu list. This makes navigation more flexible, allowing for the selection of options to be done in the same window. An example is the cascading menu, which provides secondary and even tertiary menus to appear alongside the primary active drop-down menu, enabling further related options to be selected, such as when selecting track changes from the tools menu leads to a secondary menu of three options by which to track changes in a Word document. The downside of using expanding menus, however, is that they require precise control. Users can often end up making errors, namely, overshooting or selecting the wrong options. In particular, cascading menus require users to move their cursor over the menu item, while holding the mouse or touchpad down, and then to move their cursor over to the next menu list when the cascading menu appears and select the next desired option. This can result in the user under or overshooting a menu option, or sometimes accidentally closing the entire menu. Another example of an expandable menu is a mega menu, in which many options can be displayed using a 2D drop-down layout (see Figure 7.7). This type of menu is popular with online shop-ping sites, where lots of items can be viewed at a glance on the same screen without the need to scroll. Hovering, tapping, or clicking is used to reveal more details for a selected item.
Figure 7.7 A mega menuSource: https://www.johnlewis.com
7 I N T E R F A C E S204
Collapsible menus provide an alternative approach to expanding menus in that they allow further options to be made visible by selecting a header. The headings appear adjacent to each other, providing the user with an overview of the content available (see Figure 7.8). This reduces the amount of scrolling needed. Contextual menus provide access to often-used commands associated with a particular item, for example, an icon. They provide appropri-ate commands that make sense in the context of a current task. They appear when the user presses the Control key while clicking an interface element. For example, clicking a photo on a website together with holding down the Ctrl key results in a small set of relevant menu options appearing in an overlapping window, such as open it in a new window, save it, or copy it. The advantage of contextual menus is that they provide a limited number of options associated with an interface element, overcoming some of the navigation problems associated with cascading and expanding menus.
Figure 7.8 A template for a collapsible menu Source: https://inclusive-components.design/collapsible-sections/. Reproduced with permission of Smashing Magazine
ACTIVITy 7.2 Open an application that you use frequently (for instance, a word processor, email client, or web browser) on a PC/laptop or tablet and look at the menu header names (but do not open them just yet). For each menu header—File, Edit, Tools, and so on—write down what options you think are listed under each. Then look at the contents under each header. How many options were you able to remember, and how many did you put in the wrong category? Now try to select the correct menu header for the following options (assuming that they are included in the application): Replace, Save, Spelling, and Sort. Did you select the correct header each time, or did you have to browse through a number of them?
CommentPopular everyday applications, like word processors, have grown enormously in terms of the functions they offer. The current version (2019) of Microsoft Word, for example, has 8 menu headers and numerous toolbars. Under each menu header there are on average 15 options,
7 . 2 I N T E R F A C E T y p E S 205
Icon DesignThe appearance of icons in an interface came about following the Xerox Star project. They were used to represent objects as part of the desktop metaphor, namely, folders, documents, trashcans, inboxes, and outboxes. The assumption behind using icons instead of text labels is that they are easier to learn and remember, especially for non-expert computer users. They can also be designed to be compact and variably positioned on a screen.
some of which are hidden under subheadings and appear only when they are rolled over with the mouse. Likewise, for each toolbar, there is a set of tools available, be it for Drawing, For-matting, Web, Table, or Borders. Remembering the location of frequently used commands like Spelling and Replace is often achieved by remembering their spatial location. For infrequently used commands, like sorting a list of references into alphabetical order, users can spend time flicking through the menus to find the command Sort. It is difficult to remember that the com-mand Sort should be under the Table heading, since what it is doing is not a table operation, but a tool to organize a section of a document. It would be more intuitive if the command was under the Tool header along with similar tools like Spelling. What this example illustrates is just how difficult it can be to group menu options into clearly defined and obvious categories. Some fit into several categories, while it can be difficult to group others. The placement of options in menus can also change between different versions of an application as more func-tions are added.
Research and Design ConsiderationsAn important design consideration is to decide which terms to use for menu options. Short phrases like “bring all to front” can be more informative than single words like “front.” How-ever, the space for listing menu items is often restricted, such that menu names need to be short. They also need to be distinguishable, that is, not easily confused with one another so that the user does not choose the wrong one by mistake. Operations such as Quit and Save should also be clearly separated to avoid the accidental loss of work.
The choice of which type of menu to use will often be determined by the application and type of device for which is being designed. Which is best will also depend on the number of menu options and the size of the display available in which to present them. Flat menus are best for displaying a small number of options at one time, while expanding and collapsible menus are good for showing a large number of options, such as those available in file and document creation/editing applications. Usability testing comparing drop-down menus with mega menus has shown the latter to be more effective and easier to navigate. The main reason is that megamenus enable users to readily scan many items at a glance on the same page, and in doing so find what they are looking for (Nielsen and Li, 2017).
7 I N T E R F A C E S206
Icons have become a pervasive feature of the interface. They now populate every app and operating system and are used for all manner of functions besides representing desktop objects. These include depicting tools (for example, Paint 3D), status (such as, Wi-Fi strength), categories of apps (for instance, health or personal finance), and a diversity of abstract opera-tions (including cut, paste, next, accept, and change). They have also gone through many changes in their look and feel—black and white, color, shadowing, photorealistic images, 3D rendering, and animation have all been used.
Whereas early icon designers were constrained by the graphical display technology of the day, current interface developers have much more flexibility. For example, the use of anti-aliasing techniques enables curves and non-rectilinear lines to be drawn, enabling more photo-illustrative styles to be developed (anti-aliasing means adding pixels around a jagged border of an object to smooth its outline visually). App icons are often designed to be both visually attractive and informative. The goal is to make them inviting, emotionally appealing, memorable, and distinctive.
Different graphical genres have been used to group and identify different categories of icons. Figure 7.9 shows how colorful photorealistic images were used in the original Apple Aqua set, each slanting slightly to the left, for the category of user applications (such as email) whereas monochrome straight on and simple images were used for the class of utility applica-tions (for instance, printer setup). The former has a fun feel to them, whereas the latter has a more serious look about them. While a number of other styles have since been developed, the use of slanting versus straight facing icons to signify different icon categories is still in use.
Icons can be designed to represent objects and operations in the interface using concrete objects and/or abstract symbols. The mapping between the icon and underlying object or opera-tion to which it refers can be similar (such as a picture of a file to represent the object file), ana-logical (for instance, a picture of a pair of scissors to represent cut), or arbitrary (for example, the use of an X to represent delete). The most effective icons are generally those that are isomorphic since they have a direct mapping between what is being represented and how it is represented. Many operations in an interface, however, are of actions to be performed on objects, making it more difficult to represent them using direct mapping. Instead, an effective technique is to use a combination of objects and symbols that capture the salient part of an action by using anal-ogy, association, or convention (Rogers, 1989). For example, using a picture of a pair of scissors to represent cut in a word-processing application provides a sufficient clue as long as the user understands the convention of cut for deleting text.
Figure 7.9 Two styles of Apple icons used to represent different kinds of functions
7 . 2 I N T E R F A C E T y p E S 207
Another approach that many smartphone designers use is flat 2D icons. These are simple and use strong colors and pictograms or symbols. The effect is to make them easily recogniz-able and distinctive. Examples shown in Figure 7.10a include the white ghost on a yellow background (Snapchat), a white line bubble with a solid white phone handset in a speech bubble on a lime-green background (WhatsApp), and the sun next to a cloud (weather).
Icons that appear on toolbars or palettes as part of an application or presented on small device displays (such as digital cameras or smartwatches) have much less screen real estate available. Because of this, they have been designed to be simple, emphasizing the outline form of an object or symbol and using only grayscale or one or two colors (see Figure 7.10b). They tend to convey the status, tool, or action using a concrete object (for example, the airplane symbol signaling whether the airplane mode is on or off) and abstract symbols (such as three waves that light up from none to all to convey the strength/power of the area’s Wi-Fi).
(a) (b)
Figure 7.10 2D icons designed for (a) a smartphone and (b) a smartwatch Source: (a) Helen Sharp (b) https://support.apple.com/en-ca/HT205550
ACTIVITy 7.3 Sketch simple icons to represent the following operations to appear on a digital camera screen: • Turn image 90-degrees sideways. • Crop the image. • Auto-enhance the image. • More options.
Show them to someone else, tell them that they are icons for a new digital camera intended to be really simple to use, and see whether they can understand what each represents.
7 I N T E R F A C E S208
Research and Design ConsiderationsThere are many icon libraries available that developers can download for free (for instance, https://thenounproject.com/ or https://fontawesome.com/). Various online tutorials and books on how to design icons are also available (see Hicks, 2012) together with sets of proprietary guidelines and style guides. For example, Apple provides its developers with style guides, explaining why certain designs are preferable to others and how to design icon sets. Style
CommentFigure 7.11 shows the basic Edit Photo icons on an iPhone that appear at the bottom of the screen when a user selects the edit function. The box with extended lines and two arrows is the icon for cropping an image; the three overlapping translucent circles represents “differ-ent lenses” that can be used, the wand in the top-right corner means “auto-enhance,” and the circle with three dots in it means more functions.
Figure 7.11 The basic Edit Photo icons that appear at the top and bottom of an iPhone display
7 . 2 I N T E R F A C E T y p E S 209
7.2.3 MultimediaMultimedia, as the name implies, combines different media within a single interface, namely, graphics, text, video, sound, and animation, and links them together with various forms of interactivity. Users can click links in an image or text that triggers another media such as an animation or a video. From there they can return to where they were previously or jump to another media source. The assumption is that a combination of media and interactivity can provide better ways of presenting information than can a single media, for example, just text or video alone. The added value of multimedia is that it can be easier for learning, better for understanding, more engaging, and more pleasant (Scaife and Rogers, 1996).
Another distinctive feature of multimedia is its ability to facilitate rapid access to mul-tiple representations of information. Many multimedia encyclopedias and digital libraries have been designed based on this multiplicity principle, providing an assortment of audio and visual materials on a given topic. For example, when looking to find information about the heart, a typical multimedia-based encyclopedia will provide the following:
• One or two video clips of a real live heart pumping and possibly a heart transplant operation • Audio recordings of the heart beating and perhaps an eminent physician talking about the cause of heart disease
• Static diagrams and animations of the circulatory system, sometimes with narration • Several columns of hypertext, describing the structure and function of the heart
Hands-on interactive simulations have also been incorporated as part of multimedia learning environments. An early example was the Cardiac Tutor, developed to teach students about cardiac resuscitation. It required students to save patients by selecting the correct set of procedures in the correct order from various options displayed on the computer screen (Eliot and Woolf, 1994). Other kinds of multimedia narratives and
guides are also covered in more depth in Chapter 13, “Interaction Design in Practice.” On its developers’ website (developer.apple.com), advice is given on how and why certain graphical elements should be used when developing different types of icon. Among the various guide-lines, it suggests that different categories of application (for example, Business, Utilities, Enter-tainment, and so on) should be represented by a different genre, and it recommends displaying a tool to communicate the nature of a task, such as a magnifying glass for searching or a camera for a photo editing tool. Android and Microsoft also provide extensive guidance and step-by-step procedures on how to design icons for its applications on its website.
To help disambiguate the meaning of icons, text labels can be used under, above, or to the side of their icons. This method is effective for toolbars that have small icon sets, such as those appearing as part of a web browser, but it is not as good for applications that have large icon sets, for example, photo editing or word processing, since the screen can get cluttered making it sometimes harder and longer to find an icon. To prevent text/icon clutter on the interface, a hover function can be used where a text label appears adjacent to or above an icon after the user holds the cursor over it for a second and for as long as the user keeps the cursor on it. This method allows identifying information to be temporarily displayed when needed.
7 I N T E R F A C E S210
games have also been developed to support discovery learning by encouraging children to explore different parts of the display by noticing a hotspot or other kind of link. For example, https://KidsDiscover.com/apps/ has many tablet apps that use a combination of animations, photos, interactive 3D models, and audio to teach kids about science and social studies topics. Using swiping and touching, kids can reveal, scroll through, select audio narration, and watch video tours. Figure 7.12, for example, has a “slide” mecha-nism as part of a tablet interface that enables the child to do a side-by-side comparison of what Roman ruins looks like now and in ancient Roman times.
Another example of a learning app with an interesting UI can be seen at https://www .abcmouse.com/apps.
Multimedia has largely been developed for training, educational, and entertainment pur-poses. But to what extent is the assumption that learning (such as reading and scientific inquiry skills) and playing can be enhanced through interacting with engaging multimedia interfaces true? What actually happens when users are given unlimited, easy access to mul-tiple media and simulations? Do they systematically switch between the various media and “read” all of the multiple representations on a particular subject, or are they more selective in what they look at and listen to?
Figure 7.12 An example of a multimedia learning app designed for tablets Source: KidsDiscover app “Roman Empire for iPad”
7 . 2 I N T E R F A C E T y p E S 211
ACTIVITy 7.4 Watch this video of Don Norman appearing in his first multimedia CD-ROM book (1994), where he pops up every now and again in boxes or at the side of the page to illustrate the points being discussed on that page: http://vimeo.com/18687931.
How do you think students used this kind of interactive e-textbook?
CommentAnyone who has interacted with educational multimedia knows just how tempting it is to play the video clips and animations while skimming through accompanying text or static diagrams. The former is dynamic, easy, and enjoyable to watch, while the latter is viewed as static and difficult to read from the screen. In an evaluation of the original Voyager’s “First Person: Donald Norman, Defending Human Attributes in the Age of the Machine,” students consistently admitted to ignoring the text on the interface in search of clickable icons of the author, which when selected would present an animated video of him explaining some aspect of design (Rogers and Aldrich, 1996). Given the choice to explore multimedia material in numerous ways, ironically, users tend to be highly selective as to what they actually pay atten-tion to, adopting a channel-hopping mode of interaction. While enabling the users to select the information they want to view or features to explore for themselves, there is the danger that multimedia environments may in fact promote fragmented interactions where only part of the media is ever viewed. In a review of research comparing reading from screens versus paper, Lauren Singer and Patricia Alexandra (2017) found that despite students saying they preferred reading from screens, their actual performance was worse than when using paper-based textbooks.
Hence, online multimedia material may be good for supporting certain kinds of activities, such as browsing, but less optimal for others, for instance reading at length about a topic. One way to encourage more systematic and extensive interactions (when it is considered important for the activity at hand) is to require certain activities to be completed that entail the reading of accompanying text, before the user is allowed to move on to the next level or task.
Research and Design ConsiderationsA core research question is how to encourage users to interact with all aspects of a multime-dia app, especially given the tendency to select videos to watch rather than text to read. One technique is to provide a diversity of hands-on interactivities and simulations that require the user to complete a task, solve a problem, or explore different aspects of a topic that involves reading some accompanying text. Specific examples include electronic notebooks that are integrated as part of the interface, where users can type in their own material; multiple-choice quizzes that provide feedback about how well they have done; interactive puzzles where they have to select and position different pieces in the right combination; and simulation-type games
(Continued)
7 I N T E R F A C E S212
7.2.4 Virtual RealityVirtual reality (VR) has been around since the 1970s when researchers first began devel-oping computer-generated graphical simulations to create “the illusion of participation in a synthetic environment rather than external observation of such an environment” (Gigante, 1993, p. 3). The goal was to create user experiences that feel virtually real when interacting with an artificial environment. Images are displayed stereoscopically to the users—most commonly through VR headsets—and objects within the field of vision can be interacted with via an input device like a joystick.
The 3D graphics can be projected onto Cave Automatic Virtual Environment (CAVE) floor and wall surfaces, desktops, 3D TV, headsets, or large shared displays, for instance, IMAX screens. One of the main attractions of VR is that it can provide opportunities for new kinds of immersive experiences, enabling users to interact with objects and navigate in 3D space in ways not possible in the physical world or a 2D graphical interface. Besides looking at and navigating through a 360-degree visual landscape, auditory and haptic feedback can be added to make the experience feel even more like the real world. The resulting user experi-ence can be highly engaging; it can feel as if one really is flying around a virtual world. Peo-ple can become completely absorbed by the experience. The sense of presence can make the virtual setting seem convincing. Presence, in this case, means “a state of consciousness, the (psychological) sense of being in the virtual environment” (Slater and Wilbur, 1997, p. 605), where someone behaves in a similar way to how they would if at an equivalent real event.
VR simulations of the world can be constructed to have a higher level of fidelity with the objects they represent compared to other forms of graphical interfaces, for example, mul-timedia. The illusion afforded by the technology can make virtual objects appear to be very life-like and behave according to the laws of physics. For example, landing and take-off ter-rains developed for flight simulators can appear to be very realistic. Moreover, it is assumed that learning and training applications can be improved through having a greater fidelity to the represented world.
where they have to follow a set of procedures to achieve some goal for a given scenario. Another approach is to employ dynalinking, where information depicted in one window explicitly changes in relation to what happens in another. This can help users keep track of multiple representations and see the relationship between them (Scaife and Rogers, 1996).
Specific guidelines are available that recommend how best to combine multiple media in relation to different kinds of task, for example, when to use audio with graphics, sound with animations, and so on, for different learning tasks. As a rule of thumb, audio is good for stim-ulating the imagination, movies for depicting action, text for conveying details, and diagrams for conveying ideas. From such generalizations, it is possible to devise a presentation strategy for online learning. This could be along the lines of the following:
1. Stimulate the imagination through playing an audio clip.2. Present an idea in diagrammatic form.3. Display further details about the concept through hypertext.
7 . 2 I N T E R F A C E T y p E S 213
Another distinguishing feature of VR is the different viewpoints it can offer. Players can have a first-person perspective, where their view of the game or environment is through their own eyes, or a third-person perspective, where they see the world through an avatar visually represented on the screen. An example of a first-person perspective is that experienced in first-person shooter games such as DOOM, where the player moves through the environ-ment without seeing a representation of themselves. It requires the user to imagine what they might look like and decide how best to move around. An example of a third-person perspec-tive is that experienced in Tomb Raider, where the player sees the virtual world above and behind the avatar of Lara Croft. The user controls Lara’s interactions with the environment by controlling her movements, for example, making her jump, run, or crouch. Avatars can be represented from behind or from the front, depending on how the user controls its move-ments. First-person perspectives are typically used for flying/driving simulations and games, for instance, car racing, where it is important to have direct and immediate control to steer the virtual vehicle. Third-person perspectives are more commonly used in games, learning environments, and simulations, where it is important to see a representation of self with respect to the environment and others in it. In some virtual environments, it is possible to switch between the two perspectives, enabling the user to experience different viewpoints on the same game or training environment.
In the beginning, head-mounted displays were used to present VR experiences. However, the visuals were often clunky, the headset uncomfortable to wear, and the immersive experi-ence sometimes resulting in motion sickness and disorientation. Since then, VR technology has come of age and improved greatly. There are now many off-the-shelf VR headsets (for example Oculus Go, HTC Vive, and Samsung Gear VR) that are affordable and comfortable. They also have more accurate head tracking that allow developers to create more compelling games, movies, and virtual environments.
“Out of Home Entertainment” and VR arcades have also become popular worldwide and provide a range of social VR experiences, targeted at the general public. For example, Hyper-Reality has developed a number of spooky games, for 1–4 players, such as Japanese Adventures, Escape the Lost Pyramid, and the Void. Each game lasts for about 40 minutes, where players have to carry out a set of tasks, such as finding a lost friend in a realm. The immersive entertainment is full of surprises at every turn. One moment a player might be on solid ground and the next in complete darkness. The pleasure is often in not knowing what is going to happen next and being able to recount the experiences afterward with friends and family.
Another application area is how VR can enrich the experience of reporting and wit-nessing current affairs and news, especially feelings of empathy and compassion to real-life experiences (Aronson-Rath et al., 2016). For example, the BBC together with Aardman Interactive and University College London researchers developed a VR experience called “We Wait,” where they put the viewer in a place that few foreign reporters have been, namely, on a boat with a group of refugees crossing the Mediterranean Sea (Steed et al., 2018). The goal was to let news reporters and other participants experience how it felt to be there on the boat with the refugees. They used a particular artistic polygon style rather than realism to create the characters sitting on the boat (see Figure 7.13). The characters had expressive eyes intended to convey human emotion in response to gaze interaction. The avatars were found to generate an empathic response from participants.
7 I N T E R F A C E S214
VR is also starting to be used by airlines and travel companies to enrich someone’s plan-ning experience of their travel destinations. For example, the airline KLM has developed a platform called iFly VR (https://360.iflymagazine.com/) that provides an immersive experi-ence intended to inspire people to discover more about the world. A potential danger of this approach, however, is that if the VR experience is too lifelike it might make people feel they have ‘been there, done that’ and hence don’t need to visit the actual place. However, KLM’s rationale is quite the opposite; if you make the virtual experience so compelling people will want to go there even more. Their first foray into this adventure follows the famous “Fear-less Chef” Kiran Jethwa into a jungle in Thailand to look for the world’s most remarkable coffee beans.
MagicLeap has pushed the envelope even further into new realms of virtual real-ity; combining cameras, sensors, and speakers in a headset that provides quite a different experience—one where the user can create their own worlds using various virtual tools, for example, painting a forest or building a castle—that then come alive in the actual physical space in which they reside. In this sense, it is not strictly VR, as it allows the wearer to see the virtual world and virtual objects they have created, or curated, blend with the physical objects in their living room or other space in which they are located. It is as if by magic the two are in the same world. In some ways it is a form of augmented reality (AR) – described in section 7.2.16.
Figure 7.13 Snapshot of polygon graphics used to represent avatars for the “We Wait” VR experience Source: Steed, Pan, Watson and Slater, https://www.frontiersin.org/articles/10.3389/frobt.2018.00112/full. Licensed Under CC-BY 4.0
7 . 2 I N T E R F A C E T y p E S 215
Watch this video of MagicLeap’s Create World where the virtual world meets the physical world in magical ways: https://youtu.be/K5246156rcQ.
Research and Design ConsiderationsVR has been developed to support learning and training for numerous skills. Researchers have designed apps to help people learn to drive a vehicle, fly a plane, and perform delicate surgical operations—where it is very expensive and potentially dangerous to start learning with the real thing. Others have investigated whether people can learn to find their way around a real building/place before visiting it by first navigating a virtual representation of it, see Gabrielli et al. (2000).
An early example of VR was the Virtual Zoo project. Allison et al. (1997) found that people were highly engaged and very much enjoyed the experience of adopting the role of a gorilla, navigating the environment, and watching other gorillas respond to their movements and presence.
Virtual environments (VE) have also been designed to help people practice social and speaking skills and confront their social phobias, see Cobb et al. (2002) and Slater et al. (1999). An underlying assumption is that the environment can be designed as a safe place to help people gently overcome their fears (for example, spiders, talking in public, and so forth) by confronting them through different levels of closeness and unpleasantness (such as by seeing a small virtual spider move far away, seeing a medium one sitting nearby, and then finally touching a large one). Studies have shown that people can readily suspend their disbe-lief, imagining a virtual spider to be a real one or a virtual audience to be a real audience. For example, Slater et al. (1999) found that people rated themselves as being less anxious after speaking to a virtual audience that was programmed to respond to them in a positive fashion than after speaking to virtual audiences programmed to respond to them negatively.
Core design considerations include the importance of having a virtual self-body as part of a VR experience to enhance the feeling of presence; how to prevent users from experiencing simulator sickness through experimenting with galvanic stimulation; determining the most effective ways of enabling users to navigate through them, for instance, first person versus third person; how to control their interactions and movements, for example, use of head and body movements; how best to enable users to interact with information in VR, for example, use of keypads, pointing, joystick buttons; and how to enable users to collaborate and com-municate with others in the virtual environment.
(Continued)
Peter Rubin’s (2018) guide to VR published in Wired magazine provides a sum-mary and speculation about its future: https://www.wired.com/story/wired-guide-to-virtual-reality/.
7 I N T E R F A C E S216
7.2.5 Website DesignEarly websites were largely text-based, providing hyperlinks to different places or pages of text. Much of the design effort was concerned with the information architecture, that is, how best to structure information at the interface level to enable users to navigate and access it easily and quickly. For example, Jakob Nielsen (2000) adapted his and Rolf Molich’s usabil-ity guidelines (Nielsen and Molich, 1990) to make them applicable to website design, focus-ing on simplicity, feedback, speed, legibility, and ease of use. He also stressed how critical download time was to the success of a website. Simply, users who have to wait too long for a page to appear are likely to move on somewhere else.
Since then, the goal of web design has been to develop sites that are not only usable but also aesthetically pleasing. Getting the graphical design right, therefore, is critical. The use of graphical elements (such as background images, color, bold text, and icons) can make a web-site look distinctive, striking, and pleasurable for the user when they first view it and also to make it readily recognizable on their return. However, there is the danger that designers can get carried away with the appearance at the expense of making it difficult to find something and navigate through it.
Steve Krug (2014) discusses this usability versus attractiveness dilemma in terms of the difference between how designers create websites and how users actually view them. He argues that many web designers create sites as if the user was going to pore over each page, reading the finely crafted text word for word; looking at the use of images, color, icons, and so forth; examining how the various items have been organized on the site; and then con-templating their options before they finally select a link. Users, however, often behave quite differently. They will glance at a new page, scan part of it, and click the first link that catches their interest or looks like it might lead them to what they want.
Much of the content on a web page is not read. In Krug’s words, web designers are “thinking great literature” (or at least “product brochure”), while the user’s reality is much closer to a “billboard going by at 60 miles an hour” (Krug, 2014, p. 21). While somewhat of a caricature of web designers and users, his depiction highlights the discrepancy between the
A central concern is the level of realism to target. Is it necessary to design avatars and the environments that they inhabit to be life-like, using rich graphics, or can simpler and more abstract forms be used, but which nonetheless are equally capable of engendering a sense of presence? Do you need to provide a visual representation of the arm and hands for holding objects for a self-avatar, or is it enough to have continuous movement of the object? Research has shown that it is possible for objects to appear to be moving with invisible hands as if they were present. This has been coined as the “tomato presence,” that is, where presence is maintained using a stand-in object in VR (for instance, a tomato). (See https://owlchemylabs .com/tomatopresence/.)
3D software toolkits are also available, making it much easier for developers and researchers to create virtual environments. The most popular is Unity. 3D worlds can be cre-ated using their APIs, toolkits, and physics engines to run on multiple platforms, for example, mobile, desktop, console, TV, VR, AR, and the Web.
7 . 2 I N T E R F A C E T y p E S 217
meticulous ways that designers create their websites and the rapid and less than systematic approach that users take to view them. To help navigate their way through the many choices that web developers have to make, Jason Beaird and James George (2014) have come up with a number of guidelines intended to help web developers achieve a balance between using color, layout and composition, texture, typography, and imagery. They also cover mobile and responsive web design. Other website guidelines are mentioned in Chapter 16.
Web designers now have a number of languages available to design websites, such as Ruby and Python. HTML5 and web development tools, such as JavaScript and CSS, are also used. Libraries, such as React, and open source toolkits, such as Bootstrap, enable developers to get started quickly when prototyping their ideas for a website. WordPress also provides users with an easy-to-use interface and hundreds of free templates to use as a basis when creating their own website. In addition, built-in optimization and responsive, mobile-ready themes are available. Customized web pages are available for smartphone browsers that pro-vide scrolling lists of articles, games, tunes, and so on, rather than hyperlinked pages.
Another interface element that has become an integral part of any website is breadcrumb navigation. Breadcrumbs are category labels that appear on a web page that enable users to peruse other pages without losing track of where they have come from (see Figure 7.14). The term comes from the way-finding technique that Hansel used in the Brothers Grimm fairy tale Hansel and Gretel. The metaphor conjures up the idea of leaving a path to follow back. Breadcrumbs are also used by search engine optimization tools that match up a user’s search terms with relevant web pages using the breadcrumbs. Breadcrumbs also extol usability in a number of ways, including helping users know where they are relative to the rest of the web-site, enabling one-click access to higher site levels, attracting first time visitors to continue to browse a website after having viewed the landing page (Mifsud, 2011). Therefore, using them is good practice for other web applications besides websites.
With the arrival of tablets and smartphones, web designers needed to rethink how to design web browsers and websites for them, as they realized the touchscreen affords a differ-ent interaction style than PC/laptops. The standard desktop interface was found to not work as well on a tablet or smartphone. In particular, the typical fonts, buttons, and menu tabs were too small and awkward to select when using a finger. Instead of double-clicking inter-face elements, as users do with a mouse or trackpad, tablet and smartphone screens enable finger tapping. The main methods of navigation are by swiping and pinching. A new style of website emerged that mapped better to this kind of interaction style but also one that the
Figure 7.14 A breadcrumb trail on the BestBuy website showing three choices made by the user to get to Smart Lights Source: https://www.bestbuy.ca
7 I N T E R F A C E S218
user could interact with easily when using a mouse and trackpad. Responsive websites were developed that change their layout, graphic design, font, and appearance depending on the screen size (smartphone, tablet, or PC) on which it was being displayed.
If you look at the design of many websites, you will see that the front page presents a banner at the top, a short promotional video about the company/product/service, arrows to the left or right to indicate where to flick to move through pages, and further details appear-ing beneath the home page that the user can scroll through. Navigation is largely done by swiping pages horizontally or scrolling up and down.
Tips on designing websites for tablets versus mobile phones can be found here:https://css-tricks.com/a-couple-of-best-practices-for-tablet-friendly-design/
BOX 7.3 In-Your-Face Web Ads
Web advertising has become pervasive and invasive. Advertisers realized how effective flash-ing and animated ads were for promoting their products, taking inspiration from the animated neon light advertisements used in city centers, such as London’s Piccadilly Circus. But since banner ads emerged in the 1990s, advertisers have become even more cunning in their tactics. In addition to designing even flashier banner ads, more intrusive kinds of web ads have begun to appear on our screens. Short movies and garish cartoon animations, often with audio, now pop up in floating windows that zoom into view or are tagged on at the front end of an online newspaper or video clip. Moreover, this new breed of in-your-face, often personalized web ads frequently requires the user either to wait until they end or to find a check box to close the window down. Sites that provide free services, such as Facebook, YouTube, and Gmail, are also populated with web ads. The problem for users is that advertisers pay significant revenues to online companies to have their advertisements placed on their websites, entitling them to say where, what, and how they should appear. One way users can avoid them is to set up ad blockers when browsing the web.
Research and Design ConsiderationsThere are numerous classic books on web design and usability (for example, Krug, 2014; Cooper et al., 2014). In addition, there are many good online sites offering guidelines and tips. For example, the BBC provides online guidance specifically for how to design responsive websites that includes topics such as context, accessibility, and modular design. See https://www.bbc.co.uk/gel/guidelines/how-to-design-for-the-web. Key design considerations for all websites are captured well by three core questions proposed by Keith Instone (quoted in Veen, 2001): Where am I? What’s here? Where can I go?
7 . 2 I N T E R F A C E T y p E S 219
7.2.6 Mobile DevicesMobile devices have become pervasive, with people increasingly using them in all aspects of their everyday and working lives—including phones, fitness trackers, and watches. Custom-ized mobile devices are also used by people in a diversity of work settings where they need access to real-time data or information while walking around. For example, they are now commonly used in restaurants to take orders, at car rental agencies to check in car returns, in supermarkets for checking stock, and on the streets for multiplayer gaming.
Larger-sized tablets are also used in mobile settings. For example, many airlines provide their flight attendants with one so that they can use their customized flight apps while air-borne and at airports; sales and marketing professionals also use them to demonstrate their products or to collect public opinions. Tablets and smartphones are also commonly used in classrooms that can be stored in special “tabcabbies” provided by schools for safe keeping and recharging.
Smartphones and smartwatches have an assortment of sensors embedded in them, such as an accelerometer to detect movement, a thermometer to measure temperature, and gal-vanic skin response to measure changes in sweat level on one’s skin. Other apps may be designed for fun. An example of an early app developed by magician Steve Sheraton simply for a moment of pleasure is iBeer (see Figure 7.15). Part of its success was due to the ingen-ious use of the accelerometer inside the phone. It detects the tilting of the iPhone and uses this information to mimic a glass of beer being consumed. The graphics and sounds are also very enticing; the color of the beer together with frothy bubbles and accompanying sound effects gives the illusion of virtual beer being swished around a virtual glass. The beer can be drained if the phone is tilted enough, followed by a belch sound when it has been finished.
ACTIVITy 7.5 Look at a fashion brand’s website, such as Nike, and describe the kind of interface used. How does it contravene the design principles outlined by Jeffrey Veen? Does it matter? For what type of user experience is it providing? What was your experience in engaging with it?
CommentFashion companies’ sites, like Nike, are often designed to be more like a cinematic experience and use rich multimedia elements, including videos, sounds, music, animations, and interactiv-ity. Branding is central. In this sense, it contravenes what are considered core usability guide-lines. Specifically, the site has been designed to entice the visitor to enter the virtual store and watch high-quality and innovative movies that show cool dudes wearing their products. Often, multimedia interactivities are embedded into the sites to help the viewer move to other parts of the site, for example by clicking on parts of an image or a video playing. Screen widg-ets are also provided, such as menus, skip over, and next buttons. It is easy to become immersed in the experience and forget that it is a commercial store. It is also easy to get lost and not to know—Where am I? What’s here? Where can I go? But this is precisely what companies such as Nike want their visitors to do and to enjoy: the experience.
7 I N T E R F A C E S220
Smartphones can also be used to download contextual information by scanning barcodes in the physical world. Consumers can instantly download product information by scanning barcodes using their iPhone when walking around a supermarket, including allergens, such as nuts, gluten, and dairy. For example, the GoodGuide app enables shoppers to scan prod-ucts in a store by taking a photo of their barcode to see how they rate for healthiness and impact on the environment. Others include concert tickets and location-based notifications.
Another method that provides quick access to relevant information is the use of quick response (QR) codes that store URLs and look like black-and-white checkered squares (see Figure 7.16). They work by people taking a picture using their camera phone that then takes them to a particular website. However, despite their universal appeal to companies as a way of providing additional information or special offers, not many people actually use them in practice. One of the reasons is that they can be slow, tricky, and cumbersome to use in situ. People have to download a QR reader app first, open it, and then try to hold it over the QR code to take a photo, which can take time to open up a webpage.
Figure 7.15 The iBeer smartphone app Source: Hottrix
Figure 7.16 QR code appearing on a magazine page
7 . 2 I N T E R F A C E T y p E S 221
ACTIVITy 7.6 Smartwatches, such those made by Google, Apple, and Samsung, provide a multitude of func-tions including fitness tracking, streaming music, texts, email, and the latest tweets. They are also context and location aware. For example, on detecting the user’s presence, promotional offers may be pinged to them from nearby stores, tempting them in to buy. How do you feel about this? Do you think it is the same or worse compared to the way advertisements appear on a user’s smartphone? Is this kind of context-based advertising ethical?
CommentSmartwatches are similar to smartphones in that they, too, get pinged with promotions and ads for nearby restaurants and stores. However, the main difference is that when worn on a wrist, smartwatches are ever-present; the user only needs to glance down at it to notice a new notification, whereas they have to take their phones out of their pockets and purses to see what new item has been pinged (although some people hold their smartphone permanently in their hands). This means that their attention is always being given to the device, which could make them susceptible to responding to notifications and spending more money. While some people might like to get 10 percent off on coffee if they walk into the cafe that has just sent them a digital voucher, for others such notifications may be seen as very annoying as they are constantly bombarded with promotions. Worse still, it could tempt children and vulnerable people who are wearing such a watch to spend money when perhaps they shouldn’t or to nag their parents or caretakers to buy it for them. However, smartwatch companies are aware of this potential problem, and they provide settings that the user can change in terms of the level and type of notifications they want to receive.
Research and Design ConsiderationsMobile interfaces typically have a small screen and limited control space. Designers have to think carefully about what type of dedicated hardware controls to include, where to place them on the device, and then how to map them to the software. Apps designed for mobile interfaces need to take into account the ability to navigate through content when using a mobile display is constrained, whether using touch, pen, or keypad input. The use of vertical and horizontal scrolling provides a rapid way of scanning though images, menus, and lists. A number of mobile browsers have also been developed that allow users to view and navigate the Internet, magazines, or other media in a more streamlined way. For example, Microsoft’s Edge browser was one of the first mobile browsers that was designed to make it easier to find, view, and manage content on the go. It provides a customized reading view that enables the user to re-organize the content of a web page to make it easier for them to focus on what they want to read. The trade-off, however, is that it makes it less obvious how to perform other functions that are no longer visible on the screen.
(Continued)
7 I N T E R F A C E S222
7.2.7 AppliancesAppliances include machines for everyday use in the home (for example, washing machines, microwave ovens, refrigerators, toasters, bread makers, and smoothie makers). What they have in common is that most people using them will be trying to get something specific done in a short period of time, such as starting a wash, watching a program, buying a ticket, or making a drink. They are unlikely to be interested in spending time exploring the interface or looking through a manual to see how to use the appliance. Many of them now have LED displays that provide multiple functions and feedback about a process (such as temperature, minutes remaining, and so on). Some have begun to be connected to the Internet with com-panion devices, enabling them to be controlled by remote apps. An example is a coffee maker that can be controlled to come on at a certain time from an app running on a smartphone or controlled by voice.
Another key concern for mobile display design is the size of the area on the display that the user touches to make something happen, such as a key, icon, button, or app. The space needs to be big enough for “all fingers” to press accurately. If the space is too small, the user may accidentally press the wrong key, which can be annoying. The average fingertip is between one and two centimeters wide, so target areas should be at least 7 mm to 10 mm so that they can be accurately tapped with a fingertip. Fitts’ law (see Chapter 16) is often used to help with evaluating hit area. In their developer design guidelines, Apple also suggests provid-ing ample touch targets for interactive elements, with a minimum tappable area of 44 pts. × 44 pts. for all controls.
A number of other guidelines exist providing advice on how to design interfaces for mobile devices (for instance, see Babich, 2018). An example is avoiding clutter by prioritizing one primary action per screen.
Research and Design ConsiderationsAlan Cooper et al. (2014) suggest that appliance interfaces require the designer to view them as transient interfaces, where the interaction is short. All too often, however, designers pro-vide full-screen control panels or an unnecessary array of physical buttons that serve to frus-trate and confuse the user where only a few, presented in a structured way, would be much better. Here the two fundamental design principles of simplicity and visibility are paramount. Status information, such as what the photocopier is doing, what the ticket machine is doing, and how much longer the wash is going to take should be provided in a simple form and at a prominent place on the interface. A key design question is: as soft displays increasingly become part of an appliance interface, for example, LCD and touchscreens, what are the trade-offs with replacing the traditional physical controls, such as dials, buttons, and knobs, with these soft display controls?
7 . 2 I N T E R F A C E T y p E S 223
ACTIVITy 7.7 Look at the controls on your toaster (or the one in Figure 7.17 if you don’t have one nearby) and describe what each does. Consider how these might be replaced with an LCD screen. What would be gained and lost from changing the interface in this way?
CommentStandard toasters have two main controls, the lever to press down to start the toasting and a knob to set the amount of time for the toasting. Many come with a small eject button that can be pressed if the toast starts to burn. Some also come with a range of settings for different ways of toasting (such as one side, frozen, and so forth), selected by moving a dial or press-ing buttons.
Designing the controls to appear on an LCD screen would enable more information and options to be provided, for example, only toast one slice, keep the toast warm, or automati-cally pop up when the toast is burning. It would also allow precise timing of the toasting in minutes and seconds. However, it is likely to increase the complexity of what previously was a set of logical and very simple actions. This has happened in the evolution of microwaves, washing machines, and tea kettles that have digital interfaces. They also offer many more options for warming food up, washing clothes, or the temperature to heat water. The down-side of increasing the number of choices, especially when the interface is not designed well to support this, is that it can make for a more difficult user experience for mundane tasks.
Figure 7.17 A typical toaster with basic physical controls Source: https://uk.russellhobbs.com/product/brushed-stainless-steel-toaster-2-slice
7 I N T E R F A C E S224
7.2.8 Voice User InterfacesA voice user interface (VUI) involves a person talking with a spoken language app, such as a search engine, a train timetable, a travel planner, or a phone service. It is commonly used for inquiring about specific information (for instance, flight times or the weather) or issuing a command to a machine (such as asking a smart TV to select an Action movie or asking a smart speaker to play some upbeat music). Hence, VUIs use an interaction type of command or conversation (see Chapter 3), where users speak and listen to an interface rather than click on, touch, or point to it. Sometimes, the interaction style can involve the user responding where the system is proactive and initiates the conversation, for example, asking the user if they would like to stop watching a movie or listen to the latest breaking news.
The first generation of speech systems earned a reputation for mishearing all too often what a person said (see cartoon). However, they are now much more sophisticated and have higher levels of recognition accuracy. Machine learning algorithms have been developed that are continuing to improve their ability to recognize what someone is saying. For speech out-put, actors are often used to record answers, messages, and prompts, which are much friend-lier, more convincing, and more pleasant than the artificially-sounding synthesized speech that was typically used in the early systems.
VUIs have become popular for a range of apps. Speech-to-text systems, such as Dragon, enable people to dictate rather than have to type, whether it is entering data into a spread-sheet, using a search engine, or writing a document. The words spoken appear on the screen. For some people, this mode of interaction is more efficient, especially when they are on the move. Dragon claims on their website that it is three times faster than typing and it is 99 percent accurate. Speech technology is also used by people with visual impairments, includ-ing speech recognition word processors, page scanners, web readers, and VUIs for operating home control systems, including lights, TV, stereo, and other home appliances.
One of the most popular applications of speech technology is call routing, where com-panies use an automated speech system to enable users to reach one of their services during a phone call. Callers voice their needs in their own words, for example, “I’m having problems with my Wi-Fi router,” and in response are automatically forwarded to the appropriate service (Cohen et al., 2004). This is useful for companies, as it can reduce operating costs. It can also increase revenue by reducing the number of lost calls. The callers may be happier, as their call can be routed to an available agent (real or virtual) rather than being lost or sent to voicemail.
Source: Reproduced with permission of King Features Syndicate
7 . 2 I N T E R F A C E T y p E S 225
In human conversations, people often interrupt each other, especially if they know what they want, rather than waiting for someone to go through a series of options. For example, they may stop the waitress at a restaurant in midflow when describing the specials if they know what they want, rather than let her go through the entire list. Similarly, speech technology has been designed with a feature called barge-in that allows callers to interrupt a system message and provide their request or response before the message has finished playing. This can be useful if the system has numerous options from which the caller may choose, and the chooser knows already what they want.
There are several ways that a VUI dialog can be structured. The most common is a directed dialogue where the system is in control of the conversation, asking specific questions and requiring specific responses, similar to filling in a form (Cohen et al., 2004):
System: Which city do you want to fly to?Caller: LondonSystem: Which airport: Gatwick, Heathrow, Luton, Stansted, or City?Caller: GatwickSystem: What day do you want to depart?Caller: Monday next week.System: Is that Monday, May 5?Caller: Yes
Other systems are more flexible, allowing the user to take more initiative and specify more information in one sentence (for example, “I’d like to go to Paris next Monday for two weeks”). The problem with this approach is that there is more chance for error, since the caller might assume that the system can follow all of their needs in one pass as a real travel agent would (for example, “I’d like to go to Paris next Monday for two weeks, and would like the cheapest possible flight, preferably leaving from Gatwick airport and definitely with no stop-overs …”). The list is simply too long and would overwhelm the system’s parser. Carefully guided prompts can be used to get callers back on track and help them speak appropriately (for instance, “Sorry, I did not get all that. Did you say you wanted to fly next Monday?”).
A number of speech-based phone apps exist that enable people to use them while mobile, making them more convenient to use than text-based entry. For example, people can voice queries into their phone using Google Voice or Apple Siri rather than entering text manually. Mobile translators allow people to communicate in real time with others who speak a dif-ferent language by letting a software app on their phone do the talking (for example, Google Translate). People speak in their own language using their phone while the software trans-lates what each person is saying into the language of the other one. Potentially, this means people from all over the world (there are more than 6,000 languages) can talk to one another without having to learn another language.
Voice assistants, like Amazon’s Alexa and Google Home, can be instructed by users to entertain in the home by telling jokes, playing music, keeping track of time, and enabling users to play games. Alexa also offers a range of “skills,” which are voice-driven capabilities intended to provide a more personalized experience. For example, “Open the Magic Door” is an interactive story skill that allows users to choose their path in a story by selecting differ-ent options through the narrative. Another one, “Kids court,” allows families to settle argu-ments in an Alexa-run court while learning about the law. Many of the skills are designed
7 I N T E R F A C E S226
to support multiple users taking part at the same time, offering the potential for families to play together. Social interaction is encouraged by the smart speaker that houses Alexa or Home. Smart speakers sit in a common space for all to use (similar to a toaster or refrigera-tor). In contrast, handheld devices, such as a smartphone or tablet, support only single use and ownership.
Despite advances in speech recognition, conversational interaction is limited mainly to answering questions and responding to requests. It can be difficult for VUIs to recognize children’s speech, which is not as articulate as adults. For example, Druga et al. (2017) found that young children (3–4 years old) experienced difficulty interacting with conversational and chat agents, resulting in them becoming frustrated. Also, voice assistants don’t always recognize who is talking in a group, such as a family, and always need to be called by their name each time someone wants to interact with it. There is still a way to go before voice assistant interaction resembles human conversation.
7.2.9 Pen-Based DevicesPen-based devices enable people to write, draw, select, and move objects on an interface using light pens or styluses that capitalize on the well-honed drawing and writing skills that are devel-oped from childhood. They have been used to interact with tablets and large displays, instead of mouse, touch, or keyboard input, for selecting items and supporting freehand sketching. Digital ink, such as Anoto, uses a combination of an ordinary ink pen with a digital camera that
Research and Design ConsiderationsKey research questions are what conversational mechanisms to use to structure the voice user interface and how human-like they should be. Some researchers focus on how to make it appear natural (that is, like human conversation), while others are concerned more with how to help people navigate efficiently through a menu system by enabling them to recover easily from errors (their own or the system’s), to be able to escape and go back to the main menu (similar to the undo button of a GUI), and to guide those who are vague or ambiguous in their requests for information or services using prompts. The type of voice actor, male, female, neutral, or dialect, and form of pronunciation are also topics of research. Do people prefer to listen to and are more patient with a female or male voice? What about one that is jolly versus one that is serious?
Michael Cohen et al. (2004) discuss the pros and cons of using different techniques for structuring the dialogue and managing the flow of voice interactions, the different ways of expressing errors, and the use of conversational etiquette—all still relevant for today’s VUIs. A number of commercial guidelines are available for voice interfaces. For example, Cathy Pearl (2016) has written a practical book that provides a number of VUI design principles and topics, including which speech recognition engine to use, how to measure the performance of VUIs, and how to design VUIs for different interfaces, for example, a mobile app, toy, or voice assistant.
7 . 2 I N T E R F A C E T y p E S 227
digitally records everything written with the pen on special paper (see Figure 7.18). The pen works by recognizing a special nonrepeating dot pattern that is printed on the paper. The non-repeating nature of the pattern means that the pen is able to determine which page is being written on and where on the page the pen is pointing. When writing on digital paper with a digital pen, infrared light from the pen illuminates the dot pattern, which is then picked up by a tiny sensor. The pen decodes the dot pattern as the pen moves across the paper and stores the data temporarily in the pen. The digital pen can transfer data that has been stored in the pen via Bluetooth or a USB port to a computer. Handwritten notes can also be converted and saved as standard typeface text. This can be useful for applications that require people to fill in paper-based forms and also for taking notes during meetings.
Another advantage of digital pens is that they allow users to annotate existing docu-ments, such as spreadsheets, presentations, and diagrams quickly and easily in a similar way to how they would do this when using paper-based versions. This is useful for a team who is working together and communicating to each other from different locations. One problem with using pen-based interactions on small screens, however, is that sometimes it can be difficult to see options on the screen because a user’s hand can obscure part of it when writing.
Figure 7.18 The Anoto pen being used to fill in a paper form and a schematic showing its internal components Source: www.grafichewanda.it/anoto.php?language=EN
7 I N T E R F A C E S228
7.2.10 TouchscreensSingle touchscreens, used in walk-up kiosks such as ticket machines or museum guides, ATMs, and cash registers (for instance, restaurants), have been around for a while. They work by detecting the presence and location of a person’s touch on the display; options are selected by tapping on the screen. Multitouch surfaces, on the other hand, support a much wider range of more dynamic fingertip actions, such as swiping, flicking, pinching, pushing, and tapping. They do this by registering touches at multiple locations using a grid (see Figure 7.19). This multitouch method enables devices, such as smartphones and tabletops, to recognize and respond to more than one touch at the same time. This enables users to use multiple digits to perform a variety of actions, such as zooming in and out of maps, moving photos, selecting letters from a virtual keyboard when writing, and scrolling through lists. Two hands can also be used together to stretch and move objects on a tabletop surface, similar to how both hands are used to stretch an elastic band or scoop together a set of objects.
BOX 7.4 Electronic Ink
Digital ink is not to be confused with the term electronic ink (or e-ink). Electronic ink is a dis-play technology designed to mimic the appearance of ordinary ink on paper used in e-readers, such as the Kindle. The display used reflects light like ordinary paper.
Figure 7.19 A multitouch interface Source: www.sky-technology.eu/en/blog/article/item/multi-touch-technology-how-it-works.html
7 . 2 I N T E R F A C E T y p E S 229
The flexibility of interacting with digital content afforded by finger gestures has resulted in many ways of experiencing digital content. This includes reading, scanning, zooming, and searching interactive content on tablets, as well as creating new digital content.
7.2.11 Gesture-Based SystemsGestures involve moving arms and hands to communicate (for instance, waving to say good-bye or raising an arm to speak in class) or to provide information to someone (for example, holding two hands apart to show the size of something). There has been much interest in how technology can be used to capture and recognize a user’s gestures for input by tracking them using cameras and then analyzing them using machine learning algorithms.
David Rose (2018) created a video that depicts many sources of inspiration for where gesture is used in a variety of contexts, including those made by cricket umpires, live concert signers for the deaf, rappers, Charlie Chaplin, mime artists, and Italians. His team at IDEO developed a gesture system to recognize a small set of gestures and used these to control a Philips HUE light set and a Spotify station. They found that gestures need to be sequential to be understood in the way a sentence is composed of a noun, then verb, and object plus operation. For example, for “speaker, on,” they used a gesture on one hand to designate the
Research and Design ConsiderationsTouchscreens have become pervasive, increasingly becoming the main interface that many people use on a daily basis. However, they are different from GUIs, and a central design con-cern is what types of interaction techniques to use to best support different activities. For example, what is the optimal way to enable users to choose from menu options, find files, save documents, and so forth, when using a touch interface? These operations are well mapped to interaction styles available in a GUI, but it is not as obvious how to support them on a touch interface. Alternative conceptual models have been developed for the user to carry out these actions on the interface, such as the use of cards, carousels, and stacks (see Chapter 3). The use of these objects enables users to swipe and move through digital content quickly. However, it is also easy to swipe too far when using a carousel. Typing on a virtual keyboard with two thumbs or one fingertip is also not as fast and efficient as using both hands when using a con-ventional keyboard, although many people have learned to be very adept at pecking at virtual keys on a smartphone. Predictive text can also be used to help people type faster.
Both hands may be used on multitouch tabletops to enable users to make digital objects larger and smaller or to rotate them. Dwelling touches (pressing and holding a finger down) can also be used to enable a user to perform dragging actions and to bring up pop-up menus. One or more fingers can also be used together with a dwell action to provide a wider range of gestures. However, these can be quite arbitrary, requiring users to learn them rather than being intuitive. Another limitation of touchscreens is that they do not provide tactile feedback in the same way that keys or mice do when pressed. To compensate, visual, audio, and haptic feedback can be used. See also the section on shareable interfaces for more background on multitouch design considerations.
7 I N T E R F A C E S230
noun, and another on the other hand to designate the verb. So, to change the volume, the user needs to point to a speaker with their left hand while raising their right hand to signal turn the volume up.
One area where gesture interaction has been developed is in the operating room. Sur-geons need to keep their hands sterile during operations but also need to be able to look at X-rays and scans during an operation. However, after being scrubbed and gloved, they need to avoid touching any keyboards, phones, and other nonsterile surfaces. A far from ideal workaround is to pull their surgical gown over their hands and manipulate a mouse through the gown. As an alternative, Kenton O’Hara et al. (2013) developed a touchless gesture-based system, using Microsoft’s Kinect technology, which recognized a range of gestures that sur-geons could use to interact with and manipulate MRI or CT images, including single-handed gestures for moving forward or backward through images, and two-handed gestures for zooming and panning (see Figure 7.20).
Figure 7.20 Touchless gesturing in the operating theater Source: Used courtesy of Kenton O’Hara
Watch David Rose’s inspirations for gesture video at https://vimeo.com/224522900.
7 . 2 I N T E R F A C E T y p E S 231
7.2.12 Haptic InterfacesHaptic interfaces provide tactile feedback, by applying vibration and forces to the person, using actuators that are embedded in their clothing or a device that they are carrying, such as a smartphone or smartwatch. Gaming consoles have also employed vibration to enrich the experience. For example, car steering wheels that are used with driving simulators can vibrate in various ways to provide the feel of the road. As the driver makes a turn, the steering wheel can be programmed to feel like it is resisting—in the way that a real steering wheel does.
Vibrotactile feedback can also be used to simulate the sense of touch between remote people who want to communicate. Actuators embedded in clothing can be designed to re-create the sensation of a hug or a squeeze by being buzzed on various parts of the body. Another use of haptics is to provide real-time feedback to guide people when learning a musical instrument, such as a violin or drums. For example, the MusicJacket (van der Linden et al., 2011) was developed to help novice violin players learn how to hold their instrument correctly and develop good bowing action. Vibrotactile feedback was provided via the jacket to give nudges at key places on the arm and torso to inform the student when they were either holding their violin incorrectly or their bowing trajectory had deviated from a desired path (see Figure 7.21). A user study with novice players showed that they were able to react to the vibrotactile feedback and adjust their bowing or their posture in response.
Another form of feedback is called ultrahaptics, which creates the illusion of touch in midair. It does this by using ultrasound to make three-dimensional shapes and textures that can be felt but not seen by the user (www.ultrahaptics.com). This technique can be used to create the illusion of having buttons and sliders that appear in midair. One potential use is in the automotive industry to replace existing physical buttons and knobs or touch-screens. The ultra-haptic buttons and knobs can be designed to appear next to the driver when needed, for example, when detecting the driver wants to turn down the volume or change the radio station.
Research and Design ConsiderationsA key design concern for using gestural input is to consider how a computer system recognizes and delineates the user’s gestures. In particular, how does it determine the start and end point of a hand or arm movement, and how does it know the difference between a deictic gesture (a deliberate pointing movement) and hand waving (an unconscious gesticulation) that is used to emphasize what is being said verbally?
In addition to being used as a form of input, gestures can be represented as output to show real-time avatar movement or someone’s own arm movements. Smartphones, laptops, and some smart speakers (for example, Facebook’s Portal) have cameras that can perceive three dimensions and record a depth for every pixel. This can be used to create a represen-tation of someone in a scene, for example, how they are posing and moving, and also to respond to their gestures. One design question that this raises is how realistic must the mir-rored graphical representation of the user be in order for them to be believable and for the user to connect their gestures with what they are seeing on the screen.
7 I N T E R F A C E S232
Haptics are also being embedded into clothing, sometimes called exoskeletons. Inspired by the “right trousers” in the Wallace and Gromit animated short movie, Jonathan Rossiter and his team (2018) developed a new kind of exoskeleton that can help people stand up and move around using artificial muscles that consist of air bubbles which are activated using tiny electric motors (see Figure 7.22). These are stiffened or relaxed using grapheme parts to make the trousers move. One application area is to help people who have walking difficulties and those who need to exercise but find it difficult to do so.
7.2.13 Multimodal InterfacesMultimodal interfaces are intended to enrich user experiences by multiplying the way infor-mation is experienced and controlled at the interface through using different modalities, such as touch, sight, sound, and speech (Bouchet and Nigay, 2004). Interface techniques that have been combined for this purpose include speech and gesture, eye-gaze and gesture, haptic and audio output, and pen input and speech (Dumas et al., 2009). The assumption is that multimodal interfaces can support more flexible, efficient, and expressive means of human– computer interaction that are more akin to the multimodal experiences that humans encoun-ter in the physical world (Oviatt, 2017). Different input/outputs may be used at the same time, for example, using voice commands and gestures simultaneously to move through a virtual environment, or alternately using speech commands followed by gesturing. The most common combination of technologies used for multimodal interfaces is speech and vision processing (Deng and Huang, 2004). Multimodal interfaces can also be combined with mul-tisensor input to enable other aspects of the human body to be tracked. For example, eye gaze, facial expressions, and lip movements can also be tracked to provide data about a user’s
Figure 7.21 The MusicJacket with embedded actuators that nudge the player to move their arm up to be in the correct positionSource: Helen Sharp
7 . 2 I N T E R F A C E T y p E S 233
Figure 7.22 Trousers with artificial muscles that use a new kind of bubble haptic feedback Source: Used courtesy of The Right Trousers Project: Wearable Soft Robotics for Independent Living
Research and Design ConsiderationsHaptics are now commonly used in gaming consoles, smartphones, and controllers to alert or heighten a user experience. Haptic feedback is also being developed in clothing and other wearables as a way of simulating being touched, stroked, prodded, or buzzed. A promising application area is sensory-motor skills, such as in sports training and learning to play a musi-cal instrument. For example, patterns of vibrations have been placed across snowboarders’ bodies to indicate which moves to take while snowboarding. A study reported faster reaction times than when the same instructions were given verbally (Spelmezan et al., 2009). Other uses are posture trainers that buzz when a user slouches and fitness trackers that also buzz when they detect that their users have not taken enough steps in the past hour.
(Continued)
7 I N T E R F A C E S234
attention or other behavior. This kind of sensing can provide input for customizing user inter-faces and experiences to the perceived need, desire, or level of interest.
A person’s body movement can also be tracked so that it can be represented back to them on a screen in the form of an avatar that appears to move just like them. For example, the Kinect was developed as a gesture and body movement gaming input system for the Xbox. Although now defunct in the gaming industry, it proved effective at detecting multimodal input in real time. It consisted of an RGB camera for facial and gesture recognition, a depth sensor (an infrared projector paired with a monochrome camera) for movement tracking, and downward-facing mics for voice recognition (see Figure 7.23). The Kinect looked for
Figure 7.23 Microsoft’s Xbox Kinect Source: Stephen Brashear / Invision for Microsoft / AP Images
A key design question is where best to place the actuators on the body, whether to use a single or a sequence of touches, when to activate, and at what intensity and how often to use them to make the feeling of being touched convincing (e.g., Jones and Sarter, 2008). Providing continuous haptic feedback would be simply too annoying. People would also habituate too quickly to the feedback. Intermittent buzzes can be effective at key moments when a person needs to attend to something but not necessarily tell them what to do. For example, a study by Johnson et al. (2010) of a commercially available haptic device, intended to improve posture by giving people a vibrotactile buzz when they slouched, found that while the buzzing did not show them how to improve their posture, it did improve their body awareness.
Different kinds of buzzes can also be used to indicate different tactile experiences that map to events; for example, a smartphone could transmit feelings of slow tapping to feel like water dropping, which is meant to indicate it is about to rain and transmit the sensation of heavy tapping to indicate a thunderstorm is looming.
7 . 2 I N T E R F A C E T y p E S 235
someone’s body. On finding it, it locked onto it and measured the three-dimensional position-ing of the key joints in their body. This information was converted into a graphical avatar of the user that could be programmed to move just like them. Many people readily saw them-selves as the avatar and learnt how to play games in this manner.
7.2.14 Shareable InterfacesShareable interfaces are designed for more than one person to use. Unlike PCs, laptops, and mobile devices, which are aimed at single users, shareable interfaces typically provide mul-tiple inputs and sometimes allow simultaneous input by collocated groups. These include large wall displays, for example SmartBoards (see Figure 7.24a), where people use their own pens or gestures, and interactive tabletops, where small groups can interact with informa-tion being displayed on the surface using their fingertips. Examples of interactive tabletops include Smart’s SmartTable and Circle Twelve’s DiamondTouch (Dietz and Leigh, 2001; see Figure 7.24b). The DiamondTouch tabletop is unique in that it can distinguish between dif-ferent users touching the surface concurrently. An array of antennae is embedded in the touch surface and each one transmits a unique signal. Each user has their own receiver embedded in a mat on which they’re standing or a chair in which they’re sitting. When a user touches the tabletop, very small signals are sent through the user’s body to their receiver that identifies which antenna has been touched and sends this to the computer. Multiple users can interact simultaneously with digital content using their fingertips.
An advantage of shareable interfaces is that they provide a large interactional space that can support flexible group working, enabling groups to create content together at the same time. Compared with a co-located group trying to work around a single-user PC or laptop, where typically one person takes control, making it more difficult for others to take part,
Research and Design ConsiderationsMultimodal systems rely on recognizing aspects of a user’s behavior, including handwriting, speech, gestures, eye movements, or other body movements. In many ways, this is much harder to accomplish and calibrate than single modality systems that are programmed to recognize one aspect of a user’s behavior. The most researched modes of interaction are speech, gesture, and eye-gaze tracking. A key research question is what is actually gained from combining dif-ferent input and outputs and whether talking and gesturing as humans do with other humans is a natural way of interacting with a computer (see Chapter 4). Guidelines for multimodal design can be found in Reeves et al. (2004) and Oviatt et al. (2017).
Watch this video of Circle Twelve’s demonstration of the Diamond Touch tabletop: http://youtu.be/S9QRdXlTndU.
7 I N T E R F A C E S236
multiple users can interact with large display. Users can point to and touch the informa-tion being displayed, while simultaneously viewing the interactions and having the same shared point of reference (Rogers et al., 2009). There are now a number of tabletop apps that have been developed for museums and galleries which enable visitors to learn about various aspects of the environment (see Clegg et al., 2019).
(a)
(b)
Figure 7.24 (a) A SmartBoard in use during a meeting and (b) Mitsubishi’s interactive tabletop interface Source: (a) Used courtesy of SMART Technologies Inc. (b) Mitsubishi Electric Research Labs
7 . 2 I N T E R F A C E T y p E S 237
Another type of shareable interface is software platforms that enable groups of people to work together simultaneously even when geographically apart. Early examples included shared editing tools developed in the 1980s (for example, ShRedit). Various commercial products now exist that enable multiple remote people to work on the same document at the same time (such as Google Docs and Microsoft Excel). Some enable up to 50 people to edit the same document at the same time with more watching on. These software programs provide various functions, such as synchronous editing, tracking changes, annotating, and commenting. Another collaborative tool is the Balsamiq Wireframes editor, which provides a range of shared functions, including collaborative editing, threaded comments with callouts, and project history.
Research and Design ConsiderationsEarly research on shareable interfaces focused largely on interactional issues, such as how to support electronically based handwriting and drawing, and the selecting and moving of objects around the display (Elrod et al., 1992). The PARCTAB system (Schilit et al., 1993) investigated how information could be communicated between palm-sized, A4-sized, and whiteboard-sized displays using shared software tools, such as Tivoli (Rønby-Pedersen et al., 1993). Another concern was how to develop fluid and direct styles of interaction with large displays, both wall-based and tabletop, involving freehand and pen-based gestures (see Shen et al., 2003). Current research is concerned with how to support ecologies of devices so that groups can share and create content across multiple devices, such as tabletops and wall dis-plays (see Brudy et al., 2016).
A key research issue is whether shareable surfaces can facilitate new and enhanced forms of collaborative interaction compared with what is possible when groups work together using their own devices, like laptops and PCs (see Chapter 5, “Social Interaction”). One benefit is easier sharing and more equitable participation. For example, tabletops have been designed to support more effective joint browsing, sharing, and manipulation of images during decision-making and design activities (Shen et al., 2002; Yuill and Rogers, 2012). Core design concerns include whether size, orientation, and shape of the display have an effect on collaboration. User studies have shown that horizontal surfaces compared with vertical ones support more turn-taking and collaborative working in co-located groups (Rogers and Lindley, 2004), while providing larger-sized tabletops does not necessarily improve group working but can encour-age a greater division of labor (Ryall et al., 2004).
The need for both personal and shared spaces has been investigated to see how best to enable users to move between working on their own and together as a group. Several researchers have designed cross-device systems, where a variety of devices, such as tablets, smartphones, and digital pens can be used in conjunction with a shareable surface. For example, SurfaceConstellations was developed for linking mobile devices to create novel cross-device workspace environments (Marquardt et al., 2018). Design guidelines and sum-maries of empirical research on tabletops and multitouch devices can be found in Müller-Tomfelde (2010).
7 I N T E R F A C E S238
7.2.15 Tangible InterfacesTangible interfaces use sensor-based interaction, where physical objects, such as bricks, balls, and cubes, are coupled with digital representations (Ishii and Ullmer, 1997). When a person manipulates the physical object(s), it is detected by a computer system via the sensing mecha-nism embedded in the physical object, causing a digital effect to occur, such as a sound, ani-mation, or vibration (Fishkin, 2004). The digital effects can take place in a number of media and places, or they can be embedded in the physical object itself. For example, Oren Zucker-man and Mitchel Resnick’s (2005) early Flow Blocks prototype depicted changing numbers and lights that were embedded in the blocks, depending on how they were connected. The flow blocks were designed to simulate real-life dynamic behavior and react when arranged in certain sequences.
Another type of tangible interface is where a physical model, for example, a puck, a piece of clay, or a model, is superimposed on a digital desktop. Moving one of the physical pieces around the tabletop causes digital events to take place on the tabletop. One of the earliest tangible interfaces, Urp, was built to facilitate urban planning; miniature physical models of buildings could be moved around on the tabletop and used in combination with tokens for wind and shadow-generating tools, causing digital shadows surrounding them to change over time and visualizations of airflow to vary. Tangible interfaces differ from other approaches, such as mobile, insofar as the representations are artifacts in their own right that the user can directly act upon, lift up, rearrange, sort, and manipulate.
The technologies that have been used to create tangibles include RFID tags and sensors embedded in physical objects and digital tabletops that sense the movements of objects and subsequently provide visualizations surrounding the physical objects. Many tangible systems have been built with the goal of encouraging learning, design activities, playfulness, and col-laboration. These include planning tools for landscape and urban planning (see Hornecker, 2005; Underkoffler and Ishii, 1998). Another example is Tinkersheets, which combine tangi-ble models of shelving with paper forms for exploring and solving warehouse logistics prob-lems (Zufferey, et al., 2009). The underlying simulation allows students to set parameters by placing small magnets on the form.
Tangible computing has been described as having no single locus of control or interac-tion (Dourish, 2001). Instead of just one input device, such as a mouse, there is a coordinated interplay of different devices and objects. There is also no enforced sequencing of actions and no modal interaction. Moreover, the design of the interface objects exploits their affordances to guide the user in how to interact with them. A benefit of tangibility is that physical objects and digital representations can be positioned, combined, and explored in creative ways, ena-bling dynamic information to be presented in different ways. Physical objects can also be held in both hands and combined and manipulated in ways not possible using other interfaces. This allows for more than one person to explore the interface together and for objects to be placed on top of each other, beside each other, and inside each other; the different configura-tions encourage different ways of representing and exploring a problem space. In so doing, people are able to see and understand situations differently, which can lead to greater insight, learning, and problem-solving than with other kinds of interfaces (Marshall et al., 2003).
A number of toolkits have been developed to encourage children to learn coding, electronics, and STEM subjects. These include littleBits (https://littlebits.com/), MicroBit (https://microbit.org/), and MagicCubes (https://uclmagiccube.weebly.com/). The toolkits
7 . 2 I N T E R F A C E T y p E S 239
provide children with opportunities to connect physical electronic components and sen-sors to make digital events occur. For example, the MagicCubes can be programmed to change color depending on the speed at which they are shaken; slow is blue and very fast is multicolor. Research has shown that the tangible toolkits provide many opportunities for discovery learning, exploration, and collaboration (Lechelt et al., 2018). The Cubes have been found to encourage a diversity of children, between the ages of 6 and 16, and those with cognitive disabilities, to learn through collaborating, frequently showing and telling each other and their instructors about their discoveries. These moments are facilitated by the cube’s form factor, making it easy to show off to others, for example, by waving a cube in the air (see Figure 7.25).
Tangible toolkits have also been developed for the visually impaired. For example, Torino (renamed by Microsoft to Code Jumper) was developed as a programming language for teaching programming concepts to children age 7–11, regardless of level of vision (Morrison et al., 2018). It consists of a set of beads that can be connected and manipulated to create physical strings of code that play stories or music.
Figure 7.25 Learning to code with the MagicCubes; sharing, showing, and tellingSource: Elpida Makriyannis
7 I N T E R F A C E S240
BOX 7.5 VoxBox—A Tangible Questionnaire Machine
Traditional methods for gathering public opinions, such as surveys, involve approaching peo-ple in situ, but it can disrupt the positive experience they are having. VoxBox (see Figure 7.26) is a tangible system designed to gather opinions on a range of topics in situ at an event through playful and engaging interaction (Golsteijn et al., 2015). It is intended to encour-age wider participation by grouping similar questions, encouraging completion, gathering answers to open and closed questions, and connecting answers and results. It was designed as a large physical system that provides a range of tangible input mechanisms through which people give their opinions, instead of using, for example, text messages or social media input. The various input mechanisms include sliders, buttons, knobs, and spinners about which peo-ple are all familiar. In addition, the system has a transparent tube at the side that drops a ball step by step as sets of questions are completed to act as an incentive for completion and as a progress indicator. The results of the selections are aggregated and presented as simple digital visualizations on the other side (for example, 95 percent are engaged; 5 percent are bored). VoxBox has been used at a number of events drawing in the crowds, who become completely absorbed in answering questions in this tangible format.
Figure 7.26 VoxBox—front and back of the tangible machine questionnaire Source: Yvonne Rogers
7 . 2 I N T E R F A C E T y p E S 241
7.2.16 Augmented RealityAugmented reality (AR) became an overnight success with the arrival of Pokémon Go in 2016. The smartphone app became an instant hit worldwide. Using a player’s smartphone camera and GPS signal, the AR game makes it seem as if virtual Pokémon characters are appearing in the real world—popping up all over the place, such as on buildings, on streets, and in parks. As players walk around a given place, they may be greeted with rustling bits of grass that signal a Pokémon nearby. If they walk closer, a Pokémon may pop up on their smartphone screen, as if by magic, and look as if they are actually in front of them. For exam-ple, one might be spotted sitting on a branch of a tree or a garden fence.
AR works by superimposing digital elements, like Pokémons, onto physical devices and objects. Closely related to AR is the concept of mixed reality, where views of the real world are combined with views of a virtual environment (Drascic and Milgram, 1996). To begin, augmented reality was mostly a subject of experimentation within medicine, where virtual objects, for example X-rays and scans, were overlaid on part of a patient’s body to aid the physician’s understanding of what was being examined or operated on.
AR was then used to aid controllers and operators in rapid decision-making. One exam-ple is air traffic control, where controllers are provided with dynamic information about the aircraft in their section that is overlaid on a video screen showing real planes landing, taking off, and taxiing. The additional information enables the controllers to identify planes easily,
Research and Design ConsiderationsResearchers have developed conceptual frameworks that identify the novel and specific fea-tures of a tangible interface (see Fishkin, 2004; Ullmar et al., 2005; Shaer and Hornecker, 2010). A key design concern is what kind of coupling to use between the physical action and digital effect. This includes determining where the digital feedback is provided in rela-tion to the physical artifact that has been manipulated. For example, should it appear on top of the object, beside it, or in some other place? The type and placement of the digital media will depend to a large extent on the purpose of using a tangible interface. If it is to support learning, then an explicit mapping between action and effect is critical. In contrast, if it is for entertainment purposes, for example, playing music or storytelling, then it may be better to design them to be more implicit and unexpected. Another key design question is what kind of physical artifact to use to enable the user to carry out an activity in a natural way. Bricks, cubes, and other component sets are most commonly used because of their flexibility and simplicity, enabling people to hold them in both hands and to construct new structures that can be easily added to or changed. Sticky notes and cardboard tokens can also be used for placing material onto a surface that is transformed or attached to digital content (Klemmer et al. 2001; Rogers et al., 2006).
Another research question is with what types of digital outputs should tangible interfaces be combined? Overlaying physical objects with graphical feedback that changes in response to how the object is manipulated has been the main approach. In addition, audio and haptic feedback has also been used. Tangibles can also be designed to be an integral part of a multi-modal interface.
7 I N T E R F A C E S242
which were difficult to make out—something especially useful in poor weather conditions. Similarly, head-up displays (HUDs) are used in military and civil planes to aid pilots when landing during poor weather conditions. A HUD provides electronic directional markers on a fold-down display that appears directly in the field of view of the pilot. A number of high-end cars now provide AR windshield technology, where navigation directions can literally look like they are painted on the road ahead of the driver (see Chapter 2, “The Process of Interaction Design”).
Instructions for building or repairing complex equipment, such as photocopiers and car engines, have also been designed to replace paper-based manuals, where drawings are super-imposed upon the machinery itself, telling the mechanic what to do and where to do it. There are also many AR apps available now for a range of contexts, from education to car navigation, where digital content is overlaid on geographic locations and objects. To reveal the digital information, users open the AR app on a smartphone or tablet and the content appears superimposed on what is viewed through the screen.
Other AR apps have been developed to aid people walking in a city or town. Directions (in the form of a pointing hand or arrow) and local information (for instance, the nearest bakery) are overlaid on the image of the street ahead that appears on someone’s smart-phone screen. These change as the person walks up the street. Virtual objects and infor-mation are also being combined to make more complex augmented realities. Figure 7.27 shows a weather alert with animated virtual lightning effects alongside information about a nearby café and the price of properties for sale or rent on a street. Holograms of people and other objects are also being introduced into AR environments that can appear to move
Figure 7.27 Augmented reality overlay used on a car windshieldSource: https://wayray.com
7 . 2 I N T E R F A C E T y p E S 243
and/or talk. For example, virtual tour guides are beginning to appear in museums, cities, and theme parks, which can appear to by moving, talking, or gesturing to visitors who are using an AR app.
The availability of mapping platforms, such as those provided by Niantics and Google, together with Apple’s ARKit, SparkAR Studio, and Google’s ARCore, has made it easier for developers and students alike to develop new kinds of AR games and AR apps. Another popular AR game that has emerged since Pokémon Go is Jurassic World Alive, where players walk around in the real world to find as many virtual dinosaurs as they can. It is similar to Pokémon Go but with different gaming mechanisms. For example, players have to study the dinosaurs they come across by collecting their DNA and then re-creating it. Microsoft’s Hololens toolkit has also enabled new mixed reality user experiences to be created, allowing users to create or interact with virtual elements in their surroundings.
Most AR apps use the backward-facing camera on a smartphone or tablet to overlay the virtual content onto the real world. Another approach is to use the forward-facing camera (used for selfies) to superimpose digital content onto the user’s face or body. The most popular app that has used this technique is SnapChat, which provides numerous filters with which people can experiment plus the opportunity to create their own filters. Adding accessories such as ears, hair, moving lips, and headgear enables people to trans-form their physical appearance in all sorts of fun ways.
These kinds of virtual try-ons work by analyzing the user’s facial features and build-ing a 2-D or 3-D model in real time. So, when they move their head, the make-up or acces-sories appear to move with them as if they are really on their face. Several AR mirrors now exist in retail that allow shoppers to try on sunglasses, jewelry, and make-up. The goal is to let them try on as many different products as they like to see how they look with them on. Clearly, there are advantages to virtual try-ons: it can be more convenient, engaging, and easier compared to trying on the real thing. There are disadvantages too, however, in that they only give an impression of what you look like. For example, the user cannot feel the weight of a virtual accessory on their head or the texture of virtual make-up on their face.
The same technology can be used to enable people to step into historical, famous, film, or stage characters (for instance, David Bowie or Queen Victoria). For example, a virtual try-on app that was developed as part of a cultural experience was the Magic-Face (Javornik, et al., 2017). The goal was to enable audiences to experience firsthand what it was like to try on the make-up of a character from an opera. The opera chosen was Philip Glass’s Akhnaten, set in ancient Egyptian time (see Figure 7.28a). The vir-tual make-up developed were for a Pharaoh and his wife. The app was developed by University College London researchers alongside the English National Opera and AR company, Holition. To provide a real-world context, the app was designed to run on a tablet display that was disguised as a real mirror and placed in an actor’s dressing room (see Figure 7.28b). On encountering the mirror in situ, visiting school children were fascinated by the way the virtual make-up made them look like Akhnaten and his wife, Nefertiti. The singers and make-up artists who were in the production also tried it out and saw great potential for using the app to enhance their existing repertoire of rehearsal and make-up tools.
7 I N T E R F A C E S244
(a) (b)
Figure 7.28 (a) A principal singer trying on the virtual look of Akhnaten and (b) a framed AR mirror in the ENO dressing room Source: Used courtesy of Ana Javornik
Research and Design ConsiderationsA key research concern when designing augmented reality is what form the digital augmenta-tion should take and when and where it should appear in the physical environment (Rogers et al., 2005). The information (such as navigation cues) needs to stand out but not distract the person from their ongoing activity in the physical world. It also needs to be simple and align with the real-world objects taking into account that the user will be moving. Another concern is how much digital content to overlay on the physical world and how to attract the user’s attention to it. There is the danger that the physical world becomes overloaded with digital ads and information “polluting” it to the extent that people will turn the AR app off.
One of the limitations of current AR technology is that sometimes the modeling can be slightly off so that the overlaying of the digital information appears in the wrong place or is out of sync with what is being overlaid. This may not be critical for fun applications, but it may be disconcerting if eye shadow appears on someone’s ear. It may also break the magic of the AR experience. Ambiguity and uncertainty may be exploited to good effect in mixed reality games, but it could be disastrous in a more serious context, such as the military or medical setting.
7 . 2 I N T E R F A C E T y p E S 245
7.2.17 WearablesWearables are a broad category of devices that are worn on the body. These include smart-watches, fitness trackers, fashion tech, and smart glasses. Since the early experimental days of wearable computing, where Steve Mann (1997) donned head and eye cameras to enable him to record what he saw while also accessing digital information on the move, there have been many innovations and inventions, including Google Glass.
New flexible display technologies, e-textiles, and physical computing (for example, Arduino) provide opportunities to design wearables that people will actually want to wear. Jewelry, caps, glasses, shoes, and jackets have all been the subject of experimentation designed to provide the user with a means of interacting with digital information while on the move in the physical world. Early wearables focused on convenience, enabling people to carry out a task (for example, selecting music) without having to take out and control a handheld device. Examples included a ski jacket with integrated music player controls that enabled the wearer to simply touch a button on their arm with their glove to change a music track. More recent applications have focused on how to combine textiles, electronics, and haptic technologies to promote new forms of communication. For example, CuteCircuit developed the KineticDress, which was embedded with sensors that followed the body of the wearer to capture their movements and interaction with others. These were then displayed through electroluminescent embroidery that covered the external skirt section of the dress. Depend-ing on the amount and speed of the wearer’s movement, it changed patterns, displaying the wearer’s mood to the audience and creating a magic halo around her.
Exoskeleton clothing (see section 7.2.12) is also an area where fashion meets technology in order to augment and assist people who have problems with walking by literally walk-ing or exercising the person wearing them. In this way, it combines haptics with a wear-able. Within the construction industry, exoskeleton suits have also been developed to provide additional power to workers—a bit like Superman—where metal frameworks are fitted with motorized muscles to multiply the wearer’s strength. It can make lifting objects feel lighter and in doing so protect the worker from physical injuries.
DILEMMAGoogle Glass: Seeing Too Much?
Google Glass was a wearable that went on sale in 2014 in various fashion styles (see Figure 7.29). It was designed to look like a pair of glasses, but with one lens of the glass being an interactive display with an embedded camera that could be controlled with speech input. It allowed the wearer to take photos and videos on the move and look at digital content, such as email, texts, and maps. The wearer could also search the web using voice commands, and the results would appear on the screen. A number of applications were developed beyond those for everyday use, including WatchMeTalk, which provided live captions to help the
(Continued)
7 I N T E R F A C E S246
Watch the interesting video of London through Google Glass at http://youtu.be/Z3AIdnzZUsE and the Talking Shoe concept at http://youtu.be/VcaSwxbRkcE.
hearing-impaired in their day-to-day conversations and Preview for Glass that enabled the wearer to watch a movie trailer the moment they looked at a movie poster.
However, being in the company of someone wearing a Google Glass was felt by many to be unnerving, as the wearer looked up and to the right to view what was on the glass screen rather than looking at you and into your eyes. One of the criticisms of wearers of Google Glass was that it made them appear to be staring into the distance. Others were worried that those wearing Google Glass were recording everything that was happening in front of them. As a reaction, a few bars and restaurants in the United States implemented a “no Glass” policy to prevent customers from recording other patrons.
The original Google Glass was retired after a couple of years. Since then, other types of smart glasses have come onto the market that synch a user’s smartphone with the display and camera on the glasses via Bluetooth. These include Vuzic Blade, which has a camera onboard and voice control that is connected to Amazon Echo devices, along with the provision of turn-by-turn navigation and location-based alerts; and Snap’s Spectacles, which simply allows the wearer to share photos and video with their friends on Snapchat they take when wearing these glasses.
Figure 7.29 Google GlassSource: Google Inc.
7 . 2 I N T E R F A C E T y p E S 247
7.2.18 Robots and DronesRobots have been around for some time, most notably as characters in science-fiction mov-ies, but they also play an important role as part of manufacturing assembly lines, as remote investigators of hazardous locations (for example, nuclear power stations and bomb dis-posal), and as search and rescue helpers in disasters (for instance in forest fires) or faraway places (like Mars). Console interfaces have been developed to enable humans to control and navigate robots in remote terrains, using a combination of joysticks and keyboard controls together with cameras and sensor-based interactions (Baker et al., 2004). The focus has been on designing interfaces that enable users to steer and move a remote robot effectively with the aid of live video and dynamic maps.
Domestic robots that help with the cleaning and gardening have become popular. Robots are also being developed to help the elderly and disabled with certain activities, such as picking up objects and cooking meals. Pet robots, in the guise of human companions, have been commercialized. Several research teams have taken the “cute and cuddly” approach to designing robots, signaling to humans that the robots are more pet-like than human-like. For example, Mitsubishi developed Mel the penguin (Sidner and Lee, 2005) whose role was to host events, while the Japanese inventor Takanori Shibata developed Paro in 2004, a baby harp seal that looks like a cute furry cartoon animal, and whose role was as a companion (see Figure 7.30). Sensors were embedded in the pet robots, enabling them to detect certain human behaviors and respond accordingly. For example, they can open, close, and move their eyes, giggle, and raise their flippers. The robots encourage being cuddled or spoken to,
Research and Design ConsiderationsA core design concern specific to wearable interfaces is comfort. Users need to feel comfort-able wearing clothing that is embedded with technology. It needs to be light, small, not get in the way, fashionable, and (with the exception of the displays) preferably hidden in the clothing. Another related issue is hygiene. Is it possible to wash or clean the clothing once worn? How easy is it to remove the electronic gadgetry and replace it? Where are the bat-teries going to be placed, and how long is their lifetime? A key usability concern is how does the user control the devices that are embedded in their clothing. Are touch, speech, or more conventional buttons and dials preferable?
A number of technologies can be developed and combined to create wearables including LEDs, sensors, actuators, tangibles, and AR. There is abundant scope for thinking creatively about when and whether to make something wearable as opposed to mobile. In Chapter 1, “What Is Interaction Design?” we mentioned how assistive technology can be designed to be fashionable in order to overcome stigmas of having to wear a monitoring device (for instance, for glucose levels), substitution (for example, a prosthetic) or amplifying device (for example, hearing aids).
7 I N T E R F A C E S248
as if they were real pets or animals. The appeal of pet robots is thought to be partially due to their therapeutic qualities, being able to reduce stress and loneliness among the elderly and infirm (see Chapter 6, “Emotional Interaction,” for more on cuddly robot pets). Paro has since been used to help patients with dementia to make them feel more at ease and comforted (Griffiths, 2014). Specifically, it has been used to encourage social behavior among patients who often anthropomorphize it. For example, they might say as a joke “it’s farted on me!” which makes them and others around them laugh, leading to further laughter and joking. This form of encouraging of social interaction is thought to be therapeutic.
Drones are a form of unmanned aircraft that are controlled remotely. They were first used by hobbyists and then by the military. Since then, they have become more affordable, acces-sible, and easier to fly. As a result, they have begun to be used in a wider range of contexts. These include entertainment, such as carrying drinks and food to people at festivals and par-ties; agricultural applications, such as flying them over vineyards and fields to collect data that is useful to farmers (see Figure 7.31); and helping to track poachers in wildlife parks in Africa (Preece, 2016). Compared with other forms of data collection, they can fly low and stream photos to a ground station where the images can be stitched together into maps and then used to determine the health of a crop or when is the best time to harvest the crop.
Figure 7.30 (a) Mel, the penguin robot, designed to host activities; (b) Japan’s Paro, an interactive seal, designed as a companion, primarily for the elderly and sick children Source: (a) Mitsubishi Electric Research Labs (b) Parorobots.com
Watch the video of Robot Pets of the Future at http://youtu.be/wBFws1lhuv0.
Watch the video of Rakuten delivering beer via drone to golfers on a golf course at https://youtu.be/ZameOVS2Skw.
7 . 2 I N T E R F A C E T y p E S 249
Figure 7.31 A drone being used to survey the state of a vineyard Source: Drone inspecting vineyard / Shutterstock
Research and Design ConsiderationsAn ethical concern is whether it is acceptable to create robots that exhibit behaviors that humans will consider to be human- or animal-like. While this form of attribution also occurs for agent interfaces (see Chapter 3), having a physical embodiment—as robots do—can make people suspend their disbelief even more, viewing the robots as pets or humans.
This raises the moral question as to whether such anthropomorphism should be encour-aged. Should robots be designed to be as human-like as possible, looking like us with human features, such as eyes and a mouth, behaving like us, communicating like us, and emotionally responding like us? Or, should they be designed to look like robots and behave like robots, for instance, vacuum cleaner robots that serve a clearly defined purpose? Likewise, should the interaction be designed to enable people to interact with the robot as if it were another human being, for example, by talking to it, gesturing at it, holding its hand, and smiling at it? Or, should the interaction be designed to be more like human–computer interaction, in other words, by pressing buttons, knobs, and dials to issue commands?
For many people, the cute pet approach to robotic interfaces seems preferable to one that seeks to design them to be more like fully fledged human beings. Humans know where they stand with pets and are less likely to be unnerved by them and, paradoxically, are more likely to suspend their disbelief in the companionship they provide.
Another ethical concern is whether it is acceptable to use unmanned drones to take a series of images or videos of fields, towns, and private property without permission or people knowing what is happening. They are banned from certain areas such as airports, where they present a real danger. Another potential problem is the noise they make when flying. Having a drone constantly buzzing past your house or delivering drinks to golf players or festival goers nearby can be very annoying.
7 I N T E R F A C E S250
7.2.19 Brain–Computer InterfacesBrain–computer interfaces (BCI) provide a communication pathway between a person’s brain waves and an external device, such as a cursor on a screen or a tangible puck that moves via airflow. The person is trained to concentrate on the task (for example, moving the cursor or the puck). Several research projects have investigated how this technique can be used to assist and augment human cognitive or sensory-motor functions. The way BCIs work is by detect-ing changes in the neural functioning of the brain. Our brains are filled with neurons that comprise individual nerve cells connected to one another by dendrites and axons. Every time we think, move, feel, or remember something, these neurons become active. Small electric signals rapidly move from neuron to neuron, which to a certain extent can be detected by electrodes that are placed on a person’s scalp. The electrodes are embedded in specialized headsets, hairnets, or caps.
Brain–computer interfaces have also been developed to control various games. For example, Brainball was developed as a game to be controlled by players’ brain waves in which they compete to control a ball’s movement across a table by becoming more relaxed and focused. Other possibilities include controlling a robot and being able to fly a virtual plane. Pioneering medical research, conducted by the BrainGate research group at Brown University, has started using brain-computer interfaces to enable people who are paralyzed to control robots (see Figure 7.32). For example, a robotic arm controlled by a tethered BCI has enabled patients who are paralyzed to feed themselves (see video mentioned next). Another startup company, NextMind, is developing a noninvasive brain-sensing device intended for the mass market to enable users to play games and control electronic and mobile devices in real time using just their thoughts. It is researching how to combine brain-sensing technology with innovative machine-learning algorithms that can translate brain waves into digital commands.
Watch a video of a woman who is paralyzed moving a robot with her mind at http://youtu.be/ogBX18maUiM.
Source: Tim Cordell / Cartoon Stock
7 . 2 I N T E R F A C E T y p E S 251
7.2.20 Smart InterfacesThe motivation for many new technologies is to make them smart, whether it is a smartphone, smartwatch, smart building, smart home, or smart appliance (for example smart lighting, smart speakers, or virtual assistants). The adjective is often used to suggest that the device has some intelligence and it is connected to the Internet. More generally, smart devices are designed to interact with users and other devices connected to a network, many of which are auto-mated, not requiring users to interact with them directly (Silverio-Fernández et al., 2018). The goal is to make them context-aware, that is, to understand what is happening around them and execute appropriate actions. To achieve this, some have been programmed with AI so that they can learn the context and a user’s behavior. Using this intelligence, they then change settings or switch things on according to the user’s assumed preferences. An example is the smart Nest thermostat that is designed to learn from a householder’s behavior. Rather than make the interface invisible, the designers chose to turn it into an aesthetically pleasing one that could be easily viewed (see Box 6.2).
Smart buildings have been designed to be more energy efficient, efficient, and cost effec-tive. Architects are motivated to use state-of-the-art sensor technology to control building systems, such as ventilation, lighting, security, and heating. Often, the inhabitants of such buildings are considered to be the ones at fault for wasting energy, as they may leave the lights and heating on overnight when not needed, or they forget to lock a door or window. One benefit of having automated systems take control of building services is to reduce these kinds of human errors—a phrase often used by engineers is to take the human “out of the loop.” While some smart buildings and homes have improved how they are managed and cut costs, they can also be frustrating to the user, who sometimes would like to be able to open a window to let fresh air in or raise a blind to let in natural lighting. Taking the human out of
Figure 7.32 A brain-computer interface being used by a woman who is paralyzed to select letters on a screen (developed by BrownGate)Source: Brown University
7 I N T E R F A C E S252
the loop means that these operations are no longer available. Windows are locked or sealed, and heating is controlled centrally.
Instead of simply introducing ever more automation that takes the human out of the loop further, another approach is to consider the needs of the inhabitants in conjunction with introducing smart technology. For example, a new approach that focuses on inhabitants is called human–building interaction (HBI). It is concerned with understanding and shap-ing people’s experiences with, and within, built environments (Alavi et al., 2019). The focus is on human values, needs, and priorities in addressing people’s interactions with “smart” environments.
7.3 Natural User Interfaces and Beyond
As we have seen, there are many kinds of interfaces that can be used to design for user expe-riences. The staple for many years was the GUI, then the mobile device interface, followed by touch, and now wearables and smart interfaces. Without question, they have been able to support all manner of user activities. What comes next? Will other kinds of interfaces that are projected to be more natural become more mainstream?
A natural user interface (NUI) is designed to allow people to interact with a computer in the same way that they interact with the physical world—using their voice, hands, and bodies. Instead of using a keyboard, mouse, or touchpad (as is the case with GUIs), NUIs enable users to speak to machines, stroke their surfaces, gesture at them in the air, dance on mats that detect feet movements, smile at them to get a reaction, and so on. The naturalness refers to the use of everyday skills humans have developed and learned, such as talking, writ-ing, gesturing, walking, and picking up objects. In theory, they should be easier to learn and map more readily onto how people interact with the world than compared with learning to use a GUI.
Instead of having to remember which function keys to press to open a file, a NUI means a person only has to raise their arm or say “open.” But how natural are NUIs? Is it more natural to say “open” than to flick a switch when you want to open a door? And is it more natural to raise both arms to change a channel on the TV than to press a button on a remote device or tell it what to do by speaking to it? Whether a NUI is natural depends on a number of fac-tors, including how much learning is required, the complexity of the app or device’s interface, and whether accuracy and speed are needed (Norman, 2010). Sometimes a gesture is worth a thousand words. Other times, a word is worth a thousand gestures. It depends on how many functions the system supports.
Consider the sensor-based faucets that were described in Chapter 1. The gesture-based interface works mostly (with the exception of people wearing black clothing that cannot be detected) because there are only two functions: (1) turning on the water by waving one’s hands under the tap, and (2) turning off the water by removing them from the sink. Now think about other functions that faucets usually provide, such as controlling water tem-perature and flow. What kind of a gesture would be most appropriate for changing the temperature and then the flow? Would one decide on the temperature first by raising one’s left arm and the flow by raising one’s right arm? How would someone know when to stop raising their arm to get the right temperature? Would they need to put a hand under the tap
7 . 4 W h I C h I N T E R F A C E ? 253
to check? But if they put their right hand under the tap, that might that have the effect of decreasing the flow? And when does the system know that the desired temperature and flow has been reached? Would it require having both arms suspended in midair for a few seconds to register that was the desired state? It is a difficult problem on how to provide these choices, and it is probably why sensor-based faucets in public bathrooms all have their temperature and flow set to a default.
Our overview of different interface types in this chapter has highlighted how gestural, voice, and other kinds of NUIs have made controlling input and interacting with digital content easier and more enjoyable, even though sometimes they can be less than perfect. For example, using gestures and whole-body movements have proven to be highly enjoyable as a form of input for computer games and physical exercises. Furthermore, new kinds of gesture, voice, and touch interfaces have made the web and online tools more accessible to those who are visually impaired. For example, the iPhone’s VoiceOver control features have empowered visually impaired individuals to be able to easily send email, use the web, play music, and so on, without having to buy an expensive customized phone or screen reader. Moreover, being able to purchase a regular phone means not being singled out for special treatment. And while some gestures may feel cumbersome for sighted people to learn and use, they may not be so for blind or visually impaired people. The iPhone VoiceOver press and guess feature that reads out what you tap on the screen (for example, “messages,” “calendar,” “mail: 5 new items”) can open up new ways of exploring an application while a three-finger tap can become a natural way to turn the screen off.
An emerging class of human–computer interfaces are those that rely largely on subtle, gradual, and continuous changes triggered by information obtained implicitly from the user together with the use of AI algorithms that are coded to learn about the user’s behavior and preferences. These are connected with lightweight, ambient, context-aware, affective, and augmented cognition interfaces (Solovey et al., 2014). Using brain, body, behavioral, and environmental sensors, it is now possible to capture subtle changes in people’s cognitive and emo-tional states in real time. This opens up new doors in human–computer interaction. In par-ticular, it allows for information to be used as both continuous and discrete input, potentially enabling new outputs to match and be updated with what people might want and need at any given time. Adding AI to the mix will also enable a new type of interface to emerge that goes beyond simply being natural and smart—one that allows people to develop new superpow-ers that will enable them to work synergistically with technology to solve ever-more complex problems and undertake unimaginable feats.
7.4 Which Interface?
This chapter presented an overview of the diversity of interfaces that is now available or currently being researched. There are many opportunities to design for user experiences that are a far cry from those originally developed using the command-based interfaces of the 1980s. An obvious question this raises is which one and how do you design it? In many contexts, the requirements for the user experience that have been identified will determine what kind of interface might be appropriate and what features to include. For example, if a healthcare app is being developed to enable patients to monitor their dietary intake, then a
7 I N T E R F A C E S254
mobile device that has the ability to scan barcodes and/or take pictures of food items that can be compared with a database would be a good interface to use, enabling mobility, effective object recognition, and ease of use. If the goal is to design a work environment to support collocated group decision-making activities, then combining shareable technolo-gies and personal devices that enable people to move fluidly among them would be good to consider using.
But how to decide which interface is preferable for a given task or activity? For example, is multimedia better than tangible interfaces for learning? Is voice effective as a command-based interface? Is a multimodal interface more effective than a single media interface? Are wearable interfaces better than mobile interfaces for helping people find information in for-eign cities? How does VR differ from AR, and which is the ultimate interface for playing games? In what way are tangible environments more challenging and captivating than vir-tual worlds? Will shareable interfaces, such as interactive furniture, be better at supporting communication and collaboration compared with using networked desktop technologies? And so forth. These questions are currently being researched. In practice, which interface is most appropriate, most useful, most efficient, most engaging, most supportive, and so on will depend on the interplay of a number of factors, including reliability, social acceptability, privacy, ethical, and location concerns.
In-Depth ActivityChoose a game that you or someone you know plays a lot on a smartphone (for example, Candy Crush Saga, Fortnite, or Minecraft). Consider how the game could be played using different interfaces other than the smartphone’s. Select three different interfaces (for instance, tangible, wearable, and shareable) and describe how the game could be redesigned for each of these, taking into account the user group being targeted. For example, the tangible game could be designed for children, the wearable interface for young adults, and the shareable interface for older people.1. Go through the research and design considerations for each interface and consider whether
they are relevant for the game setting and what considerations they raise.2. Describe a hypothetical scenario of how the game would be played for each of the three
interfaces.3. Consider specific design issues that will need to be addressed. For example, for the share-
able surface would it be best to have a tabletop or a wall-based surface? How will the users interact with the game elements for each of the different interfaces—by using a pen, finger-tips, voice, or other input device? How do you turn a single-player game into a multiple player one? What rules would you need to add?
4. Compare the pros and cons of designing the game using the three different interfaces with respect to how it is played on the smartphone.
F U R T h E R R E A D I N G 255
Further Reading
Many practical books have been published on interface design. Some have been revised into second editions. Publishers like New Riders and O’Reilly frequently offer up-to-date books for a specific interface area (for example web or voice). Some are updated on a regular basis while others are published when a new area emerges. There are also a number of excellent online resources, sets of guidelines, and thoughtful blogs and articles.
DASGUPTA, R. (2019) Voice User Interface Design: Moving from GUI to Mixed Modal Interaction. Apress. This is a guide that covers the challenges of moving from GUI design to mixed-modal interactions. It describes how our interactions with devices are rapidly chang-ing, illustrating this through a number of case studies and design principles of VUI design.
SummaryThis chapter provided an overview of the diversity of interfaces that can be designed for user experiences, identifying key design issues and research questions that need to be addressed. It has highlighted the opportunities and challenges that lie ahead for designers and researchers who are experimenting with and developing innovative interfaces. It also explained some of the assumptions behind the benefits of different interfaces—some that are currently supported and others that are still unsubstantiated. The chapter presented a number of interaction tech-niques that are particularly suited (or not) for a given interface type. It also discussed the dilemmas facing designers when using a particular kind of interface, for example, abstract versus realism, menu selection versus free-form text input, and human-like versus non-human-like. Finally, it presented pointers to specific design guidelines and exemplary systems that have been designed using a given interface.
Key Points • Many interfaces have emerged post the WIMP/GUI era, including voice, wearable, mobile, tangible, brain-computer, smart, robots, and drones.
• A range of design and research questions need to be considered when deciding which inter-face to use and what features to include.
• Natural user interfaces may not be as natural as graphical user interfaces—it depends on the task, user, and context.
• An important concern that underlies the design of any kind of interface is how information is represented to the user (be it speech, multimedia, virtual reality, augmented reality), so that they can make sense of it with respect to their ongoing activity, for example, playing a game, shopping online, or interacting with a pet robot.
• Increasingly, new interfaces that are context-aware or monitor people raise ethical issues concerned with what data is being collected and for what is it being used.
7 I N T E R F A C E S256
ROWLAND, C., GOODMAN, E., CHARLIER, M., LIGHT, A. and LUI, A. (2015) Designing Connected Products. O’Reilly. This collection of chapters covers the challenges of designing connected products that go beyond the traditional scope of interaction design and software development. It provides a road map and covers a range of aspects, including pairing devices, new business models, and flow of data in products.
GOOGLE Material Design https://material.io/design/ This online resource provides a living online document that visually illustrates essential interface design principles. It is beautifully laid out and very informative to click through all of the interactive examples that it provides. It shows how to add some physical properties to the digital world to make it feel more intui-tive to use across platforms.
KRISHNA, G. (2015) The Best Interfaces Are No Interfaces. New Riders. This polemical and funny book challenges the reader to think beyond the screen when designing new interfaces.
KRUG, S. (2014) Don’t Make Me Think! (3rd edn). New Riders Press. The third edition of this very accessible classic guide on web design presents up-to-date principles and examples on web design with a focus on mobile usability. It is highly entertaining with lots of great illustrations.
NORMAN, D. (2010) Natural interfaces are not natural, interactions, May/June, 6–10. This is a thought-provoking essay by Don Norman about what is natural may not appear to be natural, which is still very relevant today.
257
INTERVIEW with Leah Buechley
Leah Buechley is an independent designer, engineer, and educator. She has a PhD in computer science and a degree in physics. She began her studies as a dance major and has also been deeply engaged in theater, art, and design over the years. She was the founder and director of the high-low tech group at the MIT media lab from 2009 to 2014. She has always blended the sci-ences and the arts in her education and her career, as witnessed by her current work, which consists of computer science, indus-trial design, interaction design, art, and electrical engineering.
What is the focus of your work?I’m most interested in changing the culture of technology and engineering to make it more diverse and inclusive. To achieve that goal, I blend computation and electronics with a range of different materials and em-ploy techniques drawn from art, craft, and design. This approach leads to technol-ogies and learning experiences that appeal to a diverse group of people.
Can you give me some examples of how you mesh the digital with physical materi-als?My creative focus for the last several years has been computational design—a process in which objects are designed via an algo-rithm and then constructed with a combi-nation of fabrication and hand building. I’m especially excited about computational ceramics and have been developing a set of
tools and techniques that enable people to integrate programming and hand building with clay.
I’ve also been working on a project called LilyPad Arduino (or LilyPad) for over 10 years. LilyPad is a construction kit that enables people to embed com-puters and electronics into fabric. It’s a set of sewable electronic pieces, including mi-crocontrollers, sensors, and LEDs, that are stitched together with conductive thread. People can use the kit to make singing pil-lows, glow-in-the-dark handbags, and in-teractive ball gowns.
Another example is the work my for-mer students and I have done in paper-based computing. My former student Jie Qi developed a kit called Chibitronics circuit stickers that lets you build inter-active paper-based projects. Based on her years of research in my group at MIT, the kit is a set of flexible peel-and-stick elec-tronic stickers. You can connect ultra-thin LEDs, microcontrollers, and sensors with conductive ink, tape, or thread to quickly make beautiful electronic sketches.
The LilyPad and Chibitronics kits are now used by people around the world to learn computing and electronics. It’s been fascinating and exciting to see this research have a tangible impact.
Why would anyone want to wear a com-puter in their clothing?Computers open up new creative possibil-ities for designers. Computers are simply
(Continued)
I N T E R V I E W W I T h L E A h B U E C h L E y
7 I N T E R F A C E S258
a new tool, albeit an especially powerful one, in a designer’s toolbox. They allow clothing designers to make garments that are dynamic and interactive. Clothing that can, for example, change color in response to pollution levels, sparkle when a loved one calls you on the phone, or notify you when you blood pressure increases.
How do you involve people in your research?I engage with people in a few different ways. First, I design hardware and soft-ware tools to help people build new and different kinds of technology. The LilyPad is a good example of this kind of work. I hone these designs by teaching workshops to different groups of people. And once a tool is stable, I work hard to disseminate it to users in the real world. The LilyPad has been commercially available since 2007, and it has been fascinating and exciting to see how a group of real-world designers—who are predominantly female—is using it to build things like smart sportswear, plush video game controllers, soft robots, and in-teractive embroideries.
I also strive to be as open as possi-ble with my own design and engineering explorations. I document and publish as much information as I can about the mate-rials, tools, and processes I use. I apply an open source approach not only to the soft-ware and hardware I create but, as much as I can, to the entire creative process. I develop and share tutorials, classroom and workshop curricula, materials references, and engineering techniques.
What excites you most about your work?I am infatuated with materials. There is nothing more inspiring than a sheet of
heavy paper, a length of wool felt, a slab of clay, or a box of old motors. My thinking about design and technology is largely driven by explorations of materials and their affordances. So, materials are always delightful. For example, the shape and sur-face pattern of the cup in Figure 7.33 were computationally designed. A template of the design was then laser cut and pressed into a flat sheet or “slab” of clay. Finally, the clay was folded into shape and then fired and glazed using traditional ceramic techniques. But the real-world adoption of tools I’ve designed and the prospect this presents for changing technology culture is perhaps what’s most exciting. My most dearly held goal is to expand and diversify technology culture, and it’s tremendously rewarding to see evidence that my work is doing that.
Figure 7.33 An example of a computational cupSource: Used courtesy of Leah Buechley
Chapter 8
D A T A G A T H E R I N G
ObjectivesThe main goals of the chapter are to accomplish the following:
• Discuss how to plan and run a successful data gathering program. • Enable you to plan and run an interview. • Empower you to design a simple questionnaire. • Enable you to plan and carry out an observation.
8.1 Introduction
Data is everywhere. Indeed, it is common to hear people say that we are drowning in data because there is so much of it. So, what is data? Data can be numbers, words, measurements, descriptions, comments, photos, sketches, films, videos, or almost anything that is useful for understanding a particular design, user needs, and user behavior. Data can be quantitative or qualitative. For example, the time it takes a user to find information on a web page and the number of clicks to get to the information are forms of quantitative data. What the user says about the web page is a form of qualitative data. But what does it mean to collect these and other kinds of data? What techniques can be used, and how useful and reliable is the data that is collected?
This chapter presents some techniques for data gathering that are commonly used in interaction design activities. In particular, data gathering is a central part of discovering requirements and evaluation. Within the requirements activity, data gathering is conducted
8.1 Introduction
8.2 Five Key Issues
8.3 Data Recording
8.4 Interviews
8.5 Questionnaires
8.6 Observation
8.7 Choosing and Combining Techniques
8 D ATA G AT H E R I N G260
to collect sufficient, accurate, and relevant data so that design can proceed. Within evalua-tion, data gathering captures user reactions and their performance with a system or proto-type. All of the techniques that we will discuss can be done with little to no programming or technical skills. Recently, techniques for scraping large volumes of data from online activi-ties, such as Twitter posts, have become available. These and other techniques for managing huge amounts of data, and the implications of their use, are discussed in Chapter 10, “Data at Scale.”
Three main techniques for gathering data are introduced in this chapter: interviews, questionnaires, and observation. The next chapter discusses how to analyze and interpret the data collected. Interviews involve an interviewer asking one or more interviewees a set of questions, which may be highly structured or unstructured; interviews are usually synchronous and are often face-to-face, but they don’t have to be. Increasingly, interviews are conducted remotely using one of the many teleconferencing systems, such as Skype or Zoom, or on the phone. Questionnaires are a series of questions designed to be answered asynchronously, that is, without the presence of the investigator. These questionnaires may be paper-based or available online. Observation may be direct or indirect. Direct obser-vation involves spending time with individuals observing their activities as they happen. Indirect observation involves making a record of the user’s activity as it happens, to be studied at a later date. All three techniques may be used to collect qualitative or quanti-tative data.
Although this is a small set of basic techniques, they are flexible and can be combined and extended in many ways. Indeed, it is important not to focus on just one data gathering technique, if possible, but to use them in combination so as to avoid biases that are inherent in any one approach.
8.2 Five Key Issues
Five key issues require attention for any data gathering session to be successful: goal setting, identifying participants, the relationship between the data collector and the data provider, triangulation, and pilot studies.
8.2.1 Setting GoalsThe main reason for gathering data is to glean information about users, their behavior, or their reaction to technology. Examples include understanding how technology fits into family life, identifying which of two icons representing “send message” is easier to use, and finding out whether the planned redesign for a handheld meter reader is headed in the right direction. There are many different reasons for gathering data, and before beginning, it is important to set specific goals for the study. These goals will influence the nature of data gathering sessions, the data gathering techniques to be used, and the analysis to be performed (Robson and McCartan, 2016).
The goals may be expressed more or less formally, for instance, using some structured or even mathematical format or using a simple description such as the ones in the previous paragraph. Whatever the format, however, they should be clear and concise. In interaction design, it is more common to express goals for data gathering informally.
8 . 2 F I v E K E y I s s u E s 261
8.2.2 Identifying ParticipantsThe goals developed for the data gathering session will indicate the types of people from whom data is to be gathered. Those people who fit this profile are called the population or study population. In some cases, the people from whom to gather data may be clearly identifiable—maybe because there is a small group of users and access to each one is easy. However, it is more likely that the participants to be included in data gathering need to be chosen, and this is called sampling. The situation where all members of the target population are accessible is called saturation sampling, but this is quite rare. Assuming that only a por-tion of the population will be involved in data gathering, then there are two options: prob-ability sampling or nonprobability sampling. In the former case, the most commonly used approaches are simple random sampling or stratified sampling; in the latter case, the most common approaches are convenience sampling or volunteer panels.
Random sampling can be achieved by using a random number generator or by choosing every nth person in a list. Stratified sampling relies on being able to divide the population into groups (for example, classes in a secondary school) and then applying random sampling. Both convenience sampling and volunteer panels rely less on choosing the participants and more on the participants being prepared to take part. The term convenience sampling is used to describe a situation where the sample includes those who were available rather than those specifically selected. Another form of convenience sampling is snowball sampling, in which a current par-ticipant finds another participant and that participant finds another, and so on. Much like a snowball adds more snow as it gets bigger, the population is gathered up as the study progresses.
The crucial difference between probability and nonprobability methods is that in the former you can apply statistical tests and generalize to the whole population, while in the latter such generalizations are not robust. Using statistics also requires a sufficient number of participants. Vera Toepoel (2016) provides a more detailed treatment of sampling, particu-larly in relation to survey data.
BOX 8.1 How Many Participants Are Needed?
A common question is, how many participants are needed for a study? In general, having more participants is better because interpretations of statistical test results can be stated with higher confidence. What this means is that any differences found among conditions are more likely to be caused by a genuine effect rather than being due to chance.
More formally, there are many ways to determine how many participants are needed. Four of these are saturation, cost and feasibility analysis, guidelines, and prospective power analysis (Caine, 2016). • Saturation relies on data being collected until no new relevant information emerges, and
so it is not possible to know the number in advance of the saturation point being reached. • Choosing the number of participants based on cost and feasibility constraints is a prac-
tical approach and is justifiable; this kind of pragmatic decision is common in industrial projects but rarely reported in academic research.
(Continued)
8 D ATA G AT H E R I N G262
8.2.3 Relationship with ParticipantsOne significant aspect of any data gathering is the relationship between the person (people) doing the gathering and the person (people) providing the data. Making sure that this rela-tionship is clear and professional will help to clarify the nature of the study. How this is achieved varies in different countries and different settings. In the United States and United Kingdom, for example, it is achieved by asking participants to sign an informed consent form, while in Scandinavia such a form is not required. The details of this form will vary, but it usually asks the participants to confirm that the purpose of the data gathering and how the data will be used has been explained to them and that they are willing to continue. It usually explains that their data will be private and kept in a secure place. It also often includes a statement that participants may withdraw at any time and that in this case none of their data will be used in the study.
The informed consent form is intended to protect the interests of both the data gatherer and the data provider. The gatherer wants to know that the data they collect can be used in their analysis, presented to interested parties, and published in reports. The data provider wants reassurance that the information they give will not be used for other purposes or in any context that would be detrimental to them. For example, they want to be sure that per-sonal contact information and other personal details are not made public. This is especially true when people with disabilities or children are being interviewed. In the case of children, using an informed consent form reassures parents that their children will not be asked threat-ening, inappropriate, or embarrassing questions, or be asked to look at disturbing or violent images. In these cases, parents are asked to sign the form. Figure 8.1 shows an example of a typical informed consent form.
This kind of consent is also not generally required when gathering requirements data for a commercial company where a contract usually exists between the data collector and the data provider. An example is where a consultant is hired to gather data from company staff during the course of discovering requirements for a new interactive system to support timesheet entry. The employees of this company would be the users of the system, and the consultant would therefore expect to have access to the employees to gather data about the timesheet activity. In addition, the company would expect its employees to cooper-ate in this exercise. In this case, there is already a contract in place that covers the data
• Guidelines may come from experts or from “local standards,” for instance, from an accepted norm in the field.
• Prospective power analysis is a rigorous method used in statistics that relies on existing quantitative data about the topic; in interaction design, this data is often unavailable, mak-ing this approach infeasible, such as when a new technology is being developed.
Kelly Caine (2016) investigated the sample size (number of participants) for papers published at the international Computer-Human Interaction (CHI) conference in 2014. She found that several factors affected the sample size, including the method being used and whether the data was collected in person or remotely. In this set of papers, the sample size varied from 1 to 916,000, with the most common size being 12. So, a “local standard” for interaction design would therefore suggest 12 as a rule of thumb.
8 . 2 F I v E K E y I s s u E s 263
gathering activity, and therefore an informed consent form is less likely to be required. As with most ethical issues, the important thing is to consider the situation and make a judg-ment based on the specific circumstances. Increasingly, projects and organizations that collect personal data from people need to demonstrate that it is protected from unauthorized
You are invited to participate in a research project being conducted by the researchers listed on the bottom of the page. In order for us to be allowed to use any data you wish to provide, we must haveyour consent.In the simplest terms, we hope you will use the mobile phone, tabletop, and project website atthe University of Maryland to
• Take pictures• Share observations about the sights you see on campus• Share ideas that you have to improve the design of the phone or tabletop application or website• Comment on pictures, observations, and design ideas of others
The researchers and others using CampusNet will be able to look at your comments and pictures on the tabletop and/or website, and we may ask if you are willing to answer a few more questions (either on paper, by phone, or face-to-face) about your whole experience. You may stop participat-ing at any time.
A long version of this consent form is available for your review and signature, or you may opt to sign this shorter one by checking off all the boxes that re�ect your wishes and signing and dating the form below.___I agree that any photos I take using the CampusNet application may be uploaded to the tabletop
at the University of Maryland and/or a website now under development.___I agree to allow any comments, observations, and pro�le information that I choose to share with
others via the online application to be visible to others who use the application at the same time or after me.
___I agree to be videotaped/audiotaped during my participation in this study.___I agree to complete a short questionnaire during or after my participation in this study.
NAME[Please print]
SIGNATURE
DATE
[Contact information of Senior Researcher responsible for the project]
Crowdsourcing Design for Citizen Science Organizations
SHORT VERSION OF CONSENT FORM for participants at the University of Maryland – 18 YEARS AND OLDER
Figure 8.1 Example of an informed consent form
8 D ATA G AT H E R I N G264
access. For example, the European Union’s General Data Protection Regulation (GDPR) came into force in May 2018. It applies to all EU organizations and offers the individual unprecedented control over their personal data.
Incentives to take part in data gathering sessions may also be needed. For example, if there is no clear advantage to the respondents, incentives may persuade them to take part; in other circumstances, respondents may see it as part of their job or as a course requirement to take part. For example, if support sales executives are asked to complete a questionnaire about a new mobile sales application, then they are likely to agree if the new device will impact their day-to-day lives. In this case, the motivation for providing the required informa-tion is clear. However, when collecting data to understand how appealing a new interactive app is for school children, different incentives would be appropriate. Here, the advantage for individuals to take part is not so obvious.
8.2.4 TriangulationTriangulation is a term used to refer to the investigation of a phenomenon from (at least) two different perspectives (Denzin, 2006; Jupp, 2006). Four types of triangulation have been defined (Jupp, 2006).
• Triangulation of data means that data is drawn from different sources at different times, in different places, or from different people (possibly by using a different sampling technique).
• Investigator triangulation means that different researchers (observers, interviewers, and so on) have been involved in collecting and interpreting the data.
• Triangulation of theories means the use of different theoretical frameworks through which to view the data or findings.
• Methodological triangulation means to employ different data gathering techniques.
The last of these is the most common form of triangulation—to validate the results of some inquiry by pointing to similar results yielded through different perspectives. However, validation through true triangulation is difficult to achieve. Different data gathering methods result in different kinds of data, which may or may not be compatible. Using different theo-retical frameworks may or may not result in complementary findings, but to achieve theo-retical triangulation would require the theories to have similar philosophical underpinnings. Using more than one data gathering technique, and more than one data analysis approach, is good practice because it leads to insights from the different approaches even though it may not be achieving true triangulation.
Triangulation has sometimes been used to make up for the limitations of another type of data collection (Mackay and Anne-Laure Fayard, 1997). This is a different rationale from the original idea, which has more to do with the verification and reliability of data. Furthermore,
For more information about GDPR and data protection law in Europe and the United Kingdom, see:https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/
8 . 2 F I v E K E y I s s u E s 265
a kind of triangulation is being used increasingly in crowd sourcing and other studies involv-ing large amounts of data to check that the data collected from the original study is real and reliable. This is known as checking for “ground truth.”
8.2.5 Pilot StudiesA pilot study is a small trial run of the main study. The aim is to make sure that the proposed method is viable before embarking on the real study. For example, the equipment and instruc-tions can be checked, the questions for an interview or in a questionnaire can be tested for clarity, and an experimental procedure can be confirmed as viable. This can identify potential problems in advance so that they can be corrected. Distributing 500 questionnaires and then being told that two of the questions were very confusing wastes time, annoys participants, and is an expensive error that could be avoided by doing a pilot study.
If it is difficult to find participants or access to them is limited, asking colleagues or peers to participate can work as an alternative for a pilot study. Note that anyone involved in a pilot study cannot be involved in the main study itself. Why? Because they will know more about the study and this can distort the results.
For an example of methodological triangulation, see:https://medium.com/design-voices/the-power-of-triangulation-in-design-research-64a0957d47d2
For more information about ground truth and how ground truth databases are used to check data obtained in autonomous driving, see “The HCI Bench Mark Suite: Stereo and Flow Ground Truth with Uncertainties for Urban Autonomous Driving” at https://ieeexplore.ieee.org/document/7789500/.
BOX 8.2 Data, Information, and Conclusions
There is an important difference between raw data, information, and conclusions. Data is what you collect; this is then analyzed and interpreted and conclusions drawn. Information is gained from analyzing and interpreting the data and conclusions represent the actions to be taken based on the information. For example, consider a study to determine whether a new screen layout for a local leisure center has improved the user’s experience when booking a swimming lesson. In this case, the data collected might include a set of times to complete the booking, user comments regarding the new screen layout, biometric readings of the user’s
(Continued)
8 D ATA G AT H E R I N G266
8.3 Data Recording
Capturing data is necessary so that the results of a data gathering session can be analyzed and shared. Some forms of data gathering, such as questionnaires, diaries, interaction logging, scraping, and collecting work artifacts, are self-documenting and no further recording is necessary. For other techniques, however, there is a choice in recording approaches. The most common of these are taking notes, photographs, or recording audio or video. Often, several data recording approaches are used together. For example, an interview may be voice recorded, and then to help the interviewer in later analysis, a photograph of the interviewee may be taken to remind the interviewer about the context of the discussion.
Which data recording approaches are used will depend on the goal of the study and how the data will be used, the context, the time and resources available, and the sensitivity of the situation; the choice of data recording approach will affect the level of detail collected and how intrusive the data gathering will be. In most settings, audio recording, photographs, and notes will be sufficient. In others, it is essential to collect video data so as to record in detail the intricacies of the activity and its context. Three common data recording approaches are discussed next.
8.3.1 Notes Plus PhotographsTaking notes (by hand or by typing) is the least technical and most flexible way of record-ing data, even if it seems old-fashioned. Handwritten notes may be transcribed in whole or in part, and while this may seem tedious, it is usually the first step in analysis, and it gives the analyst a good overview of the quality and contents of the data collected. Tools exist for supporting data collection and analysis, but the advantages of handwritten notes include that using pen and paper can be less intrusive than typing and is more flexible, for example, for drawing diagrams of work layouts. Furthermore, researchers often comment that writ-ing notes helps them to focus on what is important and starts them thinking about what the data is telling them. The disadvantages of notes include that it can be difficult to capture the right highlights, and it can be tiring to write and listen or observe at the same time. It is easy to lose concentration, biases creep in, handwriting can be difficult to decipher, and the speed of writing is limited. Working with a colleague can reduce some of these problems while also providing another perspective.
heart rate while booking a lesson, and so on. At this stage, the data is raw. Information will emerge once this raw data has been analyzed and the results interpreted. For example, analyz-ing the data might indicate that users who have been using the leisure center for more than five years find the new layout frustrating and take longer to book, while those who have been using it for less than two years find the new layout helpful and can book lessons more quickly. This indicates that the new layout is good for newcomers but not so good for long-term users of the leisure center; this is information. A conclusion from this might be that a more extensive help system is needed for more experienced users to become used to the changes.
8 . 3 D ATA R E c O R D I N G 267
If appropriate, photograph(s) and short videos (captured via smartphones or other hand-held devices) of artifacts, events, and the environment can supplement notes and hand-drawn sketches, providing that permission has been given to collect data using these approaches.
8.3.2 Audio Plus PhotographsAudio recording is a useful alternative to note-taking and is less intrusive than video. Dur-ing observation, it allows observers to focus on the activity rather than on trying to capture every spoken word. In an interview, it allows the interviewer to pay more attention to the interviewee rather than trying to take notes as well as listening. It isn’t always necessary to transcribe all of the data collected—often only sections are needed, depending on the goals of the study. Many studies do not need a great level of detail, and instead recordings are used as a reminder and as a source of anecdotes for reports. It is surprising how evocative audio recordings of people or places from the data session can be, and those memories provide added context to the analysis. If audio recording is the main or only data collection tech-nique, then the quality needs to be good; performing interviews remotely, for example using Skype, can be compromised because of poor connections and acoustics. Audio recordings are often supplemented with photographs.
8.3.3 VideoSmartphones can be used to collect short video clips of activity. They are easy to use and less obtrusive than setting up sophisticated cameras. But there are occasions when a video is needed for long periods of time or when holding a phone is unreliable, for example, recording how designers collaborate together in a workshop or how teens interact in a “makerspace,” in which people can work on projects while sharing ideas, equipment, and knowledge. For these kinds of sessions, more professional video equipment that clearly captures both visual and audio data is more appropriate. Other ways of recording facial expressions together with verbal comments are also being used, such as GoToMeeting, which can be operated both in-person and remotely. Using such systems can create additional planning issues that have to be addressed to minimize how intrusive the recording is, while at the same time making sure that the data is of good quality (Denzin and Lincoln, 2011). When considering whether to use a camera, Heath et al. (2010) suggest the following issues to consider:
• Deciding whether to fix the camera’s position or use a roving recorder. This decision depends on the activity being recorded and the purpose to which the video data will be put, for example, for illustrative purposes only or for detailed data analysis. In some cases, such as pervasive games, a roving camera is the only way to capture the required action. For some studies, the video on a smartphone may be adequate and require less effort to set up.
• Deciding where to point the camera in order to capture what is required. Heath and his colleagues suggest carrying out fieldwork for a short time before starting to video record in order to become familiar with the environment and be able to identify suitable recording locations. Involving the participants themselves in deciding what and where to record also helps to capture relevant action.
• Understanding the impact of the recording on participants. It is often assumed that video recording will have an impact on participants and their behavior. However, it is worth taking an empirical approach to this issue and examining the data itself to see whether there is any evidence of people changing their behavior such as orienting themselves toward the camera.
8 D ATA G AT H E R I N G268
8.4 Interviews
Interviews can be thought of as a “conversation with a purpose” (Kahn and Cannell, 1957). How much like an ordinary conversation the interview will be depends on the type of interview. There are four main types of interviews: open-ended or unstructured, structured, semi-structured, and group interviews (Fontana and Frey, 2005). The first three types are named according to how much control the interviewer imposes on the conversation by following a predetermined set of questions. The fourth type, which is often called a focus group, involves a small group guided by a facilitator. The facilitation may be quite informal or follow a structured format.
The most appropriate approach to interviewing depends on the purpose of the interview, the questions to be addressed, and the interaction design activity. For example, if the goal is first to gain impressions about users’ reactions to a new design concept, then an informal, open-ended interview is often the best approach. But if the goal is to get feedback about a particular design feature, such as the layout of a new web browser, then a structured interview or question-naire is often better. This is because the goals and questions are more specific in the latter case.
8.4.1 Unstructured InterviewsOpen-ended or unstructured interviews are at one end of a spectrum of how much control the interviewer has over the interview process. They are exploratory and are similar to conversa-tions around a particular topic; they often go into considerable depth. Questions posed by
AcTIvITy 8.1 Imagine that you are a consultant who is employed to help develop a new augmented reality garden planning tool to be used by amateur and professional garden designers. The goal is to find out how garden designers use an early prototype as they walk around their clients’ gardens sketching design ideas, taking notes, and asking the clients about what they like and how they and their families use the garden. What are the advantages and disadvantages of the three approaches (note-taking, audio recording with photographs, and video) for data record-ing in this environment?
CommentHandwritten notes do not require specialized equipment. They are unobtrusive and flexible but difficult to do while walking around a garden. If it starts to rain, there is no equipment to get wet, but notes may get soggy and difficult to read (and write!). Garden planning is a highly visual, aesthetic activity, so supplementing notes with photographs would be appropriate.
Video captures more information, for example, continuous panoramas of the landscape, what the designers are seeing, sketches, comments, and so on, but it is more intrusive and will also be affected by the weather. Short video sequences recorded on a smartphone may be suf-ficient as the video is unlikely to be used for detailed analysis. Audio may be a good compro-mise, but synchronizing audio with activities such as looking at sketches and other artifacts later can be tricky and error prone.
8 . 4 I N T E R v I E w s 269
the interviewer are open, meaning that there is no particular expectation about the format or content of answers. For example, the first question asked of all participants might be: “What are the pros and cons of having a wearable?” Here, the interviewee is free to answer as fully or as briefly as they want, and both the interviewer and interviewee can steer the interview. For example, often the interviewer will say: “Can you tell me a bit more about . . .” This is referred to as probing.
Despite being unstructured and open, the interviewer needs a plan of the main topics to be covered so that they can make sure that all of the topics are discussed. Going into an inter-view without an agenda should not be confused with being open to hearing new ideas (see section 8.4.5, “Planning and Conducting an Interview”). One of the skills needed to conduct an unstructured interview is getting the balance right between obtaining answers to relevant questions and being prepared to follow unanticipated lines of inquiry.
A benefit of unstructured interviews is that they generate rich data that is often interre-lated and complex, that is, data that provides a deep understanding of the topic. In addition, interviewees may mention issues that the interviewer has not considered. A lot of unstruc-tured data is generated, and the interviews will not be consistent across participants since each interview takes on its own format. Unstructured interviews can be time-consuming to analyze, but they can also produce rich insights. Themes can be identified across inter-views using techniques from grounded theory and other analytic approaches, as discussed in Chapter 9, “Data Analysis, Interpretation, and Presentation.”
8.4.2 Structured InterviewsIn structured interviews, the interviewer asks predetermined questions similar to those in a questionnaire (see section 8.5, “Questionnaires”), and the same questions are used with each participant so that the study is standardized. The questions need to be short and clearly worded, and they are typically closed questions, which means that they require an answer from a predetermined set of alternatives. (This may include an “other” option, but ideally this would not be chosen often.) Closed questions work well if the range of possible answers is known or if participants don’t have much time. Structured interviews are useful only when the goals are clearly understood and specific questions can be identified. Example questions for a structured interview might be the following:
• “Which of the following websites do you visit most frequently: Amazon.com, Google.com, or msn.com?”
• “How often do you visit this website: every day, once a week, once a month, less often than once a month?”
• “Do you ever purchase anything online: Yes/No? If your answer is Yes, how often do you purchase things online: every day, once a week, once a month, less frequently than once a month?”
Questions in a structured interview are worded the same for each participant and are asked in the same order.
8.4.3 Semi-structured InterviewsSemi-structured interviews combine features of structured and unstructured interviews and use both closed and open questions. The interviewer has a basic script for guidance so that the same topics are covered with each interviewee. The interviewer starts with preplanned
8 D ATA G AT H E R I N G270
questions and then probes the interviewee to say more until no new relevant information is forthcoming. Here’s an example:
Which music websites do you visit most frequently?Answer: Mentions several but stresses that they prefer hottestmusic.comWhy?Answer: Says that they like the site layoutTell me more about the site layout.Answer: Silence, followed by an answer describing the site’s layoutAnything else that you like about the site?Answer: Describes the animationsThanks. Are there any other reasons for visiting this site so often that you haven’t mentioned?
It is important not to pre-empt an answer by phrasing a question to suggest that a par-ticular answer is expected. For example, “You seemed to like this use of color . . .” assumes that this is the case and will probably encourage the interviewee to answer that this is true so as not to offend the interviewer. Children are particularly prone to behave in this way (see Box 8.3, “Working with different kinds of users.”) The body language of the interviewer, for example whether they are smiling, scowling, looking disapproving, and so forth, can have a strong influence on whether the interviewee will agree with a question, and the interviewee needs to have time to speak and not be rushed.
Probes are a useful device for getting more information, especially neutral probes such as “Do you want to tell me anything else?” and prompts that remind interviewees if they forget terms or names help to move the interview along. Semi-structured interviews are intended to be broadly replicable, so probing and prompting aim to move the interview along without introducing bias.
BOX 8.3 Working with Different Kinds of Users
Focusing on the needs of users and including users in the design process is a central theme of this book. But users vary considerably based on their age, educational, life, and cultural experi-ences, and physical and cognitive abilities. For example, children think and react to situations differently than adults. Therefore, if children are to be included in data gathering sessions, then child-friendly methods are needed to make them feel at ease so that they will communicate with you. For very young children of pre-reading or early reading age, data gathering sessions need to rely on images and chat rather than written instructions or questionnaires. Researchers who work with children have developed sets of “smileys,” such as those shown in Figure 8.2, so that children can select the one that most closely represents their feelings (see Read et al., 2002).
Awful Not very good Really good BrilliantGood
Figure 8.2 A smileyometer gauge for early readers Source: Read et al. (2002)
8 . 4 I N T E R v I E w s 271
The examples in Box 8.3 demonstrate that technology developers need to adapt their data collection techniques to suit the participants with whom they work. As the saying goes, “One size doesn’t fit all.”
8.4.4 Focus GroupsInterviews are often conducted with one interviewer and one interviewee, but it is also com-mon to interview people in groups. One form of group interview that is sometimes used in interaction design activities is the focus group. Normally, three to ten people are involved, and the discussion is led by a trained facilitator. Participants are selected to provide a representa-tive sample of the target population. For example, in the evaluation of a university website, a group of administrators, faculty, and students may form three separate focus groups because they use the web for different purposes. In requirements activities, a focus group may be held in order to identify conflicts in expectations or terminology from different stakeholders.
Similarly, different approaches are needed when working with users from different cul-tures (Winschiers-Theophilus et al., 2012). In their work with local communities in Namibia, Heike Winschiers-Theophilus and Nicola Bidwell (2013) had to find ways of communicating with local participants, which included developing a variety of visual and other techniques to communicate ideas and collect data about the collective understanding and feelings inherent in the local cultures of the people with whom they worked.
Laurianne Sitbon and Shanjana Farhin (2017) report a study in which researchers inter-acted with people with intellectual disabilities, where they involved caregivers who knew each participant well and could appropriately make the researchers’ questions more concrete. This made it more understandable for the participants. An example of this was when the interviewer assumed that the participant understood the concept of a phone app to provide information about bus times. The caregiver made their questions more concrete for the participant by relat-ing the concept of the phone app to familiar people and circumstances and bringing in a per-sonal example (for instance, “So you don’t have to ring your mom to say ‘Mom, I am lost’”).
Another group of technology users are studied by the field of Animal-Computer Interac-tion (Mancini et al., 2017). Data gathering with animals poses additional and different chal-lenges. For example, in their study of dogs’ attention to TV screens, Ilyena Hirskyj-Douglas et al. (2017) used a combination of observation and tracking equipment to capture when a dog turns their head. But interpreting the data, or checking that the interpretation is accurate, requires animal behavior expertise.
Source: Mike Baldwin / Cartoon Stock
8 D ATA G AT H E R I N G272
The benefit of a focus group is that it allows diverse or sensitive issues to be raised that might otherwise be missed, for example in the requirements activity to understand multiple points within a collaborative process or to hear different user stories (Unger and Chandler, 2012). The method is more appropriate for investigating shared issues rather than individual experiences. Focus groups enable people to put forward their own perspectives. A preset agenda is developed to guide the discussion, but there is sufficient flexibility for the facili-tator to follow unanticipated issues as they are raised. The facilitator guides and prompts discussion, encourages quiet people to participate, and stops verbose ones from dominating the discussion. The discussion is usually recorded for later analysis, and participants may be invited to explain their comments more fully at a later date.
The format of focus groups can be adapted to fit within local cultural settings. For example, a study with the Mbeere people of Kenya aimed to find out how water was being used, any plans for future irrigation systems, and the possible role of technology in water management (Warrick et al., 2016). The researcher met with the elders from the commu-nity, and the focus group took the form of a traditional Kenyan “talking circle,” in which the elders sit in a circle and each person gives their opinions in turn. The researcher, who was from the Mbeere community, knew that it was impolite to interrupt or suggest that the conversation needed to move along, because traditionally each person speaks for as long as they want.
8.4.5 Planning and Conducting an InterviewPlanning an interview involves developing the set of questions or topics to be covered, col-lating any documentation to give to the interviewee (such as consent form or project descrip-tion), checking that recording equipment works, structuring the interview, and organizing a suitable time and place.
Developing Interview QuestionsQuestions may be open-ended (or open) or closed-ended (or closed). Open questions are best suited where the goal of the session is exploratory; closed questions are best suited where the possible answers are known in advance. An unstructured interview will usually consist mainly of open questions, while a structured interview will usually consist of closed ques-tions. A semi-structured interview may use a combination of both types.
Focus groups can be useful, but only if used for the right kind of activities. For a discussion of when focus groups don’t work, see the following links:https://www.nomensa.com/blog/2016/are-focus-groups-useful-research-technique-uxhttp://gerrymcgovern.com/why-focus-groups-dont-work/
8 . 4 I N T E R v I E w s 273
The following guidelines help in developing interview questions (Robson and McCartan, 2016):
• Long or compound questions can be difficult to remember or confusing, so split them into two separate questions. For example, instead of “How do you like this smartphone app compared with previous ones that you have used?” say, “How do you like this smartphone app?” “Have you used other smartphone apps?” If so, “How did you like them?” This is easier for the interviewee to respond to and easier for the interviewer to record.
• Interviewees may not understand jargon or complex language and might be too embar-rassed to admit it, so explain things to them in straightforward ways.
• Try to keep questions neutral, both when preparing the interview script and in conversa-tion during the interview itself. For example, if you ask “Why do you like this style of interaction?” this question assumes that the person does like it and will discourage some interviewees from stating their real feelings.
DILEMMAWhat They Say and What They Do
What users say isn’t always what they do. People sometimes give the answers that they think show them in the best light, they may have forgotten what happened, or they may want to please the interviewer by answering in the way they think will satisfy them. This may be problematic when the interviewer and interviewee don’t know each other, especially if the interview is being conducted remotely by Skype, Cisco Webex, or another digital conferenc-ing system.
For example, Yvonne Rogers et al. (2010) conducted a study to investigate whether a set of twinkly lights embedded in the floor of an office building could persuade people to take the stairs rather than the lift (or elevator). In interviews, participants told the research-ers that they did not change their behavior but logged data showed that their behavior did, in fact, change significantly. So, can interviewers believe all of the responses they get? Are the respondents telling the truth, or are they simply giving the answers that they think the inter-viewer wants to hear?
It isn’t possible to avoid this behavior, but an interviewer can be aware of it and reduce such biases by choosing questions carefully, by getting a large number of participants, or by using a combination of data gathering techniques.
8 D ATA G AT H E R I N G274
AcTIvITy 8.2 Several devices are available for reading ebooks, watching movies, and browsing photographs (see Figure 8.3). The design differs between makes and models, but they are all aimed at pro-viding a comfortable user experience. An increasing number of people also read books and watch movies on their smartphones, and they may purchase phones with larger screens for this purpose.
The developers of a new device for reading books online want to find out how appeal-ing it will be to young people aged 16–18, so they have decided to conduct some interviews. 1. What is the goal of this data gathering session?2. Suggest ways of recording the interview data.
(a) (b)
(c) (d)
Figure 8.3 (a) Sony’s eReader, (b) Amazon’s Kindle, (c) Apple’s iPad, and (d) Apple’s iPhone Source: (a) Sony Europe Limited, (b) Martyn Landi / PA Archive / PA Images, (c) Mark Lennihan / AP Images, and (d) Helen Sharp
8 . 4 I N T E R v I E w s 275
It is helpful when collecting answers to closed-ended questions to list possible responses together with boxes that can be checked. Here’s one way to convert some of the questions from Activity 8.2:
1. Have you used a device for reading books online before? (Explore previous knowledge.)Interviewer checks box: □ Yes □ No □ Don’t remember/know
2. Would you like to read a book using a device designed for reading online? (Explore initial reaction; then explore the response.)Interviewer checks box: □ Yes □ No □ Don’t know
3. Why?If response is “Yes” or “No,” interviewer asks, “Which of the following statements repre-sents your feelings best?”
3. Suggest a set of questions for use in an unstructured interview that seeks to understand the appeal of reading books online to young people in the 16–18 year old age group.
4. Based on the results of the unstructured interviews, the developers of the new device have found that an important acceptance factor is whether the device can be handled easily. Write a set of semi-structured interview questions to evaluate this aspect based on an initial prototype and run a pilot interview with two of your peers. Ask them to comment on your questions and refine them based on their comments.
Comment1. The goal is to understand what makes devices for reading books online appealing to peo-
ple aged 16–18.2. Audio recording will be less cumbersome and distracting than taking notes, and all impor-
tant points will be captured. Video recording is not needed in this initial interview as it isn’t necessary to capture any detailed interactions. However, it would be useful to take photographs of any devices referred to by the interviewee.
3. Possible questions include the following: Why do you read books online? Do you ever read print-based books? If so, what makes you choose to read online versus a print-based format? Do you find reading a book online comfortable? In what way(s) does reading online versus reading from print affect your ability to become engrossed in the story you are reading?
4. Semi-structured interview questions may be open or closed-ended. Some closed-ended questions that you might ask include the following: • Have you used any kind of device for reading books online before? • Would you like to read a book online using this device? • In your opinion, is the device easy to handle?
Some open-ended questions, with follow-on probes, include the following: • What do you like most about the device? Why? • What do you like least about the device? Why? • Please give me an example of where the device was uncomfortable or difficult to use.
8 D ATA G AT H E R I N G276
For “Yes,” interviewer checks one of these boxes: ⬜ I don’t like carrying heavy books. ⬜ This is fun/cool. ⬜ My friend told me they are great. ⬜ It’s the way of the future. ⬜ Another reason (interviewer notes the reason).
For “No,” interviewer checks one of these boxes: ⬜ I don’t like using gadgets if I can avoid it. ⬜ I can’t read the screen clearly. ⬜ I prefer the feel of paper. ⬜ Another reason (interviewer notes the reason).
4. In your opinion, is the device for reading online easy to handle or cumbersome?Interviewer checks one of these boxes:
⬜ Easy to handle ⬜ Cumbersome ⬜ Neither
Running the InterviewBefore starting, make sure that the goals of the interview have been explained to the inter-viewee and that they are willing to proceed. Finding out about the interviewee and their environment before the interview will make it easier to put them at ease, especially if it is an unfamiliar setting.
During the interview, it is better to listen more than to talk, to respond with sympathy but without bias, and to appear to enjoy the interview. The following is a common sequence for an interview (Robson and McCartan, 2016):
1. An introduction in which the interviewer introduces themselves and explains why they are doing the interview, reassures interviewees regarding any ethical issues, and asks if they mind being recorded, if appropriate. This should be exactly the same for each interviewee.
2. A warm-up session where easy, nonthreatening questions come first. These may include questions about demographic information, such as “What area of the country do you live in?”
3. A main session in which the questions are presented in a logical sequence, with the more probing ones at the end. In a semi-structured interview, the order of questions may vary between participants, depending on the course of the conversation, how much probing is done, and what seems more natural.
4. A cooling-off period consisting of a few easy questions (to defuse any tension that may have arisen).
5. A closing session in which the interviewer thanks the interviewee and switches off the recorder or puts their notebook away, signaling that the interview has ended.
8 . 4 I N T E R v I E w s 277
8.4.6 Other Forms of InterviewConducting face-to-face interviews and focus groups can be impractical, but the prevalence of Skype, Cisco WebEx, Zoom, and other digital conferencing systems, email, and phone-based interactions (voice or chat), sometimes with screen-sharing software, make remote interviewing a good alternative. These are carried out in a similar fashion to face-to-face sessions, but poor connections and acoustics can cause different challenges, and participants may be tempted to multitask rather than focus on the session at hand. Advantages of remote focus groups and interviews, especially when done through audio-only channels, include the following:
• The participants are in their own environment and are more relaxed. • Participants don’t have to travel. • Participants don’t need to worry about what they wear. • For interviews involving sensitive issues, interviewees can remain anonymous.
In addition, participants can leave the conversation whenever they want to by just cut-ting the connection, which adds to their sense of security. From the interviewer’s perspective, a wider set of participants can be reached easily, but a potential disadvantage is that the facilitator does not have a good view of the interviewees’ body language.
Retrospective interviews, that is, interviews that reflect on an activity or a data gathering session in the recent past, may be conducted with participants to check that the interviewer has correctly understood what was happening. This is a common practice in observational studies where it is sometimes referred to as member checking.
8.4.7 Enriching the Interview ExperienceFace-to-face interviews often take place in a neutral location away from the interviewee’s normal environment. This creates an artificial context, and it can be difficult for interviewees to give full answers to the questions posed. To help combat this, interviews can be enriched by using props such as personas prototypes or work artifacts that the interviewee or inter-viewer brings along, or descriptions of common tasks (examples of these kinds of props are scenarios and prototypes, which are covered in Chapter 11, “Discovering Requirements,” and Chapter 12, “Design, Prototyping, and Construction”). These props can be used to pro-vide context for the interviewees and help to ground the data in a real setting. Figure 8.4 illustrates the use of personas in a focus group setting.
For more information and some interesting thoughts on remote usability testing, see http://www.uxbooth.com/articles/hidden-benefits-remote-research/.
8 D ATA G AT H E R I N G278
As another example, Clara Mancini et al. (2009) used a combination of questionnaire prompts and deferred contextual interviews when investigating mobile privacy. A simple multiple-choice questionnaire was sent electronically to the participants’ smartphones, and they answered the questions using these devices. Interviews about the recorded events were conducted later, based on the questionnaire answers given at the time of the event.
8.5 Questionnaires
Questionnaires are a well-established technique for collecting demographic data and users’ opinions. They are similar to interviews in that they can have closed or open-ended questions, but once a questionnaire is produced, it can be distributed to a large number of participants without requiring additional data gathering resources. Thus, more data can be collected than would normally be possible in an interview study. Furthermore, participants who are located in remote locations or those who cannot attend an interview at a particular time can be involved more easily. Often a message is sent electronically to potential participants directing them to an online questionnaire.
Effort and skill are needed to ensure that questions are clearly worded and the data col-lected can be analyzed efficiently. Well-designed questionnaires are good for getting answers to specific questions from a large group of people. Questionnaires can be used on their own
Figure 8.4 Enriching a focus group with personas displayed on the wall for all participants to see
8 . 5 Q u E s T I O N N A I R E s 279
or in conjunction with other methods to clarify or deepen understanding. For example, infor-mation obtained through interviews with a small selection of interviewees might be corrobo-rated by sending a questionnaire to a wider group to confirm the conclusions.
Questionnaire questions and structured interview questions are similar, so which technique is used when? Essentially, the difference lies in the motivation of the respondent to answer the questions. If their motivation is high enough to complete a questionnaire without anyone else present, then a questionnaire will be appropriate. On the other hand, if the respondents need some persuasion to answer the questions, a structured interview format would be better. For example, structured interviews are easier and quicker to conduct if people will not stop to com-plete a questionnaire, such as at a train station or while walking to their next meeting.
It can be harder to develop good questionnaire questions compared with structured interview questions because the interviewer is not available to explain them or to clarify any ambiguities. Because of this, it is important that questions are specific; when possible, ask closed-ended questions and offer a range of answers, including a “no opinion” or “none of these” option. Finally, use negative questions carefully, as they can be confusing and may lead to false information. Some questionnaire designers, however, use a mixture of negative and positive questions deliberately because it helps to check the users’ intentions.
8.5.1 Questionnaire StructureMany questionnaires start by asking for basic demographic information (gender, age, place of birth) and details of relevant experience (the number of hours a day spent searching on the Internet, the level of expertise within the domain under study, and so on). This background information is useful for putting the questionnaire responses into context. For example, if two responses conflict, these different perspectives may be because of their level of experience—a group of people who are using a social networking site for the first time are likely to express different opinions than another group with five years’ experience of using such sites. However, only contextual information that is relevant to the study goal needs to be collected. For exam-ple, it is unlikely that a person’s height will provide relevant context to their responses about Internet use, but it might be relevant for a study concerning wearables.
Specific questions that contribute to the data-gathering goal usually follow these demo-graphic questions. If the questionnaire is long, the questions may be subdivided into related topics to make it easier and more logical to complete.
The following is a checklist of general advice for designing a questionnaire:
• Think about the ordering of questions. The impact of a question can be influenced by question order.
• Consider whether different versions of the questionnaire are needed for different populations. • Provide clear instructions on how to complete the questionnaire, for example, whether answers can be saved and completed later. Aim for both careful wording and good typography.
• Think about the length of the questionnaire, and avoid questions that don’t address the study goals.
• If the questionnaire has to be long, consider allowing respondents to opt out at different stages. It is usually better to get answers to some sections than no answers at all because of dropout.
• Think about questionnaire layout and pacing; for instance, strike a balance between using white space, or individual web pages, and the need to keep the questionnaire as compact as possible.
8 D ATA G AT H E R I N G280
8.5.2 Question and Response FormatDifferent formats of question and response can be chosen. For example, with a closed-ended question, it may be appropriate to indicate only one response, or it may be appropriate to indicate several. Sometimes, it is better to ask users to locate their answer within a range. Selecting the most appropriate question and response format makes it easier for respondents to answer clearly. Some commonly used formats are described next.
Check Boxes and RangesThe range of answers to demographic questions is predictable. Nationality, for example, has a finite number of alternatives, and asking respondents to choose a response from a prede-fined list makes sense for collecting this information. A similar approach can be adopted if details of age are needed. But since some people do not like to give their exact age, many questionnaires ask respondents to specify their age as a range. A common design error arises when the ranges overlap. For example, specifying two ranges as 15–20 and 20–25 will cause confusion; that is, which box do people who are 20 years old check? Making the ranges 15–19 and 20–24 avoids this problem.
A frequently asked question about ranges is whether the interval must be equal in all cases. The answer is no—it depends on what you want to know. For example, people who might use a website about life insurance are likely to be employed individuals who are 21–65 years old. The question could, therefore, have just three ranges: under 21, 21–65, and over 65. In contrast, to see how the population’s political views vary across generations might require 10-year cohort groups for people over 21, in which case the following ranges would be appropriate: under 21, 21–30, 31–40, and so forth.
Rating ScalesThere are a number of different types of rating scales, each with its own purpose (see Oppen-heim, 2000). Two commonly used scales are the Likert and semantic differential scales. Their purpose is to elicit a range of responses to a question that can be compared across respondents. They are good for getting people to make judgments, such as how easy, how usable, and the like.
Likert scales rely on identifying a set of statements representing a range of possible opin-ions, while semantic differential scales rely on choosing pairs of words that represent the range of possible opinions. Likert scales are more commonly used because identifying suitable statements that respondents will understand consistently is easier than identifying semantic pairs that respondents interpret as intended.
Likert ScalesLikert scales are used for measuring opinions, attitudes, and beliefs, and consequently they are widely used for evaluating user satisfaction with products. For example, users’ opinions about the use of color in a website could be evaluated with a Likert scale using a range of numbers, as in question 1 here, or with words as in question 2:
1. The use of color is excellent (where 1 represents strongly agree and 5 represents strongly disagree):
1 2 3 4 5
□ □ □ □ □
8 . 5 Q u E s T I O N N A I R E s 281
2. The use of color is excellent:
Strongly agree Agree OK Disagree Strongly disagree
□ □ □ □ □
In both cases, respondents would be asked to choose the right box, number, or phrase. Designing a Likert scale involves the following steps:
1. Gather a pool of short statements about the subject to be investigated. Examples are “This control panel is clear” and “The procedure for checking credit rating is too complex.” A brainstorming session with peers is a good way to identify key aspects to be investigated.
2. Decide on the scale. There are three main issues to be addressed here: How many points does the scale need? Should the scale be discrete or continuous? How can the scale be rep-resented? See Box 8.4 What Scales to Use: Three, Five, Seven, or More? for more on this.
3. Select items for the final questionnaire, and reword as necessary to make them clear.
In the first example above, the scale is arranged with 1 as the highest choice on the left and 5 as the lowest choice on the right. The logic for this is that first is the best place to be in a race and fifth would be the worst place. While there is no absolute right or wrong way of ordering the numbers other researchers prefer to arrange the scales the other way around with 1 as the lowest on the left and 5 as the highest on the right. They argue that intuitively the highest number suggests the best choice and the lowest number suggests the worst choice. Another reason for going from lowest to highest is that when the results are reported, it is more intuitive for readers to see high numbers representing the best choices. The important thing is to be consistent.
Semantic Differential ScalesSemantic differential scales explore a range of bipolar attitudes about a particular item, each of which is represented as a pair of adjectives. The participant is asked to choose a point between the two extremes to indicate agreement with the poles, as shown in Figure 8.5. The score for the investigation is found by summing the scores for each bipolar pair. Scores are then computed across groups of participants. Notice that in this example the poles are mixed so that good and bad features are distributed on the right and the left. In this example, there are seven positions on the scale.
UglyAttractive
ConfusingClear
ColorfulDull
BoringExciting
PleasingAnnoying
UnhelpfulHelpful
Poor Well designed
Figure 8.5 An example of a semantic differential scale
8 D ATA G AT H E R I N G282
BOX 8.4 What Scales to Use: Three, Five, Seven, or More?
Issues to address when designing Likert and semantic differential scales include the following: how many points are needed on the scale, how should they be presented, and in what form?
Many questionnaires use seven- or five-point scales, and there are also three-point scales. Some even use nine-point scales. Arguments for the number of points go both ways. Advocates of long scales argue that they help to show discrimination. Rating features on an interface is more difficult for most people than, say, selecting among different flavors of ice cream, and when the task is difficult, there is evidence to show that people “hedge their bets.” Rather than selecting the poles of the scales if there is no right or wrong, respondents tend to select values nearer the center. The counterargument is that people cannot be expected to discern accurately among points on a large scale, so any scale of more than five points is unnecessarily difficult to use.
Another aspect to consider is whether to give the scale an even or odd number of points. An odd number provides a clear central point, while an even number forces participants to decide and prevents them from sitting on the fence.
We suggest the following guidelines:
How many points on the scale?Use a small number, three, for example, when the possibilities are very limited, as in Yes/No type answers.
□ □ □Yes Don’t know No
Use a medium-sized range, five, for example, when making judgments that involve like/dislike or agree/disagree statements.
Strongly agree
Agree OK Disagree Strongly disagree
□ □ □ □ □
Use a longer range, seven or nine, for example, when asking respondents to make subtle judgments, such as when asking about a user experience dimension such as “level of appeal” of a character in a video game.
very appealing ok repulsive
Discrete or continuous?Use boxes for discrete choices and scales for finer judgments.
What order?Decide which way to order the scale, and be consistent.
8 . 5 Q u E s T I O N N A I R E s 283
8.5.3 Administering QuestionnairesTwo important issues when using questionnaires are reaching a representative sample of par-ticipants and ensuring a reasonable response rate. For large surveys, potential respondents need to be selected using a sampling technique. However, interaction designers commonly use a small number of participants, often fewer than 20 users. Completion rates of 100 percent are often achieved with these small samples, but with larger or more remote populations, ensuring that surveys are returned is a well-known problem. A 40 percent return is generally acceptable
AcTIvITy 8.3 Spot four poorly designed features in the excerpt from a questionnaire in Figure 8.6.
CommentSome of the features that could be improved upon include the following: • Question 2 requests an exact age. Many people prefer not to give this information and would rather position themselves within a range.
• In question 3, the number of hours spent searching is indicated with overlapping scales, that is, 1–3 and 3–5. How would someone answer if they spend 3 hours a day searching online?
• For question 4, the questionnaire doesn’t say how many boxes to check. • The space left for people to answer open-ended question 5 is too small, which will annoy some people and deter them from giving their opinions.
Many online survey tools prevent users from making some of these design errors. It is impor-tant, however, to be aware of such things because paper is still sometimes used.
2. State your age in years
3. How many hours a day do you spendsearching online?
4. Which of the following do you do online?
5. How useful is the Internet to you?
purchase goods
<1 hour1–3 hours3–5 hours>5 hours
send e-mailvisit chatroomsuse bulletin boardsfind informationread the news
Figure 8.6 A questionnaire with poorly designed features
8 D ATA G AT H E R I N G284
for many surveys, but much lower rates are common. Depending on your audience, you might want to consider offering incentives (see section 8.2.3, “Relationship with Participants”).
While questionnaires are often online, paper questionnaires may be more convenient in some situations, for example, if participants do not have Internet access or if it is expensive to use. Occa-sionally, short questionnaires are sent within the body of an email, but more often the advantages of the data being compiled automatically and either partly or fully analyzed make online ques-tionnaires attractive. Online questionnaires are interactive and can include check boxes, radio buttons, pull-down and pop-up menus, help screens, graphics, or videos (see Figure 8.7). They can also provide immediate data validation; for example, the entry must be a number between 1 and 20, and automatically skip questions that are irrelevant to some respondents, such as questions aimed only at teenagers. Other advantages of online questionnaires include faster response rates and automatic transfer of responses into a database for analysis (Toepoel, 2016).
The main problem with online questionnaires is the difficulty of obtaining a random sample of respondents; online questionnaires usually rely on convenience sampling, and hence their results cannot be generalized. In some countries, online questions, often delivered via smartphones, are frequently used in conjunction with television to elicit viewers’ opinions of programs and political events.
Figure 8.7 An excerpt from a web-based questionnaire showing check boxes, radio buttons, and pull-down menus
8 . 5 Q u E s T I O N N A I R E s 285
Deploying an online questionnaire involves the following steps (Toepoel, 2016, Chapter 10):
1. Plan the survey timeline. If there is a deadline, work backward from the deadline and plan what needs to be done on a weekly basis.
2. Design the questionnaire offline. Using plain text is useful as this can then be copied more easily into the online survey tool.
3. Program the online survey. How long this will take depends on the complexity of the design, for example, how many navigational paths it contains or if it has a lot of interac-tive features.
4. Test the survey, both to make sure that it behaves as envisioned and to check the ques-tions themselves. This includes getting feedback from content experts, survey experts, and potential respondents. This last group forms the basis of a pilot study.
5. Recruit respondents. As mentioned earlier, participants may have different reasons for taking part in the survey, but especially when respondents need to be encouraged, make the invitations intriguing, simple, friendly, respectful, trustworthy, motivating, interesting, informative, and short.
There are many online questionnaire templates available that provide a range of options, including different question types (for example open-ended, multiple choice), rating scales (such as Likert, semantic differential), and answer types (for example, radio buttons, check boxes, drop-down menus).
The following activity asks you to make use of one of these templates. Apart from being able to administer an online questionnaire widely, these templates also enable the question-naire to be segmented. For example, airline satisfaction questionnaires often have different sections for check-in, baggage handling, airport lounge, inflight movies, inflight food service, and so forth. If you didn’t use an airport lounge or check your baggage, you can skip those sections. This avoids respondents getting frustrated by having to go through questions that are not relevant to them. It is also a useful technique for long questionnaires, as it ensures that if a respondent opts out for lack of time or gets tired of answering the questions, the data that has been provided already is available to be analyzed.
AcTIvITy 8.4 Go to questionpro.com, surveymonkey.com, or a similar survey site and design your own questionnaire using the set of widgets that is available for a free trial period.
Create an online questionnaire for the set of questions that you developed for Activity 8.2. For each question, produce two different designs; for example, use radio buttons and drop-down menus for one question, and provide a 10-point semantic differential scale and a 5-point scale for another question.
What differences (if any) do you think the two designs will have on a respondent’s behav-ior? Ask a number of people to answer one or the other of your questions and see whether the answers differ for the two designs.
(Continued)
8 D ATA G AT H E R I N G286
BOX 8.5 Do people answer online questionnaires differently than paper and pencil? If so, why?
There has been much research examining how people respond to surveys when using a com-puter compared with paper and pencil methods. Some studies suggest that people are more revealing and consistent in their responses when using a computer to report their habits and behaviors, such as eating, drinking, and amount of exercise see Luce et al. (2003). Students have also been found to rate their instructors less favorably when online (Chang, 2004).
In a Danish study in which 3,600 people were invited to participate, the researchers concluded that although response rates for web-based invitations were lower, they were more cost-effective (by a factor of 10) and had only slightly lower numbers of missing values than questionnaires sent via paper (Ebert et al., 2018). Similarly, a study by Diaz de Rada and Dominguez-Alvarez (2014), in which the quality of the information collected from a survey given to citizens of Andalusia in Spain was analyzed, several advantages of using online versus paper-based questionnaires were identified. These included a low number of unanswered questions, more detailed answers to open-ended questions, and longer answers to questions in the online questionnaires than in the paper questionnaires. In the five open-ended questions, respondents wrote 63 characters more on average on the online questionnaires than on the paper questionnaires. For the questions in which participants had to select from a drop-down menu, there was a better response rate than when the selection was presented on paper with blank spaces.
One factor that can influence how people answer questions is the way the information is structured, such as the use of headers, the ordering, and the placement of questions. Online questionnaires provide more options for presenting information, including the use of drop-down menus, radio buttons, and jump-to options, which may influence how people read and navigate a questionnaire. But do these issues affect respondents’ answers? Smyth et al. (2005) have found that providing forced choice formats results in more options being selected. Another example is provided by Funcke et al. (2011), who found that continuous sliders ena-bled researchers to collect more accurate data because they support continuous rather than discrete scales. They also encouraged higher response rates. What can be concluded from these investigations is that the details of questionnaire design can impact how respondents react.
CommentRespondents may have used the response types in different ways. For example, they may select the end options more often from a drop-down menu than from a list of options that are cho-sen via radio buttons. Alternatively, you may find no difference and that people’s opinions are not affected by the widget style used. Some differences, of course, may be due to the variation between individual responses rather than being caused by features in the questionnaire design. To tease the effects apart, you would need to ask a large number of participants (for instance, in the range 50–100) to respond to the questions for each design.
8 . 6 O B s E R v AT I O N 287
8.6 Observation
Observation is useful at any stage during product development. Early in design, observation helps designers understand the users’ context, tasks, and goals. Observation conducted later in development, for example, in evaluation, may be used to investigate how well a prototype supports these tasks and goals.
Users may be observed directly by the investigator as they perform their activities or indi-rectly through records of the activity that are studied afterward (Bernard, 2017). Observation may also take place in the field or in a controlled environment. In the former case, individuals are observed as they go about their day-to-day tasks in the natural setting. In the latter case, individuals are observed performing specified tasks within a controlled environment such as a usability laboratory.
AcTIvITy 8.5 To appreciate the different merits of observation in the field and observation in a controlled environment, read the following scenarios and answer the questions that appear after.
Scenario 1 A usability consultant joins a group of tourists who have been given a wear-able navigation device that fits onto a wrist strap to test on a visit to Stockholm. After sight-seeing for the day, they use the device to find a list of restaurants within 2 kilometers of their current position. Several are listed, and they find the phone numbers of a few, call them to ask about their menus, select one, make a booking, and head off to the restaurant. The usability consultant observes some difficulty operating the device, especially on the move. Discussion with the group supports the evaluator’s impression that there are problems with the interface, but on balance the device is useful, and the group is pleased to get a table at a good restau-rant nearby.
Scenario 2 A usability consultant observes how participants perform a preplanned task using the wearable navigation device in a usability laboratory. The task requires the partici-pants to find the phone number of a restaurant called Matisse. It takes them several minutes to do this, and they appear to have problems. The video recording and interaction log suggest that the interface is quirky and the audio interaction is of poor quality. This is supported by participants’ answers on a user satisfaction questionnaire.1. What are the advantages and disadvantages of these two types of observation?2. When might each type of observation be useful?
Comment1. The advantages of the field study are that the observer saw how the device could be used
in a real situation to solve a real problem. They experienced the delight expressed with the overall concept and the frustration with the interface. By watching how the group used the device on the move, they gained an understanding of what the participants liked and what was lacking. The disadvantage is that the observer was an insider in the group,
(Continued)
8 D ATA G AT H E R I N G288
8.6.1 Direct Observation in the FieldIt can be difficult for people to explain what they do or to describe accurately how they achieve a task. It is unlikely that an interaction designer will get a full and true story using interviews or questionnaires. Observation in the field can help fill in details about how users behave and use technology, and nuances that are not elicited from other forms of investiga-tion may be observed. Understanding the context provides important information about why activities happen the way that they do. However, observation in the field can be complicated and harder to do well than at first appreciated. Observation can also result in a lot of data, some of which may be tedious to analyze and not very relevant.
All data gathering should have a clearly stated goal, but it is particularly important to have a focus for an observation session because there is always so much going on. On the other hand, it is also important to be prepared to change the plan if circumstances change. For example, the plan may be to spend one day observing an individual performing a task, but an unexpected meeting crops up, which is relevant to the observation goal and so it makes sense to attend the meeting instead. In observation, there is a careful balance between being guided by goals and being open to modifying, shaping, or refocusing the study as more is learned about the situation. Being able to keep this balance is a skill that develops with experience.
so how objective could they be? The data is qualitative, and while anecdotes can be very persuasive, how useful are they? Maybe they were having such a good time that their judgment was clouded and they missed hearing negative comments and didn’t notice some of the participant’s annoyance. Another study could be done to find out more, but it is not possible to replicate the exact conditions of this study. The advantages of the lab study are that it is easier to replicate, so several users could perform the same task, specific usability problems can be identified, users’ performance can be compared, and averages for such measures as the time it took to do a specific task and the number of errors can be calculated. The observer could also be more objective as an outsider. The disadvantage is that the study is artificial and says nothing about how the device would be used in the real environment.
2. Both types of study have merits. Which is better depends on the goals of the study. The lab study is useful for examining details of the interaction style to make sure that usabil-ity problems with the interface and button design are diagnosed and corrected. The field study reveals how the navigation device is used in a real-world context and how it inte-grates with or changes users’ behavior. Without this study, it is possible that developers might not have discovered the enthusiasm for the device because the reward for doing laboratory tasks is not as compelling as a good meal! In fact, according to Kjeldskov and Skov (2014), there is no definitive answer to which kind of study is preferable for mobile devices. They suggest that the real question is when and how to engage with longitudinal field studies.
8 . 6 O B s E R v AT I O N 289
Structuring Frameworks for Observation in the FieldDuring an observation, events can be complex and rapidly changing. There is a lot for observ-ers to think about, so many experts have a framework to structure and focus their observa-tion. The framework can be quite simple. For example, this is a practitioner’s framework for use in evaluation studies that focuses on just three easy-to-remember items:
The person: Who is using the technology at any particular time?The place: Where are they using it?The thing: What are they doing with it?
Even a simple framework such as this one based on who, where, and what can be surpris-ingly effective to help observers keep their goals and questions in sight. Experienced observ-ers may prefer a more detailed framework, such as the following (Robson and McCarten, 2016, p. 328), which encourages them to pay greater attention to the context of the activity:
Space: What is the physical space like, and how is it laid out?Actors: What are the names and relevant details of the people involved?Activities: What are the actors doing, and why?Objects: What physical objects are present, such as furniture?Acts: What are specific individual actions?Events: Is what you observe part of a special event?Time: What is the sequence of events?Goals: What are the actors trying to accomplish?Feelings: What is the mood of the group and of individuals?
This framework was devised for any type of observation, so when used in the context of interaction design, it might need to be modified slightly. For example, if the focus is going to be on how some technology is used, the framework could be modified to ask the following:
Objects: What physical objects, in addition to the technology being studied, are present, and do they impact on the technology use?
Both of these frameworks are relatively general and could be used in many different types of study, or as a basis for developing a new framework for a specific study.
AcTIvITy 8.6 1. Find a small group of people who are using any kind of technology, for example, smart-
phones, household appliances, or video game systems, and try to answer the question, “What are these people doing?” Watch for three to five minutes, and write down what you observe. When finished, note down how it felt to be doing this and any reactions in the group of people observed.
2. If you were to observe the group again, what would you do differently?3. Observe this group again for about 10 minutes using the detailed framework given above.
(Continued)
8 D ATA G AT H E R I N G290
Degree of ParticipationDepending on the type of study, the degree of participation within the study environment varies across a spectrum, which can be characterized as insider at one end and outsider at the other. Where a particular study falls along this spectrum depends on its goal and on the practical and ethical issues that constrain and shape it.
An observer who adopts an approach right at the outsider end of the spectrum is called a passive observer, and they will not take any part in the study environment at all. It is difficult to be a truly passive observer in the field, simply because it’s not possible to avoid interacting with the activities. Passive observation is more appropriate in lab studies.
An observer who adopts an approach at the insider end of this spectrum is called a participant observer. This means that they attempt, at various levels depending on the type of study, to become a member of the group being studied. This can be a difficult role to play since being an observer also requires a certain level of detachment, while being a participant assumes a different role. As a participant observer, it is important to keep the two roles clear and separate so that observation notes are objective while participation is also maintained. It may not be possible to take a full participant observer approach for other reasons. For example, the observer may not be skilled enough in the task at hand, the organization/group may not be prepared for an outsider to take part in their activities, or the timescale may not provide sufficient opportunity to become familiar enough with the task to participate fully. Similarly, if observing activity in a private place such as the home, full participation would be difficult even if, as suggested by some researchers (for example, Bell et al., 2005), you have spent time getting to know the family before start-ing the study. Chandrika Cycil et al. (2013) overcame this issue in their study of in-car conversations between parents and children by traveling with the families initially for a week and then asking family members to video relevant episodes of activity. In this way, they had gained an understanding of the context and family dynamics and then collected more detailed data to study activity in depth.
Comment1. What problems did this exercise highlight? Was it hard to watch everything and remember
what happened? How did the people being watched feel? Did they know they were being watched? Perhaps some of them objected and walked away. If you didn’t tell them that they were being watched, should you have?
2. The initial goal of the observation, that is, to find out what the people are doing, was vague, and chances are that it was quite a frustrating experience not knowing what was significant and what could be ignored. The questions used to guide observation need to be more focused. For example, you might ask the following: What are the people doing with the technology? Is everyone in the group using it? Are they looking pleased, frustrated, serious, happy? Does the technology appear to be central to the users’ goals?
3. Ideally, you will have felt more confident this second time, partly because it is the second time doing some observation and partly because the framework provided a structure for what to look at.
8 . 6 O B s E R v AT I O N 291
Planning and Conducting an Observation in the FieldThe frameworks introduced in the previous section are useful for providing focus and also for organizing the observation and data gathering activity. Choosing a framework is important, but there are other decisions that need to be made, including the level of participation to adopt, how to make a record of the data, how to gain acceptance in the group being studied, how to handle sensitive issues such as cultural differences or access to private spaces, and how to ensure that the study uses different perspectives (people, activities, job roles, and so forth).
One way to achieve this last point is to work as a team. This can have several benefits.
• Each person can agree to focus on different people or different parts of the context, thereby covering more ground.
• Observation and reflection can be interweaved more easily when there is more than one observer.
• More reliable data is likely to be generated because observations can be compared. • Results will reflect different perspectives.
Once in the throes of an observation, there are other issues that need to be considered. For example, it will be easier to relate to some people more than others. Although it will be tempting to pay attention to them more than others, attention needs to be paid to everyone in the group. Observation is a fluid activity, and the study will need to be refocused as it progresses in response to what is learned. Having observed for a while, interesting phenom-ena that seem relevant will start to emerge. Gradually, ideas will sharpen into questions that guide further observation.
Observing is also an intense and tiring activity, but checking notes and records and reviewing observations and experiences at the end of each day is important. If this is not done, then valuable information may be lost as the next day’s events override the previous day’s findings. Writing a diary or private blog is one way of achieving this. Any documents or other artifacts that are collected or copied (such as minutes of a meeting or discussion items) can be annotated, describing how they are used during the observed activity. Where an observation lasts several days or weeks, time can be taken out of each day to go through notes and other records.
As notes are reviewed, separate personal opinion from observation and mark issues for further investigation. It is also a good idea to check observations and interpretations with an informant or members of the participant group for accuracy.
DILEMMAWhen to Stop Observing?
Knowing when to stop doing any type of data gathering can be difficult for novices, but it is particularly tricky in observational studies because there is no obvious ending. Schedules often dictate when your study ends. Otherwise, stop when nothing new is emerging. Two indica-tions of having done enough are when similar patterns of behavior are being seen and when all of the main stakeholder groups have been observed and a good understanding of their perspectives has been achieved.
8 D ATA G AT H E R I N G292
EthnographyEthnography has traditionally been used in the social sciences to uncover the organization of societies and their activities. Since the early 1990s, it has gained credibility in interaction design, and particularly in the design of collaborative systems; see Box 8.6, “Ethnography in Requirements” and Crabtree (2003). A large part of most ethnographic studies is direct observation, but interviews, questionnaires, and studying artifacts used in the activities also feature in many ethnographic studies. A distinguishing feature of ethnographic studies com-pared with other data gathering is that a situation is observed without imposing any a priori structure or framework upon it, and everything is viewed as “strange.” In this way, the aim is to capture and articulate the participants’ perspective of the situation under study.
BOX 8.6 Ethnography in Requirements
The MERboard is a tool scientists and engineers use to display, capture, annotate, and share information in support of the operation of two Mars Exploration Rovers (MERs) on the sur-face of Mars. The MER (see Figure 8.8) acts like a human geological explorer by collecting and analyzing samples and then transmitting the results to the scientists on Earth. The scien-tists and engineers collaboratively analyze the data received, decide what to study next, create plans of action, and send commands to the robots on the surface of Mars.
The requirements for MERboard were identified partly through ethnographic field-work, observations, and analysis (Trimble et al., 2002). The team of scientists and engi-neers ran a series of field tests that simulated the process of receiving data, analyzing it, creating plans, and transmitting them to the MERs. The main problems they identified stemmed from the scientists’ limitations in displaying, sharing, and storing information (see Figure 8.9a).
Figure 8.8 Mars Exploration Rover Source: NASA Jet Propulsion Laboratory (NASA-JPL)
8 . 6 O B s E R v AT I O N 293
Ethnography has become popular within interaction design because it allows designers to obtain a detailed and nuanced understanding of people’s behavior and the use of technol-ogy that cannot be obtained by other methods of data gathering (Lazar et al., 2017). While there has been much discussion of how big data can address many design issues, big data is likely to be most powerful when combined with ethnography to explain how and why people do what they do (Churchill, 2018).
The observer in an ethnographic study adopts a participant observer (insider) role as much as possible (Fetterman, 2010). While participant observation is a hallmark of eth-nographic studies, it is also used within other methodological frameworks such as action research (Hayes, 2011), where one of the goals is to improve the current situation.
Ethnographic data is based on what is available, what is “ordinary,” what it is that people do, say, and how they work. The data collected therefore has many forms: documents, notes taken by the observer(s), pictures, and room layout sketches. Notes may include snippets of conversations and descriptions of rooms, meetings, what someone did, or how people reacted to a situation. Data gathering is opportunistic, and observers make the most of opportunities as they present themselves. Often, interesting phenomena do not reveal themselves immedi-ately but only later, so it is important to gather as much as possible within the framework of observation. Initially, spend time getting to know people in the participant group and bond-ing with them. Participants need to understand why the observers are there, what they hope
These observations led to the development of MERboard (see Figure 8.9b), which contains four core applications: a whiteboard for brainstorming and sketching, a browser for display-ing information from the web, the capability to display personal information and information across several screens, and a file storage space linked specifically to MERboard.
(a) (b)
Figure 8.9 (a) The situation before MERboard; (b) a scientist using MERboard to present information Source: Trimble et al. (2002)
8 D ATA G AT H E R I N G294
to achieve, and how long they plan to be there. Going to lunch with them, buying coffee, and bringing small gifts, for example, cookies, can greatly help this socialization process. Moreover, key information may be revealed during one of these informal gatherings.
It is important to show interest in the stories, gripes, and explanations that are provided and to be prepared to step back if a participant’s phone rings or someone else enters the workspace. A good tactic is to explain to one of the participants during a quiet moment what you think is happening and then let them correct any misunderstandings. However, asking too many ques-tions, taking pictures of everything, showing off your knowledge, and getting in their way can be very off-putting. Putting up cameras on tripods on the first day may not be a good idea. Listening and watching while sitting on the sidelines and occasionally asking questions is a better approach.
The following is an illustrative list of materials that might be recorded and collected dur-ing an ethnographic study (adapted from Crabtree, 2003, p. 53):
• Activity or job descriptions • Rules and procedures (and so on) that govern particular activities • Descriptions of activities observed • Recordings of the talk taking place between parties involved in observed activities • Informal interviews with participants explaining the detail of observed activities • Diagrams of the physical layout, including the position of artifacts • Photographs of artifacts (documents, diagrams, forms, computers, and so on) used in the course of observed activities
• Videos of artifacts as used in the course of observed activities • Descriptions of artifacts used in the course of observed activities • Workflow diagrams showing the sequential order of tasks involved in observed activities • Process maps showing connections between activities
Traditionally, ethnographic studies in this field aim to understand what people do and how they organize action and interaction within a particular context of interest to designers. However, recently there has been a trend toward studies that draw more on ethnography’s anthropological roots and the study of culture. This trend has been brought about by the perceived need to use different approaches because the computers and other digital technologies, especially mobile devices, are embedded in everyday activity, and not just in the workplace as in the 1990s.
BOX 8.7 Doing Ethnography Online
As collaboration and social activity online have increased, ethnographers have adapted their approach to study social media and the various forms of computer-mediated communication (Rotman et al., 2013; Bauwens and Genoud, 2014). This practice has various names, the most common of which are online ethnography (Rotman et al., 2012), virtual ethnography (Hine, 2000), and netnography (Kozinets, 2010). Where a community or activity has both an online and offline presence, it is common to incorporate both online and offline techniques within the data gathering program. However, where the community or activities of interest exist almost exclu-sively online, then mostly online techniques are used and virtual ethnography becomes central.
8 . 6 O B s E R v AT I O N 295
8.6.2 Direct Observation in Controlled EnvironmentsObserving users in a controlled environment may occur within a purposely built usability lab, but portable labs that can be set up in any room are quite common. Portable laboratories can mean that more participants take part because they don’t have to travel away from their normal environment. Observation in a controlled environment inevitably takes on a more formal character than observation in the field, and the user may feel more apprehensive. As with interviews, it is a good idea to prepare a script to guide how the participants will be greeted, be told about the goals of the study and how long it will last, and have their rights explained. Use of a script ensures that each participant will be treated in the same way, which brings more credibility to the results obtained from the study.
The same basic data recording techniques are used for direct observation in the labora-tory and field studies (that is, capturing photographs, taking notes, collecting video, and so on), but the way in which these techniques are used is different. In the lab the emphasis is on the details of what individuals do, while in the field the context is important, and the focus is on how people interact with each other, the technology, and their environment.
The arrangement of equipment with respect to the participant is important in a controlled study because details of the person’s activity need to be captured. For example, one camera might record facial expressions, another might focus on mouse and keyboard activity, and another might record a broad view of the participant and capture body language. The stream of data from the cameras can be fed into a video editing and analysis suite where it is coor-dinated and time-stamped, annotated, and partially edited.
Why is it necessary to distinguish between online and face-to-face ethnography? It is important because interaction online is different from interaction in person. For example, communication in person is richer (through gesture, facial expression, tone of voice, and so on) than online communication, and anonymity is more easily achieved when communicating online. In addition, virtual worlds have a persistence, due to regular archiving, that does not typically occur in face-to-face situations. This makes characteristics of the communication dif-ferent, which often includes how ethnographers introduce themselves to the community, how they act within the community, and how they report their findings. For these reasons, some researchers who work primarily online also try to meet with some of the participants face-to-face, particularly when working on sensitive topics (Lingel, 2012).
Special tools may be developed to support ethnographic data collection. Mobilab is an online collaborative platform that was developed for citizens living in Switzerland to report and discuss their daily mobility during an eight-week period using their mobile phones, tablets, and computers (Bauwens and Genoud, 2014). Mobilab enabled the researchers to more easily engage in discussion with participants on a variety of topics, including trucks parking on a bikeway.
For observational studies in large social spaces, such as digital libraries or Facebook, there are different ethical issues to consider. For example, it is unrealistic to ask everyone using a digital library to sign any kind of form agreeing to be involved in the study, yet participants do need to understand the observer’s role and the purpose of their study. The presentation of results needs to be modified too. Quotes from participants in the community, even if anonymized in the report, can easily be attributed by a simple search of the community archive or the IP address of the sender, so care is needed to protect their privacy.
8 D ATA G AT H E R I N G296
The Think-Aloud TechniqueOne of the problems with observation is that the observer doesn’t know what users are think-ing and can only guess from what they see. Observation in the field should not be intrusive, as this will disturb the context the study is trying to capture. This limits the questions being asked of the participant. However, in a controlled environment, the observer can afford to be a little more intrusive. The think-aloud technique is a useful way of understanding what is going on in a person’s head.
Imagine observing someone who has been asked to evaluate the interface of the web search engine Lycos.com. The user, who does not have much experience of web searches, is told to look for a phone for a 10-year-old child. They are told to type www.lycos.com and then proceed however they think best. They type the URL and get a screen similar to the one in Figure 8.10.
Next, they type child’s phone in the search box. They get a screen similar to the one shown in Figure 8.11. They are silent. What is going on? What are they thinking? One way around the problem of knowing what they are doing is to collect a think-aloud pro-tocol, a technique developed by Anders Ericsson and Herbert Simon (1985) for examin-ing people’s problem-solving strategies. The technique requires people to say out loud everything that they are thinking and trying to do so that their thought processes are externalized.
So, let’s imagine an action replay of the situation just described, as follows, but this time the user has been instructed to think aloud:
“I’m typing in www.lycos.com, as you told me.” <types>“Now I am typing child’s phone and then clicking the search button.<pause and silence>“It’s taking a few seconds to respond.”“Oh! Now I have a choice of other websites to go to. Hmm, I wonder which one I should select. Well, it’s for a young child so I want a ‘child-safe phone.’ This one mentions safe phones <He clicks on 7 Best Cell Phones for Kids – Mashable>“Gosh, there’s a lot more models to select from than I expected! Hmm, some of these are for older children. I wonder what I do next to find one for a 10-year-old.”
Figure 8.10 Home page of Lycos search engineSource: https://www.lycos.com
8 . 6 O B s E R v AT I O N 297
<pauses and looks at the screen>I guess I should scroll through them and identify those that might be appropriate.”<silence . . . >
Now you know more about what the user is trying to achieve, but they are silent again. They are looking at the screen, but what are they thinking now? What are they looking at?
The occurrence of these silences is one of the biggest problems with the think-aloud technique.
Figure 8.11 The screen that appears in response to searching for “child’s phone”Source: https://www.lycos.com
AcTIvITy 8.7 Try a think-aloud exercise yourself. Go to a website, such as Amazon or eBay, and look for something to buy. Think aloud as you search and notice how you feel and behave.
Afterward, reflect on the experience. Was it difficult to keep speaking all the way through the task? Did you feel awkward? Did you stop talking when you got stuck?
CommentFeeling self-conscious and awkward doing this is a common response, and some people say they feel really embarrassed. Many people forget to speak out loud and find it difficult to do so when the task becomes difficult. In fact, you probably stopped speaking when the task became demanding, and that is exactly the time when an observer is most eager to hear what’s happening.
(Continued)
8 D ATA G AT H E R I N G298
8.6.3 Indirect Observation: Tracking Users’ ActivitiesSometimes direct observation is not possible because it is too intrusive or observers cannot be present over the duration of the study, and so activities are tracked indirectly. Diaries and interaction logs are two techniques for doing this.
DiariesParticipants are asked to write a diary of their activities on a regular basis, including things like what they did, when they did it, what they found hard or easy, and what their reactions were to the situation. For example, Sohn et al. (2008) asked 20 participants to record their mobile information needs through text messages and then to use these messages as prompts to help them answer six questions on a website at the end of each day. From the data collected, they identified 16 categories of mobile information needs, the most frequent of which was “trivia.”
Diaries are useful: when participants are scattered and unreachable in person; when the activity is private, for example, in the home; or when it relates to feelings, for instance, emo-tions or motivation. For example, Jang et al. (2016) used diaries with interviews to collect data about users’ experiences with smart TVs in the home as compared to within a con-trolled lab setting. The study in the home was conducted over several weeks during which participants were asked to keep a diary of their experiences and feelings. Surveys were also collected. This mixed-methods study informed the user experience design of future systems.
Diaries have several advantages: they do not take up much researcher time to collect data; they do not require special equipment or expertise; and they are suitable for long-term studies. In addition, templates, like those used in open-ended online questionnaires, can be created online to standardize the data entry format so that the data can be entered directly into a database for analysis. However, diary studies rely on participants being reliable and remembering to complete them at the assigned time and as instructed, so incentives may be needed, and the process has to be straightforward.
Determining how long to run a diary study can be tricky. If the study goes on for too long, participants may lose interest and need incentives to continue. In contrast, if the study is too short, important data may be missed. For example, in a study of children’s experiences of a game, Elisa Mekler et al. (2014) used diaries to collect data after each gaming session in a series. After the first few sessions, all of the children in the study showed loss of motiva-tion for the game. However, by the end of the study, those who completed the game were more motivated than those who did not complete the game. Had the data been collected only once, the researchers may not have observed the impact of game completion on the children’s motivation.
If a user is silent during a think-aloud protocol, the observer could interrupt and remind them to think out loud, but that would be intrusive. Another solution is to have two people work together so that they talk to each other. Working with another person (called construc-tive interaction [Miyake, 1986]) is often more natural and revealing because participants talk in order to help each other along. This technique has proved to be particularly success-ful with children, and it also avoids possible cultural influences on concurrent verbalization (Clemmensen et al., 2008).
8 . 6 O B s E R v AT I O N 299
Another problem is that the participants’ memories of events may be exaggerated or detail is forgotten; for example, they may remember them as better or worse than they really were or as taking more or less time than they actually did take. One way of mitigating this problem is to collect other data in diaries (such as photographs including selfies, audio and video clips, and so on). Scott Carter and Jennifer Mankoff (2005) considered whether capturing events through pictures, audio, or artifacts related to the event affects the results of the diary study. They found that images resulted in more specific recall than other media, but audio was useful for capturing events when taking a photo was too awkward. Tangible artifacts, such as those shown in Figure 8.12, also encouraged discussion about wider beliefs and attitudes.
The experience sampling method (ESM) is similar to a diary in that it relies on partici-pants recording information about their everyday activities. However, it differs from more traditional diary studies because participants are prompted at random times via email, text message, or similar means to answer specific questions about their context, feelings, and actions (Hektner et al., 2006). These prompts have the benefit of encouraging immediate data capture. Niels van Berkel et al. (2017) provide a comprehensive survey of ESM and its evolution, tools, and uses across a wide range of studies.
Interaction Logs, Web Analytics, and Data ScrapingInteraction logging uses software to record users’ activity in a log that can be examined later. A variety of actions may be recorded, such as key presses and mouse or other device move-ments, time spent searching a web page, time spent looking at help systems, and task flow
Figure 8.12 Some tangible objects collected by participants involved in a study about a jazz festival Source: Carter and Mankoff (2005). Reproduced with permission of ACM Publications
8 D ATA G AT H E R I N G300
through software modules. A key advantage of logging activity is that it is unobtrusive pro-vided system performance is not affected, but it also raises ethical concerns about observing participants if this is done without their knowledge. Another advantage is that large volumes of data can be logged automatically. Visualization tools are therefore helpful for exploring and analyzing this data quantitatively and qualitatively. Algorithmic and statistical methods may also be used.
Examining the trail of activity that people leave behind when they are active on websites, Twitter, or Facebook is also a form of indirect observation. You can see an example of this by looking at a Twitter feed to which you have access, for example, that of a friend, president, prime minister, or some other leader. These trails allow examination of discussion threads on a particular topic, such as climate change, or reactions to comments made by a public figure or to a topic that is trending today. If there are just a few posts, then it is easy to see what is going on, but often the most interesting posts are those that generate a lot of comments. Examining thousands, tens of thousands, and even millions of posts requires automated tech-niques. Web analytics and data scraping are discussed further in Chapter 10.
8.7 Choosing and Combining Techniques
Combining data gathering techniques into a single data gathering program is common practice, for example, when collecting case study data (see Box 8.8). The benefit of using a combination of methods is to provide multiple perspectives. Choosing which data gathering techniques to use depends on a variety of factors related to the study goals. There is no right technique or combination of techniques, but some will undoubtedly be more appropriate than others. The decision about which to use will need to be made after taking all of the factors into account.
Table 8.1 provides an overview to help choose a set of techniques for a specific project. It lists the kind of information obtained (such as answers to specific questions) and the type of data (for example, mostly qualitative or mostly quantitative). It also includes some advantages and disadvantages for each technique. Note that different modalities can be used for some of these techniques. For example, interviews and focus groups can be conducted face-to-face, by phone, or through teleconferencing, so when considering advantages and disadvantages of the techniques, this should also be taken into account.
In addition, technique choice is influenced by practical issues.
• The focus of the study. What kind of data will support the focus and goal of the study? This will be influenced by the interaction design activity and the level of maturity of the design.
• The participants involved. Characteristics of the target user group including their location and availability.
• The nature of the technique. Does the technique require specialist equipment or training, and do the investigators have the appropriate knowledge and experience?
• Available resources. Expertise, tool support, time, and money.
8 . 7 c H O O s I N G A N D c O M B I N I N G T E c H N I Q u E s 301
Technique Good for Kind of data Advantages Disadvantages
Interviews Exploring issues
Some quantitative but mostly qualitative
Interviewer can guide interviewee if necessary. Encourages contact between developers and users.
Artificial environment may intimidate interviewee. It also removes them from the environment where work is typically being done.
Focus groups Collecting multiple viewpoints
Some quantitative but mostly qualitative
Highlights areas of consensus and conflict. Encourages contact between developers and users.
Possibility of dominant characters.
Questionnaires Answering specific questions
Quantitative and qualitative
Can reach many people with low resource requirements.
The design is key. Response rates may be low. Unless carefully designed, the responses may not provide suitable data.
Direct observation in the field
Understanding context of user activity
Mostly qualitative
Observing gives insights that other techniques don’t provide.
Very time-consuming. Huge amounts of data are produced.
Direct observation in a controlled environment
Capturing the detail of what individuals do
Quantitative and qualitative
Can focus on the details of a task without interruption.
Results may have limited use in the normal environment because the conditions were artificial.
Indirect observation
Observing users without disturbing their activity; data captured automatically
Quantitative (logging) and qualitative (diary)
User doesn’t get distracted by the data gathering; automatic recording means that it can extend over long periods of time.
A large amount of quantitative data needs tool support to analyze (logging); participants’ memories may exaggerate (diary).
Table 8.1 Overview of data gathering techniques and their use
8 D ATA G AT H E R I N G302
AcTIvITy 8.9 For each of the following products, consider what kinds of data gathering would be appropri-ate and how to use the different techniques introduced earlier. Assume that product develop-ment is just starting and that there is sufficient time and resources to use any of the techniques.1. A new software app to support a small organic produce shop. There is a system running
already with which the users are reasonably happy, but it is looking dated and needs upgrading.2. An innovative device for diabetes sufferers to help them record and monitor their blood
sugar levels.3. An ecommerce website that sells fashion clothing for young people.
Comment1. As this is a small shop, there are likely to be few stakeholders. Some period of observation
would be important to understand the context of the new and the old systems. Interview-ing the staff rather than giving them questionnaires is likely to be appropriate because there aren’t very many of them, and this will yield richer data and give the developers a chance to meet the users. Organic produce is regulated by a variety of laws, so looking at this documentation will help you understand any legal constraints that have to be taken into account. This suggests a series of interviews with the main users to understand the positive and negative features of the existing system, a short observation session to under-stand the context of the system, and a study of documentation surrounding the regulations.
2. In this case, the user group is quite large and spread out geographically, so talking to all of them is not feasible. However, interviewing a representative sample of potential users,
BOX 8.8 Collecting case study data
Case studies often use a combination of methods, for example, direct and indirect observa-tions and interviews. Although people frequently use the term case study colloquially to refer to a study that they are using as a case example, there is also a case study methodology that collects field study data over days, months, or even years. There is a body of literature that provides advice on how to do good case studies. Robert Yin (2013), for example, identifies these data collection sources: documentation, archival records, interviews, direct observations, participant observation, and physical artifacts. Case studies are good for integrating multiple per-spectives, for example, studying new technology in the wild, and for giving meaning to first impres-sions. The data collection process tends to be intensive, concurrent, interactive, and iterative.
In a study of how local communities organize and adapt technology for managing their local rivers and streams, approaching it as a case study allowed a detailed contextual analysis of events and relationships that occurred over multiple groups of volunteers during a two-year period (Preece et al., 2019). From this study, the researchers learned about the volunteers’ needs for highly flexible software to support the diverse groups of participants working on a wide range of water-related topics.
8 . 7 c H O O s I N G A N D c O M B I N I N G T E c H N I Q u E s 303
In-Depth ActivityThe aim of this in-depth activity is to practice data gathering. Assume that you have been employed to improve the user experience of an interactive product such as a smartphone app, a digital media player, a Blu-ray player, computer software, or some other type of technology. This existing product may be redesigned, or a completely new product may be created. To do the assignment, find a group of people or a single individual prepared to be the user group. These could be your family, friends, peers, or people in a local community group.For this assignment:(a) Clarify the basic goal of improving the product by considering what this means in your
circumstances.(b) Watch the group (or person) casually to get an understanding of any issues that might cre-
ate challenges for this activity and any information to help refine the study goals.(c) Explain how you would use each of the three data gathering techniques: interview, ques-
tionnaire, and observation in your data gathering program. Explain how your plan takes account of triangulation.
(d) Consider your relationship with the user group and decide if an informed consent form is required. (Figure 8.1 will help you to design one if needed.)
(e) Plan your data gathering program in detail. • Decide what kind of interview to run and design a set of interview questions. Decide how
to record the data, then acquire and test any equipment needed and run a pilot study. • Decide whether to include a questionnaire in your data gathering program, and design
appropriate questions for it. Run a pilot study to check the questionnaire. • Decide whether to use direct or indirect observation and where on the outsider/insider
spectrum should the observers be. Decide how to record the data, then acquire and test any equipment needed and run a pilot study.
(f) Carry out the study, but limit its scope. For example, interview only two or three people or plan only two half-hour observation periods.
(g) Reflect on this experience and suggest what you would do differently next time.Keep the data gathered, as this will form the basis of the in-depth activity in Chapter 9.
possibly at a local diabetic clinic, is feasible. Observing current practices to monitor blood sugar levels will help you understand what is required. An additional group of stakeholders would be those who use or have used the other products on the market. These stakeholders can be questioned about their experience with their existing devices so that the new device can be an improvement. A questionnaire sent to a wider group in order to confirm the findings from the interviews would be appropriate, as might a focus group where possible.
3. Again, the user group is quite large and spread out geographically. In fact, the user group may not be very well defined. Interviews backed up by questionnaires and focus groups would be appropriate. In this case, identifying similar or competing sites and evaluating them will help provide information for an improved product.
8 D ATA G AT H E R I N G304
Further Reading
FETTERMAN, D. M. (2010). Ethnography: Step by Step (3rd ed.) Applied Social Research Methods Series, Vol. 17. Sage. This book introduces the theory and practice of ethnography, and it is an excellent guide for beginners. It covers both data gathering and data analysis in the ethnographic tradition.
FULTON SURI, J. (2005) Thoughtless Acts? Chronicle Books, San Francisco. This intriguing little book invites you to consider how people react to their environment. It is a good intro-duction to the art of observation.
HEATH, C., HINDMARSH, J. AND LUFF, P. (2010) Video in Qualitative Research: Analyzing Social Interaction in Everyday Life. Sage. This is an accessible book that provides practical advice and guidance about how to set up and perform data gathering using video record-ing. It also covers data analysis, presenting findings, and potential implications from video research based on their own experience.
summaryThis chapter has focused on three main data gathering methods that are commonly used in interaction design: interviews, questionnaires, and observation. It has described in detail the planning and execution of each. In addition, five key issues of data gathering were presented, and how to record the data gathered was discussed.
Key Points • All data gathering sessions should have clear goals. • Depending on the study context, an informed consent form and other permissions may be needed to run the study.
• Running a pilot study helps to test out the feasibility of a planned data gathering session and associated instruments such as questions.
• Triangulation involves investigating a phenomenon from different perspectives. • Data may be recorded using handwritten notes, audio or video recording, a camera, or any combination of these.
• There are three styles of interviews: structured, semi-structured, and unstructured. • Questionnaires may be paper-based, via email, or online. • Questions for an interview or questionnaire can be open or closed-ended. Closed-ended questions require the interviewee to select from a limited range of options. Open-ended questions accept a free-range response.
• Observation may be direct or indirect. • In direct observation, the observer may adopt different levels of participation, ranging from insider (participant observer) to outsider (passive observer).
• Choosing appropriate data gathering techniques depends on the focus of the study, partici-pants involved, nature of the technique, and resources available.
F u R T H E R R E A D I N G 305
OLSON, J. S. AND KELLOGG, W. A. (eds) (2014) Ways of Knowing in HCI. Springer. This edited collection contains useful chapters on a wide variety of data collection and analysis techniques. Some topics that are particularly relevant to this chapter include: ethnography, experimental design, log data collection and analysis, ethics in research, and more.
ROBSON, C. AND McCARTAN, K. (2016) Real World Research (4th edn). John Wiley & Sons. This book provides comprehensive coverage of data gathering and analysis techniques and how to use them. Early books and related books by Robson also address topics discussed in this chapter.
TOEPOEL, V. (2016) Doing Surveys Online. Sage. This book is a “hands-on guide” for pre-paring and conducting a wide range of surveys including surveys for mobile devices, opt-in surveys, panels, polls, and more. It also discusses details about sampling that can be applied to other data gathering techniques.
Chapter 9
D A T A A N A LY S I S , I N T E R P R E T A T I O N , A N D P R E S E N T A T I O N
9.1 Introduction
9.2 Qualitative and Quantitative
9.3 Basic Quantitative Analysis
9.4 Basic Qualitative Analysis
9.5 What Kind of Analytic Framework to Use
9.6 Tools to Support Data Analysis
9.7 Interpreting and Presenting the Findings
ObjectivesThe main goals of this chapter are to accomplish the following:
• Discuss the difference between qualitative and quantitative data and analysis. • Enable you to analyze data gathered from questionnaires. • Enable you to analyze data gathered from interviews. • Enable you to analyze data gathered from observation studies. • Make you aware of software packages that are available to help your analysis. • Identify some of the common pitfalls in data analysis, interpretation, and presentation. • Enable you to interpret and present your findings in a meaningful and appropriate manner.
9.1 Introduction
The kind of analysis that can be performed on a set of data will be influenced by the goals identified at the outset and the data gathered. Broadly speaking, a qualitative analysis approach, a quantitative analysis approach, or a combination of qualitative and quantitative approaches may be taken. The last of these is very common, as it provides a more comprehensive account of the behavior being observed or the performance being measured.
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N308
Most analysis, whether it is quantitative or qualitative, begins with the initial reactions or observations from the data. This may involve identifying patterns or calculating simple numerical values such as ratios, averages, or percentages. For all data, but especially when dealing with large volumes of data (that is, Big Data), it is useful to look over the data to check for any anomalies that might be erroneous. For example, people who are 999 years old. This process is known as data cleansing, and there are often digital tools to help with the process. This initial analysis is followed by more detailed work using structured frameworks or theories to support the investigation.
Interpretation of the findings often proceeds in parallel with analysis, but there are differ-ent ways to interpret results, and it is important to make sure that the data supports any con-clusions. A common mistake is for the investigator’s existing beliefs or biases to influence the interpretation of results. Imagine that an initial analysis of the data has revealed a pattern of responses to customer care questionnaires that indicates that inquiries from customers routed through the Sydney office of an organization take longer to process than those routed through the Moscow office. This result can be interpreted in many different ways. For example, the customer care operatives in Sydney are less efficient, they provide more detailed responses, the technology supporting the inquiry process in Sydney needs to be updated, customers reaching the Sydney office demand a higher level of service, and so on. Which one is correct? To determine whether any of these potential interpretations is accurate, it would be appropri-ate to look at other data such as customer inquiry details and maybe to interview staff.
Another common mistake is to make claims that go beyond what the data can support. This is a matter of interpretation and of presentation. Using words such as many or often or all when reporting conclusions needs to be carefully considered. An investigator needs to remain as impartial and objective as possible if the conclusions are to be trusted. Showing that the conclusions are supported by the results is an important skill to develop.
Finally, finding the best way to present findings is equally skilled, and it depends on the goals but also on the audience for whom the study was performed. For example, a formal notation may be used to report the results for the requirements activity, while a summary of problems found, supported by video clips of users experiencing those problems, may be better for presentation to the team of developers.
This chapter introduces a variety of methods, and it describes in more detail how to approach data analysis and presentation using some of the common approaches taken in interaction design.
9.2 Quantitative and Qualitative
Quantitative data is in the form of numbers, or data that can easily be translated into numbers. Examples are the number of years’ experience the interviewees have, the number of projects a department handles at a time, or the number of minutes it takes to perform a task. Qualitative data is in the form of words and images, and it includes descriptions, quotes from interviewees, vignettes of activity, and photos. It is possible to express qualitative data in numerical form, but it is not always meaningful to do so (see Box 9.1).
It is sometimes assumed that certain forms of data gathering can only result in quantitative data and that others can only result in qualitative data. However, this is a fallacy. All forms of data gathering discussed in the previous chapter may result in qualitative and quantitative
9 . 2 Q u A N T I TAT I v E A N D Q u A L I TAT I v E 309
data. For example, on a questionnaire, questions about the participant’s age or number of software apps they use in a day will result in quantitative data, while any comments will result in qualitative data. In an observation, quantitative data that may be recorded includes the number of people involved in a project or how many hours someone spends sorting out a problem, while notes about feelings of frustration, or the nature of interactions between team members, are qualitative data.
Quantitative analysis uses numerical methods to ascertain the magnitude, amount, or size of something; for example, the attributes, behavior, or strength of opinion of the partici-pants. For example, in describing a population, a quantitative analysis might conclude that the average person is 5 feet 11 inches tall, weighs 180 pounds, and is 45 years old. Qualita-tive analysis focuses on the nature of something and can be represented by themes, patterns, and stories. For example, in describing the same population, a qualitative analysis might conclude that the average person is tall, thin, and middle-aged.
BOX 9.1Use and Abuse of Numbers
Numbers are infinitely malleable and can make a convincing argument, but it is important to justify the manipulation of quantitative data and what the implications will be. Before adding a set of numbers together, finding an average, calculating a percentage, or performing any other kind of numerical translation, consider whether the operation is meaningful in the specific context.
Qualitative data can also be turned into a set of numbers. Translating non-numerical data into a numerical or ordered scale is appropriate at times, and this is a common approach in interaction design. However, this kind of translation also needs to be justified to ensure that it is meaningful in the given context. For example, assume you have collected a set of interviews from sales representatives about their use of a new mobile app for reporting sales queries. One way of turning this data into a numerical form would be to count the number of words uttered by each interviewee. Conclusions might then be drawn about how strongly the sales representatives feel about the app; for example, the more they had to say about the product, the stronger they felt about it. But do you think this is a wise way to analyze the data? Does it help to answer the study questions?
Other, less obvious, abuses include translating small population sizes into percentages. For example, saying that 50 percent of users take longer than 30 minutes to place an order through an e-commerce website carries a different meaning than saying that two out of four users had the same problem. It is better not to use percentages unless the number of data points is at least 10, and even then it is appropriate to use both percentages and raw numbers to make sure that the claim is not misunderstood.
It is possible to perform legitimate statistical calculations on a set of data and still present misleading results by not making the context clear or by choosing the particular calculation that gives the most favorable result (Huff, 1991). In addition, choosing and applying the best statistical test requires careful thinking (Cairns, 2019), as using an inappropriate test can unintentionally misrepresent the data.
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N310
9.2.1 First Steps in Analyzing DataHaving collected the data, some initial processing is normally required before data analy-sis can begin in earnest. For example, audio data may be transcribed by hand or by using an automated tool, such as Dragon; quantitative data, such as time taken or errors made, is usually entered into a spreadsheet, like Excel. Initial analysis steps for data typi-cally collected through interviews, questionnaires, and observation are summarized in Table 9.1.
InterviewsInterviewer notes need to be written up and expanded as soon as possible after the interview has taken place so that the interviewer’s memory is clear and fresh. An audio or video record-ing may be used to help in this process, or it may be transcribed for more detailed analysis.
Usual raw data Example qualitative data
Example quantitative data
Initial processing steps
Interviews Audio recordings.Interviewer notes.Video recordings.
Responses to open-ended questions.Video pictures.Respondent’s opinions.
Age, job role, years of experience.Responses to close-ended questions.
Transcription of recordings.Expansion of notes.Entry of answers to close-ended questions into a spreadsheet
Question naires Written responses.Online database.
Responses to open-ended questions.Responses in “further comments” fields.Respondent’s opinions.
Age, job role, years of experience.Responses to close-ended questions.
Clean up data.Filter into different data sets.Synchronization between data recordings.
Observation Observer’s notes.Photographs.Audio and video recordings.Data logs.Think-aloudDiaries.
Records of behavior.Description of a task as it is undertaken.Copies of informal procedures.
Demographics of participants.Time spent on a task.The number of people involved in an activity.How many different types of activity are undertaken.
Expansion of notes.Transcription of recordings.
Table 9.1 Data gathered and typical initial processing steps for interviews, questionnaires, and observation
9 . 3 B A S I c Q u A N T I TAT I v E A N A LY S I S 311
Transcription takes significant effort, as people talk more quickly than most people can type (or write), and the recording is not always clear. It is worth considering whether to transcribe the whole interview or just sections of it that are relevant. Deciding what is relevant, however, can be difficult. Revisiting the goals of the study to see which passages address the research questions can guide this process.
Close-ended questions are usually treated as quantitative data and analyzed using basic quantitative analysis (see Section 9.3 “Basic Quantitative Analysis”). For example, a question that asks for the respondent’s age range can easily be analyzed to find out the percentage of respondents in each. More complicated statistical techniques are needed to identify relation-ships between responses that can be generalized, such as whether there is an interaction between the condition being tested and a demographic. For example, do people of different ages use Facebook for different lengths of time when first logging on in the morning or at night before they go to bed? Open-ended questions typically result in qualitative data that might be searched for categories or patterns of response.
QuestionnairesIncreasingly, questionnaire responses are provided using online surveys, and the data is auto-matically stored in a database. The data can be filtered according to respondent subpopula-tions (for instance, everyone under 16) or according to a particular question (for example, to understand respondents’ reactions to one kind of robot personality rather than another). This allows analyses to be conducted on subsets of the data and hence to draw specific con-clusions for more targeted goals. To conduct this kind of analysis requires sufficient data from a large enough sample of participants.
ObservationObservation can result in a wide variety of data including notes, photographs, data logs, think-aloud recordings (often called protocols), video, and audio recordings. Taken together, these different types of data can provide a rich picture of the observed activity. The difficult part is working out how to combine the different sources to create a coherent narrative of what has been recorded; analytic frameworks, discussed in section 9.5, can help with this. Initial data processing includes writing up and expanding notes and transcribing ele-ments of the audio and video recordings and the think-aloud protocols. For observation in a controlled environment, initial processing might also include synchronizing different data recordings.
Transcriptions and the observer’s notes are most likely to be analyzed using qualitative approaches, while photographs provide contextual information. Data logs and some ele-ments of the observer’s notes would probably be analyzed quantitatively.
9.3 Basic Quantitative Analysis
Explaining statistical analysis requires a whole book on its own (for example, see Cairns, 2019). Here, we introduce two basic quantitative analysis techniques that can be used effec-tively in interaction design: averages and percentages. Percentages are useful for standard-izing the data, particularly to compare two or more large sets of responses.
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N312
Averages and percentages are fairly well-known numerical measures. However, there are three different types of average, and using the wrong one can lead to the misinter-pretation of the results. These three are: mean, median, and mode. Mean refers to the commonly understood interpretation of average; that is, add together all the figures and divide by the number of figures with which you started. Median and mode averages are less well-known but are very useful. The median is the middle value of the data when the numbers are ranked. The mode is the most commonly occurring number. For example, in a set of data (2, 3, 4, 6, 6, 7, 7, 7, 8), the median is 6 and the mode is 7, while the mean is 50/9 = 5.56. In this case, the difference between the different averages is not that great. However, consider the set (2, 2, 2, 2, 450). Now the median is 2, the mode is 2, and the mean is 458/5 = 91.6!
Use of simple averages can provide useful overview information, but they need to be used with caution. Evangelos Karapanos et al. (2009) go further and suggest that averaging treats diversity among participants as error and proposes the use of a multidimensional scaling approach instead.
Before any analysis can take place, the data needs to be collated into analyzable data sets. Quantitative data can usually be translated into rows and columns, where one row equals one record, such as respondent or interviewee. If these are entered into a spreadsheet such as Excel, this makes simple manipulations and data set filtering easier. Before entering data in this way, it is important to decide how to represent the different possible answers. For example, “don’t know” represents a different response from no answer at all, and they need to be distinguished, for instance, with separate columns in the spreadsheet. Also, if dealing with options from a close-ended question, such as job role, there are two different possible approaches that affect the analysis. One approach is to have a column headed “Job role” and to enter the job role as it is given by the respondent or interviewee. The alternative approach is to have a column for each possible answer. The latter approach lends itself more easily to automatic summaries. Note, however, that this option will be open only if the original ques-tion was designed to collect the appropriate data (see Box 9.2).
Source: Mike Baldwin / Cartoon Stock
9 . 3 B A S I c Q u A N T I TAT I v E A N A LY S I S 313
BOX 9.2How Question Design Affects Data Analysis
Different question designs affect the kinds of analyses that can be performed and the kinds of conclusions that can be drawn. To illustrate this, assume that some interviews have been conducted to evaluate a new app that lets you try on virtual clothes and see yourself in real time as a 3D holograph. This is an extension of the Memory Mirror described at http://memorymirror.com.
Assume that one of the questions asked is: “How do you feel about this new app?” Responses to this will be varied and may include that it is cool, impressive, realistic, clunky, technically complex, and so on. There are many possibilities, and the responses would need to be treated qualitatively. This means that analysis of the data must consider each individual response. If there are only 10 or so responses, then this may not be too bad, but if there are many more, it becomes harder to process the information and harder to summarize the findings. This is typical of open-ended questions; that is, answers are not likely to be homogeneous and so they will need to be treated individually. In contrast, answers to a close-ended question, which gives respondents a fixed set of alternatives from which to choose, can be treated quantitatively. So, for example, instead of asking “How do you feel about the virtual try-on holograph?” assume that you have asked “In your experience, are virtual try-on holographs realistic, clunky, or distorted?” This clearly reduces the number of options and the responses would be recorded as “realistic,” “clunky,” or “distorted.”
When entered in a spreadsheet, or a simple table, initial analysis of this data might look like the following:
Respondent Realistic Clunky Distorted
A 1
B 1
C 1
. . .
Z 1
Total 14 5 7
Based on this, we can then say that 14 out of 26 (54 percent) of the respondents think virtual try-on holographs are realistic, 5 out of 26 (19 percent) think they are clunky, and 7 out of 26 (27 percent) think they are distorted. Note also that in the table, respondents’ names are replaced by letters so that they are identifiable but anonymous to any onlookers. This strategy is important for protecting participants’ privacy.
Another alternative that might be used in a questionnaire is to phrase the question in terms of a Likert scale, such as the following one. This again alters the kind of data and hence the kind of conclusions that can be drawn.
(Continued)
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N314
For simple collation and analysis, spreadsheet software such as Excel or Google Sheets is often used as it is commonly available, is well understood, and offers a variety of numerical manipulations and graphical representations. Basic analysis might involve finding out averages and identifying outliers, in other words, values that are significantly different from the majority, and hence not common. Producing a graphical representation provides an overall view of the data and any patterns it contains. Other tools are avail-able for performing specific statistical tests, such as online t-tests and A/B testing tools. Data visualization tools can create more sophisticated representations of the data such as heatmaps.
For example, consider the set of data shown in Table 9.2, which was collected during an evaluation of a new photo sharing app. This data shows the users’ experience of social media and the number of errors made while trying to complete a controlled task with the new app. It was captured automatically and recorded in a spreadsheet; then the totals and averages were calculated. The graphs in Figure 9.1 were generated using the spreadsheet package. They show an overall view of the data set. In particular, it is easy to see that there are no significant outliers in the error rate data.
Adding one more user to Table 9.2 with an error rate of 9 and plotting the new data as a scatter graph (see Figure 9.2) illustrates how graphs can help to identify outliers. Outliers are usually removed from the main data set because they distort the general patterns. However, outliers may also be interesting cases to investigate further in case there are special circum-stances surrounding those users and their session.
Virtual try-on holographs are realistic:
strongly agree agree neither disagree strongly disagree
□ □ □ □ □
The data could then be analyzed using a simple spreadsheet or table:
Respondent Strongly agree Agree Neither Disagree Strongly disagree
A 1 1
B
C 1
. . .
Z 1
Total 5 7 10 1 3
In this case, the kind of data being collected has changed. Based on this second set, nothing can be said about whether respondents think the virtual try-on holographs are clunky or dis-torted, as that question has not been asked. We can only say that, for example, 4 out of 26 (15 percent) disagreed with the statement that virtual try-on holographs are realistic, and of those, 3 (11.5 percent) strongly disagreed.
9 . 3 B A S I c Q u A N T I TAT I v E A N A LY S I S 315
These initial investigations also help to identify other areas for further investigation. For example, is there something special about users with error rate 0 or something distinctive about the performance of those who use the social media only once a month?
Social media use
User More than once a day
Once a day
Once a week
Two or three times a week
Once a month Number of errors made
1 1 4
2 1 2
3 1 1
4 1 0
5 1 2
6 1 3
7 1 2
8 1 0
9 1 3
10 1 2
11 1 1
12 1 2
13 1 4
14 1 2
15 1
16 1 1
17 1 1 0
18 1 0
Totals 4 7 2 3 2 30
Mean 1.67
(to 2 decimal places)
Table 9.2 Data gathered during a study of a photo sharing app
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N316
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 5 7 9 11 13 15 17
User
Num
ber
of e
rror
s m
ade
3
Social media use
> once a day
once a day
once a week
2 or 3 times a week
once a month
(a)
(b)
Figure 9.1 Graphical representations of the data in Table 9.2 (a) The distribution of errors made (take note of the scale used in these graphs, as seemingly large differences may be much smaller in reality). (b) The spread of social media experience within the participant group.
9 . 3 B A S I c Q u A N T I TAT I v E A N A LY S I S 317
0
2
4
6
8
10
0 5 10 15 20User
Num
ber
of e
rror
s m
ade
Figure 9.2 Using a scatter diagram helps to identify outliers in your data quite quickly
AcTIvITY 9.1 The data in the following table represents the time taken for a group of users to select and buy an item from an online shopping website.Using a spreadsheet application to which you have access, generate a bar graph and a scatter diagram to provide an overall view of the data. From this representation, make two initial observations about the data that might form the basis of further investigation.
User A B C D E F G H I J K L M N O P Q R S
Time to complete (mins)
15 10 12 10 14 13 11 18 14 17 20 15 18 24 12 16 18 20 26
CommentThe bar graph and scatter diagram are shown here.
Time to complete task
0
5
10
15
20
25
30
User
Tim
e in
min
utes
A B C F G H I J K L M N O P Q RE SD
(Continued)
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N318
It is fairly straightforward to compare two sets of results, for instance from the evalua-tion of two interactive products, using these kinds of graphical representations of the data. Semantic differential data can also be analyzed in this way and used to identify trends, pro-vided that the format of the question is appropriate. For example, the following question was asked in a questionnaire to evaluate two different smartphone designs:
For each pair of adjectives, place a cross at the point between them that reflects the extent to which you believe the adjectives describe the smartphone design. Please place only one cross between the marks on each line.
Annoying Pleasing
Di�cult to useEasy to use
ExpensiveValue-for-money
Attractive Unattractive
Not secureSecure
UnhelpfulHelpful
Lo-techHi-tech
FragileRobust
E�cientIne�cient
DatedModern
Time to complete task
A
BC
D
E FG
H
IJ
K
LM
N
O
PQ
R
S
0
5
10
15
20
25
30
0 5 10 15 20
User
Tim
e in
min
utes
From these two diagrams, there are two areas for further investigation. First, the values for user N (24) and user S (26) are higher than the others and could be looked at in more detail. In addi-tion, there appears to be a trend that the users at the beginning of the testing time (particularly users B, C, D, E, F, and G) performed faster than those toward the end of the testing time. This is not a clear-cut situation, as O also performed well, and I, L, and P were almost as fast, but there may be something about this later testing time that has affected the results, and it is worth investigating further.
9 . 3 B A S I c Q u A N T I TAT I v E A N A LY S I S 319
Table 9.3 and Table 9.4 show the tabulated results from 100 respondents. Note that the responses have been translated into five categories, numbered from 1 to 5, based on where the respondent marked the line between each pair of adjectives. It is possible that respond-ents may have intentionally put a cross closer to one side of the box than the other, but it is acceptable to lose this nuance in the data, provided that the original data is not lost, and any further analysis could refer back to it.
The graph in Figure 9.3 shows how the two smartphone designs varied according to the respondents’ perceptions of how modern the design is. This graphical notation shows clearly how the two designs compare.
1 2 3 4 5
Annoying 35 20 18 15 12 Pleasing
Easy to use 20 28 21 13 18 Difficult to use
Value-for-money 15 30 22 27 6 Expensive
Attractive 37 22 32 6 3 Unattractive
Secure 52 29 12 4 3 Not secure
Helpful 33 21 32 12 2 Unhelpful
Hi-tech 12 24 36 12 16 Lo-tech
Robust 44 13 15 16 12 Fragile
Inefficient 28 23 25 12 12 Efficient
Modern 35 27 20 11 7 Dated
Table 9.3 Phone 1
1 2 3 4 5
Annoying 24 23 23 15 15 Pleasing
Easy 37 29 15 10 9 Difficult to use
Value-for-money 26 32 17 13 12 Expensive
Attractive 38 21 29 8 4 Unattractive
Secure 43 22 19 12 4 Not secure
Helpful 51 19 16 12 2 Unhelpful
Hi-tech 28 12 30 18 12 Lo-tech
Robust 46 23 10 11 10 Fragile
Inefficient 10 6 37 29 18 Efficient
Modern 3 10 45 27 15 Dated
Table 9.4 Phone 2
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N320
Data logs that capture users’ interactions automatically, such as with a website or smart-phone, can also be analyzed and represented graphically, thus helping to identify patterns in behavior. Also, more sophisticated manipulations and graphical images can be used to highlight patterns in collected data.
9.4 Basic Qualitative Analysis
Three basic approaches to qualitative analysis are discussed in this section: identifying themes, categorizing data, and analyzing critical incidents. Critical incident analysis is a way to isolate subsets of data for more detailed analysis, perhaps by identifying themes or apply-ing categories. These three basic approaches are not mutually exclusive and are often used in combination, for example, when analyzing video material critical incidents may first be iden-tified and then a thematic analysis undertaken. Video analysis is discussed further in Box 9.3.
As with quantitative analysis, the first step in qualitative analysis is to gain an overall impression of the data and to start looking for interesting features, topics, repeated obser-vations, or things that stands out. Some of these will have emerged during data gathering, and this may already have suggested the kinds of pattern to look for, but it is important to confirm and re-confirm findings to make sure that initial impressions don’t bias analysis. For example, you might notice from the logged data of people visiting TripAdviser.com that they often look for reviews for hotels that are rated “terrible” first. Or, you might notice that a lot of respondents all say how frustrating it is to have to answer so many security questions when logging onto an online banking service. During this first pass, it is not necessary to capture all of the findings but instead to highlight common features and record any surprises that arise (Blandford, 2017).
For observations, the guiding framework used in data gathering will give some struc-ture to the data. For example, the practitioner’s framework for observation introduced in Chapter 8, “Data Gathering,” will have resulted in a focus on who, where, and what, while
Perceptions of Modern
0
5
10
15
20
25
30
35
40
45
50
1 3 5
Num
ber
of r
espo
nden
tsPhone 1
Phone 2
42
Figure 9.3 A graphical comparison of two smartphone designs according to whether they are per-ceived as modern or dated
9 . 4 B A S I c Q u A L I TAT I v E A N A LY S I S 321
using the more detailed framework will result in patterns relating to physical objects, people’s goals, sequences of events, and so on.
Qualitative data can be analyzed inductively, that is, extracting concepts from the data, or deductively, in other words using existing theoretical or conceptual ideas to categorize data elements (Robson and McCartan, 2016). Which approach is used depends on the data obtained and the goal of the study, but the underlying principle is to classify elements of the data in order to gain insights toward the study’s goal. Identifying themes (thematic analysis) takes an inductive approach, while categorizing data takes a deductive approach. In practice, analysis is often performed iteratively, and it is common for themes identified inductively then to be applied deductively to new data, and for an initial, pre-existing categorization scheme, to be enhanced inductively when applied to a new situation or new data. One of the most challenging aspects of identifying themes or new categories is determining meaningful codes that are orthogonal (that is, codes which do not overlap). Another is deciding on the appropriate granularity for them, for example at the word, phrase, sentence, or paragraph level. This is also dependent on the goal of the study and the data being analyzed.
Whether an inductive or deductive approach is used, an objective is to produce a reli-able analysis, that is, one that can be replicated by someone else if they were to use the same type of approach. One way to achieve this is to train another person to do the coding. When training is complete, both researchers analyze a sample of the same data. If there is a large discrepancy between the two analyses, either training was inadequate or the categorization is not working and needs to be refined. When a high level of reliability is reached between the two researchers, it can be quantified by calculating the inter-rater reliability. This is the percentage of agreement between the analyses of the two researchers, defined as the number of items of agreement, for example the number of categories or themes arising from the data that have been identified consistently by both researchers, expressed as a percentage of the total number of items examined. An alternative measure where two researchers have been used is Cohen’s kappa, (κ), which considers the possibility that agreement has occurred due to chance (Cohen, 1960).
Using more sophisticated analytical frameworks to structure the analysis of qualitative data can lead to additional insights that go beyond the results of these basic techniques. Section 9.5 introduces frameworks that are commonly used in interaction design.
BOX 9.3Analyzing Video Material
A good way to start a video analysis is to watch what has been recorded all the way through while writing a high-level narrative of what happens, noting down where in the video there are any potentially interesting events. How to decide which is an interesting event will depend on what is being observed. For example, in a study of the interruptions that occur in an open plan office, an event might be each time that a person takes a break from an ongoing activity, for instance, when a phone rings, someone walks into their cubicle, or email arrives. If it is a study of how pairs of students use a collaborative learning tool, then activities such as turn-taking, sharing of input devices, speaking over one another, and fighting over shared objects would be appropriate to record.
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N322
9.4.1 Identifying ThemesThematic analysis is considered an umbrella term to cover a variety of different approaches to examining qualitative data. It is a widely used analytical technique that aims to identify, analyze, and report patterns in the data (Braun and Clarke, 2006). More formally, a theme is something important about the data in relation to the study goal. A theme represents a pattern of some kind, perhaps a particular topic or feature found in the data set, which is considered to be important, relevant, and even unexpected with respect to the goals driv-ing the study. Themes that are identified may relate to a variety of aspects: behavior, a user group, events, places or situations where those events happen, and so on. Each of these kinds of themes may be relevant to the study goals. For example, descriptions of typical users may be an outcome of data analysis that focuses on participant characteristics. Although thematic analysis is described in this section on qualitative analysis, themes and patterns may also emerge from quantitative data.
After an initial pass through the data, the next step is to look more systematically for themes across participants’ transcripts, seeking further evidence both to confirm and discon-firm initial impressions in all of the data. This more systematic analysis focuses on checking for consistency; in other words, do the themes occur across all participants, or is it only one or two people who mention something? Another focus is on finding further themes that may not have been noticed first time. Sometimes, the refined themes resulting from this systematic analysis form the primary set of findings for the analysis, and sometimes they are just the starting point.
The study’s goal provides an orienting focus for the identification and formulation of themes in the first and subsequent passes through the data. For example, consider a survey to evaluate whether the information displayed on a train travel website is appropriate and sufficient. Several of the respondents suggest that the station stops in between the origin and destination stations should be displayed. This is relevant to the study’s goal and would be reported as a main theme. In another part of the survey, under further comments you might notice that several respondents say the company’s logo is distracting. Although this too is a theme in the data, it is not directly relevant to the study’s goals and may be reported only as a minor theme.
Chronological and video times are used to index events. These may not be the same, since recordings can run at different speeds from real time and video can be edited. Labels for certain routine events are also used, for instance lunchtime, coffee break, staff meeting, and doctor’s rounds. Spreadsheets are used to record the classification and description of events, together with annotations and notes of how the events began, how they unfolded, and how they ended.
Video can be augmented with captured screens or logged data of people’s interactions with a computer display, and sometimes transcription is required. There are various logging and screen capture tools available for this purpose, which enable interactions to be played back as a movie, showing screen objects being opened, moved, selected, and so on. These can then be played in parallel with the video to provide different perspectives on the talk, physical interactions, and the system’s responses that occur. Having a combination of data streams can enable more detailed and fine-grained patterns of behavior to be interpreted (Heath et al., 2010).
9 . 4 B A S I c Q u A L I TAT I v E A N A LY S I S 323
Once a number of themes have been identified, it is usual to step back from the set of themes to look at the bigger picture. Is an overall narrative starting to emerge, or are the themes quite disparate? Do some seem to fit together with others? If so, is there an over-arching theme? Can you start to formulate a meta-narrative, that is, an overall picture of the data? In doing this, some of the original themes may not seem as relevant and can be removed. Are there some themes that contradict each other? Why might this be the case? This can be done individually, but more often this is applied in a group using brainstorming techniques with sticky notes.
A common technique for exploring data, identifying themes, and looking for an overall narrative is to create an affinity diagram. The approach seeks to organize individual ideas and insights into a hierarchy showing common structures and themes. Notes are grouped together when they are similar in some fashion. The groups are not predefined, but rather they emerge from the data. This process was originally introduced into the software quality community from Japan, where it is regarded as one of the seven quality processes. The affin-ity diagram is built gradually. One note is put up first, and then the team searches for other notes that are related in some way.
Affinity diagrams are used in Contextual Design (Beyer and Holtzblatt, 1998; Holtz-blatt, 2001), but they have also been adopted widely in interaction design (Lucero, 2015). For example, Madeline Smith et al. (2018) conducted interviews to design a web app for co-watching videos across a distance, and they used affinity diagramming to identify require-ments from interviewee transcripts (see Figure 9.4). Despite the prevalence of digital col-laboration tools, the popularity of physical affinity diagramming using sticky notes drawn by hand, has persisted for many years (Harboe and Huang, 2015).
Figure 9.4 Section of an affinity diagram built during the design of a web applicationSource: Smith (2018). Used courtesy of Madeline Smith
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N324
9.4.2 Categorizing DataInductive analysis is appropriate when the study is exploratory, and it is important to let the themes emerge from the data itself. Sometimes, the analysis frame (the set of categories used) is chosen beforehand, based on the study goal. In that case, analysis proceeds deductively. For example, in a study of novice interaction designer behavior in Botswana, Nicole Lotz et al. (2014) used a set of predetermined categories based on Schön (1983)’s design and reflection cycle: naming, framing, moving, and reflecting. This allowed the researchers to identify detailed patterns in the designers’ behavior, which provided implications for education and support.
To illustrate categorization, we present an example derived from a set of studies look-ing at the use of different navigation aids in an online educational setting (Ursula Armi-tage, 2004). These studies involved observing users working through some online educational material (about evaluation methods), using the think-aloud technique. The think-aloud protocol was recorded and then transcribed before being analyzed from various perspec-tives, one of which was to identify usability problems that the participants were having with the online environment known as Nestor Navigator (Zeiliger et al., 1997). An excerpt from the transcription is shown in Figure 9.5.
I’m thinking that it’s just a lot of information to absorb from the screen. I just I don’t concentrate very well when I’m looking at the screen. I have a very clear idea of what I’ve read so far . . . but it’s because of the headings I know OK this is another kind of evaluation now and before it was about evaluation which wasn’t anyone can test and here it’s about experts so it’s like it’s nice that I’m clicking every now and then coz it just sort of organizes the thoughts. But it would still be nice to see it on a piece of paper because it’s a lot of text to read.
Am I supposed to, just one question, am supposed to say something about what I’m reading and what I think about it the conditions as well or how I feel reading it from the screen, what is the best thing really?
Observer: What you think about the information that you are reading on the screen . . . you don’t need to give me comments . . . if you think this bit fits together.
There’s so much reference to all those previously said like I’m like I’ve already forgotten the name of the other evaluation so it said unlike the other evaluation this one like, there really is not much contrast with the other it just says what it is may be . . . so I think I think of . . .
Maybe it would be nice to have other evaluations listed to see other evaluations you know here, to have the names of other evaluations other evaluations just to, because now when I click previous I have to click it several times so it would be nice to have this navigation, extra links.
Figure 9.5 Excerpt from a transcript of a think-aloud protocol when using an online educational environment. Note the prompt from the observer about halfway through.Source: Armitage (2004). Used courtesy of Ursula Armitage
To read more about the use of affinity diagrams in interaction design, see the fol-lowing page:https://uxdict.io/design-thinking-methods-affinity-diagrams-357bd8671ad4
9 . 4 B A S I c Q u A L I TAT I v E A N A LY S I S 325
This excerpt was analyzed using a categorization scheme derived from a set of negative effects of a system on a user (van Rens, 1997) and was iteratively extended to accommo-date the specific kinds of interaction observed in these studies. The categorization scheme is shown in Figure 9.6.
This scheme developed and evolved as the transcripts were analyzed and more cat-egories were identified inductively. Figure 9.7 shows the excerpt from Figure 9.5 coded using this categorization scheme. Note that the transcript is divided up using square brackets to indicate which element is being identified as showing a particular usabil-ity problem.
Having categorized the data, the results can be used to answer the study goals. In the earlier example, the study allowed the researchers to be able to quantify the number of usability problems encountered overall by participants, the mean number of problems per participant for each of the test conditions, and the number of unique problems of each type per participant. This also helped to identify patterns of behavior and recurring problems. Having the think-aloud protocol meant that the overall view of the usability problems could take context into account.
Figure 9.6 Criteria for identifying usability problems from verbal protocol transcriptionsSource: Armitage (2004). Used courtesy of Ursula Armitage
1. Interface Problems1.1. Verbalizations show evidence of dissatisfaction about an aspect of the
interface.1.2. Verbalizations show evidence of confusion/uncertainty about an
aspect of the interface.1.3. Verbalizations show evidence of confusion/surprise at the outcome of
an action.1.4. Verbalizations show evidence of physical discomfort.1.5. Verbalizations show evidence of fatigue.1.6. Verbalizations show evidence of difficulty in seeing particular aspects
of the interface.1.7. Verbalizations show evidence that they are having problems achieving
a goal that they have set themselves, or the overall task goal.1.8. Verbalizations show evidence that the user has made an error.1.9. The participant is unable to recover from error without external help
from the experimenter.1.10. The participant suggests a redesign of the interface of the elec-
tronic texts.
2. content Problems2.1. Verbalizations show evidence of dissatisfaction about aspects of the
content of the electronic text.2.2. Verbalizations show evidence of confusion/uncertainty about aspects
of the content of the electronic text.2.3. Verbalizations show evidence of a misunderstanding of the electronic
text content (the user may not have noticed this immediately).2.4. The participant suggests re-writing the electronic text content.
Identified problems should be coded as [uP, << problem no. >>].
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N326
AcTIvITY 9.2
The following is a think-aloud extract from the same study of users working through online educa-tional material. Using the categorization scheme in Figure 9.6, code this extract for usability prob-lems. It is useful to put brackets around the complete element of the extract that you are coding.
Well, looking at the map, again there’s no obvious start point, there should be something high-lighted that says ‘start here.’
Ok, the next keyword that’s highlighted is evaluating, but I’m not sure that’s where I want to go straight away, so I’m just going to go back to the introduction.
Yeah, so I probably want to read about usability problems before I start looking at evaluation. So, I, yeah. I would have thought that the links in each one of the pages would take you to the next logical point, but my logic might be different to other people’s. Just going to go and have a look at usability problems.
Ok, again I’m going to flip back to the introduction. I’m just thinking if I was going to do this myself I would still have a link back to the introduction, but I would take people through the logical sequence of each one of these bits that fans out, rather than expecting them to go back all the time.
Going back . . . to the introduction. Look at the types. Observation, didn’t really want to go there. What’s this bit [pointing to Types of UE on map]? Going straight to types of . . .
Ok, right, yeah, I’ve already been there before. We’ve already looked at usability problems, yep that’s ok, so we’ll have a look at these references.
[I’m thinking that it’s just a lot of information to absorb from the screen. uP 1.1][ I just I don’t con-centrate very well when I’m looking at the screen uP 1.1]. I have a very clear idea of what I’ve read so far . . . [but it’s because of the headings uP 1.1] I know OK this is another kind of evaluation now and before it was about evaluation which wasn’t anyone can test and here it’s about experts so it’s like it’s nice that I’m clicking every now and then coz it just sort of organises the thoughts. [But it would still be nice to see it on a piece of paper uP 1.10] [because it’s a lot of text to read UP 1.1].
Am I supposed to, just one question, am supposed to say something about what I’m reading and what I think about it the conditions as well or how I feel reading it from the screen, what is the best thing really?
Observer: What you think about the information that you are reading on the screen . . . you don’t need to give me comments . . . if you think this bit fits together.
[There’s so much reference to all those previously said uP2.1] [ like I’m like I’ve already forgotten the name of the other evaluation so it said unlike the other evaluation this one like, there really is not much contrast with the other it just says what it is may be . . . so I think I think of . . . UP 2.2]
[Maybe it would be nice to have other evaluations listed to see other evaluations you know here, to have the names of other evaluations other evaluations uP 1.10] just to, [because now when I click previous I have to click it several times uP 1.1, 1.7] [so it would be nice to have this navigation, extra links uP 1.10].
Figure 9.7 The excerpt in Figure 9.5 coded using the categorization scheme in Figure 9.6Source: Armitage (2004). Used courtesy of Ursula Armitage
9 . 4 B A S I c Q u A L I TAT I v E A N A LY S I S 327
9.4.3 Critical Incident AnalysisData gathering sessions can often result in a lot of data. Analyzing all of this data in any detail is very time-consuming and often not necessary. Critical incident analysis is one approach that helps to identify significant subsets of the data for more detailed analysis. This tech-nique is a set of principles that emerged from work carried out in the United States Army Air Forces where the goal was to identify the critical requirements of good and bad performance by pilots (Flanagan, 1954). It has two basic principles: “(a) reporting facts regarding behav-ior is preferable to the collection of interpretations, ratings, and opinions based on general impressions; (b) reporting should be limited to those behaviors which, according to compe-tent observers, make a significant contribution to the activity” (Flanagan, 1954, p. 355). In the interaction design context, the use of well-planned observation sessions satisfies the first principle. The second principle refers to critical incidents, that is, incidents that are significant or pivotal to the activity being observed, in either a desirable or an undesirable way.
In interaction design, critical incident analysis has been used in a variety of ways, but the main focus is to identify specific incidents that are significant and then to focus on these and analyze them in detail, using the rest of the data collected as context to inform interpreta-tion. These may be identified by the users during a retrospective discussion of a recent event or by an observer either through studying video footage or in real time. For example, in an
I clicked on the map rather than going back via introduction, to be honest I get fed up going back to introduction all the time.
comment
Coding transcripts takes practice, but this activity will give you an idea of the kinds of decisions involved in applying categories. Our coded extract is shown here:
[Well, looking at the map, again there’s no obvious start point uP 1.2, 2.2], [there should be something highlighted that says ‘start here’ uP 1.1, 1.10].
Ok, the next keyword that’s highlighted is evaluating, but [I’m not sure that’s where I want to go straight away uP 2.2], so I’m just going to go back to the introduction.
Yeah, so I probably want to read about usability problems before I start looking at evalua-tion. So, I, yeah. [I would have thought that the links in each one of the pages would take you to the next logical point, but my logic might be different to other people’s uP 1.3]. Just going to go and have a look at usability problems.
Ok, again I’m going to flip back to the introduction. [I’m just thinking if I was going to do this myself I would still have a link back to the introduction, but I would take people through the logical sequence of each one of these bits that fans out, rather than expect-ing them to go back all the time uP 1.10].
Going back . . . to the introduction. [Look at the types. Observation, didn’t really want to go there. What’s this bit [pointing to Types of UE on map]? uP 2.2] Going straight to types of . . .
Ok, right, yeah, I’ve already been there before. We’ve already looked at usability prob-lems, yep that’s ok, so we’ll have a look at these references.
I clicked on the map rather than going back via introduction, [to be honest I get fed up going back to introduction all the time. uP 1.1].
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N328
evaluation study, a critical incident may be signaled by times when users were obviously stuck—usually marked by a comment, silence, looks of puzzlement, and so on. These are indications only. Whether the incident is significant enough to be worthy of further investiga-tion depends on the severity of the problem identified.
Tuomas Kari et al. (2017) used the critical incident technique in a study of the location-based augmented reality game Pokémon GO. They were interested in identifying the types of behavior change that playing the game induced in players. To do this, they distributed a survey through social media channels asking experienced players to identify and describe one outstanding positive or negative experience. The 262 valid responses were themed and categorized into eight groups. Apart from expected behavior change such as increased physical activity, they also found that players were more social, found their routines more meaningful, expressed more positive emotions, and were more motivated to explore their surroundings. In another study, Elise Grison et al. (2013) used the critical incident technique to investigate specific factors that influence travelers’ choices of transport mode in Paris in order to adapt new tools and services for mobility, such as dynamic route planners. Participants were asked to report on positive and negative real events they experienced in the context of their route to work or study, and whether they regretted or were satisfied with this choice of transport. Their findings included that contextual factors have a great influence on choice, that people were more likely to choose an alternative route to return home than when setting out, and that emotional state is important when planning a route.
To read more on critical incident analysis in usability studies where the emphasis is on understanding the cause of problems, see this site: www.usabilitybok.org/critical-incident-technique.
AcTIvITY 9.3 Assign yourself or a friend the task of identifying the next available theater performance or movie that you’d like to attend in your area. As you perform this task, or watch your friend do it, make a note of critical incidents associated with the activity. Remember that a critical incident may be a positive or a negative event.
CommentInformation about entertainment may be available through the local newspaper, searching online, looking at social media to see what is recommended by friends, or contacting local cinemas or theaters directly. When this author asked her daughter to attempt this task, several critical incidents emerged, including the following:1. After checking her social media channels, nothing in the recommendations appealed to her
and so she decided to search online.2. She found that one of her all-time favorite movie classics was playing at the local movie
theater for just one week, and tickets were still available.3. When trying to buy the tickets, she discovered that she needed a credit card, which she
didn’t have, and so she had to ask me to complete the purchase!
9 . 5 W h I c h K I N D O F A N A LY T I c F R A m E W O R K T O u S E ? 329
9.5 Which Kind of Analytic Framework to Use?
There are several different analytical frameworks that can be used to analyze and interpret data from a qualitative study. In this section, six different approaches are outlined, ordered roughly in terms of their granularity, that is, the level of detail involved. For example, conversation analysis has a fine level of granularity, and it allows the details of what is said and how dur-ing a short fragment of conversation to be examined, while systems-based frameworks take a broader scope and have a coarser level of granularity, such as what happens when a new digital technology is introduced into an organization, like a hospital. Conversation analysis may result in insights related to users’ interactions through a collaboration technology, while systems-based analyses may result in insights related to changes in work practices, worker satisfac-tion, improvements in workflow, impact on an office culture, and so on. Table 9.5 lists the six approaches in terms of the main types data, focus, expected outcomes, and level of granularity.
Framework Data Focus Expected outcomes
Level of granularity
Conversation analysis
Recordings of spoken conversations
How conversations are conducted
Insights into how conversations are managed and how they progress
Word-level, or finer, for instance, pauses and inflection
Discourse analysis
Recordings of speech or writing from individuals or several participants
How words are used to convey meaning
Implicit or hidden meanings in texts
Word, phrase, or sentence-level
Content analysis
Any form of “text” including written pieces, video and audio recordings, or photographs
How often something is featured or is spoken about
Frequency of items appearing in a text
A wide range of levels from words, to feelings or attitudes, to artifacts or people
Interaction analysis
Video recordings of a naturally-occurring activity
Verbal and non-verbal interactions between people and artifacts
Insights about how knowledge and action are used within an activity
At the level of artifact, dialogue, and gesture
Grounded theory
Empirical data of any kind
Constructing a theory around the phenomenon of interest
A theory grounded in empirical data
Varying levels, depending on the phenomenon of interest
Systems-based frameworks
Large-scale and heterogeneous data
Large-scale involving people and technology, such as a hospital or airport
Insights about organizational effectiveness and efficiency
Macro-level, organizational level
Table 9.5 Overview of analytical frameworks used in interaction Design
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N330
9.5.1 Conversation AnalysisConversation analysis (CA) examines the semantics of a conversation in fine detail. The focus is on how a conversation is conducted (Jupp, 2006). This technique is used in sociological studies, and it examines how conversations start and how turn-taking is structured, together with other rules of conversation. It has been used to analyze interactions in a range of settings, and it has influenced designers’ understanding of users’ needs in these environments. It can also be used to compare conversations that take place through different media, for example, face-to-face conversations versus those conducted through social media. More recently, it has been used to analyze the conversations that take place with voice-assisted technologies and chatbots.
Voice assistants (also called smart speakers), like Amazon Echo, have become increas-ingly popular in domestic settings, providing a limited kind of conversational interaction, mainly by answering questions and responding to requests. But how do families orient and adapt to them? Does using this device change the way they talk, or do they talk to the device as if it was another human being?
Martin Porcheron et al. (2018) carried out a study examining how such devices were being used by families in their own homes, and they used conversation analysis with excerpts from selected conversations. A sample fragment of a conversation they analyzed is presented in Figure 9.8. This uses a particular type of syntax for marking up the minutiae of interac-tions and speech that took place during an approximate 10-second period. Square parenthe-ses are used to show overlapping talk, round parentheses to indicate a pause, and physical spacing to show temporal sequencing of what is said. This level of detail enables the analysis to reveal subtle cues and mechanisms that are used during the conversations.
In this fragment, Susan (who is the mother) announces to Liam (her son) and Carl (the father) her desire to play a particular game (called Beat the Intro) with their Amazon Echo. Liam does not want to play (expressed by his long “no” cry in response). Susan, however, has
01 SUS i’d like to play beat the intro in a minute02 LIA [ oh no: ]03 SUS [ alexa ][ (1.1) ] beat the in[tro04 CAR [ °yeah°; ] 05 LIA [°no:::…°06 CAR (0.6) it’s mother’s day? (0.4)07 SUS it’s ( ) yep (.) listen (.) you need to keep08 on eating your orange stuff (.) liam09 (0.7)10 CAR and your green stuff11 SUS alexa (1.3) alexa (0.5)=12 CAR =°and your brown stuff13 SUS play beat the intro
Figure 9.8 An extract of the conversation between the family and Alexa, marked up for conversa-tion analysisSource: Porcheron et al. (2018), fragment 1. Reproduced with permission of ACM Publications
9 . 5 W h I c h K I N D O F A N A LY T I c F R A m E W O R K T O u S E ? 331
already called “Alexa” to wake up the device. Carl shows his support for her, as indicated by his quick “yeah” during the pause after she says “Alexa.” Alexa, however, appears not to respond. At this point, Susan returns to the ongoing family conversation, telling Liam to keep eating his “orange stuff.” Carl also chips in after her and says to Liam that he should also eat the “green stuff.” At the same time, Susan has another go at getting Alexa to wake up. She calls out “Alexa” twice in a questioning voice. In the pause between Susan’s two calls to Alexa, Carl tells Liam to keep eating again, but this time the “brown stuff.” Having succeeded in waking up Alexa, Susan then asks it to play the game.
So, what insights does this fine level of analysis provide? Martin Porcheron et al. (2018) point out that it demonstrates how a family’s interaction with the Amazon Echo is seamlessly interwoven with other ongoing activities, in this case, the parents trying to get their child to eat his food. At a more general level, it illustrates how our conversations with each other and voice-assisted technologies interleave in nuanced ways rather than being separate conversa-tions between members of the family or the family and the device, which jump from one to another. They also show how their analysis led them to think that the term conversational interaction fails to distinguish between the interactional embeddedness of voice-assisted interfaces and human conversation. Instead, they suggest that current voice-assisted technol-ogies be designed using a conceptual model, more akin to instructing rather than conversing.
9.5.2 Discourse AnalysisDiscourse analysis focuses on dialogue, in other words, the meaning of what is said and how words are used to convey meaning. Discourse analysis is strongly interpretive, pays great attention to context, and views language not only as reflecting psychological and social aspects but also as constructing them (Coyle, 1995). An underlying assumption of discourse analysis is that there is no objective scientific truth. Language is a form of social reality that is open to interpretation from different perspectives. In this sense, the underlying philosophy of discourse analysis is similar to that of ethnography. Language is viewed as a constructive tool, and discourse analysis provides a way of focusing on how people use language to con-struct versions of their worlds (Fiske, 1994).
Small changes in wording can change meaning, as the following excerpts indicate (Coyle, 1995):
Discourse analysis is what you do when you are saying that you are doing discourse analysis. . .According to Coyle, discourse analysis is what you do when you are saying that you are doing discourse analysis. . .
By adding just three words, “According to Coyle,” the sense of authority changes, depend-ing on what the reader knows about Coyle’s work and reputation.
Discourse analysis is useful when trying to identify subtle and implicit meaning in what people are writing about, what is trending, what is fake news, and so on. It can be used with data from interviews; in social media such as Facebook, Twitter, and WhatsApp; and in emails. For example, Carlos Roberto Teixeira et al. (2018) proposed a taxonomy for analyz-ing tweets as a way of understanding the types of conversations that take place online and identifying patterns of behavior within them. The tweets they analyzed were those posted during the political scandals that were happening in Brazil during 2016. Using discourse
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N332
analysis and by understanding the cultural background and how technology was adopted and used by the Brazilian society, they were able to interpret the meaning of the tweets, beyond the words that were said.
They scraped the raw data of the tweets from the web and ranked them in order of those most retweeted. Then they selected 100 of the most influential postings, which were messages that had been retweeted between 9,000 and 47,000 times. Finally, they manually classified the tweets using Excel spreadsheets, identifying the most dominant discourse characteristics. They note, however, that since the most retweeted messages are often captured out of context (that is, outside the conversational thread), they had problems sometimes interpreting what was the context, such as the particular news story associated with a tweet.
The tweets were then classified into a number of provisional themes that were whittled down to five general ones. These were (1) “support” (tweets that promote either side of the political dispute); (2) “criticism and protest” (tweets that showed disapproval or an objec-tion); (3) “humor” (tweets that were witty and had cartoons or jokes in them); (4) “news” (tweets that refer to the news that were neutral in tone); and (5) “neutral” (tweets that were indifferent or their position could not be inferred). For reliability, two different research-ers classified each tweet using these themes. Once classified, the data was analyzed using descriptive statistics and simple visualizations, like word (or tag) clouds and pie charts. These allowed them to draw conclusions about the tweets that they had analyzed, including the relative size of the different themes and how the size changed over time. Tweets in the criti-cism and protest theme were the most popular overall, followed by humor. More generally, they discuss how this kind of discourse analysis shows that humor, protest, and criticism are highly dominant in this kind of online discourse. They suggest that people who tweet often express their feelings about ongoing political events, in terms of criticism and humor, in equal measures. Moreover, they found that tweets including images, videos, and animated GIFs were most frequently classified as humor.
This kind of analysis of public discourse, when done by hand, is extremely time-consuming. To help, there are new software tools being developed that can automatically process computer-mediated discourses (Ecker, 2016). The advantage, as you will see in the next chapter, is that much larger data sets can be analyzed. The downside is that the analyst is no longer “hands-on” and loses touch with the surrounding context, meaning different interpretations arise.
9.5.3 Content AnalysisContent analysis typically involves classifying the data into themes or categories and then studying the frequency of category occurrences (Krippendorff, 2013). The technique can be used for any text, where “text” refers to a range of media including video, newspapers, advertisements, survey responses, images, sounds, and so on. It can be used to analyze any online content, including the text of tweets, links, animated gifs, videos, and images. For example, Mark Blythe and Paul Cairns (2009) analyzed 100 videos from a YouTube search by relevance for “iPhone 3G” using content analysis. They categorized the videos into seven categories: review, reportage, “unboxing,” demonstration, satire, advertisement, and vlog commentaries (such as, complaints about queues). Similar to discourse analysis, an important aspect of content analysis is how it considers the wider context.
Content analysis is often used in conjunction with other analysis techniques as well. For example, Weixin Zhai and Jean-Claude Thill (2017) analyzed social media data from Weibo (the Chinese equivalent of Twitter) to investigate the emotions, attitudes, and views
9 . 5 W h I c h K I N D O F A N A LY T I c F R A m E W O R K T O u S E ? 333
of citizens around a rainstorm that hit Beijing on July 21, 2012, causing 79 deaths. They used content analysis alongside sentiment analysis, an approach that extracts emotional and subjective information from natural language. From their results, they found how feelings of sorrow and sadness were shared across the entire city because of the trauma associated with entrapment indoors during the deluge.
9.5.4 Interaction AnalysisInteraction analysis was developed by Brigitte Jordan and Austin Henderson (1995) as a way of investigating and understanding the interactions of human beings with each other and objects in their environment. The technique focuses on both talk and nonverbal inter-actions with artifacts and technologies, and it is based on video recordings. An underlying assumption of this approach is that knowledge and action are fundamentally social. The goal is to derive generalizations from videos of naturally occurring activities, focusing on how the people being observed make sense of each other’s actions and their collective achievements.
Interaction analysis is an inductive process, where teams of researchers suggest state-ments about general patterns from multiple examples of empirical observations. Rather than individual researchers conducting separate analyses and then comparing their results for consistency, interaction analysis is conducted collaboratively; teams discuss together their observations and interpretations of the videos being watched as they watch them. The first step involves creating a content log, comprising headings and rough summaries of what has been observed. No predetermined categories are used during this stage. Instead, they emerge from repeated playing and discussion of the video material. Hypotheses are also generated by group members about what they think is happening. This process includes suggesting the intentions, motivations, and understandings of the people who are being viewed in the vid-eos. These suggestions have to be tied to the actions of the people rather than being purely speculative. For example, if an analyst thinks someone’s motivation is to take control during a board meeting, they need to provide actual examples that demonstrate how the person is achieving this (for instance, taking over the data projector, as the locus of control, for long periods of time and presenting their ideas for others to view).
The videos are then cannibalized, as it is called, by extracting interesting materials, reclas-sifying some of them in terms of what they represent, while removing others. Instances of a salient event are assembled and played one after another to determine whether a phenomenon is a robust theme or a one-off incident. An example Brigitte Jordon and Austin Henderson (1995) use to illustrate this process is their study of people around a pregnant woman who was having her first contraction. They noticed that, at the point of the first contraction, the medical staff and family all shifted their attention away from the woman to the monitor-ing equipment. They were able to find many more examples of this phenomenon, providing strong evidence that the presence of high-tech equipment changes the practice of caregiving, specifically that caregiving is mediated by the real-time data presented through the equipment.
An example of where interaction analysis has been used in HCI is Anna Xambo et al.’s (2013) study of how groups of musicians improvise and learn to play together when using a novel collaborative tabletop technology. Video data was collected for four groups of musi-cians using this technology in a number of jamming sessions. Representative video extracts were repeatedly viewed and discussed by a team of researchers, focusing on verbal commu-nication and nonverbal communication themes. These themes were categorized into whether
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N334
they were musical, physical, or interface-related. The themes that arose during the repeated video viewing process were diverse, including error/repair situations, territory-related behav-iors, emergent coordination mechanisms, and mimicking behaviors. Anna Xambo et al. (2013) then transformed these into a set of design considerations for developing tabletop technologies to support hands-on collaborative learning.
9.5.5 Grounded TheoryThe goal of grounded theory is to develop theory from a systematic analysis and interpreta-tion of empirical data; that is, the derived theory is grounded in the data. In this respect, it is an inductive approach to developing theory. The approach was originally developed by Barney Glaser and Anselm Strauss (1967) and since then has been adopted by several researchers, with some adaptations to different circumstances. In particular, both of them have individually (and with others) developed grounded theory in slightly different ways, but the objective of this approach remains the same. Barney Glaser (1992) provides further information about the differences and areas of controversy.
In this context, theory is: “a set of well-developed concepts related through statements of relationship, which together constitute an integrated framework that can be used to explain or predict phenomena” (Strauss and Corbin, 1998). Development of a “grounded” theory progresses through alternating data collection and data analysis: first data is collected and analyzed to identify themes, then that analysis may lead to further data collection and analy-sis to extend and refine the themes, and so on. During this cycle, parts of the data may be reanalyzed in more detail. Data gathering and subsequent analysis are hence driven by the emerging theory. This approach continues until no new insights emerge and the theory is well-developed. During this process, the researcher seeks to maintain a balance between objectivity and sensitivity. Objectivity is needed to maintain accurate and impartial inter-pretation of events; sensitivity is required to notice the subtleties in the data and identify relationships between concepts.
The thrust of the analysis undertaken is to identify and define the properties and dimen-sions of relevant themes called categories in grounded theory. According to Juliet Corbin and Anselm Strauss (2014), this coding has three aspects, which are iteratively performed through the cycle of data collection and analysis:
1. Open coding is the process through which categories, their properties, and dimensions are discovered in the data. This process is similar to our discussion of thematic analysis above, including the question of granularity of coding (at the word, line, sentence, conversation level, and so on).
2. Axial coding is the process of systematically fleshing out categories and relating them to their subcategories.
3. Selective coding is the process of refining and integrating categories to form a larger theo-retical scheme. The categories are organized around one central category that forms the backbone of the theory. Initially, the theory will contain only an outline of the categories, but as more data is collected, they are refined and developed further.
Early books on grounded theory say little about what data collection techniques should be used but focus instead on the analysis. Some later books place more emphasis on data collection. For example, Kathy Charmaz (2014) discusses interviewing techniques and collection and analysis of documents for grounded theory analysis. When analyzing
9 . 5 W h I c h K I N D O F A N A LY T I c F R A m E W O R K T O u S E ? 335
data, Juliet Corbin and Anselm Strauss (2014) encourage the use of written records of analysis and diagrammatic representations of categories (which they call memos and dia-grams). These memos and diagrams evolve as data analysis progresses. Some researchers also look to digital tools such as spreadsheets and diagramming tools, but many like to develop their own physical code books such as the one Dana Rotman et al. (2014) constructed in a study to understand the motivations of citizens to contribute to citizen science projects. The data that she analyzed was derived from in-depth semistructured interviews of 33 citizen scientists and 11 scientists from the United States, India, and Costa Rica (see Figure 9.9).
The following analytic tools are used to help stimulate the analyst’s thinking and identify and characterize relevant categories:
1. The Use of Questioning: In this context, this refers to questioning the data, not your participants. Questions can help an analyst to generate ideas or consider different ways of looking at the data. It can be useful to ask questions when analysis appears to be in a rut.
2. Analysis of a Word, Phrase, or Sentence: Considering in detail the meaning of an utter-ance can also help to trigger different perspectives on the data.
3. Further Analysis Through Comparisons: Comparisons may be made between objects or between abstract categories. In either case, comparing one with the other brings alterna-tive interpretations.
Grounded theory uses thematic analysis that is, themes are identified from the data, but as data analysis informs data collection, it also relies on categorizing new data according to
Figure 9.9 Code book used in a grounded theory analysis of citizens’ motivations to contribute to citizen scienceSource: Jennifer Preece
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N336
the existing thematic set and then evolving that set to accommodate new findings. As Victoria Braun and Victoria Clarke (2006) point out, “Thematic analysis differs from other analytic methods that seek to describe patterns across qualitative data . . . such as grounded theory <which is> theoretically bounded. . . the goal of a grounded theory analysis is to generate a plausible—and useful—theory of the phenomena that is grounded in the data.”
A Grounded Theory ExampleSo-called idle games are rising in popularity (Cutting et al., 2019). Idle games are minimalist games that require little or even no interaction in order for the game to progress. For example, an idle game may involve repeating a simple action like clicking an icon to accumulate resources. Example games include the Kittens Game, which is a text-based (that is, it has no graphical user interface) game that involves managing a village of kittens, while Cookie Clicker involves baking and selling cookies. Idle games also include mechanisms to automate game play so that progress may continue for extended periods of time without the player doing anything (Purkiss and Khaliq, 2015). An extreme example studied by Joe Cutting et al. (2019) is Neko Atsume, a game about collecting cats, in which progress can be made only if the game is switched off. In their study, they were interested in the notion of “engagement” and the implications of this new genre of games for current theories of engagement.
To understand more about the idle games genre, Sultan Alharthi et al. (2018) used grounded theory, specifically to develop a taxonomy and a set of characteristics for them. By defining the essential features of this genre of games and clustering them, the authors hoped to produce design implications for each type of game.
The three stages of coding, open, axial, and selective are illustrated in Figure 9.10. Note that, in this case, the research started by the researchers playing each of the games under study.
Kongregate57 games
playing sessions
Phase 1
additional idle games
additional non-idle games
record observations,preliminary coding
open coding,form concepts
Phase 2
analyze data, discussobservations, find
relationships,categories emerged
Phase 3
Almost Idle61 games
118
146
76
70
14
14
duplicates/non-playableexcluded
final classifications anddefinitions
Figure 9.10 The process used by Alharthi et al., showing Phase 2 and Phase 3 using the three stages of grounded theory coding
9 . 5 W h I c h K I N D O F A N A LY T I c F R A m E W O R K T O u S E ? 337
Each game was played by two researchers who recorded their observations in a spread-sheet. These observations focused on gameplay, game mechanics, rewards, interactivity, pro-gress rate, and game interface. Then they rated the games using an 11-point interactivity scale (0–10) where 0 meant that play progressed without any interaction of the player, while 10 meant that the game progressed only slowly without player interaction. Progress through the levels of the game were also rated on the same scale.
At the end of each game session and observation, the researchers wrote a brief overview of the game and conducted preliminary open coding of their observations (see Figure 9.11 for an example of preliminary open coding).
Axial and selective coding progressed iteratively. The researchers held several discussion sessions to explore the relationship between the codes, the emergent concepts, and the initial categories. During this process, some of the games were re-observed, and related literature was drawn upon to help refine the concepts. For example, existing literature on game taxon-omies, prior terms, and definitions related to idle games were incorporated into the analysis. Based on this analysis, Sultan Alharthi et al. produced a taxonomy with two basic ways to characterize the games: one based on key features and one based on interactivity. From the former, they defined incremental games as idle games in which a player selects resources to generate, waits for them to accumulate, and spends the resources to automate resource gen-eration. Figure 9.12 illustrates the open codes, resulting concepts, and categories developed for incremental games. This shows that four categories of incremental games emerge from this analysis: micromanagement, derivative, single-resource, and multiplayer.
Game Feature Observations
Game name AdVenture Capitalist [G38]
Play description You start clicking on a lemonade stand and collect money. Spend money to make upgrades, increase production per click. Start hiring workers and increase production per second. When you have enough money, you can buy new businesses, automate all your businesses to increment more money, and leave the game progress.
Game mechanics Click to gain money, automate production, make upgrades to damage/sec.
Rewards One currency, which is money, is rewarded in return.
Interface Graphical
Interactivity level 7
Progress rate 9
Overview This is a single-player game, which requires long cycles ok clicking at the start, and making a number of upgrades. Production rate reaches $390/sec in less that 10 minutes and you gain 1M in cash making the game progress faster.
Figure 9.11 An illustration of preliminary open coding. Words in small capitals are identified by the researcher as potential codes.
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N338
One of the surprises from their analysis is that the resulting taxonomy is based on game rules and their basic underlying structure rather than mechanisms in the game. This is in contrast to other gaming taxonomies that feature interactivity and interaction strategies. Of course, idle games are minimal, and so they don’t have much interaction. However, the grounded theory approach allowed the development of a taxonomy that reflects the style and purpose of the genre.
9.5.6 Systems-Based FrameworksFor large projects where the researcher is interested in investigating how a new technology should best be introduced and what its impact is afterward, it is necessary to analyze many sources of data collected over a long period of time. Conducting analyses of small fragments of conversation or identifying themes from interviews may be useful for highlighting specific working practices, but understanding how a whole socio-technical system (for example a hospital, corporation, local council, or airport) works at scale requires a different kind of analytical framework. Two such frameworks are introduced next: socio-technical systems theory (Eason, 1987) and distributed cognition (Hutchins, 1995), as applied through the Distributed Cognition of Teamwork framework (Furniss and Blandford, 2006).
Socio-technical Systems TheorySocio-technical systems (STS) theory makes explicit the fact that the technology and the peo-ple in a work system are interdependent (Klein, 2014). Rather than trying to optimize either the technical system or the social system independently of each other, STS suggests that this interdependency be recognized, and the “system” be treated as a whole. The ideas behind
technology tree
economy buildingmicromanagment
derivative
single-resource
multi-player
increment generators
high interactivity
click-to-manage
click-to-progress
click together
The analysis process that developed the incremental games super-category (each category above is part of incremental games). Theprocess started with open coding of observations on idle games: multiple codes are created. Concepts are discovered through analyzingthe open codes and identifying common features. This is an iterative process, where new codes are added, combined, or deleted. Eachcode is connected to one or more games and can be combined to form new concepts. Concepts are analyzed to find common relationships,and, thus, categories emerge. In the diagram, coloration is only to aid in reading. The left grouping is to show that all contained codes arepart of click-to-manage and click-to-progress.
waiting
automate
progress
increment per second
upgrades
fight boss
multiple game levels
increment per clickone resource
clicking
increment
internal economymultiple resources
resource generator
long cycles of clickingconstantly interactconstantly return
shared resource pool
Open codes concepts categories
Figure 9.12 The grounded theory process showing the development of open coding, through concepts to categories
9 . 5 W h I c h K I N D O F A N A LY T I c F R A m E W O R K T O u S E ? 339
socio-technical theory were first conceptualized around coal mining in the 1950s (see Trist and Bamford, 1951, for example), but it also has a long history of being applied in hospitals and healthcare settings (Waterson, 2014) as well as manufacturing and social media systems. Martin Maguire (2014) highlights the importance of the socio-technical perspective with the rise of virtual organizations. Ken Eason (2014) identifies five significant and enduring aspects of STS theory (Eason, 2014):
1. Task interdependencies: If people are focused on one large task, then the division of sub-tasks between them inevitably sets up interdependencies that are critical to understand. Understanding these interdependencies is particularly useful for recognizing the implica-tions of change.
2. Socio-technical systems are “open systems”: STS are influenced by environmental fac-tors including physical disturbances and financial, market, regulatory, and technical developments.
3. Heterogeneity of system components: The overall task is undertaken by humans in the social subsystem using technical resources in the technical subsystem. Both need to be resilient. Technical components can evolve while humans can learn, develop, and change the technical components to address challenges of the future.
4. Practical contributions: STS theory is making practical contributions in analysis of existing systems, summative evaluation of a major change, through potentially predicting challenges before changes are made, and in designing socio-technical systems that are co-optimized.
5. Fragmentation of design processes: In a complex socio-technical system, there are different design processes, and these can result in fragmentation. Flexibility in speci-fication, local focus in design, user-centered design, and system evolution will help over-come these.
STS is a philosophy rather than a concrete set of methods or analytical tools. But sev-eral socio-technical design methods provide more concrete tools for using a socio-technical framework. For example, see Baxter and Sommerville (2011) and Mumford (2006).
Distributed Cognition of TeamworkDistributed cognition and Distributed Cognition of Teamwork (DiCoT) were introduced in Chapter 4, “Cognitive Aspects,” as an approach to studying the nature of cognitive phe-nomena across individuals, artifacts, and internal and external representations. Investigating how information is propagated through different media is a key goal of this approach, and while distributed cognition provides a good theoretical framework for analyzing systems, it can be difficult to apply in practice. The DiCoT framework was developed as a method to support the application of distributed cognition. It provides a framework of models that can be constructed from a set of collected data, for example ethnographic, interview transcripts, artifacts, photographs, and so on. Underlying each model is a set of principles distilled from the distributed cognition theory. The models are as follows:
• An information flow model that shows how information flows through the system and is transformed. This model captures the information channels and hubs together with the sequence of activities and communication between different team roles.
• A physical model that captures how physical structures support communication between the team roles and facilitates access to artifacts. This model helps to describe the factors that influence the performance of the system at a physical level.
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N340
• An artifact model that captures how artifacts in this system support cognition. This model can be used to represent the key characteristics of an artifact and how its design, structure, and use can support team members.
• A social structure model that examines how cognition is socially distributed. This model maps the social structures to the goal structures, shows how work is shared, and can be used to consider the robustness of the system.
• A system evolution model that depicts how the system has evolved over time. This model provides some explanation for why the work is the way it is. Any design recommendations need to take this context into account.
While the form of the models is not prescribed, the underlying principles support the models’ development. For example, underlying the physical model are principles such as the following:
• Horizon of observation: What an individual can see or hear. • Perceptual: How spatial representations aid computation. • Arrangement of equipment: How the physical arrangement of the environment affects access to information.
DiCoT has been used to understand collaborative work in remote and co-located soft-ware development teams, as shown in Figure 9.13 (Deshpande et al., 2016; Sharp et al. 2009), and has been found to be especially useful for studying how medical teams work and man-age with ever-changing technologies that are introduced into their work environment. For example, Atish Rajkomar and Ann Blandford (2012) examined how healthcare technologies are used; specifically, they examined the use of infusion pumps by nurses in an intensive care
PHPStorm
SourceTree
HipChat
Skype for Business
Appear.in
WebEx
Bitbucket
Jira
Co-locatedoffice
worker
Remoteworker
White boards Physical scrumboard
Walls of information
Suggestions, UXchanges, releaseschedule, Agile
practices relatedinformation
Non-Line ofSightconnectivity
Formalcommunication
Informal,ad-hoc
communication
Hatjitsu
Figure 9.13 An information flow diagram from a DiCoT analysis of software development remote work, based on ethnographic data
9 . 6 T O O L S T O S u P P O R T D ATA A N A LY S I S 341
unit (ICU). They gathered data through ethnographic observations and interviews, which they analyzed by constructing representational models that focused on information flows, physical layouts, social structures, and artifacts. They note that “the findings showed that there was significant distribution of cognition in the ICU: socially, among nurses; physically, through the material environment; and through technological artefacts.” Based on the results of this study, they were able to suggest changes that would improve the safety and efficiency of the nurses’ interactions with the infusion technology.
9.6 Tools to Support Data Analysis
While it is possible to perform these kinds of data analysis using only manual techniques, most people would agree that it is quicker, easier, and more accurate to use a software tool of some kind in the majority of cases. Using a simple spreadsheet application is surprisingly effective, but there are other more sophisticated tools available to support the organization, coding, and manipulation of data, and to perform statistical tests.
Tools in the former category (to support the organization of data) include facilities for cat-egorization, theme-based analysis, and quantitative analysis. These typically provide facilities to associate labels (categories, themes, and so on) with sections of data, search the data for key words or phrases, investigate the relationships between different themes or categories, and help to develop the coding scheme further. Some tools can also generate graphical representations. In addition, some provide help with techniques such as content analysis and sometimes mechanisms to show the occurrence and co-occurrence of words or phrases. In addition, searching, coding, project management, writing and annotating, and report generation facilities are common.
Two well-known tools that support some of these data analysis activities are Nvivo and Dedoose. For example, Nvivo supports the annotation and coding of data includ-ing PDF documents, photos, and video and audio files. Using Nvivo, field notes can be searched for key words or phrases to support coding or content analysis; codes and data can be explored, merged, and manipulated in several ways. The information can also be printed in a variety of forms such as a list of every occasion a word or phrase is used in the data, and a tree structure showing the relationships among codes. Like all software pack-ages, Nvivo has advantages and disadvantages, but it is particularly powerful for handling large sets of data and can generate output for statistical packages such as SAS and SPSS.
Statistical Analysis Software (SAS) and Statistical Package for the Social Sciences (SPSS) are popular quantitative analysis packages that support the use of statistical tests. SPSS, for example, is a sophisticated package offering a wide range of statistical tests such as frequency distributions, rank correlations (to determine statistical significance), regression analysis, and cluster analysis. SPSS assumes that the user knows and understands statistical analysis.
Additional tools to support the analysis of very large sets of data are discussed in Chapter 10, “Data at Scale.”
More information about software tools designed to support the analysis of quali-tative data can be found through the CAQDAS Networking Project, based at the University of Surrey.https://www.surrey.ac.uk/computer-assisted-qualitative-data-analysis
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N342
9.7 Interpreting and Presenting the Findings
Previous sections in this chapter have illustrated a range of different ways to present find-ings—as tables of numbers and text, through various graphical devices and diagrams, as a set of themes or categories, and so on. Choosing an appropriate way to present the findings of a study is as important as choosing the right analytical approach. This choice will depend on the data gathering and analysis techniques used as well as the audience and the original goals of the study. In some situations, the details of data collection and analysis will be needed, for example, when working with others to make sense of a large collection of data, or when trying to convince an audience about a controversial conclusion. This detail may include snippets of data such as photographs of the context of use or videos of participants using the product. In other situations, only the salient trends, headlines, and overall implica-tions are needed, so the style of presentation can be leaner. Where possible, a set of different complementary representations will be chosen to communicate the findings since any one representation will emphasize some aspects and de-emphasize others.
This section focuses on three kinds of presentation styles that haven’t yet been empha-sized to this point: using structured notations, stories, and summarizing.
AcTIvITY 9.4 Consumer organizations and technology companies regularly conduct investigations about technology use. One such report is from DScout investigating smartphone use, available from https://blog.dscout.com/mobile-touches. Note that the report is a downloadable PDF from this web page. Using this report or another report that you find online, look to see how the results are reported.1. What kinds of presentation are used?2. What is left out of the report?3. What is the effect of presenting findings this way?
CommentThe study from DScout found that “users tapped, swiped, and clicked a whopping 2,617 times each day on average.” Note that the web page and the report are presented differently. The web page includes more text and an edited video of participants’ responses to the findings, while the PDF report is presented in a style similar to a PowerPoint presentation. Focusing on the PDF report:1. Graphs and pie charts are used to present demographics and app use, along with lists,
tables, and typographic styles to emphasize certain findings. The report uses two other rep-resentations (see Figure 9.14): a bubble diagram to show the relative use of different apps; and a timeline to represent a “heavy user” and a “average user” and how their touches are spread across the day. These two representations illustrate that developing new (or modifying old) representations to communicate the right findings you want to highlight is acceptable.
9 . 7 I N T E R P R E T I N g A N D P R E S E N T I N g T h E F I N D I N g S 343
Chrome5%
Textra4%
Messages11%
Home screen9% Messenger
(FB)3%
Instagram3%
Google app3%
dscout3%
Facebook15%
WhatsApp3%
WordStreak
1%
Twitter1%
reddit1%Email
(native)1%Internet
1%
You-Tube1%
Hangouts2%
Gmail3%
Gallery(photo)
1%
Snap-chat1%
All other apps
12AM
Phone sessions: Average vs. heavy user
Elizabeth B.25, Chicago, IL
Lori L.45, Grandville, MI
1 2 3 4 5 6 7 8 9 10 11 12PM
1 2 3 4 5 6 7 8 9 10 11 12AM
(a)
(b)
Figure 9.14 Results representation styles used in DScout’s report about smartphone use: (a) a bubble diagram in which the size of the circle represents the number of uses and (b) a timeline across the day of touches from a average user and a heavy user
(Continued)
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N344
9.7.1 Structured NotationsA number of structured notations have been developed to analyze, capture, and present infor-mation for interaction design. These notations follow a clear syntax and semantics, which have been developed to capture particular viewpoints. Some are relatively straightforward, such as the work models promoted in contextual design (Beyer and Holtzblatt, 1998) that use simple conventions for representing flows, breakdowns, individual roles, and so on. Others, such as the modeling language Unified Modeling Language (UML), have stricter and more precise syntax to be followed and are often used to represent requirements (see Chapter 11, “Discovering Requirements”); the activity diagrams, for example, are very expressive when detailed interactions need to be captured.
Advantages of using a structured notation are that the meaning of different symbols is well-defined, and so it provides clear guidance on what to look for in the data and what to highlight and that it enforces precision in expression. Disadvantages include that by high-lighting specific elements, it inevitably de-emphasizes or ignores other aspects, and the preci-sion expressed by the notation may be lost on an audience if they don’t know the notation well. Producing diagrams or expressions in these notations may require further analysis of the findings in order to identify the specific characteristics and properties that the notation highlights. To overcome these disadvantages, structured notations are usually used in combi-nation with stories or other easily accessible formats.
9.7.2 Using StoriesStorytelling is an easy and intuitive approach for people to communicate ideas and experi-ences. It is not surprising then that stories (also called narratives) are used extensively in interaction design, both to communicate findings of investigative studies and as the basis for further development, such as product design or system enhancements.
Storytelling may be employed in three different ways. First, participants (such as inter-viewees, questionnaire respondents, and those you have observed) may have told stories of their own during data gathering. These stories can be extracted, can be compared, and may be used to communicate findings to others, for example, to illustrate points.
Second, stories (or narratives) based on observation, such as ethnographic field studies, may be employed for further data gathering. For example, Valeria Righi et al. (2017) used stories as the basis of co-design workshops in their study to explore the design and use of technologies to support older people. The scenarios were developed on the basis of ethno-graphic studies and previous co-design activities and were presented through storytelling to facilitate understanding. Note that, in this case, the audience was a group of participants in the ongoing study.
2. There is very little text or description in the PDF report and minimal details about the way in which the data was collected or analyzed. Further details are included on the web page, but not in the report.
3. The way in which the findings are presented in the report has quite an impact. The bold and clear graphical images together with minimal but highlighted text mean that the mes-sages are communicated in a straightforward fashion.
9 . 7 I N T E R P R E T I N g A N D P R E S E N T I N g T h E F I N D I N g S 345
Including specific stories gives authenticity to the findings, and it can add to its cred-ibility provided the conclusions are not overstated. Making a multimedia presentation of the story by adding video or audio excerpts and photographs will illustrate the story further. This kind of approach can also be effective if presenting data from an evaluation study that involves observation, as it is hard to contest well-chosen video excerpts of users interacting with technology or extracts from interview transcripts.
Third, stories may be constructed from smaller snippets or repeated episodes that are found in the data. In this case, stories provide a way of rationalizing and collating data to form a representative account of a product’s use or a certain type of event.
Any stories collected through data gathering may be used as the basis for construct-ing scenarios that can then be used for requirements and design activities. See Chapters 11 and 12 for more information on scenarios.
9.7.3 Summarizing the FindingsPresentation styles will usually be used in combination to produce a summary of the findings; for instance, a story may be expanded with graphical representations of activity or demo-graphics, and data excerpts from transcripts or videos may be used to illustrate particular points. Tables of numerical data may be represented as graphs, diagrams, or rigorous nota-tions, together with workflows or quotations.
Careful interpretation and presentation of the study results is just as important as choos-ing the right analysis technique so that findings are not over-emphasized, and evidence is not misrepresented. Over-generalizing results without good evidence is a common pitfall, especially with qualitative analyses’ for example, think carefully before using words such as most, all, majority, and none, and be sure that the justifications reflect the data. As discussed in Box 9.1, even statistical results can be interpreted in a misleading way. For example, if 8 out of 10 users preferred design A over design B, this does not mean that design A is 80 percent more attrac-tive than design B. If you found 800 out of 1,000 users preferred design A, then you have more evidence to suggest that design A is better, but there are still other factors to consider.
AcTIvITY 9.5 Consider each of the following findings and the associated summary statement about it. For each one, comment on whether the finding supports the statement.1. Finding: Two out of four people who filled in the questionnaire checked the box that said
they prefer not to use the ring-back facility on their smartphone.Statement: Half of the users don’t use the ring-back facility.
2. Finding: One day, Joan who works in the design department was observed walking for 10 minutes to collect printouts from the high-quality color printer.Statement: Significant time is wasted by designers who have to walk a long distance to collect printouts.
3. Finding: A data log of 1,000 hours of interaction with a website recorded during January, February, and March records eight hours spent looking at the help files.Statement: The website’s help files were used less than 1 percent of the time during the first quarter of the year.
(Continued)
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N346
Comment1. The questionnaire didn’t ask if they use the ring-back, just whether they preferred to use
the ring-back facility. In addition, two users out of four is a very small number of partici-pants, and it would be better to state the actual numbers.
2. Observing one designer on one day having to walk to get printouts does not mean that this is a general problem. There may be other reasons why this happened on this day, and other information is needed to make a clear statement.
3. This statement is justified as the log was recorded for a significant period of time and using percentages to represent this finding is appropriate as the numbers are large.
In-Depth ActivityThe goal of this in-depth activity is to practice data analysis and presentation. Assume that you are assigned to analyze and present the findings of your data gathering in-depth activity from Chapter 8 to a group of peers, for instance, via a seminar.1. Review the data that you gathered and identify any qualitative data and any quantitative
data in the data set.2. Is there any qualitative data that could sensibly and helpfully be translated into quantita-
tive measures? If so, do the translation and add this data to your quantitative set.3. Consider your quantitative data.
(a) Decide how best to enter it into spreadsheet software, for example, how to handle answers to close-ended questions. Then enter the data and generate some graphical representations. As the data set is likely to be small, think carefully about what, if any, graphical representations will provide meaningful summaries of the findings.
(b) Is there any data for which simple measures, such as percentages or averages, will be helpful? If so, calculate the three different types of average.
4. Consider your qualitative data.(a) Based on your refinement of the study question “improving the product,” identify some
themes in the qualitative data, for example, what features of the product cause people difficulties? Did any of the participants suggest alternative designs or solutions? Refine your themes and collate extracts of data that support the theme.
(b) Identify any critical incidents in the data. These may arise from interviews, question-naire responses, or observation. Describe these incidents carefully and choose one or two to analyze in more depth, focusing on the context in which they occurred.
5. Collate your findings as a presentation and deliver them to a group of peers.6. Review the presentation and any questions from the audience. Consider how to improve
the analysis and presentation.
347
Further Reading
BLANDFORD, A., FURNISS, D. and MAKRI, S. (2017) Qualitative HCI Research: Going Behind the Scenes. Morgan Claypool Publishers. This book in the form of a lecture dis-cusses the practical details behind qualitative analysis in HCI. Using the analogy of making a documentary film, the authors point out that, as with movies, qualitative analysis is often presented as a finished product while the work “behind the scenes” is rarely discussed.
SummaryThis chapter described in detail the difference between qualitative and quantitative data and between qualitative and quantitative analysis.
Quantitative and qualitative data can be analyzed for patterns and trends using simple techniques and graphical representations. Qualitative data may be analyzed inductively or deductively using a variety of approaches. Thematic analysis (an example of inductive anal-ysis) and data categorization (an example of deductive analysis) are common approaches. Analytical frameworks include conversation analysis, discourse analysis, content analysis, interaction analysis, grounded theory, and systems-based approaches.
It was noted that presenting the results is just as important as analyzing the data, hence it is important to make sure that any summary or claim arising from the analysis is carefully contextualized, and that it can be justified by the data.
Key Points • The kind of data analysis that can be done depends on the data gathering techniques used. • Qualitative and quantitative data may be collected from any of the main data gathering techniques: interviews, questionnaires, and observation.
• Quantitative data analysis for interaction design usually involves calculating percentages and averages.
• There are three different kinds of average: mean, mode, and median. • Graphical representations of quantitative data help in identifying patterns, outliers, and the overall view of the data.
• Analysis of qualitative data analysis may be inductive, in which themes or categories are extracted from the data, or deductive, in which pre-existing concepts are used to interro-gate the data.
• In practice, analysis often proceeds in iterative cycles combining inductive identification of themes and deductive application of categories and new themes.
• Which analysis approach is used is tightly coupled to the data that is collected and depends on the goals of the study.
• Several analytical frameworks exist that focus on different levels of granularity with differ-ent purposes.
F u R T h E R R E A D I N g
9 D ATA A N A LY S I S , I N T E R P R E TAT I O N , A N D P R E S E N TAT I O N348
BRAUN, V. and CLARKE, V. (2006) Using Thematic Analysis in Psychology. Qualitative Research in Psychology, 3(2), pp. 77–101. This paper focuses on thematic analysis, how it relates to other qualitative analysis approaches, and how to conduct it in a rigorous fashion. It also discusses advantages and disadvantages of the approach.
CHARMAZ, K. (2014) Constructing Grounded Theory (2nd ed.). Sage Publications. This popular book provides a useful account of how to do grounded theory.
CORBIN, J. M. and STRAUSS, A. (2014) Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage. This presents a readable and practical account of applying the grounded theory approach. It is not tailored specifically to inter-action design, and therefore it requires some interpretation. It is a good discussion of the basic approach.
HUFF, D. (1991) How to Lie with Statistics. Penguin. This wonderful little book illustrates the many ways in which numbers can be misrepresented. Unlike some (many) books on sta-tistics, the text is easy to read and amusing.
ROGERS, Y. (2012) HCI Theory: Classical, Modern, and Contemporary. Morgan and Clay-pool Publishers. This short book, in the form of a lecture, charts the theoretical developments in HCI, both past and present, reflecting on how they have shaped the field. It explains how theory has been conceptualized, the different uses it has in HCI, and which theory has made the most impact.
Chapter 10
D A T A A T S C A L E
10.1 Introduction
10.2 Approaches for Collecting and Analyzing Data
10.3 Visualizing and Exploring Data
10.4 Ethical Design Concerns
ObjectivesThe main goals of the chapter are to accomplish the following:
• Provide an overview of some of the potential impacts of data at scale on society. • Introduce key methods for collecting data at scale. • Discuss how data at scale becomes meaningful. • Review key methods for visualizing and exploring data at scale. • Introduce design principles for making data at scale ethical.
10.1 Introduction
How do you start your day? How much data do you encounter when first looking at your smartphone, switching on your laptop, or turning on another device? How much do you knowingly create and how much do you create unknowingly? Upon waking up, many people routinely will ask their personal assistant, something like, “Alexa, what is the weather today?” or “Alexa, what is the news?” or “Alexa, is the S-Bahn train to Schönefeld Airport running on time?” Or, they will ask Siri, “What is my first meeting?” or “Where is the meeting?”
Having oriented themselves for the day, people will walk a few blocks to the subway entrance, dip their Metro Card in the turnstile to pay the fare, exit the station at their stop, grab their favorite morning beverage at a nearby cafe, and proceed to their office where they check in with the employee card at a security gate and take the elevator to their floor.
These are just a few of the things that many of us do to start our workdays. Each activity involves creating, searching, and storing data in some way or another. We may know that this is happening, we may suspect that it is happening, or we may be totally unaware of the data that we are generating and with which we are interacting.
1 0 D ATA AT S C A L E350
There is also increasing concern about exactly what data is collected about us through personal assistants such as Amazon Echo, Google Home, Cortana, and Siri. We also know that many large cities, such as New York and London, have an enormous number of sur-veillance cameras (CCTV) spread around, especially in busy places such as subway stations and shopping malls. The video footage from these sources is kept for two weeks or more. Similarly, we experience being checked into an office, so we know that our movements are being tracked by security personnel. Our activities are also being tracked more surrepti-tiously through the technology that we use such as smartphones and credit cards.
What happens to all the data collected about us? How does it improve the services pro-vided by society? Does it make traveling more efficient? Does it reduce traffic congestion? Does it make the streets safer? Moreover, how much of the data collected from our smart-cards, smartphone Wi-Fi signals, and CCTV footage can be tracked back to us and pieced together to reveal a bigger picture of who we are and where we go? What might that data reveal about us?
Data at scale, or as it is often called big data, describes all kinds of data including data-bases of numbers, images of people, things and places, footage of conversations recorded, videos, texts, and environmentally sensed data (such as air quality). It is also being col-lected at an exponential rate; for example, 400 new YouTube videos are uploaded every min-ute, while millions of messages circulate through social media. Furthermore, sensors collect billions of bytes of scientific data.
Data at scale has huge potential for grounding and elucidating problems, and it can be collected, used, and communicated in a wide variety of ways. For example, it is increasingly being used for improving a whole range of applications in healthcare, science, education, city planning, finance, world economics, and other areas. It can also provide new insights into human behavior by analyzing data collected from people, such as their facial expres-sions, movements, gait, and tone of voice. These insights can be enhanced further by using machine learning and machine vision algorithms to make inferences. This includes people’s emotions, their intent, and well-being, which can then be used to inform technology inter-ventions aimed at changing or improving people’s health and well-being. However, beyond societal benefits, data can also be used in potentially harmful ways.
As mentioned in Chapter 8, “Data Gathering,” and Chapter 9, “Data Analysis,” data can be either qualitative or quantitative. Some of the methods and tools used to collect, analyze, and communicate data can be carried out manually or using quite simple tools. What makes this chapter on data at scale different is that it considers how huge volumes of data can be analyzed, visualized, and used to inform new interventions. While having access to large vol-umes of data enables analysts, designers, and researchers to address large, important issues such as climate change and world economic issues, assuming that there are tools to do this, they also raise a number of user concerns. These include whether someone’s privacy is being violated by the data being collected about them and whether the data corpora being used to make decisions about people, such as the provision of insurance and loans, are fair and transparent.
Furthermore, the combination of vast amounts of data from many sources and the avail-ability of increasingly powerful data analytic tools to analyze that data is now making it possible to discover new information that is not available from any single data source. This is enabling new kinds of research to be conducted for understanding human behavior and environmental problems.
1 0 . 2 A p p r o A C h E S T o C o L L E C T i n g A n D A n A Ly z i n g D ATA 351
10.2 Approaches to Collecting and Analyzing Data
Collecting data has never been easier. What is challenging is knowing how best to analyze, collate, and act upon the data in ways that are socially acceptable, beneficial to society, and ethically sound. Are there certain rules or policies in place on what to reveal about people or when certain patterns, anomalies, or thresholds are reached in a data stream? For example, if people-tracking technology is used at an airport, how is that revealed to those at the air-port? Is it enough only to show data that can help manage people flows and bottlenecks? For example in an airport terminal showing a public display in which one section of the terminal is detected to be much busier than another (Figure 10.1), do travelers ever stop and wonder how this data is being collected? What else is being collected about them? Do they care?
Another technique for analyzing what people are doing on websites and social media is to examine the trail of activity that they leave behind. You can see this by looking at your own Twitter feed or by looking at someone else’s whom you are following, for example, a friend’s, a political leader, or a celebrity. You can also examine discussions about a particular topic such as climate change, reactions to comments made by comedians like John Oliver or Stephen Colbert, or a topic that is trending on a particular day. If there are just a few posts, then it is easy to see what is going on, but often the most interesting posts are those that gen-erate lots of comments. When examining thousands or tens of thousands of posts, analysts use automated techniques to do this (Bostock et al., 2011; Hansen et al., 2019).
Figure 10.1 Heathrow Airport Terminal 5 Public Display in top-right corner of image showing the relative level of activity using an infographic of North vs. South SecuritySource: Marc Zakian / Alamy Stock Photo
1 0 D ATA AT S C A L E352
10.2.1 Scraping and “Second Source” DataOne way to extract data is by “scraping” it from the web (assuming that this is allowed by the application). Once the data is scraped, it can be entered into a spreadsheet for study and analyzed using data science tools. The focus from an interaction design perspective is how one can interact with that data and the way it is displayed rather than the actual scraping process per se, so that it can be analyzed and sense can be made of it.
In addition, the openly available big data that Google and other companies now provide for researchers to mine offers a “second source” methodology, meaning search terms, Face-book posts, Instagram comments, and so on. Analysis of this data can indirectly reveal new insights about the users’ concerns, desires, behaviors, and habits. For example, the Google Trends tool can be used for exploring and examining the motivation behind what people ask when they type something into Google Search. Seth Stephens-Davidowitz (2018) has used it extensively to reveal what people are interested in finding out. From his analysis of Google Search data, he discovered that people type into the search box all sorts of intimate ques-tions about their health among other topics. Moreover, he found that his analysis of search data revealed things that people would not freely admit to when asked using other research methods, such as surveys and interviews. He also makes an important assertion: to obtain new insights from big data, it requires asking the right questions of the data. Furthermore, it is not how much data can be collected or mined but what is done with the new data that has been made available. Simply mining it because there is a tool available may yield surprising results, but well-honed questions that guide and are used to interpret the data that is found will be more valuable (see Chapter 8, “Data Gathering”).
How do researchers know what are the right questions to ask of this data? This is par-ticularly pertinent for HCI researchers to understand, especially in terms of how users will relate, trust, and confide in the next generation of technologies, including domestic robots, bots, and virtual agents.
ACTiViTy 10.1 What insights do Google Trends searches tell us about ourselves?
Go to Google Trends (https://trends.google.com). Then try typing into the search box statements such as “I hate my boss,” “I feel sad,” or “I eat too much.” See how many people have typed this into Google over the last month, year, and for different countries. Then type in the opposite statements: “I love my boss,” “I am happy,” or “I never eat enough.” How do the results compare? Which is asked more often? Then type in your name and see what Google returns.
CommentIt is surprising how many people confide such personal statements in Google. Some people will tell it anything. Google Trends provides a way of comparing the search data across time, country, and other topics. When you type in your name (unless you have the same name as a famous person), it often comes back with “hmm, your search doesn’t have enough data to show here.”
1 0 . 2 A p p r o A C h E S T o C o L L E C T i n g A n D A n A Ly z i n g D ATA 353
10.2.2 Collecting Personal DataPersonal data collection started becoming popular through the quantified-self (QS) move-ment in 2008 where monthly “show and tell” meetings were organized to enable people to come together to share and discuss their various self-tracking projects. Nowadays, many apps and wearable devices exist that people can buy off the shelf, which can collect all sorts of personal data and visualize it. These results can be matched against targets reached, and recommendations, hints, or tips can also be provided about how to act upon it. Many apps now come prebundled on a smartphone or smartwatch, including those that quantify health, screen time, and sleep. Some also allow multiple activities to be tracked, aggregated, and cor-related. The most common types of apps are for physical and behavioral tracking, including mood changes, sleep patterns, sedentary behavior, time management, energy levels, mental health, exercise taken, weight, and diet. A common motivation for deciding to embark on tracking some aspect of one’s personal data over time is to see how well they are doing com-pared to a set threshold or level (that is, a set target, a comparison with the week before, and so on). The aggregate data may raise awareness and be revealing to the extent that some-one feels compelled to act upon it (for example, changing their sleeping habits, eating more healthily, or going to the gym more regularly).
Self-tracking is also increasingly being used by people who have a condition or disease as a form of self-care, such as monitoring blood glucose levels for those who have diabetes (O’Kaine et al., 2015) and the occurrence of migraine triggers (Park and Chen, 2015). This kind of self-care monitoring has been found to help people engage in reflection when look-ing at their data and then learning to associate specific indicators with patterns of behavior. Making these connections can increase self-awareness and provide them with early warning signs. It can also lead them to avoid certain events or adjust their behavior accordingly. Many people are also happy to share their tracked data with others in their social networks, which has been found to enhance their social networking and motivation (Gui et al., 2017).
Quantified-self projects generate lots of data. New kinds of health data can now be col-lected by mobile health monitors, such as heart rate, generating masses of data per person each month, which was simply heretofore unavailable. This raises questions as to how much data should be kept and for how long? Also, how can this data be used to best effect? Should it signal to the user when their heart rate deviates from normal levels? Given that masses of data are being collected from many individuals using the same devices, it is possible to col-late all of the data. Would it be useful for health clinicians and the individuals alike to have access to all of this data in order to see trends and comparisons? How can this be made to be both informative and reassuring? Translating someone’s heart rate data that is sampled many times per second along with their electroencephalogram (EEG) streamed brainwave data into an early warning sign with an appropriate form of intervention is challenging (Swan, 2013). It can easily lead to increased anxiety. Much thought needs to go into providing information into an interface that will not cause unnecessary panic. New tools should also provide flex-ibility in how the user might want to customize or annotate their data to meet their specific needs (Ayobi et al., 2018).
10.2.3 Crowdsourcing DataIncreasingly, people crowdsource information or work together using online technologies to collect and share data. The idea of a crowd working together has been taken one step further in Crowd Research, where many researchers from all over the world come together to work
1 0 D ATA AT S C A L E354
on large problems, such as climate science (Vaish et al., 2018). The goal of this approach is potentially to enable hundreds of people to contribute, through collecting data, ideating, and critiquing each other’s designs and research projects. Conducting research on a massive scale enables potentially hundreds or thousands of people to work on a single project, which can help address large problems, such as migration or climate change at scale.
There are also many citizen science and citizen engagement projects (see Chapter 5, “Social Interaction”) that crowdsource data at scale and in doing so amass billions of differ-ent types of data (photos, sensor readings, comments, and discussion), which are collected by millions of people across the world. Most of this data is stored in the cloud as well as on local machines. Examples of large citizen science projects include iSpotNature, eBird, iNaturalist,
Figure 10.2 Abundance map for the common raven. The darkest area indicates where ravens are most abundant.Source: https://ebird.org/science/status-and-trends/comrav. This link will allow you to see how abundance changes during each week of the year (purple indicates high abundance and yellow indicates low abundance).
1 0 . 2 A p p r o A C h E S T o C o L L E C T i n g A n D A n A Ly z i n g D ATA 355
and the Zooniverse. There are also thousands of much smaller projects that together generate huge amounts of data.
eBird.org, for example, collects data about bird sightings that is contributed by naturalists—amateurs ranging from beginning birders to highly experienced expert birders and professional scientists. The site was launched in 2002 as a collaboration effort between Cornell University’s Lab of Ornithology and the National Audubon Society. The data includes bird species data, the abundance of each species, geolocation data indicating where observations are made, profiles of the people who contribute, comments, and discussion. There are also smartphone apps and a website with links to many resources, including identification guides, data analysis tools, maps and visualizations, reports, and scientific articles. As of June 2018, there were more than 500 million bird observations recorded in a global database. eBird feeds data into aggregator sites such as the Global Biodiversity Information Facility (GBIF) so that it is available for scientists. It also provides several maps, many of which are interactive (see Figure 10.2) and other graphic representations of its data that are available for anyone to access.
Crowd projects raise a number of issues as to who owns and manages the data. This is especially pertinent when the data collected can be mined to unearth details about the people who contribute the data as well as about the endangered species, for example. For researchers and UX designers, there are interesting questions about how to balance making data avail-able for education and research while protecting the privacy of those contributing the data and the location of endangered species in this example. Box 10.1 discusses how one citizen science project, iNaturalist.org, tries to manage this balance.
BoX 10.1 Citizen Science and UX Design for Privacy
How privacy is interpreted by citizen scientists and their desire and need for privacy regula-tions differs across projects (Bowser et al., 2017). Being able to share citizen science data has many advantages as well as some privacy concerns. For example, participants can see what others are observing in their area. Bird enthusiasts also often like to share first sightings, for instance, when the first swallows appear in spring or the first snow geese arrive in winter. They may also want to check identifications with each other. The downside of this community inter-action is that personal profile and location data can be used to identify particular contributors and their patterns of behavior. The latter can be especially problematic, as many participants visit the same places regularly. It is, therefore, important to ask how important is privacy in citizen science compared with the benefits of community engagement? And how might UX design help protect both participants and rare species, while supporting open engagement?
Many citizen science projects and societies post privacy policies like the one shown in the following link. Other strategies involve making images and locations fuzzy so that they are not exact. This is also a good strategy for keeping the location of rare species’ observations confidential. For example, iNaturalist.org has a geoprivacy setting that can be set to “open,” “obscure,” or “private.” Obscured observations are used to hide the exact location of endan-gered species, as shown in Figure 10.3.
1 0 D ATA AT S C A L E356
The European Citizen Science Organization’s Data and Privacy Policy can be seen at https://ecsa.citizen-science.net/about-us/privacy-policy.
BoX 10.2 A Human-Data Design Approach to Sensing Data
There are a number of off-the-shelf sensor toolkits available now that can be placed in some-one’s home or local community to measure air quality or other aspects of the environment. One of the earliest open platforms developed was Smart Citizen (Diez and Posada, 2013). A compact device was built with a number of embedded sensors in it that could measure nitrogen dioxide (NO2), carbon monoxide (CO), sunlight, noise pollution, temperature, and humidity levels. The data being collected from the platform was connected to a live website that could be accessed by anyone. The various data streams were presented via a dashboard using canonical types of visualisations, such as time-series graphs (see Figure 10.4a). Data streams from other Smart Citizen devices, set up throughout the world, could also be viewed via the dashboard (https://smartcitizen.me/) making it easy to compare data collected from
Figure 10.3 iNaturalist.org geoprivacy obscures the location of an observation.In the above example: 1. EN indicates that the organism is endangered, so its location needs to be obscured, 2. indicates that obscuring is done by randomly placing the marker for the location within the broader area, and 3. allows the contributor to verify that this observation has been observed within iNaturalist.Source: https://www.inaturalist.org
1 0 . 2 A p p r o A C h E S T o C o L L E C T i n g A n D A n A Ly z i n g D ATA 357
different locations. While the mass of environmental data accumulating was fascinating to data scientists and researchers, this was not the case for many of the householders who had set up a smart citizen device in their home. They found the visualizations presented via the dashboard to be difficult to understand and were unable to connect the data being sensed with what was happening in their homes (Balestrini et al., 2015). As a result, they did not find it useful and quickly stopped looking at it.
The Physikit project took this user problem as its starting point (Houben et al., 2016). A human-data design approach was adopted; the goal is to transform sensed data being col-lected into something that is meaningful to the general public. The goal is to provide a way of enabling users to engage with their sensed data, by giving it a physical ambient presence in the location where it is being collected. A set of colorful physical cubes were designed that could light up, move parts or vibrate, depending on how they were configured (see Figure 10.4c). Householders could easily configure a set of rules to decide which cubes to connect with which data streams and what each cube should do depending on levels or thresholds being sensed. This was intended to let them select aspects of their home that they were interested in knowing more about.
For example, one of the PhysiCubes had a rotating disk on the top that could be set to move clockwise or counterclockwise and at different speeds. One household decided to use it to measure the level of humidity in their kitchen throughout the day. They placed a basil plant on top of the cube (see Figure 10.4b) as a way of visibly showing the level of humidity in the kitchen. The rule they set up for the cube was for it to rotate only if the humidity level detected was below 60 percent. At the end of each day, they could tell how humid it had been in the room by the extent to which the plant remained upright. If it was leaning toward the window, this suggested to them that the humidity level in the kitchen had been high throughout the day because the disk had not moved the plant around for the leaves to get an even amount of light. The household, had in effect, created a naturally growing physical visualization that held historical data.
(a)
(b)
(c)
1 0 D ATA AT S C A L E358
10.2.4 Sentiment AnalysisSentiment analysis is a technique that is used to infer the effect of what a group of people or a crowd is feeling or saying. The phrases that people use when offering their opinions or views are scored as being negative, positive, or neutral. The scales used vary along a con-tinuum from negative to positive, for example, –10 to +10 (where –10 is the most negative, 0 is neutral, and +10 is the most positive). Some sentiment systems provide more qualitative measures by identifying if the positive or negative sentiment is associated with a specific feel-ing, for example anger, sadness, or fear (negative feelings) or happiness, joy, or enthusiasm (positive feelings). The scores are extracted from people’s tweets and texts, online reviews, and social media contributions. Their facial expressions (see Chapter 6, “Emotional Inter-action”) when looking at ads, movies, and other digital content and customer’s voices can also be analyzed and classified using the same scales. Algorithms are then applied to the labeled data in order to identify and classify them in terms of the level of effect that has been
(a)
(b)
(c)
Figure 10.4 (a) Smart Citizen’s dashboard and visualization, (b) a PhysiMove cube set up in a householder’s home, and (c) the components of the Physikit toolkit (Houben et al., 2016).
Source: (a) https://www.citizenme.com, (b) and (c) Yvonne Rogers
1 0 . 2 A p p r o A C h E S T o C o L L E C T i n g A n D A n A Ly z i n g D ATA 359
expressed. There are a number of online tools that can be used to do this, such as DisplayR and CrowdFlower. MonkeyLearn provides a detailed tutorial on sentiment analysis (https://monkeylearn.com/sentiment-analysis/).
Sentiment analysis is commonly used by marketing and advertising companies to decide on what types of ads to design and place. In addition, it is increasingly being used in research to study social science phenomena. For example, Veronikha Effendy (2018) used sentiment analysis to study people’s opinions about the use of public transportation from their tweets. In particular, she was interested in determining what were the positive and negative opin-ions toward it, which could then be used as evidence for making a case on how to improve public transportation to increase its use in Indonesia, where there are huge traffic conges-tion problems.
However, sentiment analysis as a technique is not an exact science and should be viewed more as a heuristic than as an objective evaluation method. Giving a word a score from −10 to +10 is quite a crude way to measure. To assess how good sentiment analysis is as a method, Nicole Watson and Henry Naish (2018) compared human judgment with computer-based sentiment analysis for evaluating positive articles about the U.S. economy. They found that the computer was more often wrong than right compared with the human participants. Their analysis indicates that humans express their optimism about a topic in much richer ways. Moreover, it also shows that by focusing on emotive words in phrases, sentiment analysis misses the nuances of expression that humans understand intuitively. For example, how would sentiment analysis score the phrase written by a teen in a text to their friends that said, “I am weak”? It would probably give it a negative score. In fact, the phrase is teen slang for “That is funny,” which is completely the opposite.
10.2.5 Social Network AnalysisSocial network analysis (SNA) is a method based on social network theory (Wellman and Berkovitz, 1988; Hampton and Wellman, 2003) for analyzing and evaluating the strength of social ties within a network. While understanding social ties has been a strong interest of sociologists for many years, (for example, Hampton and Wellman, 2003; Putnam, 2000), as social media became increasingly successful, it also became a key interest for computer and information scientists (for example, Wasserman and Faust, 1994; Hansen et al., 2019). They want to understand the relationships that form among people and groups within and across different social media platforms, and with offline social networks too. Online, trillions of messages, tweets, pictures, and videos are posted and responded to every second of every day via Weibo, Tencent, Baidu, Facebook, Twitter, Instagram, and YouTube. Some examples include families posting pictures of their kids’ birthday parties and family outings, discussion of hot political issues, and friends and colleagues chatting and keeping in contact with each other’s travel experiences, hobbies, life’s challenges, and successes.
Social network analysis enables these relationships to be seen more clearly. It helps to reveal who is most active in a group, who belongs to which groups, and how the groups do or do not interact and relate to each other. Analyses can also show which topics are hot and throw light on when, how, and why some topics go viral. Managers, marketers, and politi-cians are especially interested in how these activities can influence them, their companies, and their constituents. Many other people like to try to make their posts or YouTube videos go viral, as discussed in Chapter 5.
1 0 D ATA AT S C A L E360
So, how does social network analysis work? It is a big topic, but broadly, as the name suggests, a network is a collection of things and their relationships to each other. A social net-work is a network of people and groups with relationships to each other. Human beings, like other primates, have formed networks for as long as our species has existed. Many other spe-cies, such as elephants, wolves, and meerkats, to name just a few, also rely on social networks for safety, for collaboratively rearing their young, and when foraging or hunting for food.
Two main entities make up a social network. Nodes, which are also sometimes called enti-ties or vertices, represent people and topics. The connections between the nodes are called edges, which are also known as links or ties. The edges show the connections among nodes, for exam-ple, the members of a family. They can show the direction of relationships; for instance, parents may have a line with an arrow-head that points to their children, indicating the direction of the relationship between the two nodes. Similarly, an arrow in the opposite direction indicates that children have parents. These are known as directional edges. Edges can also indicate relation-ships in both directions by having arrows at each end. Edges that do not have an arrowhead are nondirectional; that is, the direction of the relationship between two nodes is not shown.
Drawing on algorithms and based in statistics, social network analysis offers a range of metrics for describing the properties of networks. One of the most important sets of metrics for visualizing networks in big data is measures of centrality. Several different measures of centrality exist based on differen
