Creating Effective Decision Aids for Complex Tasks
Caroline Clarke Hayes and Farnaz Akhavi
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 152-172
Abstract
Engineering design tasks require designers to continually compare, weigh, and choose among many complex alternatives. The quality of these selection decisions directly impacts the quality, cost, and safety of the final product. Because of the high degree of uncertainty in predicting the performance of alternatives while they are still just sketches on the drawing board, and the high cost of poor choices, mathematical decision methods incorporating uncertainty have long held much appeal for product designers, at least from a theoretical standpoint. Yet, such methods have not been widely adopted in practical settings. The goals of this work are to begin understanding why this is so and to identify future questions that may lead to solutions. This paper summarizes the results of several studies by the authors: two laboratory studies in which we asked product designers to use various mathematical models to compare and select design alternatives, and a set of ethnographic studies in which we observed product designers as they worked so that we could better understand their actual practices and needs during decision making. Based on these studies, we concluded that the mathematical models, as formulated, are not well suited to designers' needs and approaches. We propose a research agenda for developing new approaches that combine decision theoretic and user-centered methods to create tools that can make product designers' decision making work easier, more systematic, more effective, and more reportable.
Practitioner's Take Away
This article looks at some of the issues in designing and developing tools for complex problem solving in work domains such as mechanical design, logistical planning, and medical decision making. It is particularly challenging to develop tools (software or otherwise) to assist in these tasks because so much of the work is cognitive. The steps are often internalized, highly nuanced, and dependant on a body of personal experience, rather than well-defined processes. Tools to support decision making must often cater to the needs of a diverse group of users who may range from domain novices to domain experts. Additionally, the tasks themselves and the knowledge associated with them may change rapidly with technological advances making incorporation of extensive volumes of complex knowledge in a tool impractical. The lessons learned from the work reported in this paper can be applied to many other complex domains.
- In order to design tools to fit users' needs in complex domains, it is important to understand (a) how they solve problems, (b) where their strengths and weaknesses lie, and (c) what type of challenges, constraints, and conditions exist in their actual work environments. Observational and measurement techniques for understanding work and problem solving, such as ethnographic studies, protocol analysis, and laboratory studies (in addition to usability studies of the evolving tool), may be even more important in complex, expert domains than they are in other tasks.
- Domain experts, intermediates, and novices may not all have the same needs, nor do they have the same knowledge, strengths, and weaknesses. While it is desirable to design a tool to assist all of these different types of users, it may not be practical or possible to do so. Tool designers may need to choose to design specifically for users at a specific range of domain expertise levels.
- Tools must respect users' problem solving approaches. For example, in the study reported in this paper, users employed flexible approaches, with many short cuts such as only searching for additional information if it was needed to distinguish between two top design alternatives.
- Users were impatient with having to gather, estimate, or make guesses pertaining to information that might not actually be necessary for a decision and viewed such tasks as unnecessary or "busy work." (For example, user may feel, "I can choose the best alternative in my head much more quickly without having to specify all this information for the computer, so why should I bother with the computer tool?"). Unless users perceive that the work of using a tool will directly benefit them, they are likely to view these data entry tasks as unnecessary chores and are unlikely to adopt the tool.
- Old style "expert systems" were not entirely successful because they attempted to encode complex, highly nuanced, and highly contextual expert knowledge in software. Such systems were expensive to build, as well as brittle and hard to maintain. It can be more cost effective and successful to provide tools that can help problem solvers explore, organize, and visualize problem-relevant data to which they can apply their own knowledge and judgment powers.
- Similarly, tools should avoid requiring users to articulate and enter large quantities of their knowledge or judgments on a problem-by-problem basis. Unless users perceive this work as directly benefiting them in their current task, they are likely to perceive such data entry tasks as unnecessary chores.
Article Contents
Download Article
Creating Effective Decision Aids for Complex Tasks