Thumbnail: Rolf Molich

By Cliff Anderson

Rolf Molich is “curious.” He cites that curiosity as the impetus behind his CUE, or Comparative Usability Evaluation, studies (www.DialogDesign.dk/cue.html). These now-world-famous studies look at how usability tests and expert reviews are actually carried out in practice – and about how reproducible they really are. There’s a lot more to Rolf, though, than his CUE studies.

Early Years
Rolf was curious from the beginning. Though his background is technical (he has a Masters in software engineering), when he first happened upon usability in the 1980s, he was hooked. In fact, like a number of early practitioners, he found he was doing something very similar on his own: “I was just talking to users, though. Testing, at that time, was totally unknown.”

Running into Jakob Nielsen also helped. The two worked on a project for the Danish government together. Out of that came a course, a book, and a lifelong friendship.

Others he ran into in those early years included Clayton Lewis, Jef Raskin, Ben Shneiderman, and Patricia Wright. He was an inveterate conference-goer, especially when “the IT field was so small you could have one conference and cover everything.” He also brought some of the speakers at these conferences back to Denmark.

After many years with a number of different companies, Rolf “dared to take the risk of becoming independent.” In 1997, he started Dialog Design. In addition to “wanting to work full-time with usability,” he also was intrigued by the free time and independence owning his own company might give him. And that – and his curiosity – led him to his first CUE study.

CUE
“Lots of people were talking about how wonderful usability studies were” Rolf points out. “I tended to agree with them because I had seen myself the enormous political impact a usability study can have. But then I was curious and said, ‘Are they reproducible?’”

He shared his thoughts on a usability list serv, got four volunteers, then launched CUE-1. This study looked at a Windows application, having the participants test it in any way they saw fit to try and identify usability problems.

The results were presented at UPA 1998. Though “everyone in the crowd thought it was interesting,” there was also some resistance, something that Rolf would become used to over the years. He notes, “People said, ‘This just can’t be true. The results just can’t be this diverse.”

But they were. In CUE-2, Rolf found that 75% of the problems generated by nine teams running simultaneous, independent tests of Hotmail were unique, something very similar to what he had found in CUE-1.

The next study, CUE-3, took a different approach. Instead of doing tests, he had 12 teams do an expert review on a car reservation site (avis.com). Though this study “has not been published in any great detail,” it did serve as a springboard for CUE-4.

That study, the largest to date, was part of a workshop at CHI 2003. It divided up 17 teams, with half doing tests and half doing expert reviews. Rolf notes, “One of the things we wanted to find out was, ‘Is there a difference?’ Tests are more expensive, but do they give better results – more quality, more reliability, more reproducible results? We found that not to be the case.”

He continues, “It would be very difficult, if not impossible, to make a distinction based on what the teams came up with. There was one difference though. The teams that used an expert review were a bit more efficient. They invested fewer resources. Otherwise, there were few differences.”

CUE-5 was done at User Interface Engineering’s 2005 User Interface Conference. It pitted 13 teams against an Ikea tool that lets users customize their own furniture. This time, though, not all the teams were experts (who Rolf had relied on in previous studies). Nonetheless, except for “some who quite clearly didn’t have the time to do a thorough review, some of them did as well as the experts in CUE-4.”

The last study, CUE-6, was also done as part of a UIE conference, this past October. Thirteen teams took a look at another car reservation site (enterprise.com) before the conference. Rolf then discussed the results with the participating teams at a Master Workshop at the conference.

An Ounce of Prevention
Undoubtedly, the most important finding of the CUE studies has been the diversity of the results. In explanation, Rolf points to the simple number of usability problems in the average system: “If you have 500 problems to choose from on a non-trivial website, and the usability test basically finds 40 problems more or less at random, then it’s no longer a surprise when you get these different problems.”

What’s the solution? Interestingly, Rolf doesn’t see it as increasing the number of users, or other methodological changes. “The solution is a much more radical one,” he states firmly. “It’s prevention. Many of the usability problems we have seen should have not occurred on the website in the first place.”

But how to provide that prevention? Rolf recommends that “the development team know certain basic rules, like the heuristics that Jakob and I have been teaching for 30 years now.” Indeed, though many now associate those famous heuristics with Nielsen, they first appeared in print in a jointly written paper at CHI 90.

Testing
So, where does that leave testing? “I still think usability testing is a very important method,” Rolf acknowledges, “but not to find usability problems. Its most important role is to make people understand the need for the prevention of usability problems. Usability tests are a very, very important political instrument. It’s an absolutely unique method for convincing product team members that usability problems exist in their product. Your baby is ugly, and something needs to be done about it. But the method is much too expensive to eradicate all usability problems or even just all serious usability problems.”

Rolf sees another possible problem with testing. Much too often, project teams do not actually implement the agreed-upon recommendations. Once again, Rolf was curious: “Why is it that much of what usability professionals say and can demonstrate tends to get ignored? During and immediately after the test sessions, everyone’s very enthusiastic, then they turn to other things, and the results tend to be forgotten. But it’s a very important message – usability problems don’t correct themselves.”

Rolf thinks he has an idea of where part of the problem may lie – with the usability reports themselves. “Sometimes I get shocked when I see some of the reports being produced from a professional usability test,” he confides. “They deal with usability, but the usability report itself is unusable, which I think is not a very good idea.”

Rolf recommends “making sure your presentations are extremely usable, so the project team doesn’t have that excuse.” Some things he recommends include prioritizing issues (he counsels 30 to 40), including a one-page executive summary (“I’ve seen executive summaries of 10 pages, which I think is ridiculous”), and making sure to include positive results (“to show that you’re not only critical, but you can also administer praise”).

The Future
He goes on in a more general vein: “Our profession is suffering by some people who are running around and selling voodoo. Quality is a very important parameter in our profession.” As to how to get that quality, Rolf points to “broad knowledge, experience, and flair for the whole subject area. Experience is definitely very important.”

He continues: “The things that usability professionals have to promote themselves on these days are that they have the knowledge. It’s really a specialist job. It’s something that requires a considerable time investment to keep abreast of developments in the field. It also requires special skills and you have to master these skills.” Being curious probably helps too.