Things we've learnt about UI evaluation

One thing that really stands out after reading the comments on our Google+ evaluation is that we had to be more explicit about our underlying assumptions. For example we didn’t indicate how we invited respondents for our QUIS survey or for the user interviews. We did say why we chose the QUIS 5.0 questionnaire, but we didn’t mention it in the extensive document. We didn’t indicate why we chose the actions executed by the test users in the usability tests either.

Also, choosing a general survey always carries the risk of having some questions that are not a 100% applicable to the specific setting. So we sometimes tried to extract information from questions that were not really relevant.

We noticed the importance of good data representation, which helps the reader understand and verify our the results of our tests. In this regard, we think the box plot with the scores of all the QUIS questions was a good choice, but it would have been more informative if we would have placed the questions instead of numbers on the x-axis.

We already realized beforehand that 11 users is a small test group the extract some representative results from a questionnaire. Maybe we should have thought of some additional ways to reach more respondents.