Develop a Set of Criteria to Evaluate a Website of a Holiday Booking Site
The exponential rise in internet usage has made websites more pervasive [Wilson et al, 1997]. As more and more people around the world are using websites on a regular basis for various reasons, the need for ongoing evaluation of current Websites and those under development has become increasingly evident.
According to Rubin (1994) there are four main factors for which evaluation of websites should be undertaken. The websites should not only be useful but also effective and learnable, and one that generates high user satisfaction [Rubin, 1994]. In simple terms evaluation of the websites will show whether the goal/s of creating the website are achieved or not. The idea is to uncover problems or stumbling blocks that may interfere with high utilization of the website. The appropriateness of the website design as per the target user is also determined through the evaluation.
Website Evaluation Techniques
Different types of evaluation methods are valid and useful at different points during the development of a product such as a website. McGrath (2000) explains four strategies to evaluate a website:
Field Strategies: In field strategy, the website is observed in lieu of a field experiment or field study such as beta testing. It gives the usability engineer a much better idea about the context in which the system will be operating.
Experimental Strategies: Experimental method is a formal method and it can be expensive because it requires controlled environment and skilled personnel to administer them. Therefore it is important to ensure that the purpose of the evaluation is clearly understood and the experiment has been properly designed.
Website usability testing falls under the category of experimental strategy because it involves observing a user in a controlled setting such as a laboratory. Usability testing allows testers to collect both qualitative and quantitative data as the end user performs a real task or set of tasks. Talk Aloud/Think Aloud Method In this evaluation method users are asked to think aloud while they carry out their activities with the system [Lewis, 1982]. This way valuable insight can be gathered about how people are operating with the system and their strategies for carrying out the task.
Cooperative Evaluation Method Cooperative method is a method of evaluation which is used to identify and rectify problems with the prototype. (Monk et al, 1983) It encourages users to explain their behavior while performing a particular task. This method can be considered as the next development of the talk aloud method and has a major advantage of creating less artificial situations. Another advantage of this method is as a result of its use in the early stages of the design process which enables the creation of better prototypes after every evaluation.
The question asking protocol It actively seeks feedback from the users by asking direct questions about the task being performed on the website. These processes can be performed during any phase of website usability testing.
Respondent Strategies: In the respondent strategy, user’s opinions are documented via a survey, a questionnaire or interview. Empirical Evaluation Method falls under the category of respondent’s strategy. This method consists of an analysis of user performance in relation to the proposed system (Dumas and Redish, 1999). Empirical methods involve working with the users and gathering data that will have to be analyzed. Empirical methods include the use of questionnaires, Interview, focus groups, Usage logging, user feedback and observation. Some of the main advantages can be listed as: its user experience probing nature, ease of repetition, ease of finding the most important parts of the system and tracking changes in user requirements [Nielsen, 1993]. Empirical methods can be categorized into two formative and summative methods.
Formative Evaluation Method
The purpose of formative evaluation is to help design the process [Hewitt, 1986]. It is used to refine and formulate the design. It involves working closely with the users gathering feedback using questionnaires, focus groups and interviews about their opinions of the system. Some of the advantages of this method are that it elicit opinions about how the users feel about the system, what problems they are experiencing, any changes they feel might be needed, what are the strengths of the system and so on. This form of evaluation starts at an early stage since users need to be involved in the design process as soon as possible.
Summative Evaluation Method Summative evaluation requires quantitative data. Hewitt (1986) suggest the purpose of summative evaluation as being about assessing the impact, usability, and effectiveness of the system the overall performance of the system. This type of evaluation is more useful when the design is complete and is done using questionnaires, interviews, user feedback etc. Advantages of summative evaluation include detailed feedback about the system on the whole and an overall estimate of the representative population.
Theoretical and Expert Review Strategies
A website evaluated using a theoretical strategy involves a computer simulation such as a cognitive walkthroughs. Heuristic Evaluation This method is used to evaluate the system from the perspective of an expert. The expert or interface design specialist judges the various properties which may lead to problems in future based on a set of principals or heuristics. The evaluator should ideally be an expert but this evaluation can also be done by non-experts (Nielsen, 1993). [Nielsen, J] suggests the use of ten heuristics or general principles/guidelines to develop a good user Interface design. These include:
1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation
The interface is checked for conformance with the organisational or other guidelines document. Generally guidelines documents contain thousands of items; therefore, it may take the expert reviewers some time to master the guidelines.
Consistency Inspection The experts verify consistency across a family of interfaces, checking for consistency of terminology, colour, layout, input and output formats, training materials and online help.
Cognitive walkthrough The experts simulate users walking through the interface to carry out typical tasks. Cognitive walkthroughs were developed for interfaces that can be learned by exploratory browsing (Wharton et al., 1994) but they are useful even for interfaces that require substantial training.
Formal Usability Inspection: The experts hold courtroom-style meeting with a moderator or judge, to present the interface and to discuss its merits and demerits.
Problems with the evaluation Techniques
The techniques mentioned above are used for website evaluation at different levels and evaluated different aspects of the website. Each of the evaluation techniques mentioned above has it own drawbacks. Field studies cannot provide the true picture of how the website is actually used by the end user as the user would have done in reality. This happens because the users get influenced by the fact that they are being observed by the evaluators.
Dawkins (1998) explains that a major drawback of the experimental method is to ensure that all extraneous factors are ruled out or prevented from influencing the results. The experiment must prove that the independent variable is responsible for the observed behavior. Secondly, the order in which things are presented to the subjects can affect the performance of the website to a great extent. For e.g., if Memorability was being tested then it could be possible that one task seemed to be easier to remember than another simply because it was tackled second. With respect to Talk/Think Aloud method Boren and Ramey (2000) feel that evaluators need to be aware that talk aloud in not always easy and need to develop methods to get the users to speak and not feel embarrassed.
One of the main drawbacks of the respondent strategy is that it can be expensive both in terms of money and time. Shneiderman (1998) adds that observing without participating takes a lot of time and we do not have much time in this competitive world. Some of the other limitations of the methods used in this strategy with reference to formative and summative evaluations are the need of an unbiased group of users to collect data (which is difficult to find) and statistical expertise to analyze the collected data.
Some of the main drawbacks of the expert review strategies are that the experts may not have an adequate understanding of the user or task domain. If a group of experts come from different backgrounds then they might give conflicting suggestions based on their evaluations creating further confusion. Shneiderman (1998) adds that even expert evaluators find it difficult to know how a typical or first time user will behave while interacting with the system. Shneiderman further suggests that the experts should be from the target user group and is generally hard to find. Also a specific limitation of heuristic evaluation is that it does not provide a systematic solution to the problems that were discovered during the evaluation
Final analysis
From the above discussion which elaborates some of the important evaluation methods based on certain criteria, one might question the necessity of creating additional set of criteria for evaluating holiday booking websites when so many are already existing. This point is well addressed by Kim and Lee (2002) who mentions that the effectiveness of specific evaluation criteria may vary significantly between different types of websites. This point of view is also supported by Perry and Bodkin (2000), Kwon et al (2002) who think that the set of criteria for evaluating a website will vary largely depending on several factors like target user domain, industry specific requirements etc.
From the findings of the literature review that has been summarized earlier and the above discussion it may be said that a combination of respondent evaluation strategy (Summative technique) along with Expert review (Specifically Heuristic method) will be best suited for this piece of research. One of the primary reasons for choosing these two methods is that the websites under consideration have already passed its design phase and have been implemented. The experimental and field strategies are primarily used during design phase for further enhancing the evolving prototypes.
Tags: analysis, empirical methods, evaluation, holiday booking, respondent strategy, usability, website
