1 Introduction *

2 Problems *

3 Summary *

4 Recommendations *

5 References *

1 Evaluator Report *

2 Evaluator Report *

3 Evaluator Report - PC *

4 Evaluator Report *

1. Introduction

This report contains the results of an evaluation of mmcr. The report is based on two separate evaluations:

1. A heuristic evaluation [1] of mmcr carried out in July 1997 as part of the work in WP4 of the Merci project. The heuristic evaluation was performed by four HCI trained people. Their individual reports are attached at the end.
2. Reports from three users who have used mmcr for recording Mbone sessions as part of their research since November 1996.

The report contains three sections:

1. The first section, which is also the main section, is a list of usability problems, software bugs and comments, interspersed with recommendations for how the problems could be rectified. The problems are listed according to the following headings:

Window management
Help system
Consistency
Other
Main Window
Windows Relating to Play functionality
Conference Browser window
Conference details window
Select the media and their sources window
New media details window
Play window
Windows Relating to Record Functionality
Record window
Documentation
Tasks
General Comments

2. This is followed by a short summary drawing out the main problems found in the evaluation.
3. In the final section we outline our recommendations for how the tool could be improved - both a short term solution and a long-term solution.

The aim of this report is to assess the usability of mmcr and to provide pointers to changes that would improve the usability of the tool.

1. Problems
1. Window management

New windows and dialogue boxes always appear in the top left-hand corner. Each new box obscures its predecessor. This predictability is an asset but having to move windows, e.g. help windows so it is possible to see the help window and the mmcr window that the help window refers to, can be an irritation.

If the user chooses to move windows to somewhere else on the screen, new message windows can easily be overlooked as they appear away from the main application windows.

Behaviour of message windows containing help is not consistent. Some help message windows must be dismissed before the user can carry on with her task, others can be left open.

• Ensure that windows appear where they are needed.
• Message boxes, like ‘Really quit VCR’ which must be responded to in order to do anything else, should appear on top of the window that caused the message window to appear.
• Help windows should appear near the windows where they are needed.
• If previous windows are no longer needed, they should be closed.
• Help windows should be allowed to stay open so the user can refer to them while continuing with her task.
• Provide a Dismiss or Cancel button in every window.

1. Help system

It is evident that an effort has been made to prevent errors, to provide helpful error messages, to give feedback to users. Much of the labelling is also in clear English and makes the function of a window or button clear. However, further work, including a careful analysis of the language used, is needed to make this consistent.

For example: Too much information is given in too short a space so that it sounds incoherent. And also the user must hold a series of instructions in memory - or refer back repeatedly to the HELP menu in the main window. To give all instructions in one go also involves referring to dialogue boxes and buttons that the user has not yet seen at the time help is sought.

The help text is sometimes not precise when referring to buttons. For instance, in the HELP menu’s Recording option, there is a reference to the recorder’s ‘record’ button. In fact the button is labelled Start Recording. In the main window, still visible at this stage, there is a Record button. This could result in confusion.

Help is accessed via a menu. On the main window this offers help on Startup, Recording, Playback. Help is also available on some of the pop-up windows that appear during the record and playback process. This is an approach which ought to help users because it provides information at the moment when they are actually using a given part of the interface. To work really well, however, it needs to be consistent: all the windows should have this feature. This is, in effect, like providing balloon help, and some users might find such a two-level help system more useful than what is provided at the moment - with balloon help as the first line and more detail available via menus.

A review of what information is needed, and when, would also be a good idea. Is it necessary, for example, to refer to the ‘connections list’ under Startup, when at this stage the list would probably be empty?

There is also a tendency to assume knowledge that the user may not have: for example, the maximum number of recorders and players is specified, but without explaining the basic idea that mmcr consists of these objects.

• Make the help system consistent by providing access to help in each window.
• Make sure that references to button names and menu names are accurate.
• Explain the basic knowledge needed in each help window. Users should not be expected to know anything about the multicasting. Users who join sessions from sdr, for example, can do so without knowing about multicast addresses.

1. Consistency

‘Standardisation’ is a key HCI principle to increase interface comprehension - there are some problems here:

1. The use of terms within the application are not standardised. For example, exit buttons are called: Dismiss, Cancel, Quit, Cancel playback, Quit recording. Any of these terms are good, but they should not all appear in the same application.
2. The labelling of buttons and windows is not consistent, especially regarding the use of capital and small letters.

More specifically, functionality which appears both in the play and the record part of mmcr should look alike in both parts. The Record window has one button Start Recording which, when pressed, changes names to Pause Recording and if that is pressed, it changes name again to Resume Recording. The Play window has got three different buttons, PLAY, PAUSE and REWIND. Why not have separate Play and Pause buttons in the Record window as well? Or have two buttons in the Play window, one that changes names from Play, Pause and Resume Play and one called Rewind?

The concept of having Player(s) and Recorder(s) seems like a good one and easy to grasp. It is not, however, reflected in the labels on the windows - which read Record and Play. This is a possible obstacle to user understanding of the tool. And whilst Record brings up a Record window, Play brings up a Conference Browser. This is not simple to solve, but making the names reflect the design might help.

• Make the use of labels that cover the same thing consistent. Use Dismiss and Cancel throughout the application. Only use the labels Play and Record for when you actually start playing and recording. Re-label the buttons Record and Play in the main window to Start up Recorder and Start up Player or the like.
• Provide a stop button in the Play and Record windows.
• Have a Dismiss or Quit button in clear view, not embedded in the Options menu, and have them appear in the same place in similar windows.
• Use capital and small letters consistently.

1. Other

It is assumed that words like ‘media’ and ‘source’ will be interpreted correctly by the user. This may not be so for someone new to this field. In addition, Add media option in the Recorder and the Proceed button in the Player both bring up a dialogue box labelled New media details. These boxes look very similar. This could lead to confusion since many users may not see the processes (selecting and giving addresses for the media to be recorded and giving the destination addresses for playing back each medium) as being the same.

• Provide some explanatory text so it is clear what addresses need to be entered and where the user can find these addresses.

1. Main Window

The two main functions are presented nice and clearly. Good presentation of an exit button in clear view

On the face of it, the first window is well designed. The three buttons and the menu together represent the four tasks that the user is likely to want to make use of: Record, Play and Quit and Help. However, it turns out to be more complicated than that. It is possible to record and/or play up to six different sessions at any one time. Hence the Active Connections box which will list the various sessions being recorded or played. Pressing Play or Record will start up individual little record or play windows. This means that if the user chooses to play, she first has to press Play in the main window and then, once she has chosen which conference to play and which addresses to play it on, she gets another window which contains a play button. It is not at all obvious which of the play buttons to press to start the playback. The problem is that it is not clear that the main window is only there to allow the user to start up recording tools or a playback tools which then take on lives of their own.

• The basic idea of the tool needs to be explained, either by changing the labels on the buttons to Start up Recorder and Start up Player as suggested earlier, or by making it more obvious that this particular VCR can create multiple instances of players and recorders by labelling the subsequent windows, mmcr1, mmcr2 etc.

1. Windows Relating to Play Functionality

The sequence of actions is longer for play than for record - unlike a home VCR. This is because as well as selecting what to play from the list of conferences, addresses and ttl have to be specified for playout. The tool’s flexibility adds complexity too; being able to select only certain streams to play back adds another stage and means that you cannot simply choose your conference and then play it. For some users this flexibility may not be needed. The number of choices makes the process more complicated than it needs to be and makes navigation harder.

0. Conference Browser Window

This window contains a scrollable list of conference id/Title pairs. The aim of this is presumably to help the user select the correct conference. If this is so, then all the information displayed ought to help the user to identify this. Conference id is a label assigned by the recorder and is in a form that is not easy for a person to memorise or to distinguish (conf863287079 and conf 863434927 are examples). There is no apparent order of the conferences in the list. It would be useful to have them listed in some recognisable order.

• Order conferences according to time and date or according to title, and display title and start of description and date.

Search is case sensitive. Maybe it ought not to be.

• Change the search algorithm so that it is not case sensitive.

Result of the search is another conference browser, containing a modified list. One button is different; Search has become Undo search (quite a good way to indicate status - but an odd concept when you first come across it - possibly Show full list might be clearer).

• Change the name of Undo search to Show full list.

From here you can select conference details - as with the first conference browser. Doing so sometimes gives you an error message, saying that you need to select a conference from the list. This happens (correctly) when you have not previously selected one and at times when you have. Undo search takes you back to the full list (in a smaller window!).

Whatever you do, eventually you have to select the Conference details button. Why does the Play conference button (greyed out) never become available?

• Automatically display conference details when a conference has been selected, or give the user the option of skipping that particular step. Viewing conference details is not always necessary and the user should not be forced to do so.

1. Conference details window

The Conference details window has three buttons: Play, View Sources, and Dismiss. Pressing Play and View Sources produces nearly the same window, a window listing the sources and the index numbers of the media streams. The only difference is that View Sources only allows the user to view the sources and she must return to the Conference details window, whereas the Play will allow the user to select media streams for playback. The View Sources option seems unnecessary, in view of the fact that the user is forced to view sources anyway when she selects Play.

When Play is selected the player disappears (disconcerting - the user is left wondering if something went wrong - but probably a good idea as there were four windows visible.)

• Do away with the View Sources button and functionality.
• Do not use the label Play here as it does not actually cause any data to be played yet.
• This window, again, does not actually do anything. Maybe it should be combined with the next window, the Select the media and their sources window, where the user has to make some decisions.

1. Select the media and their sources window

The Select the media and their sources window enables the user to select the media streams that she wants to play back. The information is clearly laid out in columns and where possible the ‘sources’ are given as person-names, which is helpful. Select all seems to highlight all the sources. If she wants to select them herself, she can click on the name, to select all media from that source, or she can pick her way through the boxes to choose the streams she wants. It is assumed that the user knows it is possible to replay only some of the streams.

To be sure of seeing all the media streams available it is necessary to scroll. This is a possible source of errors, particularly where the various sources are using different numbers of media (for example, where one or two are not sending video).

It would be nice for the user to be able to reverse step through this process to allow for changes of mind. This is not always possible. When she selects Cancel playback she is returned to the start of this process; that is, only the main mmcr window is visible and she has to begin again by pressing the Play button - she has to create a new player. It is, however, possible to deselect a media stream, by clicking again on the item. This is a useful feature, though it is one that did not always seem to work at first.

Again, the windows does not tell the user what to do. The window has three buttons: Select all, Proceed, and Cancel Playback. Clicking Proceed without choosing any data streams causes a window to pop up explaining that data streams have to be chosen in order to proceed.

• The user should be asked to choose which of the media she wants to play. Help on this subject should be available.
• The Proceed button should be greyed out until it can be pressed.
• The index numbers are meaningless to the user and should not be displayed.
• The window should contain a Dismiss button rather than a Cancel Playback to allow the user to reverse the process step by step.
• Ensure that all sources and all data streams are displayed in the window without having to scroll (within limits, of course).

1. New media details window

Once the data streams have been selected and Proceed has been chosen in the Select the media and their sources window, New media details windows pop up, one for each of the media that are to be played back. The address, port and ttl for each medium must be entered. It is not reasonable to expect the user to know what these mean. And to some it might seem that the same information is being requested yet again. There is no information to tell the user what the addresses, ports and ttls are and where she can find more information. The language used in these windows presupposes intimate knowledge of multicasting. If an invalid multicast address, for example 22.2.2 is entered, the error message tells the user that ‘‘22.2.2’ is not a multicast address’. It does not say what a correct multicast address is.

• Either the user must be guided by access to extensive help or by doing away with the window all together. Setting mmcr up to play data stream to particular addresses could be an option - default could be for mmcr to start up the relevant tools up on random addresses so the user does not have to worry about addresses and ttls (look in sdr, using normal interface, for inspiration).
• Position the cursor in the address box, ready for the user to type in an address, rather than in the greyed out box containing the name of the media.

1. Play window

The Play window has three buttons: PLAY, PAUSE and REWIND. Furthermore it has three menus, one for adding and muting media, another for hiding the Play window and quitting and the last is a help menu. The buttons are greyed out when it does not make sense to press them, i.e. unless PLAY has been pressed, PAUSE and REWIND cannot be chosen. Why are the labels written with capital letters? And why is there no STOP button? This is the third time that the user encounters a play button. How is she to know that this is the final one? Play is not a good label for the first two instances, in the main window and in the Conference details window.

Quit should have been a separate Cancel or Dismiss button somewhere in the window rather than option in a menu. It is not clear that choosing Quit will only quit this particular instance of a player and not the entire application.

The screen gets crowded during the process. At one point there are four windows on the screen.

• Remove windows when not needed.
• Provide a Stop button.
• Provide a Quit, Dismiss or Cancel button in clear view.

1. Windows Relating to Record Functionality
0. Record Window

The Record window starts up indicating that the user must fill out two fields called, Title and Description. Once these have been filled out, a New media details window (very like the one in the player) pops up and the user must enter Media name, Address, Port and select Data Type. When the media has been entered (if the user gets that far) the details are displayed in the Record window, and there is no indication of further steps to be taken in recording process. Often a user will want to record both audio and video, but the user will only be prompted to enter one media for recording. There should be an obvious selection procedure on the screen.

• The application should prompt the user about the number of media she wants to record.

A New media details window must be opened for each media that the user wants to record. However, after the first window which pops up automatically after having filled in the Description field, the user must open other New media details window by choosing the Add media option in the Media menu. Embedded menus are not a viable option if the drop down menu has only one option, as in this case.

• If having a Media menu is part of a long term plan where more options will go in the menu, add the menu items now, but greyed out. If not, Add media should be a button or appear in the Options menu.

Hitting the Return key after filling in the Description field automatically calls up New media details window. The same result is obtained by selecting Add media, from the Media menu. If Return were a shortcut to the next stage throughout the application, then this would be a good feature. As it happens, this is not consistently so. To get a second media box you have to use the Add media menu item.

• Ensure that the Return key always performs the next default choice of the user.
• The application should guide the user through the entire process and not leave her to guess what the next step should be. When the user has filled out the first New media details window, the user should be asked if more media are to be recorded.

It is not possible to back track. If a wrong address has been entered, the recorder has to be abandoned a new one set up from the beginning.

• Provide means of back tracking step by step.

The Hide window option in the Options menu of the Record window does not provide clear feedback on how the window can be un-hidden. Is this function really needed as windows can be minimised under the window system?

• Do away with the Hide Window function or provide the user with a way of retrieving the window.

As in the player, there is no Stop button.

• Provide a Stop button.

In the New media details window, the user is allowed to record whiteboard as rtp, though it cannot be played back.

• Warn users of unsuitable combinations of media and format.

The user is not prompted to select the Set duration option - quite rightly, since it is an option. But it is a useful one and users may want to know it exists. Perhaps a reference in the Help->Recording text would be a good idea.

• Mention the Set duration option in the help system.

If it turns out that conferences on certain addresses will be recorded regularly many users would find it useful to be able to keep their own list of such conferences and addresses, and be able to select them by name, and skip the media/address input stage of recording. At present there is a considerable memory load imposed on the user - remembering address/port combinations for all the media.

• Provide the user with the option of having a private address book where aliases can be stored for frequently used address/port combinations - like most email systems.

There is no easy way of assuring that the tool is actually recording, other than checking all the addresses. It might be nice with some kind of visual indication (like the red light on a VCR) when data is actually being recorded.

1. Documentation

There is no documentation available for how to use the mmcr apart from the little help windows that can be opened from within the tool.

There are a couple of web pages - one is an installation guide and that other is a FAQ page, but none of them resembles a user guide.

• Provide a simple guide for users who are not familiar with multicasting.

1. Tasks

Basic functions that are not supported are: fast forward and rewind. At the moment a conference can only be viewed from the beginning. Once started, it can be paused or rewound right to the beginning. This is extremely unsatisfying. A timer should also be available so it is possible to make references to specific points in the conferences. Being able to ‘bookmark’ certain places would be desirable as well.

• Providing fast forward or go to and rewind (not all the way to the beginning) functionality should be high priority. The tool is tedious to use without it.

Unless the user has a PC where the config files can be changed, mmcr cannot be used for viewing conferences which have been recorded on boom.

• Allow the user to choose which server to use.

The user cannot delete recordings that she has made. And once a recording has been started, it cannot be stopped without saving the recordings. This means that if a user realises that she has typed in a wrong address or port, and has to restart the recording, she has to choose a new name for the recording.

• The user should be allowed to delete her own recordings.
• The user should be able to abandon a recording without saving it.

1. General comments

Only machines that can run software for Suns are supported. This seems particularly strange as the application is written in java which is renowned for platform independence.

Some of the problems reported by users who have used mmcr to record conferences as part of their research, are not interface problems. Examples of other problems are:

• The tool was not always available when it was needed.
• The tool was not tested with the older version of rat and vic which are used in the ReLaTe interface.
• The tool seemed not have been fully tested to deal encryption.

This has led to faulty recording which in the case of the ReLaTe project has substantially diminished the number of recordings available for analysis. Out of around 30 recordings attempted, only about six are complete. This is due to a number of problems, of which the ones mentioned above, played a part. But, lack disk space and other factors also played a substantial role in the lack of successful recordings.

1. Software Bugs

• One of the evaluators gave up trying to get mmcr to run on an sgi, but has managed to run it on a PC. This has brought about some interesting bugs (some of the bugs may also exist in the Sun version): Menus do not appear on the windows, windows do not disappear when no longer in use, windows do not change size so information gets lost etc. Other bugs include: if an address outside the range of valid multicast addresses is typed by mistake when wanting to play back a recording, the application crashes and has to be restarted. When typing in addresses to be recorded, an error box pops up after the address for each media - even though no errors have been made. If you press conference details without having selected a conference, a box pops up telling you that you have not selected a conference. You press ok, select a conference, but the box keeps popping up. The only way of getting round it is to double click on the conference.
• Windows do not appear on top when the title bar is clicked. To see underlying windows, the ones on top need to be moved out of the way (on PCs).
• The interface often freezes and the tool needs to be restarted - it nearly always crashes when doing anything beyond an ideal session, i.e. typing mistakes, changing your mind about addresses etc. can cause it to crash.
• Windows often appear partly outside the screen (on Ultras) and are often not updated properly which leaves an impression of modern art rather than a user interface (on PCs).
• Highlighted buttons do not respond to <Enter>.
• When entering media name, address and port, no matter whether correctly input or not, a Message window pops up to tell the user that the input was incorrect. Eventually, if the user is persistent enough and keeps pressing OK the input is accepted (on PCs).
• Furthermore, even though the cursor is positioned in the first box, ready to input the media’s name, the user has to click in the box before being able to write in it (on PCs).
• The Play conference button in the Conference Browser window, remains greyed out at all times.
• One of the evaluators was not able to create multiple Player instances.
• Attempting to get out of the window by pressing Cancel media causes the tool to crash (on PCs).
• The Select the media and their sources window sometimes appears without the Select all button (on PCs).

1. Summary

The evaluation revealed that the tool has some serious usability problems. In fact, one evaluator, despite using other Mbone tools on a regular basis, never managed to play or record anything. The problems can be grouped together into problems relating to:

• Affordance: The interface does not guide the user through the processes of recording and playing. The user is required to guess what the next step is towards fulfilling her goal of either recording or playing a conference, and must rely on message boxes telling her that the step she chose was not the right one. The interface should guide the user through the steps necessary to complete her task.
• Help: The help provided is both inconsistent and inadequate. The help assumes that the user already knows the underlying philosophy of the tool, and goes straight to explain the more intricate parts of it. Some help messages have to be dismissed before they can be acted on, making it difficult to remember the contents of them. And finally, the provision of help in the different windows is not consistent, i.e. the user cannot rely on its being available.
• Consistency: Buttons doing the same thing (exit/cancel/quit/ and play, pause etc) should be called the same and appear in the same place in all the windows where the function is available.
• Bugs: The tool is still unstable. Minor user errors can cause the tool to crash.
• Features: The tool lacks basic features which makes it impossible for some and difficult for others to use as it is at the moment.
• Users who are not multicast experts and do not know about multicast addresses and ports and ttls, cannot use the tool. When mmcr is made to interface with sdr, recording a session will be easy. However, the user should still be able to play back a conference without knowing about multicast addresses by having mmcr starting up the tools automatically.
• The process of recording and playing conferences could be cut down substantially by providing an address book, so the user can ask to record from or play back to the ‘Merci’ addresses for example, rather than having to type them in every time. This would also eliminate the risk of typing in a wrong address or port by mistake.

1. Recommendations

1. The most important thing is to make the tool more robust. It crashes far too often which makes using it an aggravating experience.
2. In the short term, it would not take much work on the user interface to achieve a huge improvement. Most of the action points listed in section 1.1 involve relatively little work and would improve the tool considerably.
3. In the long term, we would recommend a complete redesign of the interface using conceptual design along the lines of sdr [2]. In particular, we would like to see an mmcr with two user interfaces, one for novice users where the user can select a conference, press play and the tools would start up and start playing; and one for expert users allowing the user to choose which addresses and tools to use, in the same way as sdr has a normal and a technical interface. This would, however, require an HCI expert to prepare and evaluate the design for implementation.