Young Professionals of Akron Usability Study

Young Professionals of Akron Usability Study

people

Project Team

face

Responsibilities
  • Usability Study

build

Tools Used
  • Tobii (Eyetracking)

Goals of Testing

The primary goals of conducting usability testing on the Young Professionals of Akron website were to:

Findings

As a result of the tests conducted and the data gathered, these general findings were identified and are discussed further within this report:

Methodology

Standard usability methodologies were followed as much as possible. The following sections indicate specific methodologies and indicate any variations from standard procedures.

Participants

For the purposes of this study, a characterization of an expected user was documented and a screening questionnaire was developed to identify potential participants who met the criteria identified in the characterization. The characterization was as follows.

Goals and Interests

Although the screening questionnaire was thoughtfully developed, the time and location constraints involved with this study caused us to use readily-available participants who may or may not have might the characterization criteria identified.

The participants used in the test were made up of a convenient sample of people within the Kent State IAKM program’s Usability I class. The four participants used to test the YPA site were randomly selected from that population. For background on each participant, a questionnaire was used to determine their familiarly with computers and Internet-related tasks.

This table indicates responses to the pre-test questionnaire. For the first four questions, participants were asked to indicate their level on a scale of 1 to 5 (1 = low use, 5 = regular use) the results were as follows:

  Participant 1 Participant 2 Participant 3 Participant 4
Level of use with computers and related hardware 5 4 5 5
Level of use with Internet browsing software 5 5 5 5
Level of use of email 5 5 5 5
Level of use of online forms 4 3 5 4
Previously participated in networking groups? Yes Yes Yes Yes
Receive regular email updates or e-newsletters? Yes Yes Yes Yes
Special assistance required? No No No No

These results indicate that each participant was familiar with computers, hardware, Internet browsing, email and online forms. In addition to standard computer use skills, each of the available participants was familiar with usability studies methodologies and the Tobii software used to conduct the test. This was because of their participation in the Usability I course that they participated in.

Test Sessions

For each test session, participants were greeted and invited to wait in a ‘green room’ that was far enough away from the testing rooms to ensure they were not exposed to tests in progress. This was done to protect them from gathering prior knowledge that might influence test results.

While participants waited in the green room, the pre-test questionnaire was conducted and assessed by a tester to determine if special accommodations for testing were required (for example, enlarged text). None of the participants required special accommodations except for one participant who was left-handed and required the mouse be placed of the left side of the computer instead of the pre-arranged right side. No other changes were made during his test session.

During each test, a pre-developed script was used by the moderator so that test sessions were conducted as similarly as possible. The script included an explanation of how the test would be conducted, a brief explanation of the purpose of the Young Professionals of Akron website, and encouragement for participants to ‘think-out-loud’ as they completed the tasks that comprised the usability test.

In addition, the script contained the statements that indicated each task to be completed and questions for the moderator to ask after the completion of each task. When the moderator indicated the beginning of a new task and explained the requested task action, a slip of paper with the task listed in large print (size 16 font) was provided to the participant for use as a reminder when completing the task.

While moderators typically followed the script, there were some deviations when participants’ comments or actions warranted additional questions and data gathering on the part of the moderator. In such cases, the moderator would ask unplanned questions and then return to the script when the questioning and related conversation was complete.

After each participant completed the three tasks and testing-related conversations were complete, the participant was thanked for his/her time and escorted out of the testing area.

Data Collection

To begin the testing process, each participant was asked to fill out a questionnaire to capture data in regards to their familiarity with computers and the Internet related tasks. The results were cataloged in a simple spread sheet shown in the Participants section.

To record each participant’s experience while completing the usability test of the YPA website, each member of the group took notes during the test. Also noted were significant comments made by the participant and any interesting occurrences observed by the team members.

Through the use of Tobii Studio software, use data was collected from four participants. Each session was recorded using Tobii software and records of participants’ eye movements and visual heat maps were created. These data results were analyzed and compared. Each session also collected data in the form of screen captures, audio, face shots, keystrokes, and mouse clicks.

In addition, participants were asked to speak aloud their thought processes in navigating through the site. They were also asked questions while they performed each task. Their comments and answers were recorded. Upon the completion of the test, participants were also asked follow-up questions about their overall experience.

Findings

All of the data was collected and assessed for usefulness. As a result, not all of the data was used to create this report of findings, but is available for use if needed at a later date.

Success and Time

In general, the participants were able to successfully complete the three tasks. This table indicates successful completion of tasks for each participant.

check = Successful close = Unsuccessful
  Task 1: Locate descriptions and pictures of past events Task 2: Locate the newsletter sign-up form Task 3: Email YPA a question using the site
Participant 1 check check check
Participant 2 check check check
Participant 3 close check check
Participant 4 check close check
Percentage of Participants Successful 75% 75% 100%

Although participants were generally successful in completing each of the tasks, the difficulty with task 1 is discernable when the amount of time spent on the tasks is viewed. This table indicates the amount of time spent on each task by each participant.

  Task 1: Locate descriptions and pictures of past events Task 2: Locate the newsletter sign-up form Task 3: Email YPA a question using the site
Participant 1 0:40 0:24 0:09
Participant 2 2:58 0:15 0:16
Participant 3 4:59 1:10 0:04
Participant 4 3:10 1:32 0:02
Mean Time to Completion 2:57 0:50 0:08
Median Time to Completion 3:04 0:47 0:07
Range 0:40 – 4:59 0:15 - 1:32 0:02 - 0:16
Standard Deviation 1:47 0:21 0:06

As you can see from the large amounts of time that participants required to complete task 1 (locate descriptions and pictures of past events), participants encountered some difficulty with this task. Task 2 (locate the newsletter and event notification sign-up form) also caused some difficulty for users as evidenced by the time-on-task measurements. Task 3 was easily and successfully completed by all four participants.

Time-on-task measurements are not fully accurate, however, because the participants were encouraged to talk and ‘think-out-loud’ while they completed the tasks. Their sharing of comments may have inflated the amount of time required to complete each task.

Evaluations

Task 1

While there were a small number of participants in our study, some common trends were evident. The first task, to locate a description and pictures of a past YPA event, was the hardest task for participants to complete. It took the most time to complete, with an average completion of just less than three minutes.

The biggest barrier to successful completion of this task can be easily identified as a labeling issue in the global navigation bar. Most participants did not easily identify that descriptions and photos of past events would be located in the section labeled The Library. All of the participants first looked in other sections of the site before trying The Library section, which was usually attempted in desperation.

Here are the paths that each participant followed to find the descriptions and photos of past events.

Participant 1 Participant 2 Participant 3 Participant 4
Home
arrow_downward
Clicked the event listed below News and Events
arrow_downward
See All 2010 Events
arrow_downward
The Library
Home
arrow_downward
Save the Date
arrow_downward
See All 2010 Events
arrow_downward
Date on Calendar
arrow_downward
Back
arrow_downward
Community Happenings
arrow_downward
The Library
Home
arrow_downward
Attempted to click News and Events (not a link)
arrow_downward
We Are YPA
arrow_downward
Welcome
arrow_downward
Community Happenings
arrow_downward
Free Tour (sublink)
arrow_downward
Welcome
arrow_downward
Save the Date
arrow_downward
Welcome
arrow_downward
Aborted task
Home
arrow_downward
Attempted to click News and Events (not a link)
arrow_downward
Save the Date
arrow_downward
Calendar scrolled
arrow_downward
Links to other sites
arrow_downward
Welcome
arrow_downward
News Articles
arrow_downward
Community Happenings
arrow_downward
The Library
Time: 40 s Time: 2 m 58 s Time: 4 m 49 s Time: 3 m 10 s

Here are some of the comments that participants made while completing task 1:

Since the YPA is a group that presents itself as an organization focused on participating in networking events, more effort should be made to expose the available events and past events to the intended audience. The only place on the home page that explicitly mentions events is in the side bar on the right side of the page, a place that by convention is usually relegated to advertising and other off-topic content.

The subsequent tasks were performed much more quickly. This may mean that once the participants have gotten familiar with the way the site is presented they can navigate more efficiently. For example, with task 2, locate the sign-up for the e-newsletter, participants were able to complete the task in about a minute.

Task 2

There were some navigational issues with task 2, though not as severe as with task 1. Two of the four participants attempted to locate the sign-up for the e-newsletter in the Contact Us section of the website. After scanning the Contact Us page and not finding the e-newsletter sign-up, these two participants then tried the Get Connected links.

Task 2 also helped identify another issue with the overall site – the Flash-based graphic located above the global navigation bar. None of the participants immediately recognized the link in the graphic.

In fact, most of the participants disregarded the Flash-based graphic upon first view of the site. A heat map (below) that shows where the participants focused their attentions supports the finding that the Flash graphic was nearly ignored. As you can see, the graphic was not the subject of much attention (the red hotspots indicate the most focused attention).

Figure 1: Heatmap showing inattention to Flash graphics Comments made by participants specific to the Flash-based graphic include:

Task 3

Task 3, email YPA to ask a question, was not problematic for any of the participants and most completed the task in about 10 seconds. The only comment made related to the task of emailing a question was specific to email address, where the participant indicated that she questioned the likelihood of receiving a response because it was a Yahoo address that “probably no one is even checking.”

Other Findings (Not Specific to Tasks)

Through observation, note-taking, and reviewing the recordings of the test sessions, we found other issues not directly related to the completion of tasks. These general findings are listed below.

Recommendations

There seemed to be three main areas of trouble for the participants in our test:

  1. Unfamiliar labels in the global navigation - The navigation labels on the YPA website did more to confuse participants than to help them find the content they were looking for. As noted, the labels The Library, “Save the Date” and “Community Happenings” were especially confusing to the participants. Perhaps these labels could be replaced with something resembling “Past Events”, “Future Events” and “Other Events in the Area,” respectively. Changing these labels would help users be able to understand exactly what to expect in each of those sections of the website before visiting those pages.
  2. Web conventions not followed - There is quite a large amount of non-black colored text on the YPA website. Conventionally, blue text represents a link when presented in content. On the YPA website, however, blue text almost never denotes a link; instead, links are most often represented by orange colored text. This web faux pas had participants clicking text expecting to be directed to their desired content but instead left them disappointed and searching for alternate methods of reaching their target. To remedy this situation, the use of blue text should be reevaluated across all pages on the website. Text links should be changed from orange to blue and existing non-link blue text should be changed to black or some other non-blue color.
  3. Elements intended to be prominent were instead ignored - The Flash animation on the homepage and above the navigation on other pages of the website was ignored by every participant in our tests until they were probed by the test facilitator. Users noted that they automatically associate moving images that they are not expecting to see with advertisements.

Suggestions for Further Investigation

There were some aspects of the YPA website that our group did not examine during our usability tests. Of these, there were two opportunities we noted for further investigation: how users find an event to attend and how users expect to be able to RSVP for an event.

Currently, scheduled events are located on their own page of the YPA website. Users must navigate a calendar interface to find an event to attend. Since the YPA averages only one event per month, it’d be interesting to investigate if there could be a better way to present the upcoming events so users could more easily find all planned events without navigating month to month through a full calendar.

A comment from one of our test participants stating that there was distrust in using the YPA’s Yahoo email address highlights the need to test the RSVP process on the website. Today, all RSVPs to events are handled by sending an email to the YPA. It would be interesting to see if there would be increased incentive to RSVP and attend events if the process were reworked so users could perhaps enter their information into a form or even use their Facebook login to RSVP instead.