11.1. Beta Testing a Course

When a course is being designed in Studio, the course team can choose View Live to assess their progress in designing the course, test their work, find gaps and errors, and mimic the student experience. To gather information about the overall experience of taking a course, you can also conduct a beta test of the course.

11.1.1. The Beta Testing Process

During a beta test, you gather information about the experience of taking your course in an effort to improve its quality. The phases of the beta testing process, and some questions that you may find useful to consider as you prepare for each phase, follow.

  1. Planning: What are your objectives for the test? What is the schedule for testing?
  2. Recruitment: How many testers do you need? How much time do you expect testers to spend? When do you need them to complete their feedback?
  3. Provide access to your course: Can testers access your entire course immediately, or are its sections and subsections available at different times? How will that affect the schedule?
  4. Collect feedback: How do you want testers to provide feedback? Who checks for feedback, and how often?
  5. Evaluate feedback: How is feedback evaluated? Who decides what changes to make as a result, and on what schedule?
  6. Conclusion: How do you acknowledge the efforts of your testers? What do you share with them about the results of their efforts?

There is no one answer to any of these questions. They are included here as background on the role that beta testing can play in the preparation of your course.

11.1.2. The Beta Testing Role

Beta testers have early access to the course. Beta testers are not members of the course team or staff: they don’t have information about “how it’s supposed to work”. They use their own computers and Internet connections to view videos, follow links, and complete problems. They interact with the course as students will to find, and make, mistakes.

However, beta testers are not the same as other students in the course, either. They have privileged access to the course and have more time to review and complete the course materials than the enrolled students do. Course discussions are not open before the course start date, so beta testers cannot participate in community conversations. As a result of these differences, beta testers do not receive certificates when they complete a course.

Note

If one of your beta testers wishes to earn a certificate for the course, she must create a separate user account with a different username and email address. She can then use that separate, non-privileged user account to enroll in the course and repeat the work, completing assignments and exams when they are available to all students.

11.1.2.1. Qualities of Good Beta Testers

A beta test is valuable in part because it is unscripted. Your beta testers are not following a predetermined series of steps, or using a particular computer environment, when they access your course. When you recruit beta testers, however, you may find these skills and characteristics to be helpful.

  • Curiosity.
  • Attention to detail for identifying problems and inconsistencies.
  • Solid communication skills for reporting problems to course staff.

Your beta testers should also have varying levels of knowledge about the course content:

  • To recognize when material is wrong or missing, at least one tester should know the content well.
  • To identify material that is unclear, at least one tester should be less familiar with the content.

Depending on the objectives you set for your beta test, you may want to consider recruiting testers who use assistive technologies, who have different native languages, or who have varying levels of familiarity with computer software.

11.1.2.2. Using Course Staff as Beta Testers

Course staff can provide valuable feedback about your course. However, they are typically stakeholders in the success of your course and have a significant amount of knowledge about it. As a result, they can be too close to the course to interact with it in the same way as students will. They can also be either reluctant to provide feedback, or overly zealous.

If you do want a staff member to be a beta tester, a different, second email address must be used for this additional role. The privileges of the course staff role override those of a beta tester, so a second registration on the site, activation, and enrollment in the course are necessary using this second email address. The staff member must log in using the second email address (with the beta tester role) in order to experience the course as a student.

11.1.3. What to Test

Beta testers should interact with everything in the course.

  • Click all links.
  • Watch all videos.
  • Download video transcripts and other files.
  • Complete all problems.

As they work, beta testers log issues and questions for the course team.

Note

Beta testers cannot read or make discussion posts or contribute to the course wiki.

11.1.4. How Beta Testers See Course Content

To beta test a course, you:

  • Define a number of days before section and subsection release days for testing to begin.
  • Identify beta testers.

Designated beta testers see course content before students can. The matrix that follows shows the course content that beta testers can access earlier than other students.

  Yes No
Before the Course Enrollment Date X  
Before the Course Start Date X  
Before the section Release Day X  
Before the subsection Release Day X  
Before the unit is Public   X
Before a draft replaces a live unit   X

The course team can continue to add content in Studio after the beta test begins. When new content is ready for testing, be sure to publish the unit.

11.1.4.1. Define When the Beta Test Starts

To define the start of the beta test, you use Studio to specify a number of days before the release day that is defined for each section and subsection in your course.

  1. In Studio, from the Settings menu select Advanced Settings.
  2. In the field for the Days Early for Beta Users policy key, enter the number of days before each section and subsection release day for beta testers to have access.
  3. Click Save Changes.

11.1.4.2. Example

Beta tester access to courseware for a course with Days Early for Beta Users = 20 and 2 sections:

Course Enrollment Date = 31 August 31 August; earlier if enrolled by course staff
Course Start Date = 15 September 26 August
section 1 Release Day = 15 September 26 August
section 2 Release Day = 22 September 2 September
subsection 1 Release Day = 22 September 2 September
subsection 2 Release Day = 24 September 4 September
subsection 2, unit 1 Visibility = Public 4 September
subsection 2, unit 2 Visibility = Public; draft in progress 4 September for Public version; No access to draft
subsection 2, unit 3 Visibility = Private No access

In this example, the beta testers cannot access all of the courseware when the beta test starts on 26 August: they can access section 1 but not section 2. You may want to provide a schedule of section availability to your testers so that they can plan their time appropriately. Future release dates do not display to the beta testers in the courseware.

11.1.5. Adding Beta Testers

Before you can add beta testers:

  • You need the email address or username of each beta tester.
  • Each beta tester must register and activate a user account for that email address/username.

When you add beta testers, note the following.

  • If the beta test starts before the course Enrollment Start Date, testers cannot enroll themselves in your course. However, you can enroll the beta testers prior to the Enrollment Start Date.
  • If you add the beta testers after the defined start of the beta test, and they are enrolled in the course, they see your course on their Current Courses dashboards when they log in.
  • If you add beta testers before the test starts, or if they are not enrolled, they do not see your course on their dashboards. You can enroll the beta testers in the course yourself, and you may want to send an email message to remind them when the test begins.

11.1.5.1. Add Multiple Beta Testers

If you have a number of beta testers that you want to add, you can use the “batch add” option to add them all at once, rather than individually. With this feature, you have options to enroll the beta testers in the course (before or after the Enrollment Start Date) and send an email message to notify the beta testers that they have been added.

To add multiple beta testers:

  1. View the live version of your course.
  2. Click Instructor, and then click Membership.
  3. In the Batch Beta Tester Addition section of the page, enter one or more addresses or usernames separated by commas or line feeds (the Enter key). You can also copy data from a CSV file of email addresses and paste it here.
  4. To enroll the beta testers in your course, leave Auto Enroll selected. If the beta test starts before the course Enrollment Start Date, testers cannot enroll themselves in your course.
  5. To send an email message to the beta testers, leave Notify users by email selected. An example of the message that is sent to a beta tester who is not enrolled (or auto enrolled) in the course follows.
"You have been invited to be a beta tester for {course name} at {URL}. Visit {link} to join the course and begin the beta test."
  1. Click Add beta testers.

To remove the Beta Tester role from one or more users, enter their email addresses in the Batch Add Beta Testers field and then click Remove beta testers.

Note

The Auto Enroll option has no effect when you click Remove beta testers. The user’s role as a beta tester is removed; course enrollment is not affected.

11.1.5.2. Add Beta Testers Individually

To add a single beta tester:

  1. View the live version of your course.

  2. Click Instructor then Membership.

  3. In the Administration List Management section, use the drop-down list to select Beta Testers.

  4. Under the list of users who currently have that role, enter an email address or username and click Add Beta Tester.

    If the beta test starts before the Enrollment Start Date of your course, you can also enroll the beta tester. See Enrollment.

To remove the Beta Tester role from users individually, find the user in the list of beta testers, and then click Revoke access to the right of that user’s email address.

11.1.6. Reporting Issues During a Course

Despite the efforts of the course team and the beta testers, additional problems, questions, and issues can occur while a course is running.

  • Often, students report issues by asking questions in a discussion. Your discussion administration team can watch for posts that indicate problems.
  • To let students know the best way to report problems, you can post information on the Course Info page or post it in a discussion.