12.1. Beta Testing a Course

This topic describes how to beta test your course.

12.1.1. Overview

When you are designing a course in Studio, you can select View Live to test your work work, find gaps and errors, and mimic the learner experience. To gather information about the overall experience of taking a course, you can also conduct a beta test of the course.

12.1.2. The Beta Testing Process

During a beta test, you gather information about the experience of taking your course in an effort to improve its quality. The phases of the beta testing process, and some questions that you may find useful to consider as you prepare for each phase, follow.

  1. Planning: What are your objectives for the test? What is the schedule for testing?
  2. Recruitment: How many testers do you need? How much time do you expect testers to spend? When do you need them to complete their feedback?
  3. Provide access to your course: Can testers access your entire course immediately, or are its sections and subsections available at different times? How will that affect the schedule?
  4. Collect feedback: How do you want testers to provide feedback? Who checks for feedback, and how often?
  5. Evaluate feedback: How is feedback evaluated? Who decides what changes to make as a result, and on what schedule?
  6. Conclusion: How do you acknowledge the efforts of your testers? What do you share with them about the results of their efforts?

There is no one answer to any of these questions. They are included here as background on the role that beta testing can play in the preparation of your course.

12.1.3. The Beta Testing Role

Beta testers have early access to the course. Beta testers are not members of the course team: they do not have information about “how it is supposed to work”. They use their own computers and Internet connections to view videos, follow links, and complete problems. They interact with the course as learners will to find, and make, mistakes.

However, beta testers are not the same as other learners in the course, either. They have privileged access to the course and have more time to review and complete the course materials than the enrolled learners do. Course discussions are not open before the course start date, so beta testers cannot participate in community conversations. As a result of these differences, beta testers do not receive certificates when they complete a course.

Note

If one of your beta testers wishes to earn a certificate for the course, she must create a separate user account with a different username and email address. She can then use that separate, non-privileged user account to enroll in the course and repeat the work, completing assignments and exams when they are available to all students.

12.1.3.1. Qualities of Good Beta Testers

A beta test is valuable in part because it is unscripted. Your beta testers are not following a predetermined series of steps, or using a particular computer environment, when they access your course. When you recruit beta testers, however, you may find these skills and characteristics to be helpful.

  • Curiosity.
  • Attention to detail for identifying problems and inconsistencies.
  • Solid communication skills for reporting problems to the course team.

Your beta testers should also have varying levels of knowledge about the course content:

  • To recognize when material is wrong or missing, at least one tester should know the content well.
  • To identify material that is unclear, at least one tester should be less familiar with the content.

Depending on the objectives you set for your beta test, you might want to consider recruiting testers who use assistive technologies, who have different native languages, or who have varying levels of familiarity with computer software.

12.1.3.2. Using Course Team Members as Beta Testers

The course team can provide valuable feedback about your course. However, they are typically stakeholders in the success of your course and have a significant amount of knowledge about it. As a result, they can be too close to the course to interact with it in the same way as learners will. They can also be either reluctant to provide feedback, or overly zealous.

If you do want a team member to be a beta tester, note that the privileges of the Staff or Admin role override those of a beta tester. The team member must use a different, second email address to create an additional edX account and enroll in the course, and the course team must assign only the beta tester role to that edX account. The course team member experiences the course as a learner only when she signs in to edX using the beta tester account.

12.1.4. What to Test

Beta testers should interact with everything in the course.

  • Select all links.
  • Watch all videos.
  • Download video transcripts and other files.
  • Complete all problems.

Beta testers can use a desktop computer to access an edX website, the edX mobile app on a smartphone, or both to complete their tests. As they work, beta testers log issues and questions for the course team.

12.1.5. How Beta Testers See Course Content

To beta test a course, you:

  • Define a number of days before section and subsection release days for testing to begin.
  • Identify beta testers.

Designated beta testers see course content before students can. The matrix that follows shows the course content that beta testers can access earlier than other students.

  Yes No
Before the course enrollment date X  
Before the course start date X  
Before the section release day X  
Before the subsection release day X  
Before the unit is published   X
Before a draft replaces a live unit   X

The course team can continue to add content in Studio after the beta test begins. When new content is ready for testing, be sure to publish the unit.

Important

When you set up the beta test and define when it starts, verify that the first section, subsection, and unit in your course are available to be tested. If that first unit is not available, an error occurs when testers attempt to access any other content in the course.

12.1.5.1. Define When the Beta Test Starts

To define the start of the beta test, you use Studio to specify a number of days before the release day that is defined for each section and subsection in your course.

  1. In Studio, from the Settings menu select Advanced Settings.
  2. In the field for the Days Early for Beta Users policy key, enter the number of days before each section and subsection release day for beta testers to have access.
  3. Select Save Changes.

12.1.5.2. Example

Beta tester access to courseware for a course with Days Early for Beta Users = 20 and 2 sections:

Course enrollment date = 31 August 31 August; earlier if enrolled by course team
Course start date = 15 September 26 August
Section 1 release day = 15 September 26 August
section 2 release day = 22 September 2 September
Subsection 1 release day = 22 September 2 September
Subsection 2 release day = 24 September 4 September
Subsection 2, unit 1 visibility = public 4 September
Subsection 2, unit 2 visibility = public; draft in progress 4 September for public version; no access to draft
Subsection 2, unit 3 visibility = private No access

In this example, the beta testers cannot access all of the courseware when the beta test starts on 26 August: they can access section 1 but not section 2. You may want to provide a schedule of section availability to your testers so that they can plan their time appropriately. Future release dates do not display to the beta testers in the courseware.

12.1.6. Adding Beta Testers

Before you can add beta testers:

  • You need the email address or username of each beta tester.
  • Each beta tester must register and activate a user account for that email address/username.

When you add beta testers, note the following.

  • If the beta test starts before the course Enrollment Start Date, testers cannot enroll themselves in your course. However, you can enroll the beta testers prior to the Enrollment Start Date.
  • If you add the beta testers after the defined start of the beta test, and they are enrolled in the course, they see your course on their Current Courses dashboards when they log in.
  • If you add beta testers before the test starts, or if they are not enrolled, they do not see your course on their dashboards. You can enroll the beta testers in the course yourself, and you may want to send an email message to remind them when the test begins.

12.1.6.1. Add Multiple Beta Testers

If you have a number of beta testers that you want to add, you can use the “batch add” option to add them all at once, rather than individually. With this feature, you have options to enroll the beta testers in the course (before or after the Enrollment Start Date) and send an email message to notify the beta testers that they have been added.

To add multiple beta testers:

  1. View the live version of your course.

  2. Select Instructor, and then select Membership.

  3. In the Batch Beta Tester Addition section of the page, enter one or more addresses or usernames separated by commas or line feeds (the Enter key). You can also copy data from a CSV file of email addresses and paste it here.

  4. To enroll the beta testers in your course, leave Auto Enroll selected. If the beta test starts before the course Enrollment Start Date, testers cannot enroll themselves in your course.

  5. To send an email message to the beta testers, leave Notify users by email selected. An example of the message that is sent to a beta tester who is not enrolled in the course follows.

    ::

    Dear betatester,

    You have been invited to be a beta tester for course at edge.edx.org by a member of the course team.

    Visit https://edge.edx.org/courses/course-name/about to join the course and begin the beta test.

  6. Select Add beta testers.

To remove the Beta Tester role from one or more users, enter their email addresses in the Batch Add Beta Testers field and then select Remove beta testers.

Note

The Auto Enroll option has no effect when you select Remove beta testers. The user’s role as a beta tester is removed; course enrollment is not affected.

12.1.6.2. Add Beta Testers Individually

  1. View the live version of your course.

  2. Select Instructor then Membership.

  3. In the Administration List Management section, use the drop-down list to select Beta Testers.

  4. Under the list of users who currently have that role, enter an email address or username and click Add Beta Tester.

    If the beta test starts before the Enrollment Start Date of your course, you can also enroll the beta tester. See Enrollment.

To remove the Beta Tester role from users individually, find the user in the list of beta testers, and then select Revoke access to the right of that user’s email address.

12.1.7. Reporting Issues During a Course

Despite the efforts of the course team and the beta testers, additional problems, questions, and issues can occur while a course is running.

  • Often, learners report issues by asking questions in a discussion. Your discussion moderation team can watch for posts that indicate problems.
  • To let learners know the best way to report problems, you can post information on the Course Info page or post it in a discussion.